WO2021075317A1

WO2021075317A1 - Ar system and terminal

Info

Publication number: WO2021075317A1
Application number: PCT/JP2020/037875
Authority: WO
Inventors: 泰士山本; 林　宏樹
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2019-10-18
Filing date: 2020-10-06
Publication date: 2021-04-22
Also published as: JP7355840B2; JPWO2021075317A1

Abstract

An augmented reality (AR) system comprising: a storage unit for storing map data; a positioning unit for estimating the global position information of a communication terminal at the time of imaging in the communication terminal; a position derivation unit for successively acquiring local position information and deriving the position information of the communication terminal on the basis of the global position information and the local position information; a display form determination unit for determining a display form, with respect to AR content that corresponds to the position information of the communication terminal, in accordance with the position information of the communication terminal; and a display unit for displaying the AR content in the display form determined by the display form determination unit. The display form determination unit determines the display form so that, when the display position of the AR content being displayed is to be changed in accordance with the position information of the communication terminal, the display position changes in a prescribed slow change mode in which a change of the display position is not immediately completed.

Description

AR system and terminal

One aspect of the present invention relates to an AR system and a terminal.

Document 1 describes a technique for moving AR (Augmented Reality) content, which is a specific image in virtual space, and displaying it (according to the position of the user terminal) in response to the movement of the user terminal in the real space. Are listed.

Japanese Unexamined Patent Publication No. 2018-158120

Here, in displaying the AR content as described above, for example, a self-position recognition technique for estimating the position / direction of the camera from the image captured by the user terminal is used. In such a technology, specifically, for example, self-position recognition (global positioning) of real-world coordinates is performed from an image captured by a user terminal on the server side and map data, and global positioning on the server is performed on the user terminal side. The self-position recognition of the user terminal is performed based on the result of and the local tracking result. In this way, when performing the process of converting (correcting) the local tracking result held in the user terminal to the global coordinate system, for example, due to the influence of the error of the global positioning result (camera position estimation result), it is global. Even though the position of the user terminal has not changed before and after positioning, the position of the AR content being drawn may be misaligned and displayed (displayed with a clutter). This may worsen the AR user experience.

One aspect of the present invention has been made in view of the above circumstances, and an object of the present invention is to improve the user experience of AR.

The AR system according to one aspect of the present invention is an AR system that displays AR contents according to the position of a terminal, and is associated with a feature amount of a feature point included in image data acquired in advance and the feature point. Based on the storage unit that stores the map data associated with the global position information, which is the absolute position information, the image data captured by the terminal, and the map data stored in the storage unit, the terminal The global position estimation unit that estimates the global position information of the terminal at the time of imaging and the local position information that is the relative position information of the terminal with respect to a certain point are continuously acquired and used as the global position information and the local position information. Based on this, the position deriving unit that derives the position information of the terminal, the display mode determining unit that determines the display mode according to the position information of the terminal, and the display mode determining unit determine the AR content according to the position information of the terminal. A display unit that displays AR content in a display mode is provided, and when the display mode determination unit changes the display position of the AR content being displayed according to the position information of the terminal, the change in the display position is immediately changed. The display mode is determined so that the display position changes in a predetermined slow change mode that is not completed.

In the AR system according to one aspect of the present invention, when displaying AR content according to the position information of the terminal, when the display position is changed, the change of the display position is not immediately completed and the AR content is displayed in a predetermined slow change mode. The position can be changed. When deriving the location information of the terminal based on the global location information and the local location information, for example, when the global location information is newly acquired, the position of the terminal actually changes before and after the reacquisition of the global location information. The position of the AR content being drawn is misaligned and displayed due to the influence of the error of the global position information (error of the estimation result of the camera position of the terminal) even though it is not (or only a slight change). It may be displayed). That is, there is a problem that the AR content is displayed unnaturally at the timing of reacquiring the global location information. In this respect, in the AR system according to one aspect of the present invention, when the display position of the AR content is changed in consideration of the global position information, the display position can be changed in the slow change mode in which the display position is not changed immediately. , It is possible to prevent the AR content from moving immediately and being displayed in a cluttered manner each time the global location information changes, and the AR content can be displayed in a more natural manner. As described above, according to the AR system according to one aspect of the present invention, the user experience of AR can be improved.

The terminal according to one aspect of the present invention is a terminal that transmits captured image data to a server and displays AR contents according to a position in cooperation with the server, and is a relative of the terminals with respect to a certain point. For the location derivation unit that continuously acquires local location information, which is the location information, and derives the location information of the terminal based on the global location information estimated by the server based on the image data and the local location information, and the location information of the terminal. The display mode determination unit includes a display mode determination unit that determines the display mode according to the position information of the terminal and a display unit that displays the AR content in the display mode determined by the display mode determination unit. When the display position of the AR content being displayed is changed according to the position information of the terminal, the display mode is changed so that the display position changes in a predetermined slow change mode that does not immediately complete the change of the display position. decide.

According to one aspect of the present invention, the user experience of AR can be improved.

It is a figure explaining the outline of the AR system which concerns on this embodiment. It is a block diagram which shows the functional structure of the AR system which concerns on this embodiment. It is a figure explaining the display position movement of AR content. It is a figure which shows the example of the movement mode of AR content. It is a figure explaining the movement mode of the AR content during display and non-display. It is a flowchart which shows the process performed by an AR system. It is a figure which shows the hardware configuration of the positioning server, the communication terminal, and the content server included in an AR system.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the description of the drawings, the same reference numerals are used for the same or equivalent elements, and duplicate description is omitted.

FIG. 1 is a diagram for explaining the outline of the AR (Augmented Reality) system 1 according to the present embodiment. The AR system 1 shown in FIG. 1 is a system that displays AR contents according to the position of the communication terminal 50 (terminal). The AR system 1 includes a positioning server 10, a content server 30, and a communication terminal 50.

As shown in FIG. 1, in the AR system 1, the image data captured by the communication terminal 50 is transmitted to the positioning server 10. In FIG. 1, an image of a building is exemplified as image data captured by the communication terminal 50. The positioning server 10 acquires global position information based on the image data captured by the communication terminal 50 (details will be described later), and transmits the global position information to the communication terminal 50. The global position information is position information (absolute position information) indicated by a common coordinate system that can be used by any device, and may include position / direction / tilt information. The communication terminal 50 acquires the position and the like of itself (specifically, its own camera that captures image data) by acquiring global position information from the positioning server 10. Then, the communication terminal 50 transmits the acquired global location information to the content server 30. The content server 30 acquires the AR content according to the global location information (details will be described later), and transmits the AR content according to the global location information to the communication terminal 50. In FIG. 1, an image of a girl is illustrated as AR content according to global location information. By performing the above processing, the AR system 1 displays the AR content according to the position of the communication terminal 50.

FIG. 2 is a block diagram showing the functional configuration of the AR system 1. As described above, the AR system 1 includes a positioning server 10, a content server 30, and a communication terminal 50. Although only one communication terminal 50 is shown in FIG. 1, a plurality of communication terminals 50 may actually be included.

The positioning server 10 estimates global position information, which is the absolute position information of the communication terminal 50 at the time of imaging, based on the image data captured by the communication terminal 50. The positioning server 10 has a storage unit 11 and a positioning unit 12 (global position estimation unit).

The storage unit 11 is a map in which the feature amount (for example, the brightness direction vector) of the feature point included in the image data acquired in advance and the global position information which is the absolute position information associated with the feature point are associated with each other. Data 100 is stored. Such map data 100 is generated based on a large amount of image data previously captured by, for example, a stereo camera (not shown) capable of simultaneously capturing an object from a plurality of different directions (3D point cloud). is there. The feature point is a point that is prominently detected in the image, and is, for example, a point whose brightness (intensity) is larger (or smaller) than that of other regions. The global position information of the feature points is the global position information set in association with the feature points, and is the global position information in the real world about the region indicated by the feature points in the image. It should be noted that the association of the global position information with each feature point can be performed by a conventionally known method.

The storage unit 11 stores three-dimensional position information as global position information of the feature points of the map data 100. The storage unit 11 stores, for example, the latitude, longitude, and height of the feature points as three-dimensional global position information of the feature points. The storage unit 11 may store a plurality of divided map data divided for each fixed area according to the global position information of the map data 100.

The positioning unit 12 is based on the image data captured by the communication terminal 50 and the map data 100 stored in the storage unit 11, and the global position information (three-dimensional) of the communication terminal 50 at the time of imaging by the communication terminal 50. Position information) is estimated. Specifically, the positioning unit 12 matches the feature points of the map data 100 with the feature points of the image data captured by the communication terminal 50, and sets the region of the map data 100 corresponding to the captured image data. Identify. Then, the positioning unit 12 estimates the imaging position of the image data (that is, the global position information of the communication terminal 50 at the time of imaging) based on the global position information associated with the feature points of the map data 100 related to the specified area. To do. The positioning unit 12 transmits the positioning result to the communication terminal 50. The positioning result may include information on the direction estimated from the image data (direction in the three-dimensional coordinates of roll, pitch, and yaw) in addition to the global position information. The positioning unit 12 may acquire global position information based on image data newly captured by the communication terminal 50 at predetermined time intervals, or may newly acquire the global position information at the timing of receiving an instruction from the communication terminal 50. Global position information may be acquired based on the image data captured in.

The content server 30 derives AR content corresponding to the global location information based on the global location information transmitted from the communication terminal 50, and transmits the AR content to the communication terminal 50. The content server 30 has a storage unit 31 and a content output unit 32. The storage unit 31 stores the content data 300 (3D content space) in which the global position information and the AR content are associated with each other. The content output unit 32 identifies AR content according to the global location information of the communication terminal 50 based on the global location information transmitted from the communication terminal 50 and the content data 300 stored in the storage unit 31. Then, the content output unit 32 transmits (outputs) the specified AR content to the communication terminal 50. The content server 30 may periodically specify the AR content based on the global position information of the communication terminal 50 at predetermined time intervals, or the global position of the communication terminal 50 may be specified at the timing when the instruction from the communication terminal 50 is received. AR content may be specified based on the information.

The communication terminal 50 is, for example, a terminal capable of wireless communication, such as a smartphone, a tablet terminal, a PC, or the like. The communication terminal 50 receives the AR content provided by the content server 30, for example, by the user using the implemented application. The communication terminal 50 transmits the captured image data to the positioning server 10, and cooperates with the positioning server 10 and the content server 30 to display AR content according to the position. In the communication terminal 50, for example, when an application is executed, imaging (continuous imaging) by the mounted camera is started. Then, the communication terminal 50 acquires the global position information from the positioning server 10 according to the captured image data, and receives the content including the AR content from the content server 30 based on the global position information. The communication terminal 50 includes an imaging unit 51, a storage unit 52, a position deriving unit 53, a display mode determining unit 54, and a display unit 55.

The image pickup unit 51 is a function of controlling the image capture of image data by the camera. The image pickup unit 51 starts capturing image data by the camera, for example, at the timing when the execution of the application for receiving the AR content is started. When the imaging unit 51 starts imaging, the imaging unit 51 continuously performs imaging by the camera until the execution of the application is completed. The captured image data is stored in the storage unit 52, and is used for estimating the global position information by the positioning server 10 and deriving the position information of the communication terminal 50 by the position deriving unit 53.

The storage unit 52 includes an imaging result by the imaging unit 51, an estimation result of global position information by the positioning server 10, position information of the communication terminal 50 derived by the position deriving unit 53, AR content received from the content server 30, and This is a database that stores information related to the display mode determined by the display mode determination unit 54.

The position derivation unit 53 continuously acquires the local position information which is the relative position information of the communication terminal 50 with respect to a certain point, and derives the position information of the communication terminal 50 based on the global position information and the local position information. To do. A certain point is, for example, a point at which the execution of the application (application receiving the provision of AR content) is started on the communication terminal 50. The position derivation unit 53 uses, for example, the three-dimensional latitude, longitude, and height of the point at which the execution of the application is started as a reference (coordinates (x, y, z) are (0,0,0)), and from that point. By continuously acquiring the amount of change in the three-dimensional latitude, longitude, and height of the above, the local position information, which is the relative position information of the communication terminal 50 with respect to the relevant point, is tracked (local tracking). ..

Then, the position derivation unit 53 derives the position information of the communication terminal 50 based on the global position information and the local position information. The position deriving unit 53 derives the position information of the communication terminal 50 from the global position information as a starting point in consideration of the amount of movement from the global position information (the amount of movement shown in the local position information). Specifically, the position derivation unit 53 starts from the global position information, the local position information at the time when the image data related to the derivation of the global position information is captured, and the current local position information. The movement amount is derived, and the position information of the communication terminal 50 is derived.

The position derivation unit 53 compares the positions of the feature points included in the image data between the image data continuously imaged by the image pickup unit 51, for example, and displaces the feature points (how much the same feature points move between the image data). Local location information may be acquired based on the information indicating the displacement). Such local tracking is performed by, for example, the SLAM (Simultaneous Localization and Mapping) engine of the communication terminal 50. That is, the position derivation unit 53 compares the feature points of the image data captured at a certain time point with the feature points of the image data one frame before by the function of the SLAM engine, and changes the position, number, and density of the feature points. By measuring, changes in the position and angle of the communication terminal 50 are measured. The SLAM engine generates map data in which the feature amount of a feature point (for example, a brightness direction vector) and the position information associated with the feature point are mapped in real time based on continuously captured image data. .. The position deriving unit 53 repeatedly executes such mapping and local tracking.

In the following description, the position derivation unit 53 will perform local tracking using the above-mentioned image data, but the position derivation unit 53 continuously acquires three-dimensional local position information from the sensor mounted on the communication terminal 50. It may be a thing. In this case, the sensor mounted on the communication terminal 50 is not limited, but for example, a Visual SLAM (Simultaneous Localization and Mapping) using an inertial measurement unit (IMU), a camera sensor, a general acceleration sensor, or a gyro sensor. Etc. can be used.

The display mode determination unit 54 responds to the position information of the communication terminal 50 with respect to the AR content (specifically, the AR content according to the global position information used for deriving the position information) according to the position information of the communication terminal 50. To determine the display mode. The display mode determination unit 54 is based on a process (first process) for determining the display mode based on the newly acquired global position information, or based on the global position information that has already been used for determining the display mode. One of the processes (second process) for determining the display mode is executed.

In the second process, the display mode determination unit 54 considers the position information (global position information and local position information) of the communication terminal 50 derived by the position derivation unit 53 for the AR content corresponding to the position information of the communication terminal 50. The display mode is determined so that it is displayed at the position related to (position information). In this case, the display mode determination unit 54 determines the display mode so that the change (movement) of the display position of the AR content is immediately completed even when the local position information changes and the display position of the AR content changes. To do. Immediately completing the change in the display position of the AR content means, for example, that the display position of the AR content changes according to the step function immediately after the start of the change, as shown in FIG. 4 (c).

In the first process, the display mode determination unit 54 converts the AR content corresponding to the position information of the communication terminal 50 into the position information (newly acquired global position information) of the communication terminal 50 derived by the position derivation unit 53. The display mode is determined so that it is displayed at such a position. In this case, when the display mode determination unit 54 changes the display position of the AR content being displayed according to the position information (newly acquired global position information) of the communication terminal 50, the change in the display position is immediately changed. The display mode is determined so that the display position changes in a predetermined slow change mode that is not completed.

The reason for changing the display position of the AR content in the slow change mode in the first process will be described with reference to FIG. In FIG. 3, the upper figure shows the position information of the communication terminal 50 before and after the acquisition of the new global position information (the upper left figure is before the acquisition, the upper right figure is after the acquisition), and the lower figure shows the communication in the state corresponding to the upper figure. The display image of the terminal 50 is shown. The terms "up / down / left / right" here are up / down / left / right when viewed from a direction in which the reference numerals in FIG. It is assumed that there are contents CX and CY as AR contents that can be displayed at the position of the camera 500 as shown in the upper figure, and only the contents CX are displayed in the display range IA of the camera 500 as shown in the lower figure. Now, even though the position of the camera 500 of the communication terminal 50 has not actually changed (or has changed only slightly), due to an error in the global position information (an error in the estimation result of the position of the camera 500), As shown in the upper figure of FIG. 3, it is assumed that it is recognized that the position of the camera 500 has changed at the timing when the global position information is newly acquired. Specifically, as shown in the upper right figure of FIG. 3, it is assumed that it is recognized that the position of the camera 500 has changed from the position indicated by the broken line to the position indicated by the solid line. In this case, as shown in the lower right figure of FIG. 3, the display position of the content CX suddenly changes from the position indicated by the broken line to the position indicated by the solid line in the display range IA of the camera. In this case, although the position of the communication terminal 50 has not actually changed (or has changed only slightly) before and after the reacquisition of the global position information, the error of the global position information (camera of the communication terminal 50) has not changed. Due to the influence of the error of the estimation result of the position of 500), the position of the AR content drawn is shifted and displayed (displayed with a click). This may make the display of the AR content unnatural and give the user a sense of discomfort. In the AR system according to the present embodiment, the display mode determining unit 54 determines the display mode so that the display position of the AR content changes in the slow change mode described above, thereby suppressing the stuttering of the AR content described above. ing.

Specifically, the display mode determination unit 54 determines the display mode so that the display position of the AR content gradually changes with the passage of a predetermined time, for example. Such a change in the display position is different from the mode in which the AR content changes according to the step function immediately after the start of the change as in the first process described above (see FIG. 4C), and the display position of the AR content is changed. This is a display position change in the slow change mode in which the change of is not completed immediately. In the display position change in the slow change mode, the coordinates of the camera position are input smoothly instead of the step function, so that it is possible to reduce the clutter of the AR content.

As shown in FIG. 4A, the display mode determining unit 54 may determine the display mode so that the AR content moves at a constant speed and the display position of the AR content gradually changes. Alternatively, as shown in FIG. 4B, the display mode determining unit 54 may perform the movement start period and the movement end period during the movement start period at which the movement of the AR content is started and the movement end period at which the movement ends. The display mode may be determined so that the moving speed of the AR content is slower than the period of. The movement start period is a predetermined or arbitrary period including, for example, a time point at which the movement is started. The movement end period is, for example, a predetermined or arbitrary period including a time point at which the movement is completed. Specifically, as shown in FIG. 4B, the display mode determining unit 54 determines the display mode so that the AR content moves smoothly according to the function approximated by the quadratic function.

Further, for example, the display mode determination unit 54 displays the AR content being displayed again after the AR content is hidden according to the position information of the communication terminal 50 without changing the display position during the display. The display mode is determined so that the display position changes at the time of operation. Such a change in the display position is different from the mode in which the AR content changes according to the step function immediately after the start of the change as in the first process described above (see FIG. 4C), and the display position of the AR content is changed. This is a display position change in the slow change mode in which the change of is not completed immediately (the display position does not change until it is displayed again). FIG. 5 is a diagram showing how the AR contents being displayed and hidden are moved. As shown in FIG. 5, it is assumed that the content CX is currently drawn in the display range IA of the camera 500, and the content CY is outside the display range IA. In this case, when the position of the content CX outside the display range IA (not being displayed) changes based on the re-acquired global position information of the communication terminal 50, the display mode determination unit 54 immediately positions the content CX. The correction is performed, and the display mode is determined so that when the content CY is drawn in the display range IA, it is displayed at the position after the position correction. On the other hand, when the position of the content CX drawn (displayed) in the display range IA changes based on the re-acquired global position information of the communication terminal 50, the display mode determination unit 54 specifically When the position shown by the solid line in FIG. 5 changes to the position shown by the broken line, the display position is not changed without performing the position correction during the display, and the content CX is out of the display range IA. Position correction (coordinate conversion) is performed, and then the display mode is determined so that the display position is changed when the content CX is drawn in the display range IA again.

The display unit 55 displays the AR content in the display mode determined by the display mode determination unit 54. When the display mode is determined by the display mode determination unit 54 by the second process described above, the display unit 55 immediately completes the change (movement) of the display position of the AR content. When the display mode is determined by the display mode determination unit 54 by the first process described above, the display unit 55 changes the display position of the AR content in a predetermined slow change mode that does not immediately complete the change in the display position (the change in the display position of the AR content ( Move) is completed.

Next, the processing performed by the AR system 1 according to the present embodiment, specifically, the processing related to the display of AR contents will be described with reference to FIG. FIG. 6 is a flowchart showing the processing performed by the AR system 1.

As shown in FIG. 6, in the AR system 1, first, in the communication terminal 50, whether the global position information is newly acquired or the display mode is determined based on the newly acquired global position information. Whether or not it is determined (step S1). In step S1, the display mode is not determined based on the newly acquired global position information, but the display mode is determined based on the global position information that has already been used for determining the display mode. If it is determined, the communication terminal 50 performs the second process described above (step S2). That is, the communication terminal 50 determines the display mode so that the AR content is displayed at the position related to the position information (position information in consideration of the global position information and the local position information) of the communication terminal 50. In this case, the communication terminal 50 determines the display mode so that the change (movement) of the display position of the AR content is completed immediately even when the display position of the AR content changes due to the change of the local position information. The AR content is displayed.

On the other hand, when it is determined in step S1 that the global position information is newly acquired, whether or not the position information of the communication terminal 50 has changed before and after the acquisition of the global position information in the communication terminal 50. Is determined (step S3). If it is determined in step S3 that the position information has not changed, the process of step S1 is executed again.

On the other hand, when it is determined in the process of step S3 that the position information has changed, the communication terminal 50 does not immediately complete the change in the display position of the AR content being displayed. The display mode is determined so that the display position changes in the mode, and the display position of the AR content is changed in the slow change mode (step S4). Specifically, for example, the communication terminal 50 gradually changes the display position of the AR content with the passage of a predetermined time (see FIGS. 4A and 4B). Alternatively, for example, when the AR content being displayed is displayed again after the AR content is hidden according to the position information of the communication terminal 50 without changing the display position during the display. Change the display position to.

Finally, in the communication terminal 50, it is determined whether or not the execution of the application receiving the AR content is finished (step S5), and when the execution of the application is finished, the process is finished, and when it is not finished, the process is finished. It is executed again from the process of step S1.

Next, the operation and effect of the AR system 1 according to the present embodiment will be described.

The AR system 1 according to the present embodiment is a system that displays AR contents according to the position of the communication terminal 50, and is associated with the feature amount of the feature points included in the image data acquired in advance and the feature points. A storage unit 11 that stores map data 100 associated with global position information that is absolute position information, image data captured by the communication terminal 50, and map data 100 stored in the storage unit 11. Based on the above, the positioning unit 12 that estimates the global position information of the communication terminal 50 at the time of imaging by the communication terminal 50 and the local position information that is the relative position information of the communication terminal 50 with respect to a certain point are continuously obtained. The position deriving unit 53, which is acquired and derives the position information of the communication terminal 50 based on the global position information and the local position information, and the AR content corresponding to the position information of the communication terminal 50 are displayed according to the position information of the communication terminal 50. A display mode determining unit 54 for determining the mode and a display unit 55 for displaying AR content in the display mode determined by the display mode determining unit 54 are provided, and the display mode determining unit 54 determines the AR content being displayed. When the display position is changed according to the position information of the communication terminal 50, the display mode is determined so that the display position changes in a predetermined slow change mode in which the change of the display position is not completed immediately.

In the AR system 1 according to the present embodiment, when displaying AR content according to the position information of the communication terminal 50, when the display position is changed, the change of the display position is not completed immediately in a predetermined slow change mode. The display position can be changed. In the case of deriving the position information of the communication terminal 50 based on the global position information and the local position information, for example, when the global position information is newly acquired, the communication terminal 50 is actually before and after the reacquisition of the global position information. Even though the position has not changed (or only a slight change), the position of the AR content being drawn is displaced due to the influence of the global position information error (error of the estimation result of the camera position of the terminal). May be displayed (displayed with a clutter). That is, there is a problem that the AR content is displayed unnaturally at the timing of reacquiring the global location information. In this regard, in the AR system 1 according to the present embodiment, when the display position of the AR content is changed in consideration of the global position information, the display position is changed in the slow change mode in which the display position is not changed immediately. Every time the global location information changes, it is possible to prevent the AR content from moving immediately and being displayed in a cluttered manner, and the AR content can be displayed in a more natural manner. As described above, according to the AR system 1 according to the present embodiment, the user experience of AR can be improved. In addition, the technical effect that the processing load can be reduced by not changing the display position unnecessarily is also achieved.

The display mode determination unit 54 determines the display mode so that the display position of the AR content gradually changes with the passage of a predetermined time. According to such a configuration, when the display position of the AR content is changed, the movement of the AR content is not completed immediately according to the step function, but the movement of the AR content is gradually completed over time. be able to. As a result, it is possible to prevent the AR content from moving immediately and being displayed in a cluttered manner each time the global location information changes, and the AR content can be displayed in a more natural manner.

The display mode determination unit 54 determines the display mode so that the AR content moves at a constant speed and the display position of the AR content changes. By moving the AR content at a constant speed, the AR content can be moved in a manner that is not unnatural to the user.

In the movement start period at which the movement of the AR content is started and the movement end period at which the movement ends, the display mode determining unit 54 makes the movement speed of the AR content slower than the period between the movement start period and the movement end period. As described above, the display mode is determined. By making the movement speed at the start and end of the movement slower than the movement speed during the movement, it becomes difficult for the user to be aware of the movement of the AR content, and the AR content is moved in a manner that does not make it more unnatural for the user. be able to.

When the display mode determining unit 54 displays the AR content being displayed again after being hidden according to the position information of the communication terminal 50 without changing the display position during the display. The display mode is determined so that the display position changes. In this way, by not moving the AR content during display (not changing the display position), the AR content is displayed in a more natural manner without making the user aware of the movement of the AR content. be able to.

Finally, the hardware configurations of the positioning server 10, the content server 30, and the communication terminal 50 included in the AR system 1 will be described with reference to FIG. 7. The positioning server 10, the content server 30, and the communication terminal 50 physically include a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like. It may be configured as.

In the following explanation, the word "device" can be read as a circuit, device, unit, etc. The hardware configuration of the positioning server 10, the content server 30, and the communication terminal 50 may be configured to include one or more of the devices shown in the figure, or may be configured not to include some of the devices. You may.

For each function of the positioning server 10, the content server 30, and the communication terminal 50, the processor 1001 performs calculations by loading predetermined software (programs) on hardware such as the processor 1001 and the memory 1002, and the communication device. It is realized by controlling communication by 1004 and reading and / or writing of data in the memory 1002 and the storage 1003.

Processor 1001 operates, for example, an operating system to control the entire computer. The processor 1001 may be composed of a central processing unit (CPU: Central Processing Unit) including an interface with a peripheral device, a control device, an arithmetic unit, a register, and the like. For example, the control function of the positioning unit 12 and the like of the positioning server 10 may be realized by the processor 1001.

Further, the processor 1001 reads a program (program code), a software module, and data from the storage 1003 and / or the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described in the above-described embodiment is used. For example, the control function of the positioning unit 12 or the like of the positioning server 10 may be realized by a control program stored in the memory 1002 and operated by the processor 1001, or may be realized in the same manner for other functional blocks. Although it has been described that the various processes described above are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. Processor 1001 may be mounted on one or more chips. The program may be transmitted from the network via a telecommunication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one such as a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EPROM (Electrically Erasable Programmable ROM), and a RAM (Random Access Memory). May be done. The memory 1002 may be referred to as a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store a program (program code), a software module, or the like that can be executed to carry out the wireless communication method according to the embodiment of the present invention.

The storage 1003 is a computer-readable recording medium, and is, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, an optical magnetic disk (for example, a compact disk, a digital versatile disk, or a Blu-ray). It may consist of at least one (registered trademark) disk), smart card, flash memory (eg, card, stick, key drive), floppy (registered trademark) disk, magnetic strip, and the like. The storage 1003 may be referred to as an auxiliary storage device. The storage medium described above may be, for example, a database, server or other suitable medium containing memory 1002 and / or storage 1003.

The communication device 1004 is hardware (transmission / reception device) for communicating between computers via a wired and / or wireless network, and is also referred to as, for example, a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may have an integrated configuration (for example, a touch panel).

Further, each device such as the processor 1001 and the memory 1002 is connected by the bus 1007 for communicating information. Bus 1007 may be composed of a single bus, or may be composed of different buses between devices.

Further, the positioning server 10, the content server 30, and the communication terminal 50 include a microprocessor, a digital signal processor (DSP: Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable). It may be configured to include hardware such as Gate Array), and a part or all of each functional block may be realized by the hardware. For example, processor 1001 may be implemented on at least one of these hardware.

Although the present embodiment has been described in detail above, it is clear to those skilled in the art that the present embodiment is not limited to the embodiment described in the present specification. This embodiment can be implemented as a modified or modified mode without departing from the spirit and scope of the present invention determined by the description of the claims. Therefore, the description of the present specification is for the purpose of exemplifying explanation, and does not have any restrictive meaning to the present embodiment.

For example, the AR system 1 has been described as being configured to include the communication terminal 50, the positioning server 10, and the content server 30, but the present invention is not limited to this, and each function of the AR system 1 is realized only by the communication terminal. You may.

Each aspect / embodiment described in the present specification includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA. (Registered Trademarks), GSM (Registered Trademarks), CDMA2000, UMB (Ultra Mobile Broad-band), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-Wide) Band), Bluetooth®, and other systems that utilize suitable systems and / or extended next-generation systems based on them may be applied.

The order of the processing procedures, sequences, flowcharts, etc. of each aspect / embodiment described in the present specification may be changed as long as there is no contradiction. For example, the methods described herein present elements of various steps in an exemplary order, and are not limited to the particular order presented.

The input / output information and the like may be saved in a specific location (for example, memory) or may be managed by a management table. Input / output information and the like can be overwritten, updated, or added. The output information and the like may be deleted. The input information or the like may be transmitted to another device.

The determination may be made by a value represented by 1 bit (0 or 1), by a boolean value (Boolean: true or false), or by comparing numerical values (for example, a predetermined value). It may be done by comparison with the value).

Each aspect / embodiment described in the present specification may be used alone, in combination, or switched with execution. Further, the notification of predetermined information (for example, the notification of "being X") is not limited to the explicit one, but is performed implicitly (for example, the notification of the predetermined information is not performed). May be good.

Software, whether referred to as software, firmware, middleware, microcode, hardware description language, or by any other name, is an instruction, instruction set, code, code segment, program code, program, subprogram, software module. , Applications, software applications, software packages, routines, subroutines, objects, executable files, execution threads, procedures, features, etc. should be broadly interpreted.

In addition, software, instructions, etc. may be transmitted and received via a transmission medium. For example, the software uses wired technology such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and / or wireless technology such as infrared, wireless and microwave to websites, servers, or other When transmitted from a remote source, these wired and / or wireless technologies are included within the definition of transmission medium.

The information, signals, etc. described herein may be represented using any of a variety of different techniques. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description are voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may be represented by a combination of.

Note that the terms explained in the present specification and / or the terms necessary for understanding the present specification may be replaced with terms having the same or similar meanings.

Further, the information, parameters, etc. described in the present specification may be represented by an absolute value, a relative value from a predetermined value, or another corresponding information. ..

User terminals may be mobile communication terminals, subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, etc. It may also be referred to as a mobile device, wireless device, remote device, handset, user agent, mobile client, client, or some other suitable term.

The terms "determining" and "determining" used herein may include a wide variety of actions. "Judgment", "decision" is, for example, calculating, computing, processing, deriving, investigating, looking up (eg, table, database or another). It can include searching in the data structure), and assuming that the confirmation (ascertaining) is regarded as "judgment" or "decision". Also, "judgment" and "decision" are receiving (for example, receiving information), transmitting (for example, transmitting information), input (input), output (output), and access. (Accessing) (for example, accessing data in memory) may be regarded as "judgment" or "decision". In addition, "judgment" and "decision" mean that the things such as solving, selecting, choosing, establishing, and comparing are regarded as "judgment" and "decision". Can include. That is, "judgment" and "decision" may include considering some action as "judgment" and "decision".

The phrase "based on" as used herein does not mean "based on" unless otherwise stated. In other words, the statement "based on" means both "based only" and "at least based on".

When the terms "first", "second", etc. are used herein, any reference to that element does not generally limit the quantity or order of those elements. These designations can be used herein as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements can be adopted there, or that the first element must somehow precede the second element.

As long as "include", "including", and variations thereof are used within the scope of the present specification or claims, these terms are similar to the term "comprising". Is intended to be inclusive. Moreover, the term "or" as used herein or in the claims is intended not to be an exclusive OR.

In the present specification, a plurality of devices shall be included unless the device has only one device, which is clearly technically or technically present.

In the whole of this disclosure, if it does not clearly indicate the singular from the context, it shall include more than one.

10 ... Positioning server (server), 11 ... Storage unit, 12 ... Positioning unit (global position estimation unit), 50 ... Communication terminal (terminal), 53 ... Position derivation unit, 54 ... Display mode determination unit, 55 ... Display unit , 100 ... Map data.

Claims

It is an AR system that displays AR content according to the position of the terminal.
A storage unit that stores map data in which the feature amount of the feature point included in the image data acquired in advance and the global position information which is the absolute position information associated with the feature point are associated with each other.
A global position estimation unit that estimates the global position information of the terminal at the time of imaging at the terminal based on the image data captured by the terminal and the map data stored in the storage unit.
A position deriving unit that continuously acquires local position information that is relative position information of the terminal with respect to a certain point and derives the position information of the terminal based on the global position information and the local position information.
With respect to the AR content corresponding to the position information of the terminal, a display mode determining unit that determines the display mode according to the position information of the terminal, and
A display unit that displays the AR content in a display mode determined by the display mode determination unit is provided.
When the display mode determining unit changes the display position of the AR content being displayed according to the position information of the terminal, the display position changes in a predetermined slow change mode that does not immediately complete the change of the display position. An AR system that determines the display mode as described above.
The AR system according to claim 1, wherein the display mode determining unit determines the display mode so that the display position of the AR content gradually changes with the passage of a predetermined time.
The AR system according to claim 2, wherein the display mode determining unit determines the display mode so that the AR content moves at a constant speed and the display position of the AR content changes.
In the movement start period at which the movement of the AR content is started and the movement end period at which the movement ends, the display mode determining unit has a movement speed of the AR content rather than a period between the movement start period and the movement end period. The AR system according to claim 2, wherein the display mode is determined so that the display mode is slowed down.
The display mode determining unit displays the AR content being displayed when it is displayed again after being hidden according to the position information of the terminal without changing the display position during the display. The AR system according to any one of claims 1 to 4, wherein the display mode is determined so that the position changes.
A terminal that transmits captured image data to a server and displays AR content according to the position in cooperation with the server.
Local position information, which is the relative position information of the terminal with respect to a certain point, is continuously acquired, and the global position information estimated by the server based on the image data and the local position information of the terminal are used. A position derivation unit that derives position information and
With respect to the AR content corresponding to the position information of the terminal, a display mode determining unit that determines the display mode according to the position information of the terminal, and
A display unit that displays the AR content in a display mode determined by the display mode determination unit is provided.
When the display mode determining unit changes the display position of the AR content being displayed according to the position information of the terminal, the display position changes in a predetermined slow change mode that does not immediately complete the change of the display position. A terminal that determines the display mode as described above.