JP2024060347A - Information processing device - Google Patents

Abstract

[Problem] To provide a technology that allows a user to easily retrace the path he or she has taken even in a situation where data communication and GPS radio wave reception are unavailable. [Solution] The information processing device of the present invention is characterized by having an acquisition means for acquiring an image corresponding to the user's field of view, a detection means for detecting landmarks that are real objects from the image, a registration means for registering the detected landmarks, and a control means for controlling, when a registered landmark is detected, to display a virtual object in association with the landmark in the field of view. [Selected Figure] Figure 1

Description

The present invention relates to an information processing device, and in particular to a technology for providing route guidance.

Portable information terminals, such as smartphones, provide route guidance using the Global Positioning System (GPS) and map information. Route guidance is provided not only in urban areas, but also in mountainous areas (route guidance for mountain climbing), allowing users to easily enjoy mountain climbing even without knowledge of how to read maps or topographical maps. Even if data communication is not possible, as long as GPS signals can be received, route guidance can be provided by displaying the route and current position on a map by storing map information in the terminal in advance, thereby preventing users from getting lost or stranded. Patent Document 1 discloses route guidance that uses GPS information and the device's orientation.

JP 2004-28916 A

If you get lost while hiking, it is important to retrace your steps and return to the correct route. However, when using GPS route guidance, if the GPS signal reception deteriorates due to terrain or weather conditions, the correct current location is not displayed (the correct current location cannot be determined), making it difficult to return to the correct route.

The present invention aims to provide a technology that allows users to easily retrace their steps even in situations where data communication and GPS signal reception are not possible.

A first aspect of the present invention is an information processing device characterized by having an acquisition means for acquiring an image corresponding to a user's field of view, a detection means for detecting landmarks that are real objects from the image, a registration means for registering the detected landmarks, and a control means for controlling the display of a virtual object in association with the landmark in the field of view when a registered landmark is detected.

A second aspect of the present invention is a method for controlling an information processing device, comprising the steps of acquiring an image corresponding to a user's field of view, detecting a landmark that is a real object from the image, registering the detected landmark, and, when a registered landmark is detected, controlling to display a virtual object in association with the landmark in the field of view.

A third aspect of the present invention is a program for causing a computer to function as each of the means of the information processing device.

According to the present invention, even in a situation where data communication and GPS signal reception are not possible, the user can easily retrace his/her steps.

FIG. 1 is a block diagram showing a configuration of an apparatus according to a first embodiment; Flowchart showing the flow of processing according to the first embodiment FIG. 1 is a diagram showing priority information according to the first embodiment; FIG. 1 is a diagram for explaining a method for selecting a real object according to the first embodiment; FIG. 1 is a diagram for explaining a method for selecting a real object according to the first embodiment; FIG. 1 is a diagram showing landmark information according to the first embodiment; FIG. 1 is a diagram showing a virtual object according to the first embodiment; FIG. 1 is a block diagram showing a modified example of the configuration of the device according to the first embodiment. FIG. 13 is a diagram showing a modified example of landmark information according to the first embodiment; FIG. 13 is a diagram illustrating a modification of a virtual object according to the first embodiment; FIG. 11 is a block diagram showing the configuration of an apparatus according to a second embodiment. Flowchart showing the flow of processing according to the second embodiment FIG. 13 is a diagram showing travel route information for each section according to the second embodiment; FIG. 13 is a diagram showing a virtual object according to the second embodiment; FIG. 13 is a diagram showing a modification of a virtual object according to the second embodiment; FIG. 1 is a block diagram showing a configuration of an information processing apparatus according to a first and second embodiment;

First Embodiment
A first embodiment of the present invention will be described. Fig. 16 is a block diagram showing the configuration of an information processing device 1600 according to the first embodiment. The information processing device 1600 has a CPU 1601, a ROM 1602, a RAM 1603, an operation unit 1605, an imaging unit 1606, a display unit 1607, a movement detection sensor 1608, and a 3D shape measurement sensor 1609, which are connected to a bus 1604.

The CPU 1601 is a computing device that controls the entire information processing device 1600, and executes various programs stored in the ROM 1602 to perform various processes. The ROM 1602 is a read-only non-volatile memory that stores various information that does not require modification (e.g., various programs and various parameters). For example, the ROM 1602 stores image processing programs and initial data. The RAM 1603 is a memory that temporarily stores various information such as input information from various devices and calculation results in image processing, and also provides a working area for the CPU 1601.

The operation unit 1605 is an interface that accepts operations performed by the user using operating members such as switches and controllers.

The imaging unit 1606 is a camera that captures images (video) of real space.

The display unit 1607 is a display capable of displaying virtual objects (imaginary objects, for example, various types of text information) in the user's field of vision. For example, the display unit 1607 is a transparent display, and displays virtual objects on the display surface of the display unit 1607. This allows the user to see the virtual objects as if they were superimposed on the user's field of vision (real space). The virtual objects may be displayed at any position of the user's choice.

The movement detection sensor 1608 is a sensor capable of detecting the movement and rotation of the information processing device. For example, the movement detection sensor 1608 acquires information on six degrees of freedom, including forward/backward movement, left/right movement, up/down movement, yaw rotation, pitch rotation, and roll rotation.

The 3D shape measurement sensor 1609 is a sensor capable of measuring the 3D shape of an object. For example, the 3D shape measurement sensor 1609 acquires point cloud information representing the 3D shape. As the 3D shape measurement sensor 1609, for example, a LIDAR sensor that measures the reflection of irradiated electromagnetic waves can be used.

The imaging unit 1606, the display unit 1607, and the movement detection sensor 1608 are often used in general augmented reality devices. The information processing device 1600 may be a glasses-type device (head-mounted display device, head-mounted display (HMD)) that is worn on the user's head. In the case of a head-mounted display device, the movement and rotation detected by the movement detection sensor 1608 can also be considered as the movement and rotation of the user's head. The head-mounted display device may be an optical see-through type or a video see-through type. The head-mounted display device may be a device for both eyes or a device for one eye.

Fig. 1 (A) is a block diagram showing the configuration of an HMD 100 according to the first embodiment. The HMD 100 is an example of the information processing device 1600 in Fig. 16. The HMD 100 has an imaging unit 101, a display unit 102, a movement detection sensor 103, a real object detection unit 104, a landmark determination unit 105, a landmark feature acquisition unit 106, a relative position acquisition unit 107, a landmark memory unit 108, and a landmark determination unit 109.

The imaging unit 101 is a camera that captures an image (field of view image, field of view information) corresponding to the user's field of view, and corresponds to the imaging unit 1606 in FIG. 16.

The display unit 102 is a display capable of displaying virtual objects in the user's field of vision, and corresponds to the display unit 1607 in FIG. 16.

The movement detection sensor 103 is a sensor that detects the movement and rotation of the HMD 100 (user's head) in real time, and corresponds to the movement detection sensor 1608 in FIG. 16. For example, the movement detection sensor 103 obtains information on six degrees of freedom, including forward/backward movement, left/right movement, up/down movement, yaw rotation, pitch rotation, and roll rotation. This makes it possible to know in which direction the user's head is facing in 3D space, and how far it has moved in the 3D space.

The real object detection unit 104, the landmark determination unit 105, the landmark feature acquisition unit 106, the relative position acquisition unit 107, and the landmark determination unit 109 correspond to (are realized by) the CPU 1601 in FIG. 16.

The real object detection unit 104 and the landmark determination unit 105 detect landmarks, which are real objects, from the field of view image. The real object detection unit 104 acquires a field of view image from the imaging unit 101, and detects real objects from the field of view image. The image processing for detecting real objects from the field of view image is, for example, image processing that uses artificial intelligence techniques such as machine learning and deep learning. The landmark determination unit 105 determines a landmark that will serve as a landmark for route guidance from one or more real objects detected by the real object detection unit 104.

The landmark feature acquisition unit 106 is a sensor that detects the features of the landmark determined by the landmark determination unit 105, and corresponds to the 3D shape measurement sensor 1609 in FIG. 16. The feature information (information representing the features of the landmark) acquired by the landmark feature acquisition unit 106 is not particularly limited. As the feature information, 3D shape information (for example, the above-mentioned point cloud information) representing the 3D shape (three-dimensional shape) of the landmark may be acquired. As the feature information, 2D shape information representing the 2D shape (two-dimensional shape) of the landmark may be acquired. As the feature information, one of the 2D shape information and the 3D shape information may be acquired, or both may be acquired. In the first embodiment, it is assumed that both the 2D shape information and the 3D shape information are acquired as the feature information. The feature information may be identification information (for example, a building name) that can identify the landmark. One or both of the feature information and the identification information may be acquired.

The relative position acquisition unit 107 acquires information on the user's relative position (current relative position) with respect to a specific position based on the information (movement and rotation information) acquired by the movement detection sensor 103. For example, the specific position is the user's position when use of the HMD 100 begins, or the user's position when a landmark is first registered (detected) after use of the HMD 100 begins. Hereinafter, the relative position with respect to the specific position will simply be described as the relative position.

Landmarks (e.g., 3D shapes of landmarks) are registered in the landmark storage unit 108. In the first embodiment, when a detected landmark is registered, the relative position of the user is associated with the landmark and registered. For this reason, the landmark storage unit 108 stores characteristic information of the landmark and relative position information of the user (relative position information). The landmark storage unit 108 corresponds to the RAM 1603 in FIG. 16.

The landmark determination unit 109 determines whether or not a registered landmark is present in the user's field of view (current field of view). This determination is made, for example, based on information stored in the landmark memory unit 108, the processing result of the real object detection unit 104, and the processing result of the relative position acquisition unit 107. This determination can be considered as a determination of whether or not a registered landmark has been detected, or as a determination of whether or not a detected real object (landmark) has been registered. When the landmark determination unit 109 determines that a registered landmark is present in the user's field of view, it controls the display unit 102 so that a virtual object is displayed in association with the landmark in the user's field of view.

The HMD 100 may be divided into multiple devices. For example, as shown in FIG. 1B, the HMD 100 may be transformed into a system having an HMD 110 and an information processing device 111.

Figure 2 is a flowchart showing the processing flow performed by the HMD 100.

In step S201, the real object detection unit 104 detects real objects from the field of view image (the user's current field of view) acquired by the imaging unit 101.

In step S202, the landmark determination unit 109 determines whether or not a registered landmark is present in the user's current field of view. Details of this determination will be described later. If it is determined that a registered landmark is present in the user's current field of view, processing proceeds to step S203, and if it is determined that a registered landmark is not present in the user's current field of view, processing proceeds to step S204.

In step S203, the landmark determination unit 109 controls the display unit 102 so that the virtual object is displayed in association with a registered landmark that exists in the user's current field of view. However, immediately after starting to use the HMD 100, there are no registered landmarks. Therefore, in this example, the process proceeds from step S202 to step S204.

In step S204, the landmark determination unit 105 determines whether the current timing is the registration timing (timing to register a landmark). For example, the landmark determination unit 105 determines that the current timing is the registration timing immediately after the use of the HMD 100 is started in order to set the position where the use of the HMD 100 is started as the starting point of the route. Thereafter, the landmark determination unit 105 determines whether the current timing is the registration timing in accordance with a predetermined registration rule. For example, the landmark determination unit 105 determines whether the current timing is the registration timing in accordance with at least one of the following multiple registration rules. The user may be able to select one or more registration rules to be used from the multiple registration rules.
- Registering landmarks at a specified time interval - Registering a landmark every time the user moves a specified distance - Registering a landmark when the user's direction of travel changes by an amount greater than a threshold - When a distinctive real object (a real object that meets specified conditions) is detected, that real object is registered as a landmark - Registering a new landmark when the user is likely to lose sight of the previously registered landmark

The user may be able to specify the registration timing (landmarks may be registered at a timing specified by the user). Furthermore, the landmark determination unit 105 may adjust the registration rules according to the characteristics of the user. The landmark determination unit 105 may set athletic abilities such as the level of mountain climbing experience and stride length by having the user input and adjust the registration rules according to the set athletic abilities. For example, the landmark determination unit 105 may adjust the registration rules so that landmarks are registered at longer time intervals as the stride length becomes shorter. The landmark determination unit 105 may adjust the registration rules according to the location where the HMD 100 is used, such as urban areas and mountainous areas.

If it is determined that the current timing is not the registration timing, the process returns to step S201, and if it is determined that the current timing is the registration timing, the process proceeds to step S205.

In step S205, the landmark determination unit 105 determines a landmark from among one or more real objects detected in step S201. It is desirable to store all real objects in the user's current field of view as landmarks, but there are constraints due to the computing power and storage capacity of the HMD 100. For this reason, the landmark determination unit 105 may determine only some real objects as landmarks. For example, the landmark determination unit 105 excludes real objects whose positions may change (e.g., people, animals, vehicles such as automobiles and bicycles, and luggage carried by humans) from the one or more real objects detected in step S201. Then, the landmark determination unit 105 determines a landmark from among the remaining real objects. Note that it is also possible to prevent real objects whose positions may change from being detected in step S201.

In the first embodiment, the landmark determination unit 105 determines (detects) landmarks according to a predetermined determination rule. For example, the landmark determination unit 105 stores priority information as shown in FIG. 3 in advance. The priority information is information indicating the priority of a real object. In the example of FIG. 3, a priority is determined for each combination of the type and size of a real object. In addition, in the example of FIG. 3, it is also determined that one man-made object of "medium" size cannot be determined as a landmark. In the case of a man-made object of "medium" size, multiple man-made objects can be determined as one landmark. Then, the landmark determination unit 105 determines a predetermined number or less of real objects with higher priorities as landmarks from among one or more detected real objects (real objects remaining after excluding real objects whose positions may change). For example, when the number of real objects is a predetermined number or more, the predetermined number of real objects are determined as landmarks, and when the number of real objects is less than the predetermined number, all real objects are determined as landmarks. All real objects with a priority higher than a threshold value may be determined as landmarks.

The landmark determination unit 105 may adjust the decision rules (e.g., priority or granularity of the types of real objects) according to the characteristics of the user. The landmark determination unit 105 may set the athletic ability such as the level of mountain climbing experience and stride length by having the user input, and adjust the decision rules according to the set athletic ability. For example, the landmark determination unit 105 may adjust the decision rules so that the less experience a user has in mountain climbing, the more finely granular the types of real objects are classified. The landmark determination unit 105 may adjust the registration rules according to the location where the HMD 100 is used, such as urban areas and mountainous areas.

The landmark determination unit 105 may also determine a real object selected by the user as a landmark. In that case, for example, as shown in FIG. 4, the display unit 102 is controlled so that a virtual object (a virtual object emphasizing each real object, a dashed frame) indicating each real object is displayed. Then, as shown in FIG. 5, the user selects a real object using a hand ray 502 and a cursor 503 that can be operated with the user's hand 501 in the augmented reality space. The hand ray 502 is a virtual object like a ray of light emitted from the fingertip, and the cursor 503 is a virtual object displayed at the tip of the hand ray 502 (for example, the position where the hand ray 502 hits the real object).

In step S206, the landmark feature acquisition unit 106 determines whether or not a landmark was determined in step S205. If it is determined that a landmark was not determined, the process returns to step S201, and if it is determined that a landmark was determined, the process proceeds to step S207.

In step S207, the landmark feature acquisition unit 106 registers the landmark determined in step S206 in the landmark storage unit 108. For example, the landmark feature acquisition unit 106 acquires feature information of the landmark determined in step S206 (e.g., landmark type, size, 2D shape information, and 3D shape information) and records it in the landmark storage unit 108. In addition, the relative position acquisition unit 107 acquires the user's current relative position information (information of the current relative position) and records it in the landmark storage unit 108. The feature information and the relative position information are recorded in association with each other. As a result, the landmark storage unit 108 stores information (landmark information) as shown in FIG. 6.

Next, we explain what happens when a registered landmark is in the user's current field of view.

As described above, in step S202, the landmark determination unit 109 determines whether or not a registered landmark is present in the user's current field of view. This determination is made based on the landmark information stored in the landmark storage unit 108, the processing result of the real object detection unit 104, and the processing result of the relative position acquisition unit 107. For example, the landmark determination unit 109 extracts (picks up) one or more landmarks around the user from the landmark information (registered landmarks) as shown in FIG. 6 based on the user's current relative position. Then, the landmark determination unit 109 compares the extracted one or more landmarks with one or more real objects detected in step S201 to determine whether or not a registered landmark is present in the user's current field of view.

For example, the landmark feature acquisition unit 106 acquires feature information of the real object detected in step S201. Then, the landmark determination unit 109 matches the feature information of the real object detected in step S201 with the feature information of the extracted landmark. If there is a landmark whose feature information is equivalent to the feature information of the real object detected in step S201 among the one or more extracted landmarks, the landmark determination unit 109 determines that the registered landmark is present in the user's current field of view. At this time, the landmark determination unit 109 determines that the real object detected in step S201 is the landmark whose feature information is equivalent (most similar) to the feature information of the real object among the one or more extracted landmarks. Then, the process proceeds to step S203. If there is no landmark whose feature information is equivalent to the feature information of the real object detected in step S201 among the one or more extracted landmarks, the landmark determination unit 109 determines that the registered landmark is not present in the user's current field of view. Then, the process proceeds to step S204.

In step S203, the landmark determination unit 109 controls the display unit 102 so that a virtual object is displayed in association with a registered landmark that exists in the user's current field of view. For example, as shown in FIG. 7, virtual objects 701 and 702 (ribbon-shaped icons) are displayed for a real object that has been determined to be a registered landmark. The virtual objects 701 and 702 indicate that the real object is a landmark. Note that the virtual object is not limited to the ribbon-shaped icon, and may be another virtual object that emphasizes the real object (for example, a star-shaped icon, the outline of a real object, or a frame surrounding a real object).

According to the configuration described above, even in a situation where data communication and GPS signal reception are not possible, the user can easily retrace his/her steps by relying on the landmarks indicated by the virtual objects. This in turn reduces the possibility of the user getting lost and stranded.

Note that when a user realizes that he or she is lost and uses a landmark to return to the normal route, the time (date and time, registration time) at which the landmark was registered may be important. Therefore, as shown in FIG. 8(A), a time acquisition unit 812 that acquires information on the current time may be added to the HMD 100. The time acquisition unit 812 corresponds to the CPU 1601 in FIG. 16. When a detected landmark is registered, the current time (registration time) may be associated with the landmark and registered. In this way, the landmark storage unit 108 stores information (landmark information) as shown in FIG. 9. As shown in FIG. 10, virtual objects 1001 and 1002 may be displayed for real objects that have been determined to be registered landmarks. The virtual objects 1001 and 1002 have the registration times of the landmarks drawn based on the landmark information (the registration times of the landmarks). The registration times of the landmarks can also be considered as the time at which the landmarks were first detected. As shown in FIG. 8(B), a time acquisition unit 812 may be added to the information processing device 111.

Second Embodiment
A second embodiment of the present invention will be described. In the first embodiment, an example of displaying a virtual object that serves as a landmark for a real object on the way of a route will be described. In the second embodiment, an example of displaying a virtual object that represents a route taken by a user will be described. Hereinafter, the route taken by the user will be referred to as a travel route. Note that, in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and a description will be given of points that are different from the first embodiment.

Fig. 11(A) is a block diagram showing the configuration of an HMD 1100 according to the second embodiment. The HMD 1100 has a configuration in which a movement path acquisition unit 1112, a movement path storage unit 1113, a path object generation and display unit 1114, and an operation mode setting unit 1115 are added to the configuration of the HMD 100 according to the first embodiment. The movement path acquisition unit 1112, the movement path storage unit 1113, the path object generation and display unit 1114, and the operation mode setting unit 1115 correspond to the CPU 1601 in Fig. 16.

The movement path acquisition unit 1112 acquires the user's movement path information (information on the user's movement path) based on the relative position information acquired by the relative position acquisition unit 107. The movement path storage unit 1113 stores the movement path information acquired by the movement path acquisition unit 1112. The path object generation and display unit 1114 generates a virtual object (path object) representing the user's movement path based on the movement path information stored in the movement path storage unit 1113. Then, the path object generation and display unit 1114 controls the display unit 102 so that the path object is displayed in the user's field of vision. The operation mode setting unit 1115 sets the path recording mode or the path display mode as the operation mode of the HMD 1100. For example, the operation mode setting unit 1115 sets the path recording mode or the path display mode in response to an instruction from the user. The route recording mode is a mode in which movement path information is recorded, and the route display mode is a mode in which a path object is displayed.

Note that, like the HMD 100, the HMD 1100 may be divided into multiple devices. For example, as shown in FIG. 11(B), the HMD 1100 may be modified into a system having an HMD 1110 and an information processing device 1111.

Figure 12 is a flowchart showing the processing flow performed by the HMD 1100.

In step S1201, similar to S201 in FIG. 2, the real object detection unit 104 detects a real object from the field of view image (the user's current field of view) acquired by the imaging unit 101.

In step S1202, the operation mode setting unit 1115 determines whether the current operation mode is the route recording mode or the route display mode. If it is determined that the current operation mode is the route recording mode, processing proceeds to step S1203, and if it is determined that the current operation mode is the route display mode, processing proceeds to step S1208. For example, when moving toward a destination, the user instructs the HMD 1100 to set the route recording mode. Then, when the user realizes that he or she is lost and wants to retrace his or her steps, the user instructs the HMD 1100 to set the route display mode.

First, we will explain the process in route recording mode.

In step S1203, similarly to step S204 in Fig. 2, the landmark determination unit 105 determines whether or not the current timing is the registration timing (timing to register a landmark). In the second embodiment, in order to register a landmark when the user gets lost, the following timings may be used as the registration timing.
The timing when the operation mode is switched to the route display mode. The timing when it is determined in the route display mode (step S1210 described later) that the user has deviated from his/her own movement route.

If it is determined that the current timing is not the registration timing, processing proceeds to step S1204, and if it is determined that the current timing is the registration timing, processing proceeds to step S1205.

In step S1204, the travel route acquisition unit 1112 acquires travel route information between the previously registered landmark and the landmark to be registered next, based on the current relative position information acquired by the relative position acquisition unit 107. The travel route acquisition unit 1112 records the travel route information from the previously registered landmark in the travel route storage unit 1113, and repeatedly updates the travel route information until the next landmark is registered. The travel route information is, for example, a group of information obtained by sampling relative coordinate information at a predetermined period. As a result, the travel route storage unit 1113 stores information such as that shown in FIG. 13 (travel route information for each section).

In step S1205, similar to step S205 in FIG. 2, the landmark determination unit 105 determines a landmark from among one or more real objects detected in step S1201.

In step S1206, similar to step S206 in FIG. 2, the landmark feature acquisition unit 106 determines whether or not a landmark was determined in step S1205. If it is determined that a landmark was not determined, the process returns to step S1201, and if it is determined that a landmark was determined, the process proceeds to step S1207.

In step S1207, similar to step S207 in FIG. 2, the landmark feature acquisition unit 106 registers the landmark determined in step 1205 in the landmark storage unit 108.

Next, we will explain the processing in route display mode.

In step S1208, similar to step S202 in FIG. 2, the landmark determination unit 109 determines whether or not the registered landmark is present in the user's current field of view. If it is determined that the registered landmark is present in the user's current field of view, processing proceeds to step S1209, and if it is determined that the registered landmark is not present in the user's current field of view, processing proceeds to step S1210.

In step S1209, similar to step S203 in FIG. 2, the landmark determination unit 109 controls the display unit 102 so that a virtual object is displayed in association with a registered landmark that exists in the user's current field of view. For example, as shown in FIG. 14, a virtual object 1401 (a ribbon-shaped icon) is displayed for a real object that has been determined to be a registered landmark.

In step S1210, the route object generation and display unit 1114 compares the current relative position information acquired by the relative position acquisition unit 107 with the information stored in the movement route storage unit 1113 to determine the user's position. Here, it is determined whether the user is on his/her own movement route. If the user is on his/her own movement route, the section in which the user is located is identified from among multiple sections corresponding to the multiple pieces of movement route information.

For example, the route object generating and displaying unit 1114 selects a relative position that is closest to the user's current relative position from among multiple relative positions stored in the moving route storage unit 1113. Then, the route object generating and displaying unit 1114 determines that the user is on his/her own moving route when the difference between the selected relative position and the user's current relative position is equal to or smaller than a threshold, and determines that the user has deviated from his/her own moving route when the difference is greater than the threshold. When the difference between the selected relative position and the user's current relative position is equal to or smaller than a threshold, the route object generating and displaying unit 1114 identifies a section associated with the selected relative position (a moving route including the selected relative position) as a section in which the user is located.

If it is determined that the user is on his/her own travel route, processing proceeds to step S1211; if it is determined that the user has deviated from his/her own travel route, processing proceeds to step S1212.

In step S1211, the route object generation and display unit 1114 controls the display unit 102 based on the travel route information of the section where the user is located so that a virtual object (route object) representing the travel route of the section is displayed in the user's field of vision. For example, as shown in FIG. 14, a virtual object 1402 (curved arrow) is displayed. The virtual object 1401 is a virtual object displayed in association with a registered landmark that exists in the user's current field of vision, and the virtual object 1402 can also be considered as a virtual object displayed in association with the landmark. The virtual object may be displayed semi-transparently so as not to impede the user's walking, and the user may be able to switch between displaying and hiding the virtual object.

In step S1212, the route object generation and display unit 1114 controls the display unit 102 to provide a predetermined notification (warning that the user has deviated from the travel route). For example, the route object generation and display unit 1114 controls the display unit 102 to display a warning. This makes it possible to prompt the user to retrace his or her steps. Suppose that the user strays from the travel route of the first section, is warned, and attempts to retrace his or her steps. At this time, a landmark is registered, and the travel route of the second section that is connected to the first section is displayed. This allows the user to easily retrace his or her steps to the travel route of the first section.

After step S1211 or step S1212 is performed, processing proceeds to step S1203, where landmarks are registered and travel route information is obtained, as in the case of the route recording mode.

According to the above-mentioned configuration, the user's travel route is displayed, so the user can more easily retrace his or her travel route. This further reduces the possibility of the user getting lost or stranded.

The route object generation and display unit 1114 may generate a virtual object as shown in FIG. 15 and control the display unit 102 so that the virtual object is displayed in the user's field of vision. The virtual object in FIG. 15 is a map representing a plurality of landmarks and the user's movement route between the plurality of landmarks. The virtual object in FIG. 15 can be generated based on landmark information (information on a plurality of registered landmarks) stored in the landmark memory unit 108 and information (movement route information for each section) stored in the movement route memory unit 1113.

Although an example in which an image whose entirety corresponds to the user's field of view has been described, an image whose part corresponds to the user's field of view may be used. For example, a spherical image captured in a 360° direction from the user may be used. A plurality of images including an image whose entirety or part corresponds to the user's field of view may be used. For example, an image captured behind the user, an image captured to the right of the user, and an image captured to the left of the user may be used. By using these images, it is possible to improve the accuracy of matching feature information of real objects that look different depending on the direction (for example, real objects that look different on the way there and back).

Although an example of registering landmarks has been described, the landmarks may be registered in advance. For example, data on mountain climbing courses may be obtained from the Internet and stored in the information processing device (HMD) in advance. In this way, the user can proceed along the correct route from the beginning (before getting lost) by relying on the landmarks or virtual objects representing the correct route.

The above embodiment (including modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above embodiments.

<Other embodiments>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The disclosure of this embodiment includes the following configuration, method, and program.
(Configuration 1)
An acquisition means for acquiring an image corresponding to a user's field of view;
A detection means for detecting a landmark, which is a real object, from the image;
A registration means for registering the detected landmarks;
and a control means for controlling, when a registered landmark is detected, to display a virtual object in association with the landmark in the field of view.
(Configuration 2)
2. The information processing apparatus according to configuration 1, wherein the registration means registers the detected landmark at a timing designated by a user.
(Configuration 3)
3. The information processing apparatus according to configuration 1 or 2, wherein the registration means registers the detected landmark at a timing according to a predetermined rule.
(Configuration 4)
4. The information processing apparatus according to configuration 3, wherein the registration means registers the detected landmarks at predetermined time intervals.
(Configuration 5)
5. The information processing apparatus according to claim 3, wherein the registration means registers a detected landmark every time the user moves a predetermined distance.
(Configuration 6)
The information processing device according to any one of configurations 3 to 5, wherein the registration means registers the detected landmark at a timing when the user's traveling direction changes by an amount of change greater than a threshold value.
(Configuration 7)
7. The information processing apparatus according to any one of configurations 3 to 6, wherein the registration means adjusts the predetermined rule in accordance with the characteristics of the user.
(Configuration 8)
The information processing device described in any one of configurations 1 to 7, characterized in that the detection means detects one or more real objects from the image and determines, among the one or more real objects, a real object selected by a user as a landmark.
(Configuration 9)
9. The information processing device according to any one of configurations 1 to 8, wherein the detection means detects landmarks according to a predetermined rule.
(Configuration 10)
Priorities of real objects are determined in advance;
The information processing device according to configuration 9, characterized in that the detection means detects one or more real objects from the image, and determines as landmarks a predetermined number or less of the one or more real objects having a higher priority from among the one or more real objects.
(Configuration 11)
11. The information processing device according to configuration 9 or 10, wherein the detection means adjusts the predetermined rule in accordance with a characteristic of the user.
(Configuration 12)
12. The information processing device according to any one of configurations 1 to 11, wherein the registration means registers a three-dimensional shape of the detected landmark.
(Configuration 13)
The device further includes a time acquisition unit for acquiring time information,
The registration means registers a time in association with the detected landmark when registering the landmark,
The information processing device according to any one of configurations 1 to 12, characterized in that when a registered landmark is detected, the control means controls to display a virtual object in which the registration time of the landmark is drawn, associated with the landmark in the field of view.
(Configuration 14)
The method further includes a position acquisition means for acquiring information on a relative position of the user with respect to a specific position,
The information processing device according to any one of configurations 1 to 13, wherein the registration means, when registering a detected landmark, registers a relative position of the user with respect to the specific position in association with the landmark.
(Configuration 15)
The control means
Extracting one or more landmarks in the vicinity of the user from a plurality of registered landmarks based on a relative position of the user with respect to the specific position;
15. The information processing device according to configuration 14, characterized in that the extracted one or more landmarks are compared with a detected landmark to determine whether the detected landmark has been registered.
(Configuration 16)
16. The information processing device according to configuration 14 or 15, wherein the specific location is the location of the user when use of the information processing device is started.
(Configuration 17)
16. The information processing device according to configuration 14 or 15, wherein the specific location is the location of the user when a landmark is first registered after use of the information processing device is started.
(Configuration 18)
The method further comprises: recording means for recording information on a route taken by the user based on a relative position of the user with respect to the specific position;
The information processing device according to any one of configurations 14 to 17, characterized in that the control means controls to display a second virtual object representing the route in the field of view based on the recorded information of the route.
(Configuration 19)
The device further includes a means for setting a route recording mode or a route display mode,
The recording means records information about the route in the route recording mode,
19. The information processing apparatus according to configuration 18, wherein the control means controls so as to display the second virtual object in the route display mode.
(Configuration 20)
The control means
comparing the location of the user relative to the particular location with the route to determine whether the user has deviated from the route;
20. The information processing device according to configuration 18 or 19, characterized in that, when it is determined that the user has deviated from the route, a predetermined notification is given to the user.
(Configuration 21)
The information processing device according to any one of configurations 18 to 20, wherein the control means controls to display a third virtual object, which is a map representing a plurality of registered landmarks and the route, in the field of view based on information of the plurality of registered landmarks and information of the recorded route.
(Method)
acquiring an image corresponding to a user's field of view;
detecting landmarks, which are real objects, from the image;
registering the detected landmarks;
and when a registered landmark is detected, controlling so as to display a virtual object in association with the landmark in the field of view.
(program)
A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 21.

104: Real object detection unit 105: Landmark determination unit 106: Landmark feature acquisition unit 109: Landmark determination unit

Claims (23)

An acquisition means for acquiring an image corresponding to a user's field of view;
A detection means for detecting a landmark, which is a real object, from the image;
A registration means for registering the detected landmarks;
and a control means for controlling, when a registered landmark is detected, to display a virtual object in association with the landmark in the field of view. 2. The information processing apparatus according to claim 1, wherein the registration means registers the detected landmark at a timing designated by a user. 2. The information processing apparatus according to claim 1, wherein the registration means registers the detected landmark at a timing according to a predetermined rule. 4. The information processing apparatus according to claim 3, wherein said registration means registers the detected landmarks at predetermined time intervals. 4. The information processing apparatus according to claim 3, wherein the registration means registers the detected landmark every time the user moves a predetermined distance. The information processing apparatus according to claim 3 , wherein the registration means registers the detected landmark at a timing when the moving direction of the user changes by an amount of change larger than a threshold value. 4. The information processing apparatus according to claim 3, wherein said registration means adjusts said predetermined rule in accordance with a characteristic of said user. The information processing apparatus according to claim 1 , wherein the detection means detects one or more real objects from the image, and determines, as the landmark, a real object selected by a user from the one or more real objects. 2. The information processing apparatus according to claim 1, wherein said detection means detects the landmarks in accordance with a predetermined rule. Priorities of real objects are determined in advance;
The information processing apparatus according to claim 9 , wherein the detection means detects one or more real objects from the image, and determines, as landmarks, a predetermined number or less of the one or more real objects having a higher priority from among the one or more real objects. 10. The information processing apparatus according to claim 9, wherein the detection means adjusts the predetermined rule in accordance with a characteristic of the user. 2. The information processing apparatus according to claim 1, wherein said registration means registers a three-dimensional shape of the detected landmark. The device further includes a time acquisition unit for acquiring time information,
The registration means registers a time in association with the detected landmark when registering the landmark,
The information processing device according to claim 1 , characterized in that, when a registered landmark is detected, the control means controls to display a virtual object in which the registration time of the landmark is drawn, in association with the landmark in the field of view. The method further includes a position acquisition means for acquiring information on a relative position of the user with respect to a specific position,
The information processing apparatus according to claim 1 , wherein, when registering the detected landmark, the registration means registers the relative position of the user with respect to the specific position in association with the landmark. The control means
Extracting one or more landmarks in the vicinity of the user from a plurality of registered landmarks based on a relative position of the user with respect to the specific position;
15. The information processing apparatus according to claim 14, further comprising: comparing the one or more extracted landmarks with a detected landmark to determine whether the detected landmark has been registered. 15. The information processing apparatus according to claim 14, wherein the specific location is the location of the user when use of the information processing apparatus is started. The information processing apparatus according to claim 14 , wherein the specific position is a position of the user when a landmark is first registered after use of the information processing apparatus is started. The method further comprises: recording means for recording information on a route taken by the user based on a relative position of the user with respect to the specific position;
The information processing apparatus according to claim 14 , wherein the control means performs control so as to display a second virtual object representing the route in the field of view based on the recorded information of the route. The device further includes a means for setting a route recording mode or a route display mode,
The recording means records information about the route in the route recording mode,
19. The information processing apparatus according to claim 18, wherein the control means performs control so as to display the second virtual object in the route display mode. The control means
comparing the location of the user relative to the particular location with the route to determine whether the user has deviated from the route;
20. The information processing apparatus according to claim 18, further comprising control for issuing a predetermined notification to the user when it is determined that the user has deviated from the route. 19. The information processing device according to claim 18, wherein the control means controls to display a third virtual object, which is a map representing a plurality of registered landmarks and the route, in the field of view based on information of the plurality of registered landmarks and information of the recorded route. acquiring an image corresponding to a user's field of view;
detecting landmarks, which are real objects, from the image;
registering the detected landmarks;
and when a registered landmark is detected, controlling so as to display a virtual object in association with the landmark in the field of view. A program for causing a computer to function as each of the means of an information processing device according to any one of claims 1 to 21.

Images