JP2021144010A - Portable terminal, navigation method, and computer program - Google Patents

Abstract

To enable navigation in places where it is difficult to receive signals from positioning satellites.SOLUTION: A portable terminal equipped with an imaging device acquires an imaged video having been imaged by the imaging device, specifies the reference bearing and position of the imaging device at navigation start time on the basis of the imaged video in which an installed object having been installed at a prescribed position is imaged, specifies a route from the position of the installed object to a destination, detects the bearing change amount and movement amount of the imaging device on the basis of the movement of a feature point with the passage of time extracted from the imaged video, specifies the current bearing and position of the imaging device on the basis of bearing change amount and movement amount of the imaging device detected by a motion detection unit after having specified the reference bearing and position of the imaging device, generates a navigation image that indicates a route on the basis of the specified current bearing and position of the imaging device, and displays the generated navigation image by superimposition on the current imaged video.SELECTED DRAWING: Figure 2

Description

The present disclosure relates to mobile terminals, navigation methods, and computer programs.

Patent Document 1 discloses a mobile terminal having a navigation function, which includes a camera and a display for displaying an image captured by the camera. The mobile terminal is provided by a first detecting means for detecting the current position of the mobile terminal, a second detecting means for detecting the direction of the camera, a setting means for setting a destination, and a setting means. Based on the set destination and the current position detected by the first detection means, the acquisition means for acquiring the information on the route from the current position to the destination from the server, the information on the route acquired by the acquisition means, and the information on the route acquired by the acquisition means. Based on the orientation of the camera detected by the second detection means, a calculation means for calculating the direction to go on the image captured by the camera and an index indicating the direction to go calculated by the calculation means on the image captured by the camera are provided. It is provided with a means for additionally displaying on a display.

Japanese Unexamined Patent Publication No. 2006-194665

However, since the mobile terminal disclosed in Patent Document 1 uses a positioning signal from a positioning satellite represented by a GPS satellite to determine the current position of the mobile terminal, the mobile terminal is located in a department store, a shopping mall, a station yard, or an underground shopping mall. In places where it is difficult to receive positioning signals such as, the current position cannot be measured and navigation is difficult.

An object of the present disclosure is to provide a technique that enables navigation in a place where it is difficult to receive a positioning signal from a positioning satellite.

A mobile terminal including the image pickup device according to the present disclosure is navigated based on an image acquisition unit that acquires an image captured by the image pickup device and the image capture image of an installation object installed at a predetermined position. The time lapse of the reference specifying unit that specifies the reference orientation and position of the imaging device at the start, the route specifying unit that specifies the route from the position of the installation object to the destination, and the feature points extracted from the captured image. A motion detection unit that detects the amount of change in direction and the amount of movement of the image pickup device based on the movement accompanying the image pickup device, and the image pickup device that is detected by the motion detection unit after specifying the reference orientation and position of the image pickup device. A position specifying unit that specifies the current direction and position of the image pickup device based on the amount of change in direction and the amount of movement, and a navigation image showing the route based on the current direction and position of the specified image pickup device are generated. A navigation display unit is provided, which superimposes and displays the generated navigation image on the current captured image.

In the navigation method executed by the mobile terminal including the imaging device according to the present disclosure, the mobile terminal equipped with the imaging device acquires the captured image captured by the imaging device, and the installed object installed at a predetermined position captures the image. Based on the captured image, the reference orientation and position of the image pickup device at the start of navigation are specified, the route from the position of the installation object to the destination is specified, and the feature points extracted from the captured image. The amount of change in direction and the amount of movement of the image pickup device are detected based on the movement over time, the reference direction and position of the image pickup device are specified, and then the change in direction of the image pickup device detected by the motion detection unit. The current orientation and position of the imaging device are specified based on the amount and the amount of movement, and a navigation image showing the route is generated based on the identified current orientation and position of the imaging device, and the generated navigation The image is superimposed and displayed on the current captured image.

The computer program executed by the mobile terminal including the imaging device according to the present disclosure acquires the captured image captured by the imaging device, and is based on the captured image captured by the installation object installed at a predetermined position. , The reference orientation and position of the imaging device at the start of navigation, the route from the position of the installation object to the destination, and the movement of the feature points extracted from the captured image over time. , The amount of change in direction and the amount of movement of the image pickup device are detected, the reference direction and position of the image pickup device are specified, and then the amount of change in direction and the amount of movement of the image pickup device detected by the motion detection unit are used. The current orientation and position of the imaging device are specified, a navigation image showing the route is generated based on the identified current orientation and position of the imaging device, and the generated navigation image is used as the current captured image. The mobile terminal is made to execute the process of superimposing and displaying.

It should be noted that these comprehensive or specific aspects may be realized by a system, a device, a method, an integrated circuit, a computer program or a recording medium, and any of the system, a device, a method, an integrated circuit, a computer program and a recording medium. It may be realized by various combinations.

According to the present disclosure, navigation is possible even in a place where it is difficult to receive a signal from a positioning satellite.

The figure for demonstrating the outline of AR (Augmented Reality) navigation which concerns on Embodiment 1. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 1. Diagram showing an example of shooting a mark on a digital signage with a camera Diagram showing a display example of AR navigation Flowchart showing an example of processing at the start of AR navigation Flowchart showing an example of AR navigation processing Diagram for explaining further functions of the navigation display Diagram for explaining an example of how to update the target node The figure which shows the configuration of the mobile terminal which concerns on Embodiment 2 and the example of the mark detection processing at the start of navigation. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 2 and the continuation example of FIG. 9 of the mark detection processing at the start of navigation. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 2 and the example of the regeneration processing of the navigation image with the movement of a user. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 2 and the example of processing when a user deviates from a route. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 2 and the example of the process which displays caution information when a user deviates from a route. The figure which shows the configuration example of the mobile terminal which concerns on Embodiment 2 and the example of the processing when the user who deviated from the route returns to the route. The figure which shows the hardware composition of the computer which realizes the function of the mobile terminal which concerns on this disclosure by a program.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed explanation than necessary may be omitted. For example, detailed explanations of already well-known matters and duplicate explanations for substantially the same configuration may be omitted. This is to avoid unnecessary redundancy of the following description and to facilitate the understanding of those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter of the claims.

(Embodiment 1)
<Overview of AR navigation>
FIG. 1 is a diagram for explaining an outline of AR (Augmented Reality) navigation according to the first embodiment.

In places where it is difficult to receive positioning signals transmitted from positioning satellites, such as in department stores, shopping malls, train stations, and underground malls, the mobile terminal 100 cannot determine the current position, so it is not possible to navigate user 2 to the destination. difficult. The present embodiment describes the mobile terminal 100 capable of navigating the user 2 to the destination even in a place where it is difficult to receive the positioning signal transmitted from the positioning satellite.

In the first embodiment, an electronic signboard 10 capable of transmitting an optical signal 12 typified by LinkRay (registered trademark) is installed at a predetermined place in a department store, a shopping mall, a station yard, an underground shopping mall, or the like. For example, the electronic signboard 10 capable of transmitting the optical signal 12 can transmit predetermined information (hereinafter referred to as "optical signal information") by modulating the blinking of the light source of the LED (Light Emitting Diode).

The user 2 who owns the mobile terminal 100 provided with the camera 90 points the camera 90 at the digital signage 10. The mobile terminal 100 receives optical signal information from the digital signage 10 through the camera 90. The optical signal information includes an ID (Identification) of the digital signage 10.

The mobile terminal 100 transmits an ID to a predetermined server 20 through a wireless communication network such as a mobile communication network (for example, LTE, 4G, 5G) or WiFi (registered trademark). Then, the mobile terminal 100 receives the route information 202C (see FIG. 2) indicating the route 30 for reaching a predetermined destination from the server 20 starting from the location of the digital signage 10 indicated by the ID. As a result, the mobile terminal 100 can specify the departure place without receiving the positioning signal from the positioning satellite. Examples of predetermined destinations are station ticket gates, transfer destination stations, bus stations, underground mall entrances and exits, stores displayed in advertisements for digital signage 10, and the like.

The mobile terminal 100 executes AR navigation from the departure point to the destination based on the received route information 202C. In AR navigation, the mobile terminal 100 visualizes the route information 202C with respect to the image captured by the image sensor of the camera 90 (hereinafter referred to as “image captured image”) 201 (see FIG. 2), and the navigation image 205 (FIG. 2, FIG. (See FIG. 4) is superimposed and displayed on the display.

In AR navigation, since the navigation image 205 is superimposed and displayed in accordance with the captured image 201 of the actual landscape, the user 2 can arrive at the destination without hesitation as compared with the case of viewing the route displayed on the two-dimensional map. Hereinafter, a detailed description will be given.

<Configuration of mobile terminal>
FIG. 2 is a diagram showing a configuration example of the mobile terminal 100 according to the present embodiment. FIG. 3 is a diagram showing an example in which the mark 11 of the digital signage 10 is photographed by the camera 90. FIG. 4 is a diagram showing a display example of AR navigation. Next, the configuration of the mobile terminal 100 will be described with reference to FIGS. 2 to 4.

As shown in FIG. 2, the mobile terminal 100 includes a control unit 110 and a storage unit 200. The control unit 110 is realized by the CPU 1003 (and GPU 1004) shown in FIG. The storage unit 200 is realized by the RAM 1006 and the storage device 1007 shown in FIG. The control unit 110 can write information to the storage unit 200 and read information from the storage unit 200.

The control unit 110 includes a video acquisition unit 111, an optical signal reception unit 112, an information acquisition unit 113, a destination list display unit 114, a reference identification unit 115, a route identification unit 116, a motion detection unit 117, a position identification unit 118, and the like. The navigation display unit 119 is included.

The image acquisition unit 111 acquires the image captured image 201 captured by the image sensor of the camera 90 from the camera 90, which is an example of the image pickup device 1010 (see FIG. 15), and writes it in the storage unit 200.

The optical signal receiving unit 112 receives the optical signal 12 represented by LinkRay. For example, the optical signal receiving unit 112 detects the optical signal 12 from the captured image 201 obtained by capturing the electronic signboard 10, and extracts the optical signal information from the optical signal 12. The optical signal information includes at least an ID.

The information acquisition unit 113 transmits an ID to the server 20, and receives the electronic signage information 202A, the destination list information 202B, and the route information 202C corresponding to the ID from the server 20. The digital signage information 202A includes information regarding the orientation and position of the digital signage 10. In addition, "direction" may be read as other terms such as "direction", "direction" or "direction". In addition, "position" may be read as other terms such as "location", "coordinates" or "longitude and latitude". The destination list information 202B includes information on at least one destination starting from the position of the digital signage 10. For example, the destination list information 202B includes information indicating the name of the destination, the position of the destination, and the distance from the departure point to the destination. The route information 202C includes information indicating a route from the starting point to each destination.

The destination list display unit 114 displays a list of destinations based on the destination list information 202B. User 2 can select one desired destination from the displayed list of destinations.

The reference specifying unit 115 is based on the distortion and / or size of the square mark 11 included in the captured image 201, and is the reference direction (hereinafter referred to as “reference direction”) and position (hereinafter referred to as “reference direction”) of the camera 90 at the start of navigation. "Reference position") is specified. Then, the reference specifying unit 115 writes the information 203 indicating the reference direction and the reference position of the specified camera 90 to the storage unit 200.

The reference specifying unit 115 may specify the reference orientation based on the orientation of the camera 90 in which the optical axis of the camera 90 is perpendicular to the display surface (the surface perpendicular to the floor surface) of the electronic signage 10. For example, as shown in FIG. 3, the reference specifying unit 115 uses the orientation of the camera 90 when the distortion of the square mark 11 in the captured image 201 is minimized as the reference orientation. Here, since the orientation of the display surface of the electronic signboard 10 is known from the electronic signage information 202A, the reference specifying unit 115 can specify the reference orientation by this method. For example, when the orientation of the display surface of the digital signage 10 is "south", the reference specifying unit 115 can specify the reference orientation as "north" facing the display surface of the electronic signage 10. The acquisition of the reference direction may be estimated by calculation from the distortion of the square mark 11. Since the viewing angle and lens distortion parameters of the camera 90 are known, the relative angle between the camera 90 and the mark 11 can be determined by the appearance of the mark 11 which should be a square and the calculation using these parameters. Can be calculated. Therefore, the reference direction can be obtained by correcting the current direction of the camera 90 with the calculated angle. Further, the shape of the mark 11 does not necessarily have to be square. However, since the distortion of the shape of the mark 11 is used to specify the reference direction, it is preferable to make the shape look different depending on the shooting angle.

Further, for example, as shown in FIG. 3, the reference specifying unit 115 uses the position of the camera 90 when the square mark 11 in the captured image 201 matches the predetermined frame 121 as the reference position. Here, since the position of the electronic signboard 10 is known from the electronic signage information 202A, the reference specifying unit 115 can specify the reference position by this method. For example, when it is known in advance that the square mark 11 in the captured image 201 matches the predetermined frame 121 at a position 1 m south of the position of the digital signage 10, the reference specifying unit 115 uses the reference. The position can be specified as a position 1 m south of the position of the digital signage 10.

The route identification unit 116 identifies the route from the departure point to the destination selected by the user 2 based on the route information 202C. A route may consist of a plurality of nodes indicating a starting point, a destination and a passing point, and an edge connecting the two nodes. Further, the route specifying unit 116 may update the target node, which is an intermediate target point on the route, at any time as the user 2 moves. The details of updating the target node will be described later (see FIG. 8).

As shown in FIG. 4, the motion detection unit 117 extracts a plurality of feature points 211 from the current (latest) captured image 201 by using a known technique for extracting the feature points 211 from the captured image 201. Then, the motion detection unit 117 detects the amount of change in direction and the amount of movement of the camera 90 based on the movement of the extracted feature points 211 with the passage of time. For example, when the feature point 211 moves to the left, the motion detection unit 117 detects that the direction of the camera 90 has turned to the right by the amount of the movement direction change. For example, when the feature point 211 moves downward, the motion detection unit 117 detects that the camera 90 has moved forward by the amount of the movement. When the mobile terminal 100 includes a 6-axis sensor, the amount of change in the orientation of the camera 90 may be calculated using the measured value of the 6-axis sensor.

The position specifying unit 118 is based on the amount of change in direction and the amount of movement of the camera 90 specified by the motion detection unit 117, and the current (latest) direction (hereinafter referred to as “current direction”) and position (hereinafter referred to as “current direction”) of the camera 90. "Position") is specified. Then, the position specifying unit 118 writes the information 204 indicating the current direction and the current position of the specified camera 90 to the storage unit 200.

For example, when the reference orientation of the camera 90 and the time at which the reference position is specified are set to t ₀ , the position specifying unit 118 sets the time based on the amount of change in the orientation and the amount of movement of the camera 90 after the lapse of Δt time from the time t _0. The current direction and the current position of the camera 90 at t ₁ (= t _{0 + Δt) are specified.} Similarly, the position specifying unit 118 determines the current direction and the current position of the camera 90 at _{time t 2} (= t ₁ + Δ t) based on the amount of change in direction and the amount of movement of the camera 90 after the lapse of Δt time from _{time t 1.} Identify. By repeating this, the position specifying unit 118 can specify the current direction and the current position of the camera 90 starting from the reference direction and the reference position. As a result, the mobile terminal 100 can specify the current direction and the current position of the camera 90 even in a place where it is difficult to receive the positioning signal from the positioning satellite.

The navigation display unit 119 generates a navigation image 205 showing a route based on the current direction and the current position of the camera 90. Then, the navigation display unit 119 superimposes the generated navigation image 205 on the current captured image 201 to generate and display an AR navigation image as shown in FIG.

For example, as shown in FIG. 4, the navigation display unit 119 detects a floor surface from the captured image 201 using a predetermined image analysis technique, and the navigation image 205 exists at a predetermined height from the detected floor surface. An image of AR navigation may be generated and displayed so that it can be seen. At this time, the navigation display unit 119 may detect the inclination of the camera 90 with respect to the floor surface (plane) by using the measured value of the 6-axis sensor. Further functions of the navigation display unit 119 will be described later (see FIG. 7).

<Operation of AR navigation>
FIG. 5 is a flowchart showing an example of processing at the start of AR navigation.

The user 2 points the camera 90 of the mobile terminal 100 toward the digital signage 10 (S101).

The optical signal receiving unit 112 receives the optical signal 12 from the electronic signboard 10 directed to the camera 90, and extracts the optical signal information including the ID from the optical signal 12 (S102).

The information acquisition unit 113 transmits the ID included in the received optical signal information to the server 20, and receives the electronic signage information 202A, the destination list information 202B, and the route information 202C corresponding to the ID from the server 20 (S103). ).

The destination list display unit 114 displays a list of destinations included in the destination list information 202B on the display (S104).

The user 2 selects one destination from the list of destinations displayed on the display, which is an example of the output device 1002 (see FIG. 15), via the touch panel, which is an example of the input device 1001 (see FIG. 15). Is selected (S105).

As shown in FIG. 3, the user 2 points the camera 90 at the mark 11 displayed on the digital signage 10, and moves the camera 90 back and forth and up and down so that the mark 11 in the captured image 201 fits in the frame 121. Move slowly (S106).

The reference specifying unit 115 specifies the reference direction and the reference position of the camera 90 based on the distortion and size of the mark 11 in the captured image 201 (S107).

The route identification unit 116 identifies the route from the departure point to the destination selected in S105 based on the route information 202C (S108).

The mobile terminal 100 starts the AR navigation process (S109). The details of the AR navigation process will be described later (see FIG. 6).

By the above processing, the reference direction and the reference position of the camera 90 at the start of navigation can be specified, and the AR navigation processing can be started.

FIG. 6 is a flowchart showing an example of AR navigation processing. The flowchart corresponds to a detailed description of the process of S109 in FIG.

The image acquisition unit 111 acquires the current captured image 201 from the camera 90 (S201).

The motion detection unit 117 detects the motion of the feature point 211 between the previous captured image 201 and the current captured image 201, and detects the amount of change in direction and the amount of movement of the camera 90 (S202).

The position specifying unit 118 is the current direction (current direction) and position (current position) of the camera 90 based on the previous direction and position of the camera 90 and the amount of change and movement of the camera 90 detected in S202. ) Is specified (S203).

The navigation display unit 119 generates a navigation image 205 based on the current direction and the current position of the camera 90, and the navigation display unit 119 superimposes the generated navigation image 205 on the current captured image 201 to perform AR. Generate and display a navigation image (S204).

The position specifying unit 118 determines whether or not the current position of the camera 90 is near the destination (S205). For example, it is determined whether or not the distance between the current position of the camera 90 and the destination is less than a predetermined threshold value.

When it is determined that the current position of the camera 90 is not near the destination (S205: NO), the mobile terminal 100 returns to the process of S201. When it is determined that the current position of the camera 90 is near the destination (S205: YES), the mobile terminal 100 notifies the user 2 that the camera 90 has arrived near the destination, and ends the AR navigation process.

By the above processing, even in a place where it is difficult to receive the positioning signal transmitted from the positioning satellite, the mobile terminal 100 allows the user 2 to be moved by AR navigation from the departure point where the digital signage 10 is located. You can navigate to the selected destination.

The reference direction and the reference position of the camera 90 may be corrected by the mark 11 displayed on the electronic signboard 10 in the middle of the route. For example, suppose that the user 2 points the camera 90 at the mark 11 displayed on the electronic signboard 10 in the middle of the route while moving according to the AR navigation, and performs the same operation as in S106. In this case, the reference specifying unit 115 may respecify the reference direction and the reference position of the camera 90 based on the mark 11 displayed on the electronic signboard 10 in the middle of the route by the same processing as in S107. As a result, it is possible to correct the error of the current direction and the current position of the camera 90 that may be accumulated during the movement.

<Function of navigation display>
FIG. 7 is a diagram for explaining a further function of the navigation display unit 119.

Unlike a vehicle that is premised on traveling on a road, the walking user 2 does not always continue walking along the route shown by the navigation image 205 displayed first. For example, the user 2 may avoid people, shift the course to the left or right in order to follow the flow of people, or short-cut the route shown by the navigation image 205 in the middle of walking. In such a case, if the navigation image 205 initially displayed remains as it is, the navigation image 205 may be removed from the display and become invisible. Further, for example, when the specified route passes through the center of the road but the sidewalk is only at the end, or the route shown by the navigation image 205 deviates from the display due to other factors such as accumulation of errors. There is also. Further, regardless of whether or not the navigation image 205 is displayed, the display position or direction of the navigation image 205 may deviate from the actual traveling direction of the user 2 due to the limit of the accuracy of the position and the direction. be. In this case, the user 2 may not be able to move as instructed by the navigation image 205. Further, when the route recognized by the mobile terminal 100 and the route actually being traveled by the user 2 are supposed to match, but are deviated due to an error or the like, as shown in the navigation image 205. If the user moves, the deviation may become larger. Therefore, the navigation display unit 119 has a function of appropriately adjusting the display of the navigation image 205 according to the movement of the user 2. Hereinafter, a detailed description will be given.

For example, when the user 2 who has been traveling along the path M0 along the navigation image 205 advances to the path M1 which is slightly deviated from the path M0, the navigation image 205 is displayed at the left end of the display unless the display of the navigation image 205 is changed. I can't see it because I'm close to it. Therefore, in the navigation display unit 119, the target direction V1 indicated by the route to the target node N and the current direction U1 of the camera 90 are substantially the same (for example, the angle θ ₀ formed by V1 and U1 is less than a predetermined threshold value). When the position of the camera 90 moves by a predetermined distance or more, the navigation image 205A from the current position of the camera 90 toward the target node N is regenerated. As a result, even if the user 2 advances to the path M1 slightly deviated from the original path M0, the navigation image 205A is redisplayed near the center of the display accordingly. Therefore, even when the user 2 advances to the path M1 slightly deviated from the original path M0, it is possible to prevent the navigation image 205 from being disengaged from the display and disappearing. Here, the difference between the target direction V1 and the current direction U1 corresponds to the difference between the route recognized by the system as correct and the traveling direction of the user 2. Therefore, the fact that these directions move by a predetermined distance or more in substantially the same direction indicates that the user 2 is continuously moving along the route shown by the navigation image 205. In such a situation, there are various possible causes for the navigation image 205 to deviate from the display, but in any case, it is strictly determined that the moving direction of the user 2 is incorrect. , It confuses user 2 who intends to move in the correct direction. Therefore, in the present embodiment, by redisplaying the navigation image 205A near the center of the display, the recognition of the user and the displayed navigation image 205A are apparently matched, thereby suppressing the occurrence of such confusion. ing. Further, the correction of the display position may be adjusted suddenly near the center of the display or may be adjusted gently.

Further, if the determination of whether or not there has been a movement of a predetermined distance or more is omitted and the target direction V1 indicated by the route to the target node N and the current direction U1 of the camera 90 are substantially the same, the navigation image 205A is redisplayed at any time. You may. In this case, the direction recognized by the mobile terminal 100 and the actual traveling direction of the user 2 are substantially the same. Further, the process of redisplaying the navigation image 205A even if the navigation image 205A is not moved by the predetermined distance or more and the process of redisplaying the navigation image 205A when the movement is performed by the predetermined distance or more may be used in combination. In this case, the process of redisplaying the navigation image 205A _{even if there is no movement of a predetermined distance or more is executed when, for example, the angle θ 0} formed by V1 and U1 is small, and the navigation is performed when there is a movement of a predetermined distance or more. The process of redisplaying the image 205A _{may be executed when θ 0} is a large angle. As a result, the deviation that seems to be caused by the error of the system is corrected at any time, and the large deviation that the traveling direction itself of the user 2 may be wrong can be corrected after a certain verification period.

Further, when the navigation image 205A is regenerated, the route itself may be corrected instead of modifying only the display position of the navigation image 205A. In this case, for example, not only the target node N but all the nodes may be moved or rotated. As a result, the route itself recognized by the mobile terminal 100 is corrected, so that an error is less likely to occur in future movements. However, if this correction is made, even if the user 2 is moving on the wrong route, it may be recognized that he / she is always walking on the correct route, so the position is compared with the original route. Alternatively, if the direction changes too much, the route correction may not be performed and it may be determined that the movement of the user 2 is incorrect. When moving or rotating the node, the adjustment may be made suddenly or slowly. Further, although the adjustment of the node itself is immediately reflected, the display of the navigation image 205A may be adjusted gently. By doing so, the route recognized internally by the mobile terminal 100 can be quickly adjusted to the actual movement of the user 2, and the possibility of surprise the user due to the sudden movement of the navigation image 205A can be reduced. can.

Further, in the above case, in the modification of the display position of the navigation image 205A or the node constituting the route, the deviation between the target direction V1 indicated by the route to the target node N and the current direction U1 of the camera 90 exceeds a predetermined threshold value. In some cases, it may not be performed. As a result, it is possible to reduce the possibility that the correction will occur even when an obvious error has occurred, such as when the user 2 is reversing the route.

Further, the modification of the display position of the navigation image 205A or the node constituting the route may be suppressed when the distance to the corner is less than a predetermined distance. On routes such as corners, cranks, and U-shapes, user 2 is less likely to follow the navigation image 205A because it is easy to shortcut or make a large turn, and the navigation image 205A or the nodes that make up the route. This is because an excessive number of corrections may occur, and as a result, an inconvenient correction result may occur.

Further, the modification of the nodes constituting the display position or the route of the navigation image 205A may be corrected based on the point where the distance in the three-dimensional space is the shortest, regardless of the target node N. However, when there are a plurality of close edges, the degree of coincidence between the edge direction and the movement direction may be taken into consideration when selecting. By doing so, it is possible to cope with a deviation when performing a three-dimensional movement such as a spiral staircase.

Further, in the navigation display unit 119, the target node N is not included in the viewing angle of the camera 90, and the target direction V1 and the current direction U1 of the camera 90 are significantly different (for example, the angle θ _{0 formed by V1 and U1).} When it is detected that the position of the camera 90 has moved by a predetermined distance or more in a state (above a predetermined threshold value), the navigation image 205 is hidden and information that alerts the user 2 that the camera 90 is off the route (hereinafter, """Cautioninformation") may be displayed on the display. As a result, when the user 2 advances to a course M2 that is significantly different from the original course M0, the navigation display unit 119 can display caution information and can omit the unnecessary navigation image 205 generation process.

The navigation display unit 119 may output a voice that calls attention or a vibration pattern that calls attention together with or instead of displaying the caution information.

Further, the navigation display unit 119 may end the AR navigation when the state of displaying the caution information continues for a predetermined time or more. This is because it is difficult to continue AR navigation and the processing load and battery consumption of the mobile terminal 100 are suppressed.

Further, in the navigation display unit 119, the target node N is included in the viewing angle of the camera 90 in the state of displaying the caution information, and the target azimuth V1 and the current azimuth U1 of the camera 90 are substantially the same (for example, V1). _{When it is detected that the angle θ 0} formed by U1 is less than a predetermined threshold, the caution information is hidden, and the navigation image 205B from the current position of the camera 90 toward the target node N is regenerated and redisplayed. good. As a result, the navigation display unit 119 can redisplay the navigation image 205 when the user 2 advances to the path M2 which is significantly different from the original path M0 and then proceeds to the path M3 which is slightly deviated from the original path M0. Also, not only when the target node N is within the viewing angle, but also when equipment (signboards and posters) or terrain (slopes, corners, etc.) recorded as existing near the target node are in the field of view. , It may be determined that the user 2 has returned to the vicinity of the route, and the navigation image 205 may be regenerated and redisplayed.

Further, the navigation display unit 119 is based on the AR space coordinates of the reference point (for example, the position of the departure point or destination of the route, the origin of the AR space) and the AR space coordinates of the position of the camera 90 at a certain time. , The longitude and latitude of the position of the camera 90 at a certain time may be calculated. Then, the navigation display unit 119 may write the longitude and latitude of the position of the camera 90 at the calculated time in the log information (not shown). As a result, the longitude and latitude indicating the position of the camera 90 at each time are recorded in the log information. That is, the movement route of the camera 90 is recorded in the log information. The log information may be output as a file or uploaded to the server 20. By using this log, the longitude and latitude of the camera 90 recognized by the mobile terminal 100 at the reference point can be recorded, so that the recorded longitude and latitude can be reused at the same reference point. can. As a result, when there is a deviation between the reference point and the position recognized by the mobile terminal 100, the deviation can be corrected. This deviation may be due to the nature of the mobile terminal 100 or the nature of the reference point. An example of the former is caused by distortion of the image captured by the camera 90, and an example of the latter is considered to be a case where the reference point itself is realized by a movable electronic signage. Therefore, by recording the type or ID of the mobile terminal as a log and analyzing the log, it is possible to identify a model or a signboard that is likely to be displaced.

<How to update the target node>
FIG. 8 is a diagram for explaining an example of a method of updating the target node.

A route is composed of multiple nodes and edges connecting adjacent nodes. For example, let the Nth node from the origin node be the first target node N. The first target node N may be a node that satisfies certain conditions. For example, nodes in a predetermined order from the node of the departure place may be set as the first target node N. Alternatively, among the nodes separated from the starting point by a predetermined distance or more, the node closest to the starting point may be set as the first target node N.

_{Here, let θ 1} be the angle formed by the edge E1 connecting the node N-1 and the target node N and the line F1 connecting the current position of the camera 90 and the target node N. _{Let θ 2} be the angle formed by E1 connecting the node N-1 and the target node N and the edge E2 connecting the target node N and the node N + 1.

Route specification unit 116, when it becomes theta _1> and (θ _2/2), good node N + 1 as the next target node. The route identification unit 116 may repeat this until it reaches the destination. As a result, when the current position of the camera 90 is sufficiently close to the target node N or has passed the target node N, the target node is updated to the previous node. Therefore, at any position, the navigation display unit 119 can generate an appropriate navigation image 205 toward the target node.

<Modification example>
In the above description, an example in which the camera 90 receives the optical signal 12 transmitted from the digital signage 10 and acquires the ID has been described. However, the mobile terminal 100 may acquire the ID by another method. For example, instead of the digital signage 10, a poster on which a two-dimensional code is printed may be installed at a predetermined place. Then, the mobile terminal 100 may read the two-dimensional code with the camera 90 and acquire the ID from the two-dimensional code. In this case, the predetermined mark 11 may be printed on the poster. Further, even when the optical signal 12 is used, the source of the signal does not have to be in the form of an electronic signboard, and may be, for example, a lighting fixture. Further, the lighting fixture that emits the optical signal 12 may be used as a backlight, and the signboard may be covered with a cover or a poster on which the mark 11 is printed. That is, in the present disclosure, the object to be photographed for specifying the reference position and the reference direction may be an arbitrary installation object. Further, each information may be acquired by using different installation objects, such as acquiring the position from the lighting equipment and acquiring the direction from the digital signage.

Further, in the above description, the information transmitted from the digital signage 10 is an ID, and other information is acquired from the server 20 by using the ID. However, the information is regulated in the digital signage 10 itself, and the information is regulated by wireless communication or the like. All information or some information may be transmitted to the mobile terminal 100.

Further, in the above description, the present disclosure has been described as a system for indoor navigation, but it may be used for outdoor navigation. Further, when the present disclosure is applied to a route that moves both outdoors and indoors, the current position may be corrected by using the position and direction calculated using information that can be acquired outdoors such as GPS.

In the above description, the mobile terminal 100 receives the ID from the digital signage 10 and then acquires the information including the destination list information and the route information from the server 20 by using the ID, but the present invention is limited to this. is not it. The acquisition timing may be divided such that the destination list information is acquired from the server 20 when the ID is received, and the route information corresponding to the destination is acquired from the server 20 when the destination is selected. By doing so, it is not necessary to acquire unnecessary route information, so that the amount of communication can be reduced. Further, when the ID is received with the destination set in the mobile terminal 100 in advance, the acquisition of the destination list information is omitted and the route information corresponding to the set destination is acquired. May be good. By doing so, it is not necessary to acquire unnecessary destination list information, so that the amount of communication can be reduced.

(Embodiment 2)
In the second embodiment, a modified example of the configuration of the mobile terminal 100 that realizes the contents described in the first embodiment will be described.

FIG. 9 is a diagram showing an example of the configuration of the mobile terminal and the mark detection process at the start of navigation according to the second embodiment.

FIG. 10 is a diagram showing a configuration example of the mobile terminal according to the second embodiment and a continuation example of FIG. 9 of the mark detection process at the start of navigation.

FIG. 11 is a diagram showing a configuration example of a mobile terminal according to the second embodiment and an example of a navigation image regeneration process accompanying the movement of the user 2.

FIG. 12 is a diagram showing a configuration example of the mobile terminal according to the second embodiment and an example of processing when the user 2 deviates from the route.

FIG. 13 is a diagram showing a configuration example of a mobile terminal according to the second embodiment and an example of a process of displaying caution information when the user 2 deviates from the route.

FIG. 14 is a diagram showing a configuration example of a mobile terminal according to the second embodiment and an example of processing when a user 2 who has deviated from the route returns to the route.

The mobile terminal 100 can also realize the contents described in the first embodiment by the configuration and processing shown in FIGS. 9 to 14.

<Hardware configuration>
The embodiments according to the present disclosure have been described in detail with reference to the drawings. The function of the mobile terminal 100 described above can be realized by a computer program.

FIG. 15 is a diagram showing a hardware configuration of a computer that realizes the functions of the mobile terminal 100 by a program.

The computer 1000 includes an input device 1001, an output device 1002, a CPU (Central Processing Unit) 1003, a GPU (Graphics Processing Unit) 1004, a ROM (Read Only Memory) 1005, a RAM (Random Access Memory) 1006, a storage device 1007, and a read device. A device 1008, a transmission / reception device 1009, and an image pickup device 1010 are provided, and each device 1001 to 1010 is connected to a bus 1011 capable of bidirectional communication.

An example of the input device 1001 is a keyboard, mouse or touchpad. An example of the output device 1002 is a display or a speaker. The CPU 1003 may be read as another term such as a processor, a controller, an integrated circuit, and a PLD (programmable logic device). An example of the storage device 1007 is an SSD (Solid State Drive) or a flash memory. The reading device 1008 reads information from a recording medium such as a USB (Universal Serial Bus) memory. The transmission / reception device 1009 wirelessly communicates via a communication network. The image pickup device 1010 includes an image pickup device and generates a captured image (moving image and still image). The generated captured image is stored in the storage device 1007.

The reading device 1008 reads the program from the recording medium on which the program for realizing the functions of the above devices is recorded, and stores the program in the storage device 1007. Alternatively, the transmission / reception device 1009 communicates with a server connected to the communication network, and stores a program or application downloaded from the server for realizing the functions of the devices in the storage device 1007. Then, the CPU 1003 copies the program stored in the storage device 1007 to the RAM 1006, and sequentially reads and executes the instructions included in the program from the RAM 1006, thereby realizing the functions of the respective devices.

Each functional block used in the description of the above embodiment is typically realized as an LSI which is an integrated circuit. These may be individually integrated into one chip, or may be integrated into one chip so as to include a part or all of them. Although it is referred to as LSI here, it may be referred to as IC, system LSI, super LSI, or ultra LSI depending on the degree of integration.

Furthermore, if an integrated circuit technology that replaces an LSI appears due to advances in semiconductor technology or another technology derived from it, it is naturally possible to integrate functional blocks using that technology.

(Summary of this disclosure)
The mobile terminal (100) including the image pickup device (90) according to the present disclosure includes an image acquisition unit (111) that acquires an image pickup image (201) captured by the image pickup device, and an installation object (111) installed at a predetermined position. Based on the captured image captured by 10), the reference specifying unit (115) that specifies the reference orientation and position of the image pickup device at the start of navigation, and the route specification that specifies the route from the position of the installation object to the destination. A motion detection unit (117) that detects the amount of change in direction and the amount of movement of the image pickup device based on the movement of the unit (116) and the feature point (211) extracted from the captured image over time, and a reference of the image pickup device. Based on the amount of change in direction and the amount of movement of the image pickup device detected by the motion detection unit after specifying the direction and position of the image pickup device, the position identification section (118) for specifying the current direction and position of the image pickup device is specified. It includes a navigation display unit (119) that generates a navigation image (205) indicating a route based on the current direction and position of the image pickup apparatus and superimposes and displays the generated navigation image on the current captured image.

According to this configuration, the mobile terminal can specify the position (reference position) of the image pickup device (mobile terminal) at the start of navigation without receiving the positioning signal, and at the start of navigation without using an electronic compass. The orientation (reference orientation) of the imaging device in the above can be specified. Then, by the subsequent motion detection, the current position (current position) and orientation (current orientation) of the image pickup apparatus can be specified without receiving the positioning signal and without using the electronic compass. In addition, a navigation image can be generated based on the current position and current orientation of the imaging device, and the generated navigation image can be superimposed and displayed on the current captured image. Therefore, AR navigation can be realized even in a place where it is difficult to receive a positioning signal.

Further, the route may be composed of a plurality of nodes indicating a predetermined passing point. Then, the navigation display unit may generate a navigation image from the current position toward the position indicated by the target node, which is one of the plurality of nodes. Further, the navigation display unit so that the display position of the navigation image is substantially in the center of the display when the deviation between the target direction toward the position indicated by the target node and the current direction from the current position is less than a predetermined threshold value. You may adjust.

According to this configuration, the navigation display unit regenerates the navigation image according to the change in the current position of the image pickup device. Therefore, for example, when the user (2) deviates slightly from the route, the navigation image is carried. It is possible to prevent the terminal from being removed from the display and disappearing.

In addition, the navigation display unit displays cautionary information indicating that the vehicle is off the route when the deviation between the target direction from the current position to the position indicated by the target node and the current direction is equal to or greater than a predetermined threshold value. And the navigation image may be hidden.

According to this configuration, when the orientation of the imaging device deviates significantly from the orientation of the original route, caution information can be displayed to notify the user that the orientation deviates significantly from the route.

Further, the navigation display unit may hide the caution information and redisplay the navigation image when the deviation between the target direction and the current direction becomes less than the threshold value while displaying the caution information.

According to this configuration, navigation can be resumed when the image pickup device, which has been largely deviated from the original route orientation, returns to the original route orientation.

Further, the navigation display unit may end the navigation to the destination when the state in which the deviation between the target direction and the current direction is equal to or greater than the threshold value continues for a predetermined time or longer while displaying the caution information.

According to this configuration, the navigation can be ended when the direction of the image pickup apparatus deviates greatly from the direction of the original route and the state continues for a while.

Although the embodiments have been described above with reference to the accompanying drawings, the present disclosure is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications, modifications, substitutions, additions, deletions, and equality within the scope of the claims. It is understood that it belongs to the technical scope of the present disclosure. Further, each component in the above-described embodiment may be arbitrarily combined as long as the gist of the invention is not deviated.

The technology of the present disclosure can be used for navigation using a mobile terminal.

2 User 10 Electronic signboard 11 Mark 12 Optical signal 20 Server 30 Route 90 Camera 100 Mobile terminal 110 Control unit 111 Video acquisition unit 112 Optical signal reception unit 113 Information acquisition unit 114 Destination list display unit 115 Reference identification unit 116 Route identification unit 117 Motion detection unit 118 Position identification unit 119 Navigation display unit 121 Frame 200 Storage unit 201 Captured video 202A Electronic signboard information 202B Destination list information 202C Route information 203 Information indicating the reference orientation and reference position of the camera 204 Current orientation and current position of the camera Information indicating 205, 205A, 205B Navigation image 211 Feature point 1000 Computer 1001 Input device 1002 Output device 1003 CPU
1004 GPU
1005 ROM
1006 RAM
1007 Storage device 1008 Reading device 1009 Transmission / reception device 1010 Imaging device 1011 Bus

Claims (8)

A mobile terminal equipped with an imaging device
An image acquisition unit that acquires an image captured by the image pickup device, and an image acquisition unit.
Based on the captured image of the installation object installed at the predetermined position, the reference specifying unit that specifies the reference orientation and position of the imaging device at the start of navigation, and the reference specifying unit.
A route identification unit that identifies the route from the position of the installation to the destination,
A motion detection unit that detects the amount of change in direction and the amount of movement of the image pickup device based on the movement of feature points extracted from the captured image over time.
A position specifying unit that specifies the current direction and position of the image pickup device based on the amount of change and movement of the direction of the image pickup device detected by the motion detection unit after specifying the reference direction and position of the image pickup device. When,
A navigation display unit that generates a navigation image indicating the route based on the current orientation and position of the specified imaging device and superimposes and displays the generated navigation image on the current captured image.
Mobile terminal. The route is composed of a plurality of nodes indicating a predetermined passing point.
The navigation display unit generates the navigation image from the current position to the position indicated by the target node, which is one of the plurality of nodes.
The mobile terminal according to claim 1. In the navigation display unit, when the deviation between the target direction toward the position indicated by the target node and the current direction from the current position is less than a predetermined threshold value, the display position of the navigation image is substantially in the center of the display. Adjust so that
The mobile terminal according to claim 2. Note that the navigation display unit deviates from the route when the deviation between the target direction from the current position to the position indicated by the target node and the current direction is equal to or greater than a predetermined threshold value. Display information and hide the navigation image,
The mobile terminal according to claim 2 or 3. The navigation display unit hides the caution information and redisplays the navigation image when the deviation between the target direction and the current direction becomes less than the threshold value while displaying the caution information. do,
The mobile terminal according to claim 4. While displaying the caution information, the navigation display unit ends navigation to the destination when the state in which the deviation between the target direction and the current direction is equal to or greater than the threshold value continues for a predetermined time or longer. do,
The mobile terminal according to claim 4 or 5. A navigation method performed by a mobile terminal equipped with an imaging device.
The captured image captured by the imaging device is acquired, and the image is captured.
Based on the captured image of the installation object installed at the predetermined position, the reference orientation and position of the imaging device at the start of navigation are specified.
Identify the route from the location of the installation to the destination
Based on the movement of the feature points extracted from the captured image with the passage of time, the amount of change in orientation and the amount of movement of the imaging device are detected.
After specifying the reference orientation and position of the imaging device, the current orientation and position of the imaging device are specified based on the amount of change and movement of the orientation of the imaging device detected by the motion detection unit.
A navigation image showing the route is generated based on the current orientation and position of the specified imaging device, and the generated navigation image is superimposed and displayed on the current captured image.
Navigation method. A computer program executed by a mobile terminal equipped with an image pickup device.
The captured image captured by the imaging device is acquired, and the image is captured.
Based on the captured image of the installation object installed at the predetermined position, the reference orientation and position of the imaging device at the start of navigation are specified.
Identify the route from the location of the installation to the destination
Based on the movement of the feature points extracted from the captured image with the passage of time, the amount of change in orientation and the amount of movement of the imaging device are detected.
After specifying the reference orientation and position of the imaging device, the current orientation and position of the imaging device are specified based on the amount of change and movement of the orientation of the imaging device detected by the motion detection unit.
A navigation image showing the route is generated based on the current orientation and position of the specified imaging device, and the generated navigation image is superimposed and displayed on the current captured image.
A computer program that causes the mobile terminal to execute processing.

Images