JP5174942B2 - Terminal device, image display method and image display program executed by terminal device - Google Patents

Description

  The present invention relates to a terminal device having a route guidance function.

  This type of technique is described in Patent Documents 1 and 2, for example. Patent Document 1 describes a technology for selectively activating a navigation function in a mobile terminal device with a navigation function when the mobile terminal device is connected to a hands-free device installed in a vehicle. Patent Document 2 describes a technology for automatically switching which of a map image using map information or a live-action image showing a state outside a vehicle is preferentially displayed according to a situation outside the vehicle. . The situation outside the vehicle includes the degree of shielding by front obstacles (vehicles, etc.), external brightness, rain, fog, distance to the preceding vehicle, road attributes, and landmarks (signals, convenience stores, etc.) The presence or absence is listed.

JP 2007-163386 A WO2007-129382

  In recent years, a technique has been proposed in which a portable terminal device such as a high-function mobile phone called “smartphone” is installed in a vehicle via a holding device called “cradle” and used. In addition, navigation called “AR navigation (AR: Augmented Reality)” using a photographed image by a smartphone camera has been proposed. In AR navigation, an image for route guidance such as a direction and a distance to a destination is superimposed and displayed on a real image taken by a camera. Therefore, when using AR navigation, it can be said that it is desirable that the shooting direction of the camera matches the traveling direction of the vehicle. That is, when the shooting direction of the camera is deviated from the traveling direction of the vehicle, it is considered difficult to perform AR navigation appropriately.

  For this reason, it is considered difficult to suitably apply the techniques described in Patent Documents 1 and 2 described above to a system having a smartphone and a cradle. Specifically, when the technique described in Patent Document 1 is applied, AR navigation is activated when the smartphone is connected to the cradle. At this time, the shooting direction of the camera is the traveling direction. If it is deviated from, it can be said that AR navigation cannot be executed properly. In addition, when the technique described in Patent Document 2 is applied, it is determined whether or not to display the AR navigation preferentially based on the situation outside the vehicle. Since the situation that deviates from the traveling direction of the vehicle is not taken into consideration, it can be said that the AR navigation may not be executed appropriately.

  The above-mentioned thing is mentioned as an example as a subject which the present invention tends to solve. The present invention provides a terminal device and a terminal device capable of appropriately switching which of a live-action guide image and a map guide image is to be preferentially displayed based on a relationship between a shooting direction of a camera and a traveling direction of a vehicle. An object is to provide an image display method and an image display program to be executed.

The invention according to claim 1 is a terminal device that can be held by the terminal holding device, and includes a photographing unit, a live-action guidance image for route guidance using a photographed image photographed by the photographing unit, and a map image Display control means for switching and displaying each of the map guidance images for route guidance using the display means, and when the display control means is removed from the terminal holding device, the live-action guidance The display is switched from the image display to the map image display.

The invention according to claim 6 is an image display method that can be held by the terminal holding device and executed by the terminal device having the photographing unit, and provides route guidance using the photographed image photographed by the photographing unit. A display control step of switching and displaying each of a live-action guide image and a map guide image for route guidance using the map image, and the display control step is removed from the terminal holding device. In this case, the display of the photograph guide image is switched to the display of the map image.

The invention according to claim 7 is an image display program that can be held by a terminal holding device and that is executed by a terminal device that has a photographing unit and a computer, and uses a photographed image photographed by the photographing unit. The display control means causes the computer to function as a display control means for switching and displaying each of a live-action guidance image for route guidance and a map guidance image for route guidance using a map image. When the camera is removed from the terminal holding device, the display of the live-action guide image is switched to the display of the map image.

The terminal device in the state hold | maintained at the terminal holding | maintenance apparatus is shown. The example of the state which rotated the terminal holder is shown. The schematic structure of a terminal device is shown. An example of the terminal holding | maintenance apparatus and terminal device in the state installed in the vehicle interior is shown. The figure for demonstrating the specific example of the method of judging the shift | offset | difference of an imaging | photography direction and a traveling direction is shown. A processing flow executed when a navigation application is started is shown. The processing flow performed during execution of AR navigation is shown. The figure for demonstrating an example of the method of judging the shift | offset | difference of an imaging | photography direction and an advancing direction is shown. The figure for demonstrating the modification 5 is shown.

  In one aspect of the present invention, a terminal device attached to a moving body is a captured image captured by the imaging unit based on a relationship between the imaging unit and the imaging direction of the imaging unit and the traveling direction of the mobile unit. A determination unit that determines which of the live-action guide image using the map information and the map guide image using the map information is displayed with priority, and based on the determination by the determination unit, Display control means for performing control to display one of the map guidance images.

  The above terminal device is attached to the moving body, and images the front of the moving body by imaging means such as a camera. Further, the terminal device has a function of performing route guidance (navigation) from the current location to the destination. Based on the relationship between the shooting direction of the shooting means and the traveling direction of the moving body, the determination means determines whether the shooting guide image using the shot image shot by the shooting means or the map guide image using the map information is used. Judge whether to display with priority. Specifically, the determination unit makes such a determination by determining a deviation between the shooting direction and the traveling direction. Then, the display control means performs control to display one of the photograph guide image and the map guide image based on the determination result by the determination means. According to the above terminal device, it is possible to appropriately switch the guide image to be displayed among the live-action guide image and the map guide image.

  In one aspect of the terminal device, the determination unit determines to display the live-action guide image preferentially when the shift between the shooting direction and the traveling direction is within a predetermined range, and the shift is If it is outside the predetermined range, it is determined that the map guidance image is displayed with priority.

  According to this aspect, it is possible to suppress an inappropriate live-action guide image from being displayed in a situation where the shooting direction is deviated from the traveling direction. That is, only in a situation where an appropriate live-action guide image can be displayed, the live-action guide image can be preferentially switched and displayed.

  Suitably, the said determination means can determine the shift | offset | difference of the said imaging | photography direction and the said advancing direction based on the image of the white line contained in the said captured image.

  Preferably, the determination unit acquires an output of a sensor provided in the terminal device and / or a holding device configured to hold the terminal device, and the imaging is performed based on the output of the sensor. The deviation between the direction and the traveling direction can be determined.

  Further, preferably, the determination unit can determine the difference between the shooting direction and the traveling direction in consideration of both the sensor output and the white line image included in the shot image. This makes it possible to accurately determine the deviation between the shooting direction and the traveling direction.

  In another aspect of the terminal device, the display control unit displays the map guidance image when a destination for route guidance is not set. Thereby, the user can set the destination using the map guidance image.

  In another aspect of the terminal device, the display control unit displays the map guidance image while the determination unit is performing the determination. In this aspect, in a situation where it is uncertain whether or not an appropriate live-action guide image can be displayed, the map guide image can be displayed without displaying the live-action guide image for the convenience of the user. it can.

  In another aspect of the above terminal device, the display control means switches from the photographed guide image to the map guide image when an operation is performed on the terminal device while the photographed guide image is displayed. . In this aspect, when an operation is performed on the terminal device, the shooting direction tends to change, and there is a possibility that an appropriate live-action guidance image may not be displayed. Therefore, the display image can be switched to the map guidance image.

  In another aspect of the present invention, an image display method attached to a moving body and executed by a terminal device having a photographing unit is based on a relationship between a photographing direction of the photographing unit and a traveling direction of the moving body. A determination step for determining which of a live-action guide image using a photographed image photographed by the photographing means and a map guide image using map information is to be displayed preferentially; and the determination in the determination step And a display control step for performing control to display one of the photograph guide image and the map guide image.

  In another aspect of the present invention, an image display program that is attached to a moving body and includes a photographing unit and is executed by a terminal device that includes a computer, the computer displays the photographing direction of the photographing unit and the moving body. Determining means for determining whether to preferentially display a live-action guidance image using a photographed image photographed by the photographing means or a map guidance image using map information based on a relationship with a traveling direction Based on the determination by the determination unit, the display unit functions as a display control unit that performs control to display one of the photograph guide image and the map guide image.

  According to the image display method and the image display program described above, it is possible to appropriately switch the guide image to be displayed among the live-action guide image and the map guide image.

  In still another aspect of the present invention, the terminal device includes the photographing unit, the detecting unit that detects the tilt of the terminal device, and the photographing based on the relationship between the photographing direction of the photographing unit and the tilt of the terminal device. Based on the determination by the determination means, the determination means for determining which of the live-action guide image using the photographed image taken by the means and the map guide image using the map information is to be displayed preferentially Display control means for performing control to display one of the photograph guide image and the map guide image with priority.

  According to said terminal device, when a user carries and uses a terminal device (for example, when a pedestrian uses route guidance using a terminal device), it is displayed among a live-action guidance image and a map guidance image. The guide image can be switched appropriately.

  In one aspect of the terminal device, the detection unit detects an inclination of the shooting direction of the shooting unit with respect to a horizontal plane, and the determination unit has a tilt of the shooting direction within a predetermined range with respect to the horizontal plane. In addition, the photograph guide image is preferentially displayed, and it is determined that the map guide image is preferentially displayed when the inclination of the photographing direction is out of a predetermined range with respect to the horizontal plane.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

[Device configuration]
First, the configuration of the terminal device according to the present embodiment will be described.

  FIG. 1 shows the terminal device 2 held by the terminal holding device 1. Fig.1 (a) has shown the front view, FIG.1 (b) has shown the side view, FIG.1 (c) has shown the rear view.

  The terminal holding device 1 mainly includes a base 11, a hinge 12, an arm 13, a substrate holder 15, and a terminal holder 16. The terminal holding device 1 functions as a so-called cradle, and a terminal device 2 such as a smartphone is attached.

  The base 11 functions as a base when the terminal holding device 1 is attached to a moving body such as a vehicle. For example, an adhesive tape, a suction cup, or the like is provided on the lower surface of the base 11, and the base 11 is fixed to the installation surface 5 such as a dashboard of the vehicle by the adhesive tape.

  The arm 13 is fixed to the hinge 12 and is rotatably attached to the base 11. As the hinge 12 rotates, the arm 13 rotates in the front-rear direction of the terminal device 2, that is, in the directions of the arrows 41 and 42 in FIG. That is, the installation angle of the substrate holder 15 and the terminal holder 16 with respect to the installation surface 5 can be adjusted by rotating the arm 13 via the hinge 12 with respect to the base 11 fixed to the installation surface 5 of the vehicle. .

  The substrate holder 15 includes a cover 15a, a ball link 15b, a sensor substrate 15c, and a sensor 15d. The ball link 15 b is attached to the upper end of the arm 13, and holds the substrate holder 15 at an arbitrary angle with respect to the arm 13. The cover 15 a is provided at the lower end of the substrate holder 15 and has a role of regulating the rotation of the substrate holder 15 with respect to the arm 13. A sensor substrate 15c is provided inside the substrate holder 15, and a sensor 15d is provided on the sensor substrate 15c. A suitable example of the sensor 15d is a gyro sensor that detects at least one of a horizontal angular velocity and acceleration of the moving body.

  The terminal holder 16 is a holder that holds the terminal device 2. The terminal holder 16 includes a connector 16a and a wiring 16b. The connector 16 a is provided on the front surface of the terminal holder 16, that is, the bottom of the surface on which the terminal device 2 is installed, and is connected to the connector of the terminal device 2 when the terminal device 2 is installed on the terminal holder 16. The connector 16a is electrically connected to the sensor substrate 15c by the wiring 16b. Therefore, the detection signal from the sensor 15d is supplied to the terminal device 2 through the sensor substrate 15c, the wiring 16b, and the connector 16a.

  The terminal device 2 includes a front surface 2a which is the front surface side of the terminal device 2 main body and has a display unit 25 such as a liquid crystal display panel, and a back surface 2b on the back surface side of the terminal device 2 main body. Normally, the terminal device 2 is configured in a rectangular flat plate shape, and the front surface 2a and the back surface 2b are configured substantially in parallel.

  The terminal holder 16 has a contact surface 16c on the front side. When the terminal device 2 is attached to the terminal holder 16, the contact surface 16 c abuts on the back surface 2 b of the terminal device 2 and supports the back surface 2 b of the terminal device 2. In the example illustrated in FIG. 1, the contact surface 16 c of the terminal holder 16 is configured such that the entire surface thereof is in contact with the back surface 2 b of the terminal device 2. Instead of this, one or several of the contact surfaces 16c may be partially protruded, and only the protruded portion may be in contact with the back surface 2b of the terminal device 2.

  A camera 29 is provided on the back surface 2 b of the terminal device 2. In addition, a hole 17 is formed in the terminal holder 16 of the terminal holding device 1 at a position facing the camera 29 in a state where the terminal device 2 is held by the terminal holding device 1. The hole 17 is configured to have a diameter larger than the diameter of the lens of the camera 29. Thereby, in a state where the terminal device 2 is held by the terminal holding device 1, the camera 29 can photograph the back side of the terminal holder 16 without being obstructed by the outer wall of the terminal holder 16. Specifically, the camera 29 captures the outside of the vehicle.

  In the example illustrated in FIG. 1, the terminal holder 16 is configured to cover substantially the entire back surface 2 b of the terminal device 2, and the hole portion 17 is formed at a position facing the camera 29 of the terminal device 2. Instead, the terminal holder 16 may be configured to cover only the surface below the position where the camera 29 is provided in the terminal device 2 in a state where the terminal device 2 is held by the terminal holding device 1. it can. In one example, the contact surface 16c of the terminal holder 16 extends to a position below the position where the camera 29 of the terminal device 2 is provided (in other words, the camera 29 of the terminal device 2 is provided). The contact surface 16c does not exist above the position). In such another example, it is not necessary to form the hole 17 in the terminal holding device 1.

  In the example shown in FIG. 1, the camera 29 is provided on a substantially center line in the left-right direction of the back surface 2 b of the terminal device 2, but the camera 29 is not limited to being provided at such a position. For example, the camera 29 may be provided at a position somewhat away from the center line in the left-right direction of the back surface 2b. In this case, instead of forming the hole 17 in the terminal holder 16, the terminal device 2 is held by the terminal holding device 1, and the terminal device 2 is cut into a portion including the position where the camera 29 is provided. It is good also as forming a notch.

  Next, the rotation function of the terminal holder 16 with respect to the substrate holder 15 will be described. The terminal holder 16 holding the terminal device 2 can be rotated by 90 degrees with respect to the substrate holder 15. That is, when the state of FIG. 1A is set to a rotation angle of 0 degrees, the terminal holder 16 is rotated in four angles of 0 degrees, 90 degrees, 180 degrees, and 270 degrees clockwise or counterclockwise. It is possible to fix. The reason why the rotation angle can be fixed every 90 degrees is that the user normally uses the display unit 25 in a vertically or horizontally arranged state when viewing the terminal device 2. As described above, the terminal device 2 usually has a rectangular flat plate shape, and “arranged vertically” means an arrangement in which the longitudinal direction of the display unit 25 is vertical. The “arrangement” is an arrangement in which the longitudinal direction of the display unit 25 is horizontal.

  FIG. 2 shows an example of a state in which the terminal holder 16 is rotated. When the terminal holding device 1 is viewed from the front side, when the terminal holder 16 is rotated 90 degrees in the direction of the arrow from the state of FIG. 2A, the state shown in FIG. Further, when the terminal holding device 1 is viewed from the back side, when the terminal holder is rotated 90 degrees in the direction of the arrow from the state of FIG. 2C, the state shown in FIG.

  In terms of structure, for example, a rotation shaft (not shown) is provided in the approximate center of the substrate holder 15, and the terminal holder 16 can be rotated relative to the substrate holder 15 by fixing the terminal holder 16 to the rotation shaft. It can be. Further, on the surface where the substrate holder 15 and the terminal holder 16 are in contact with each other, the concave and convex portions or grooves and protrusions that are mutually fitted every rotation angle of 90 degrees are provided, so that the terminal holder 16 is moved at the rotation angle position of every 90 degrees. Can be fixed. Such a structure is merely an example, and other structures may be adopted as long as the terminal holder 16 can be fixed to the sensor substrate 15c at every rotation angle of 90 degrees.

  Next, FIG. 3 schematically shows the configuration of the terminal device 2. As illustrated in FIG. 3, the terminal device 2 mainly includes a CPU 21, a ROM 22, a RAM 23, a communication unit 24, a display unit 25, a speaker 26, a microphone 27, an operation unit 28, and a camera 29. Have The terminal device 2 is a portable terminal device having a call function such as a smartphone. For example, the terminal device 2 is installed at a position on the dashboard where the driver of the vehicle can visually recognize the display unit 25 while being held by the terminal holding device 1.

  A CPU (Central Processing Unit) 21 controls the entire terminal device 2. For example, the CPU 21 acquires map information and executes processing for performing route guidance (navigation) to the destination. In this case, the CPU 21 causes the display unit 25 to display a guidance image for performing route guidance. Examples of the guide image include a live-action guide image or a map guide image described later.

  A ROM (Read Only Memory) 22 includes a nonvolatile memory (not shown) in which a control program for controlling the terminal device 2 is stored. A RAM (Random Access Memory) 23 stores data set by the user via the operation unit 26 so as to be readable, and provides a working area to the CPU 21. Note that a storage unit other than the ROM 22 and the RAM 23 may be provided in the terminal device 2 and various data used for route guidance processing such as map information and facility data may be stored in the storage unit.

  The communication unit 24 is configured to be able to perform wireless communication with other terminal devices 2 via a communication network. The communication unit 24 is configured to be able to perform wireless communication with a server such as a VICS center. The communication unit 24 can receive data such as map information and traffic jam information from such a server.

  The display unit 25 is configured by a liquid crystal display, for example, and displays characters, images, and the like to the user. The speaker 26 outputs sound to the user. The microphone 27 collects sound emitted by the user.

  The operation unit 28 can be configured by an operation button or a touch panel type input device provided on the casing of the terminal device 2, and various selections and instructions by the user are input. When the display unit 25 is a touch panel system, the touch panel provided on the display screen of the display unit 25 also functions as the operation unit 28.

  The camera 29 is constituted by a CCD camera, for example, and is provided on the back surface 2b of the terminal device 2 as shown in FIG. Basically, the direction of the optical axis of the camera 29 (the axis extending in the vertical direction from the center of the lens) coincides with the vertical direction (in other words, the normal direction) of the back surface 2b of the terminal device 2. Note that the camera 29 may be provided not only on the back surface 2 b of the terminal device 2 but also on the front surface 2 a of the terminal device 2.

  The camera 29 corresponds to an example of a photographing unit in the present invention, and the CPU 21 corresponds to an example of a determination unit and a display control unit in the present invention (details will be described later).

  Next, FIG. 4 shows an example of the terminal holding device 1 and the terminal device 2 that are installed in the vehicle interior of the vehicle 3. As shown in FIG. 4, the terminal holding device 1 is fixed to an installation surface 5 such as a dashboard of the vehicle 3, and the terminal device 2 is held by the terminal holding device 1 in such a fixed state. Further, as indicated by a broken line in FIG. 4, the terminal device 2 captures the traveling direction of the vehicle 3 with the camera 29.

  In this specification, the “shooting direction” of the camera 29 means a direction in which the camera 29 is facing, and more specifically corresponds to the direction of the optical axis of the lens of the camera 29. Further, in this specification, the “traveling direction” of the vehicle 3 means the front-rear direction (specifically, the forward direction) of the vehicle 3. This “traveling direction” includes not only the direction in which the vehicle 3 actually travels but also the direction in which the vehicle 3 will travel (the direction in which the vehicle 3 is expected to travel). That is, in defining the “traveling direction”, the vehicle 3 does not necessarily have to travel, and the vehicle 3 may stop.

[Display control method]
Next, a display control method according to the present embodiment will be described. In the present embodiment, the CPU 21 in the terminal device 2 performs a route guidance to the destination, a live-action guide image using a photographed image (actual image) by the camera 29, and a map guide image (using map information). Hereinafter, a process of switching the display image is also performed. In other words, when performing route guidance, the CPU 21 performs between AR navigation using an image captured by the camera 29 and normal navigation using map information (hereinafter also simply referred to as “normal navigation”). Switch the type of navigation to be performed. In this case, the CPU 21 performs such switching based on the relationship between the shooting direction of the camera 29 and the traveling direction of the vehicle 3.

  The “map guide image (normal map image)” corresponds to a map image around the position of the vehicle 3 that is generated based on the map information. In addition, the “map guidance image (normal map image)” is an image in which an image for route guidance (for example, an image showing a searched road so as to stand out) is displayed on the map image, and It is assumed that such an image for route guidance is not displayed, and both the image on which the map image is displayed are included.

  Here, the reason for performing the above switching will be briefly described. As described above, with the terminal device 2 installed on the vehicle 3 via the terminal holding device 1, “AR Navi” that performs route guidance using an image in front of the vehicle taken by the camera 29 of the terminal device 2 is used. Are known. The AR navigation is to display an image for route guidance such as the direction and distance to the destination on the image taken by the camera 29 (this display image is the above-mentioned “actual guide image”). ”). Therefore, it can be said that it is desirable that the shooting direction of the camera 29 coincides with the traveling direction of the vehicle 3 in order to perform AR navigation appropriately. That is, when the shooting direction of the camera 29 is deviated from the traveling direction of the vehicle 3, it is considered difficult to perform AR navigation appropriately.

  In the present embodiment, in consideration of the contents described above, it is possible to determine that the shooting direction of the camera 29 is deviated from the traveling direction of the vehicle 3 in a situation where AR navigation cannot be performed appropriately. In a situation, AR navigation is not executed, that is, a live-action guide image is not displayed. In order to realize this, the CPU 21 in the terminal device 2 determines which of the live-action guide image and the map guide image is to be preferentially displayed based on the difference between the shooting direction of the camera 29 and the traveling direction of the vehicle 3. It is determined, in other words, whether AR navigation or normal navigation is to be prioritized. Specifically, when the deviation between the shooting direction and the traveling direction is determined to be within a predetermined range, the CPU 21 determines to display the live-action guide image with priority, and the deviation between the shooting direction and the traveling direction is determined. If it is determined that it is out of the predetermined range, it is determined that the map guidance image is preferentially displayed. The “predetermined range” used for the determination is set in advance based on, for example, whether AR navigation can be performed appropriately.

  Next, a specific example of a method for determining the difference between the shooting direction of the camera 29 and the traveling direction of the vehicle 3 will be described.

  The CPU 21 in the terminal device 2 performs image processing on an image captured by the camera 29 to recognize a white line image on the road in the captured image, and based on the white line image, the shooting direction of the camera 29 is recognized. And the deviation of the traveling direction of the vehicle 3 is determined. In one example, the CPU 21 uses a plurality of captured images obtained after traveling a certain distance after the vehicle 3 starts traveling, and based on changes in white line images in the plurality of captured images, Judgment about the deviation from the direction of travel. In this example, when the white line image hardly changes in the plurality of captured images (for example, when the amount of change such as the position and angle of the white line is less than a predetermined value), the CPU 21 changes the shooting direction from the traveling direction. Judge that there is almost no deviation. In this case, the CPU 21 determines that the deviation between the shooting direction and the traveling direction is within a predetermined range, and determines to display the photographed guide image with priority.

  On the other hand, when the white line image is changed in a plurality of photographed images (for example, when the amount of change such as the position and angle of the white line is a predetermined value or more), the photographing direction is changed from the traveling direction. Judge that it is shifted. Further, when the white line image is not included in the plurality of captured images, the CPU 21 determines that the capturing direction is deviated from the traveling direction. In such a case, the CPU 21 determines that the deviation between the shooting direction and the traveling direction is out of the predetermined range, and determines to display the map guidance image with priority.

  Next, with reference to FIG. 5, an application example of the method for determining the deviation between the shooting direction and the traveling direction will be described. FIGS. 5A and 5B are diagrams illustrating an example of an image captured by the camera 29. FIG. Specifically, FIG. 5A illustrates a case where the shooting direction of the camera 29 is not substantially deviated from the traveling direction of the vehicle 3 (that is, the shooting direction of the camera 29 substantially matches the traveling direction of the vehicle 3). FIG. 5B shows an example of a captured image captured when the shooting direction of the camera 29 is deviated from the traveling direction of the vehicle 3.

  When a photographed image as shown in FIG. 5A is obtained, the white line image 50 in the photographed image hardly changes. Therefore, the CPU 21 determines that the deviation between the photographing direction and the traveling direction is within a predetermined range. To do. On the other hand, when a captured image as shown in FIG. 5B is obtained, since the white line image is not included in the captured image, the CPU 21 has a difference between the capturing direction and the traveling direction. Judged outside the predetermined range. Note that the captured images as shown in FIGS. 5A and 5B are used to determine the difference between the shooting direction and the traveling direction, and basically the determination is made. During the time, it is not displayed on the display unit 25.

  As described above, according to the present embodiment, the guide image to be displayed among the photographed guide image and the map guide image is determined by appropriately determining the deviation between the shooting direction of the camera 29 and the traveling direction of the vehicle 3. It can be switched appropriately. Thereby, in a situation where the shooting direction of the camera 29 is deviated from the traveling direction of the vehicle 3, it is possible to suppress an inappropriate live-action guidance image from being displayed. That is, according to the present embodiment, the live-action guide image can be preferentially switched and displayed only in a situation where an appropriate live-action guide image can be displayed.

  Note that, as described above, the present invention is not limited to determining the difference between the shooting direction and the traveling direction based on changes in white lines in a plurality of captured images. In another example, it is possible to determine a deviation between the shooting direction and the traveling direction based on the position and angle of the white line in the captured image. In this example, the CPU 21 determines that the deviation between the shooting direction and the traveling direction is within the predetermined range when the white line is located within the predetermined range of the captured image or when the inclination of the white line is an angle within the predetermined range. To do. On the other hand, the CPU 21 determines that the deviation between the shooting direction and the traveling direction is out of the predetermined range when the white line is not located within the predetermined range of the captured image or when the inclination of the white line is not an angle within the predetermined range. To do.

  Note that, as described above, even when the display is preferentially displayed based on the difference between the shooting direction and the traveling direction, the display is preferentially determined depending on the setting by the user. In some cases, the guidance image is not displayed. For example, even when the CPU 21 determines that priority is to display the live-action guide image because the deviation between the shooting direction and the traveling direction is within a predetermined range, the setting for automatically switching to the AR navigation is turned off. If it is, the map guide image is displayed without displaying the live-action guide image.

[Processing flow]
Next, a processing flow executed by the CPU 21 in this embodiment will be described with reference to FIGS.

  FIG. 6 shows a processing flow executed when a navigation (AR navigation or normal navigation) application is activated in this embodiment. The processing flow is realized by the CPU 21 in the terminal device 2 executing a program stored in the ROM 22 or the like.

  First, in step S101, the CPU 21 displays a normal map image on the display unit 25. Specifically, the CPU 21 generates a normal map image based on the map information acquired from the server via the communication unit 24 or the map information stored in the storage unit, and causes the display unit 25 to display the normal map image. The reason why the normal map image is displayed instead of the live-action guidance image at the start of the processing flow is to allow an operation such as setting a destination to be performed on the normal map image. In addition, it is considered that there is no need to display a live-action guide image at the start of the processing flow. After step S101, the process proceeds to step S102.

  In step S <b> 102, the CPU 21 determines whether or not the terminal device 2 is attached to the terminal holding device 1. For example, a sensor that detects attachment and detachment of the terminal device 2 is provided in the terminal holding device 1 or the like, and the CPU 21 can obtain an output signal from the sensor and perform the determination in step S102. If the terminal device 2 is attached to the terminal holding device 1 (step S102; Yes), the process proceeds to step S103. If the terminal device 2 is not attached to the terminal holding device 1 (step S102; No), the process is performed. The process returns to step S102.

  In step S103, the CPU 21 determines whether a destination has been set. Specifically, the CPU 21 determines whether or not the user inputs a destination by operating the operation unit 28 or the like. This determination is performed because the destination setting is one of the conditions for starting route guidance. If the destination has been set (step S103; Yes), the process proceeds to step S106. If the destination has not been set (step S103; No), the process returns to step S103.

  Note that the order of executing the determination in step S102 and the determination in step S103 may be reversed. That is, whether or not the terminal device 2 is attached to the terminal holding device 1 after determining whether or not the destination has been set (specifically, when it is determined that the destination has been set). It is good also as judging.

  In step S106, the CPU 21 determines whether or not the AR navigation automatic switching setting is on. That is, the CPU 21 determines whether or not the user has been set to automatically switch to AR navigation. If the AR navigation automatic switching setting is on (step S106; Yes), the process proceeds to step S107.

  In step S <b> 107, the CPU 21 controls the camera 29 to perform shooting. And CPU21 acquires the picked-up image image | photographed with the camera 29. FIG. Thereafter, the process proceeds to step S108. Note that the CPU 21 internally performs image processing on the captured image without displaying the captured image on the display unit 25 until the AR navigation is activated. In other words, the captured image is used when processing for determining a shift between the capturing direction of the camera 29 and the traveling direction of the vehicle 3 to be described later, while the CPU 21 captures the image during such processing. Do not display images. During this time, the CPU 21 displays a normal map image.

  In step S108, the CPU 21 starts route guidance with normal navigation. Specifically, the CPU 21 performs a route search from the current location to the destination based on the map information and the like, and causes the display unit 25 to display a map guidance image (normal map image) based on the searched route. As described above, although the AR navigation automatic switching setting is ON, the route guidance is started by the normal navigation, and it is uncertain whether or not the AR navigation can be appropriately performed at this stage. Because. In other words, in a situation in which it is uncertain whether AR navigation can be performed properly, it is preferable to display a normal map guidance image rather than displaying a live-action guidance image for the convenience of the user. Because. After step S108, the process proceeds to step S109.

  Note that the order in which the process of step S107 and the process of step S108 are executed may be reversed, or the process of step S107 and the process of step S108 may be performed simultaneously. In other words, the route may be taken with the camera 29 after starting the route guidance with the normal navigation, or the route may be taken with the camera 29 simultaneously with the start of the route guidance with the normal navigation.

  In step S <b> 109, the CPU 21 determines whether or not the shooting direction of the camera 29 substantially matches the traveling direction of the vehicle 3. In other words, the CPU 21 determines whether or not the deviation between the shooting direction of the camera 29 and the traveling direction of the vehicle 3 is within a predetermined range. For example, as described above, the CPU 21 recognizes a white line image existing on a road in the captured image by performing image processing on the captured image, and determines the shooting direction and the traveling direction based on the white line image. Judgment is made on the deviation. In this example, the CPU 21 uses a plurality of captured images obtained when the vehicle 3 travels a certain distance, and based on the change in the white line in the plurality of captured images, the difference between the capturing direction and the traveling direction. Judgment is made. When the white line images in the plurality of captured images are hardly changed, the CPU 21 determines that the shooting direction is substantially coincident with the traveling direction (step S109; Yes), in other words, the shooting direction and the traveling direction. Is determined to be within a predetermined range. In this case, the CPU 21 determines that the AR navigation can be properly performed, and activates the AR navigation (step S111). Specifically, the CPU 21 causes the display unit 25 to display a live-action guidance image in which an image for route guidance is superimposed on an image captured by the camera 29. Then, the process ends.

  On the other hand, when the white line image has changed in a plurality of captured images, the CPU 21 determines that the capturing direction is deviated from the traveling direction (step S109; No), in other words, the capturing direction and the progress. It is determined that the deviation from the direction is out of the predetermined range. In this case, the CPU 21 continues route guidance using normal navigation (step S110). In other words, the CPU 21 continuously displays the normal map image. Then, the process returns to step S109. That is, the processes in steps S109 and S110 are repeatedly executed until the shooting direction substantially matches the traveling direction, specifically, until the shooting direction substantially matches the traveling direction by the user adjusting the shooting direction. If the normal map image is displayed continuously even though the AR navigation automatic switching setting is set to ON, the user determines that the shooting direction does not substantially match the traveling direction, and the shooting is performed. The direction can be adjusted. That is, the user can adjust the shooting direction while checking the type of the displayed guidance screen.

  On the other hand, when the AR navigation automatic switching setting is not on (step S106; No), the process proceeds to step S112. In step S112, CPU21 starts route guidance by normal navigation in the same procedure as said step S108. Then, the process ends. Such normal navigation is executed until the vehicle 3 arrives at the destination.

  Next, with reference to FIG. 7, a processing flow performed during execution of AR navigation will be described. Specifically, the processing flow is executed after step S111 described above. The processing flow is also realized by the CPU 21 in the terminal device 2 executing a program stored in the ROM 22 or the like.

  First, in step S201, the CPU 21 determines whether or not an operation has been performed on the terminal device 2 by the user. That is, the CPU 21 determines whether or not the user has performed an operation on the operation unit 28 or the like during execution of AR navigation. For example, it is determined whether or not an operation of depressing a switch button for switching from a live-action guide image to a normal map image or an operation of depressing a button for resetting a destination has been performed. When operation with respect to the terminal device 2 is performed (step S201; Yes), a process progresses to step S202.

  In step S202, the CPU 21 ends the AR navigation and switches the display image from the live-action guide image to the normal map image. The reason for this is as follows. First, when the switch button for switching from the live-action guide image to the normal map image is pressed, it is considered that the real-life guide image should be switched to the normal map image immediately. In addition, if the button for resetting the destination is pushed down instead of the switching button, it is considered desirable to have the operation such as resetting the destination on the normal map image. is there. Furthermore, as can be said when all the buttons on the terminal device 2 are operated, when the operation on the terminal device 2 is performed, the shooting direction of the camera 29 changes, and the shooting direction tends to deviate from the traveling direction. It is. That is, there is a possibility that an appropriate live-action guidance image cannot be displayed.

  After step S202, the process proceeds to step S103 shown in FIG. In this case, the processing after step S103 is performed in the same procedure as that shown in FIG. This is because, when an operation is performed on the terminal device 2 as described above, it is determined whether or not the destination has been set (step S103), and the shooting direction of the camera 29 is the progress of the vehicle 3. This is because it is desirable to determine again whether or not the direction substantially matches (step S109). That is, when an operation is performed on the terminal device 2 as described above, the user is again made to set the destination, adjust the tilt of the terminal holding device 1, adjust the shooting direction of the camera 29, and the like. This is because it is desirable.

  On the other hand, when operation with respect to the terminal device 2 is not performed (step S201; No), a process progresses to step S203. In step S <b> 203, the CPU 21 determines whether or not the terminal device 2 has been removed from the terminal holding device 1. For example, a sensor that detects attachment and detachment of the terminal device 2 is provided in the terminal holding device 1 or the like, and the CPU 21 can obtain an output signal from the sensor and perform the determination in step S203. When the terminal device 2 is removed from the terminal holding device 1 (step S203; Yes), the process proceeds to step S204.

  In step S204, the CPU 21 ends the AR navigation and switches the display image from the live-action guide image to the normal map image. This is because when the terminal device 2 is detached from the terminal holding device 1, it is difficult for the user to use the route guidance with reference to the live-action guide image, that is, the live-action guide image is displayed. This is because there seems to be no necessity.

  After step S204, the process proceeds to step S102 shown in FIG. That is, it is determined again whether or not the terminal device 2 is attached to the terminal holding device 1 (step S102). And when the terminal device 2 is attached to the terminal holding device 1 (step S102; Yes), the process after step S103 is performed in the procedure similar to the procedure shown in FIG. This is because if the terminal device 2 is attached to the terminal holding device 1 after being detached from the terminal holding device 1, it is determined whether or not the destination has been set (step S103) It is determined whether or not the apparatus 1 is substantially horizontal or substantially perpendicular to the ground (step S104), and whether or not the shooting direction of the camera 29 substantially matches the traveling direction of the vehicle 3 (step S109). This is because it is desirable to repeat the above. That is, when the terminal device 2 is attached to the terminal holding device 1 after being detached from the terminal holding device 1, the user can again adjust the tilt of the terminal holding device 1 and the shooting direction of the camera 29 to the user. This is because it is desirable to do this.

  On the other hand, when the terminal device 2 is not removed from the terminal holding device 1 (step S203; No), the process proceeds to step S205. In step S205, the CPU 21 determines whether or not the vehicle 3 has arrived at the destination. When arriving at the destination (step S205; Yes), the CPU 21 ends the AR navigation, and switches the display image from the photographed guide image to the normal map image (step S206). Thereafter, the process ends. On the other hand, when it has not arrived at the destination (step S205; No), a process returns to step S201.

  According to the processing flow described above, it is possible to appropriately switch the guide image to be displayed among the photograph guide image and the map guide image (normal map image). Specifically, an appropriate guidance screen corresponding to the situation can be automatically switched and displayed with priority without the user performing a switching operation.

[Modification]
Hereinafter, modifications of the above-described embodiment will be described.

(Modification 1)
In the above, an example in which the deviation between the shooting direction and the traveling direction is determined based on the white line image on the road in the shot image. In the first modification, instead of using the white line in the captured image, the difference between the capturing direction and the traveling direction is determined based on the ratio of the road image in the captured image. Specifically, in the first modification, the CPU 21 analyzes the captured image to obtain the ratio of the road image in the captured image, and compares the determined ratio with a predetermined value to determine the capturing direction. And the deviation from the direction of travel. If the calculated ratio is equal to or greater than the predetermined value, the CPU 21 determines that the deviation between the shooting direction and the traveling direction is within the predetermined range and determines to display the live-action guide image with priority. On the other hand, if the calculated ratio is less than the predetermined value, the CPU 21 determines that the shift between the shooting direction and the traveling direction is out of the predetermined range and determines to display the map guidance image with priority.

(Modification 2)
In Modification 2, instead of using the white line in the photographed image or the ratio of the road in the photographed image, the difference between the photographing direction and the traveling direction is determined based on the position of the road image in the photographed image. . Specifically, in the second modification, the CPU 21 recognizes a road image in the captured image by performing image analysis on the captured image, and determines whether or not the road image is within a predetermined range of the captured image. The deviation between the shooting direction and the traveling direction is determined. When the road image is located within a predetermined range of the photographed image, for example, when the road image is located in a substantially central region of the photographed image, the CPU 21 determines that the deviation between the photographing direction and the traveling direction is within the predetermined range. Therefore, it is determined that the live-action guide image is displayed with priority. On the other hand, when the road image is not located within the predetermined range of the photographed image, for example, when the road image is located at the end region of the photographed image, the CPU 21 has a deviation between the photographing direction and the traveling direction within the predetermined range. It is determined that the map guidance image is to be displayed preferentially.

(Modification 3)
In the third modification, instead of determining the difference between the photographing direction and the traveling direction by performing image analysis on the photographed image as in the above-described embodiment and the first and second modifications, the terminal device 2 and / or the terminal holding device 1 is used. Based on the output of the sensor provided in, a deviation between the shooting direction and the traveling direction is determined. Specifically, in Modification 3, the CPU 21 determines the difference between the shooting direction and the traveling direction based on the output of a sensor that detects the traveling state (speed, acceleration, position, etc.) of the vehicle 3. In one example, the CPU 21 is provided in the terminal holding device 1 and can detect a speed in at least a two-dimensional direction (not limited to a sensor that directly detects a speed, but also a sensor that can detect a speed indirectly). The travel direction is determined based on the output of the image including the travel direction to determine the deviation between the shooting direction and the travel direction.

  Here, with reference to FIG. 8, an example of a method for determining the deviation between the shooting direction and the traveling direction will be described.

  FIG. 8A shows a view of the terminal device 2 held by the terminal holding device 1 as viewed from above. In FIG. 8A, for convenience of explanation, the terminal holding device 1 and the terminal device 2 are illustrated in a simplified manner. As illustrated in FIG. 8A, a sensor 15 d is provided in the substrate holder 15 of the terminal holding device 1. The sensor 15d is an acceleration sensor (in other words, a G sensor) configured to be able to detect acceleration in a two-dimensional direction. Hereinafter, “sensor 15d” is referred to as “acceleration sensor 15d”. As described above, in a state where the terminal device 2 is held by the terminal holding device 1 (specifically, a state where the connector of the terminal device 2 and the connector 16a in the terminal holder 16 are connected), the output signal of the acceleration sensor 15d is , And supplied to the terminal device 2 through the sensor substrate 15c in the substrate holder 15, the wiring 16b in the terminal holder 16, and the connector 16a. In this case, the CPU 21 in the terminal device 2 acquires the output signal of the acceleration sensor 15d.

  Specifically, the acceleration sensor 15d detects the acceleration in the X direction and the acceleration in the Y direction as shown in FIG. Since the acceleration sensor 15d is fixed to the terminal holding device 1 and the positional relationship with the camera 29 of the terminal device 2 attached to the terminal holding device 1 is constant, the acceleration sensor 15d detects the acceleration in the X and Y directions. There is a fixed relationship with the shooting direction of the camera 29. In addition, as shown to Fig.8 (a), it shall be comprised so that a X direction and an imaging | photography direction may correspond.

  FIG. 8B shows the terminal device 2 in a state of being held by the terminal holding device 1 as in FIG. 8A, but here the terminal device 2 faces the traveling direction of the vehicle 3. The figure of the state which is not, ie, the state which the imaging | photography direction of the camera 29 does not correspond with the advancing direction of the vehicle 3 is shown. In a state where the terminal device 2 is held by the terminal holding device 1, the direction of the terminal holding device 1 matches the direction of the terminal device 2. Therefore, it can be said that the acceleration sensor 15d in the terminal holding device 1 can appropriately detect the direction of the terminal device 2 (specifically, the shooting direction of the camera 29 in the terminal device 2).

FIG. 8 (c) shows only the acceleration sensor 15d in FIG. 8 (b). The acceleration sensor 15d detects acceleration in a two-dimensional direction with respect to the X direction and the Y direction as shown in FIG. The X direction corresponds to the shooting direction of the camera 29. If the shooting direction of the camera 29 is deviated from the traveling direction of the vehicle 3, the shooting of the camera 29 in the traveling direction of the vehicle 3 is determined from the ratio of the acceleration in the X direction and the acceleration in the Y direction detected by the acceleration sensor 15d. The shift angle θ in the direction (X direction) can be calculated. The shift angle θ can be calculated from the following equation (1).
Deviation angle θ = arctan (acceleration in the Y direction / acceleration in the X direction) Equation (1)
Specifically, the deviation angle θ is calculated by the CPU 21 in the terminal device 2. In this case, the CPU 21 acquires output signals corresponding to the acceleration in the X direction and the acceleration in the Y direction detected by the acceleration sensor 15d, and calculates the shift angle θ based on the output signal.

  When the deviation angle θ is less than the predetermined angle, the CPU 21 determines that the deviation between the shooting direction of the camera 29 and the traveling direction of the vehicle 3 is within a predetermined range, and the deviation angle θ is equal to or greater than the predetermined angle. If it is, it is determined that the deviation between the shooting direction of the camera 29 and the traveling direction of the vehicle 3 is outside the predetermined range.

  Note that, as described above, the present invention is not limited to determining the difference between the shooting direction and the traveling direction based only on the output of the sensor such as the acceleration sensor 15d. The difference between the photographing direction and the traveling direction may be determined based on the result of image analysis of the photographed image as in FIG. In other words, even if the output of the sensor is combined with the result of image analysis of the photographed image by any one or more of the embodiment and the first and second modifications, the deviation between the photographing direction and the traveling direction is determined. good. By doing so, the shooting direction is generally coincident with the traveling direction, but it is prevented that the shooting guide image is erroneously switched to the map guidance image when there is an obstacle in front of the camera 29. Is possible.

(Modification 4)
In the fourth modification, the CPU 21 determines the difference between the shooting direction and the traveling direction periodically (that is, repeatedly at a predetermined cycle) during execution of the AR navigation, thereby obtaining the photographed guide image and the map guide image. Perform display control to switch. Thereby, when the shift | offset | difference of an imaging | photography direction and a advancing direction arises, it can switch to a map guidance image from a live-action guidance image rapidly.

(Modification 5)
In the above-described embodiment, the present invention is applied to the terminal device 2 mounted on the terminal holding device 1 (that is, the terminal device 2 mounted on the mobile body via the terminal holding device 1). It was what was applied. On the other hand, the modified example 5 is applied to the terminal device 2 that the user simply carries. For example, the modified example 5 is applied when a pedestrian uses route guidance using the terminal device 2.

  With reference to FIG. 9, the modification 5 is demonstrated concretely. As shown in FIG. 9A, when the user uses AR navigation based on a live-action guide image while walking or the like, it is desirable that the shooting direction of the camera 29 is substantially horizontal. It tends to be vertical. That is, the user tends to use the terminal device 2 with the inclination of the terminal device 2 substantially perpendicular to the ground. On the other hand, as shown in FIG. 9B, when the user uses normal navigation based on the map guidance image while walking or the like, the map guidance image is easy to see (for other reasons, as shown in FIG. For example, the user tends to get tired if it is held vertically as shown in FIG. 9 (a)). That is, the user tends to use the terminal device 2 in a state where the inclination of the terminal device 2 is close to the ground level.

  From the above, in Modification 5, the CPU 21 of the terminal device 2 preferentially displays either the live-action guide image or the map guide image based on the relationship between the shooting direction of the camera 29 and the tilt of the terminal device 2. In other words, it is determined which of the AR navigation and the normal navigation should be prioritized. Specifically, when the tilt of the shooting direction of the camera 29 is within a predetermined range with respect to the horizontal plane, the CPU 21 determines to display the live-action guide image with priority, and tilts the shooting direction of the camera 29. Is outside the predetermined range with respect to the horizontal plane, it is determined that the map guidance image is preferentially displayed.

  Note that the “predetermined range” used for such determination is set in advance in consideration of the inclination of the terminal device 2 when the AR navigation and the normal navigation are used by an actual pedestrian. Further, the CPU 21 obtains the inclination of the shooting direction of the camera 29 based on the output of the sensor 15d (gyro sensor) that detects at least one of the angular velocity or acceleration around the horizontal of the moving body.

(Modification 6)
Although the example which applies this invention to a vehicle was shown above, application of this invention is not limited to this. The present invention can be applied to various mobile objects such as ships, helicopters, and airplanes in addition to vehicles.

  As described above, the embodiments are not limited to the above-described embodiments, and can be appropriately changed without departing from the gist or concept of the invention that can be read from the claims and the entire specification.

  INDUSTRIAL APPLICABILITY The present invention can be used for a mobile phone having a call function and a navigation device that provides route guidance.

1 terminal holding device 2 terminal device 3 vehicle 15 substrate holder 16 terminal holder 21 CPU
25 Display unit 28 Operation unit 29 Camera

Claims (7)

A terminal device that can be held by the terminal holding device;
Photographing means;
Display control means for switching and displaying each of a photograph guide image for route guidance using a photographed image photographed by the photographing means and a map guidance image for route guidance using a map image; ,
Have
The terminal device characterized in that the display control means switches from displaying the live-action guide image to displaying the map image when it is detached from the terminal holding device. Setting means for setting which of the route guidance using the photograph guide image or the map guide image is performed;
Even if the display control means is set to perform route guidance using the live-action guide image by the setting means, the display control means is provided when the display control means is removed from the terminal holding device during the route guidance. The terminal device according to claim 1, wherein the display is switched from display of a guide image to display of the map image.   The display control means switches between the live-action guide image and the map guide image based on the relationship between the moving direction of the terminal device and the photographing direction by the photographing means, and causes the display means to display the switching means. Item 3. The terminal device according to Item 1 or 2. The display control means is set to have a route guidance destination and set to provide route guidance using the live-action guidance image, and the photographing direction of the imaging means is the terminal. 4. The live-action guide image is displayed on the display means when the terminal device is attached to the terminal holding device in a state that coincides with the moving direction of the device. The terminal device as described in any one.   The terminal device according to claim 1, wherein the terminal device is mounted on a moving body via a terminal holding device. An image display method that can be held by a terminal holding device and executed by a terminal device having a photographing unit,
A display control step of switching between a live-action guidance image for route guidance using a photographed image photographed by the photographing means and a map guidance image for route guidance using a map image, and displaying on the display means; Have
The image display method according to claim 1, wherein the display control step switches from displaying the live-action guide image to displaying the map image when the terminal holding device is detached. An image display program that can be held by a terminal holding device and that is executed by a terminal device having a photographing unit and a computer,
Display control means for switching between a live-action guidance image for route guidance using a photographed image photographed by the photographing means and a map guidance image for route guidance using a map image to display on the display means Make the computer function,
The image display program according to claim 1, wherein the display control means switches from displaying the live-action guide image to displaying the map image when removed from the terminal holding device.

Images