JP2006101013A - Guide information display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to simply and easily grasp in which a direction the user should advance to a destination on the display unit of a portable terminal with improved visibility. <P>SOLUTION: A synthesized image where an arrow for indicating the direction of the destination and a photographed image transmitted from the portable terminal are synthesized is generated at a server side for transmitting to the portable terminal. At the portable terminal, the received synthesized image is displayed on the display unit for guiding the direction to the destination. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a guide information display method for a mobile terminal such as a camera-equipped mobile phone or a camera-equipped PDA (Personal Digital Assistant), and in particular, on an image of a landscape at the current position photographed by the owner of the mobile terminal. The present invention relates to a guide information display method in which destination direction arrows are superimposed and displayed.

  The in-vehicle navigation system has the function to display the route from the departure point to the destination on the map with an arrow, and the positioning function using the GPS (Global Positioning Sestem) satellite The position is recognized (for example, refer to Patent Document 1).

Also, in a mobile terminal such as a mobile phone or PDA, the current position is measured using either an artificial satellite, a pseudo satellite, or a radio base station, and an arrow indicating the current position and the direction to the destination is displayed on a map or a schematic diagram. There is also a known technology that displays
JP 2004-233230 A JP 2004-156998 A

  In the prior art that displays the guidance information to the destination on the map, the map information is stored in advance in an in-vehicle navigation device, or a portable terminal such as a mobile phone or a PDA that is limited in storage capacity Then, the map information is downloaded. However, unless the map information is always updated to the latest information, it is not suitable for the current situation in a region where stores are frequently replaced. In addition, in the destination display on the map, it is necessary to understand the correlation between the map and the actual landscape, so it is possible to determine which direction the direction of travel should actually proceed with respect to the direction of travel on the map. There is a problem that it is difficult to grasp intuitively with good visibility.

  In addition, it is impractical to install a positioning function that measures the position of a device such as a mobile phone that requires miniaturization and weight reduction. For example, if a GPS positioning function is installed, a GPS antenna and reception for that purpose There is a problem that a machine or the like is required, which hinders downsizing and weight reduction and leads to an increase in cost. Also, GPS cannot be used indoors.

  The present invention has been made in view of the above points. The purpose of the present invention is to determine the direction in which the portable terminal holder should proceed with respect to the destination in the display on the display unit of the portable terminal. The purpose is to make it easy and easy to grasp the direction. In addition, an object of the present invention is to provide a map to the owner of the mobile terminal even if the mobile terminal does not have a positioning function and does not hold or download the map information in advance. It is intended to be able to display a guidance in a destination direction in a very easy-to-understand form that is not used.

In order to achieve the above-described object, in a representative invention according to the present application, a portable terminal, a server for providing guidance information that can communicate with the portable terminal via a network, and various places, the captured images are arranged. In the guide information display method in the system using the sign marker with the information that can identify the location by analyzing the image,
The mobile terminal includes a camera unit, a transmission / reception unit capable of transmitting / receiving information to / from the outside via a network, a storage unit for storing various data, a display unit, an input unit for performing operation input, and an entire mobile terminal A control unit that controls the control,
The server includes a transmission / reception unit capable of transmitting / receiving information to / from the outside via a network, a map information database storing map information, and identification information of a mobile terminal received from the mobile terminal or from a dedicated input terminal A storage table that stores destination information in association with each other, a captured image that includes the sign marker received from the mobile terminal, and a location of the photographed sign marker and a photographing direction in which the sign marker is photographed Based on the received destination information and information on the location of the specified sign marker, the route to the destination is searched with reference to the contents of the map information database. Based on the route direction search unit and the identified shooting direction, the destination image is indicated in the shot image received from the mobile terminal. A synthetic image generation unit which synthesizes a mark has,
When the owner of the portable terminal accesses the server using the portable terminal or the dedicated input terminal, and transmits the identification information of the portable telephone and the information of the destination, the server Storing identification information and destination information of the mobile terminal in the storage table;
The user of the portable terminal takes an image including the sign marker with the camera unit, and transmits the taken image to the server, whereby the server is taken by the position / photographing direction recognition unit. After specifying the location of the sign marker and the shooting direction in which the sign marker was shot, the route search unit searches for the route to the destination, and the synthesized image generation unit sends the destination image to the shot image received from the portable terminal Generating a composite image obtained by combining the arrows indicating, and transmitting the generated composite image to the mobile terminal;
The mobile terminal displaying the composite image received from the server on the display unit;
By providing, a composite image in which an arrow indicating a destination direction to the destination is synthesized on a photographed image taken by the mobile terminal is displayed on the display unit of the mobile terminal, and guidance to the destination is performed. To be.

  According to the present invention, it is possible to easily and easily comprehend with good visibility, which direction the traveling direction that the user should travel with respect to the destination is the display on the display unit of the mobile terminal. Also, even if the mobile device does not have a positioning function, and the map information is not stored or downloaded in advance, the destination of the mobile device owner in a very easy-to-understand form that does not use a map A direction indication can be presented.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram showing an overview of a system for realizing a guidance information display method according to an embodiment of the present invention (hereinafter referred to as the present embodiment). In FIG. 1, 1 is a mobile terminal (camera mobile terminal) such as a camera-equipped mobile phone or camera-equipped PDA, and 2 is a mobile terminal 1 and a network (public network (telephone network, Internet), wireless LAN, etc.) 5 A server for providing guidance information that can be communicated via the mobile phone 3 is fixedly installed in a station or a downtown area, and is dedicated for a user (here, the owner of the mobile terminal 1) to perform an input operation. An input terminal (stationary input terminal) communicates with the server 2 via the dedicated network 6 or public network. In addition, 4 is a sign marker (a sign marker for image recognition) arranged in a key point in a facility, an underground mall, or a downtown area, and preferably has its own coloring to stand out to the human eye, It is marked and placed at a position higher than the height of the adult, and provided with consideration that the surrounding scenery (real space) is photographed together with the sign marker 4. The service range covered by the server 2 can be an area of any size and an arbitrary number of areas. However, for convenience of explanation, the service range covered by the server 2 is a certain specific area. Suppose that it is in a facility.

  FIG. 2 is a block diagram illustrating a functional configuration of each of the components 1 to 4 in the system of FIG. In the mobile terminal 1 of FIG. 2, 101 is a control unit that controls the overall control of the mobile terminal 1, 102 is a videophone function unit (videophone transmission / reception function unit) that exchanges video and audio in both directions, and 103 is e-mail. E-mail function unit (e-mail transmission / reception function unit) for transmitting and receiving, 104 a camera unit (camera function unit) for photographing by the holder of the mobile phone 1, and 105 for displaying a photographed image and various guidance images Part (thin image display panel), 106 is an input operation part for the owner of the mobile phone 1 to perform various operations such as transmission / reception operation, photographing operation, character input, etc. 107 is a telephone number or an e-mail address of the mobile phone 1 The non-contact type RFID (Radio Frequency Identification) tag 108 that stores information is connected to the server 2 or the like via the Internet by a communication protocol conforming to WEB. WEB connection function of, 109 stores various data and a storage unit used as a work area.

  In the server 2 of FIG. 2, 201 is a control unit that controls the overall control of the server 2, 202 is an ID of a member having a membership card and identification information (telephone number, E) of the mobile terminal 1 owned by the member. The member ID / mobile terminal identification information correspondence database 203 that stores data in which the email address is associated is received from the input terminal 3 or the identification information of the mobile terminal 1 and the destination information received from the mobile terminal 1 The portable terminal identification information / destination information correspondence table 204 that holds data associated with destination information for a predetermined period, 204 is a marker ID unique to each sign marker 4 marked on the sign marker 4, and Data in which position information indicating the location of the sign marker 4 and information on the direction (direction) of the display surface of the sign marker 4 is stored is stored An in-marker information database.

  Further, in the server 2 in FIG. 2, reference numeral 205 denotes an image portion of the photographed sign marker 4 extracted from the photographed image including the sign marker 4 received from the mobile terminal 1, and the extracted image portion is analyzed by image processing. A position / photographing direction recognition unit (image analysis determination processing unit) that identifies the location of the photographed sign marker 4 and the photographing direction in which the sign marker 4 was photographed. When the marker ID is marked on the photographed sign marker 4, the position / photographing direction recognition unit 205 first recognizes the marker ID portion and interprets / specifies the marker ID. Then, the position / photographing direction recognition unit 205 refers to the content of the sign marker information database 204 based on the read / specified marker ID, and the orientation of the sign marker 4 on the display surface of the sign marker 4. And the shooting direction in which the sign marker 4 is shot in the opposite direction.

  Alternatively, when the marker ID is marked on the photographed sign marker 4, the position / photographing direction recognition unit 205 refers to the contents of the sign marker information database 204 based on the identified marker ID, and The location of the marker 4 is specified, and the shooting direction in which the sign marker 4 is shot is specified by a three-dimensional recognition process from the shot image including the received sign marker 4 (for example, the left side of the rectangular sign marker 4 is from the right side) If it is larger by a predetermined amount, it is determined that the sign marker 4 is photographed from a predetermined angle on the left front side, etc. It should be noted that such a three-dimensional recognition processing program is known and can be easily obtained through the Internet. is there).

  Alternatively, the position / photographing direction recognition unit 205 does not mark the photographed sign marker 4 with the marker ID, and the photographed sign marker 4 itself has information indicating the location and the direction of the display surface. When the information indicating (azimuth) is marked, the image portion of the photographed sign marker 4 is analyzed by image processing, and the location information marked on the sign marker 4 and the direction in which the display surface faces Is obtained, and the location of the sign marker 4 and the shooting direction in which the sign marker that is opposite to the orientation of the display surface of the sign marker 4 is shot are specified. Alternatively, the position / photographing direction recognizing unit 205 does not mark the photographed sign marker 4 with the marker ID, and only photographed sign marker 4 itself is marked with information indicating the location. The image part of the photographed sign marker 4 is analyzed by image processing to obtain information on the location marked on the sign marker 4 and the received photographed image including the sign marker 4 is subjected to three-dimensional recognition processing. The shooting direction in which the sign marker 4 is shot is specified.

  Note that information identification processing such as color extraction processing, contour extraction processing, intersection extraction processing, and the like is known, and a number of programs for realizing it are also implemented.

  Furthermore, in the server 2 of FIG. 2, 206 is a map information database storing the latest map information, and 207 is a route search unit for searching for a route to the destination based on the destination information and the current position information. The route search unit 207 refers to the portable terminal identification information / destination information correspondence table 203 based on the identification information of the portable terminal 1 that has transmitted the captured image including the sign marker 4. The destination position desired by the owner and the location of the sign marker 4 identified by the position / capturing direction recognition unit 205 from the captured image including the sign marker 4 received from the mobile terminal 1 (the current position of the owner of the mobile terminal 1 ) And the contents of the map information database 206, the shortest or easy-to-understand route from the current position of the owner of the portable terminal 1 to the destination is searched by a predetermined search algorithm.

  Further, in the server 2 of FIG. 2, reference numeral 208 denotes a 3D arrow display generation unit that generates an arrow image to be added to the captured image received from the mobile terminal 1 to indicate the traveling direction to the destination, and 209 denotes the mobile terminal 1. A comment / advice generating unit 210 that generates a comment image or a character image of advice to be added to a captured image received from the mobile phone 1, an arrow image generated by the 3D arrow display generating unit 208, or a comment A composite image generation unit that generates a composite image by combining the character images generated by the advice generation unit 209. Note that the 3D arrow display generation unit 208 receives the captured image received from the mobile terminal 1 based on the shooting direction of the sign marker 4 specified by the position / shooting direction recognition unit 205 and the search result information searched by the route search unit 207. An arrow image of a three-dimensional expression to be added to is generated. Further, the comment / advice generation unit 209, based on the search result information (route information) searched by the route search unit 207, for example, “turn right at the next corner and shoot the sign marker 4 after turning right”. A guidance message significant for the owner of the mobile terminal 1 is generated.

  Further, in the server 2 of FIG. 2, 211 is a videophone function unit (videophone transmission / reception function unit) that exchanges video and audio in both directions, and 212 is an e-mail function unit that transmits and receives e-mail (email transmission / reception). (Function unit) 213 is a WEB connection function unit accessible by the portable terminal 1 through the Internet, and 214 is a storage unit that stores various data and is used as a work area.

  Also, in the input terminal 3 in FIG. 2, 301 is a control unit that performs overall control of the input terminal 3, 302 is a large-screen display unit (thin image display panel) that displays various guidance images, and 303 is a display unit. A touch panel (for example, an infrared touch panel) 304 as an operation input unit integrally provided on 301 is installed on the back side of the display unit 302, and is used as an RFID tag 107 built in the mobile terminal 1 or a member card possessed by a member. An RFID reader that reads the content recorded in the built-in RFID tag 107 in a non-contact manner, 305 is a connection function unit (for example, a WEB connection function unit) with the server 2, 306 stores various data, and serves as a work area It is a storage unit used.

  Further, in the sign marker 4 in FIG. 2, 401 is a marker ID expression unit that represents a marker ID unique to each sign marker. The sign marker 4 is not marked with a marker ID (marker ID expression unit 401). Instead, information indicating the location of the sign marker 4 and information indicating the direction (orientation) of the display surface are provided. Can be expressed directly. Further, the sign marker 4 has a three-dimensional recognition for specifying the photographing direction in which the sign marker 4 is photographed by the three-dimensional recognition process performed by the position / photographing direction recognition unit (image analysis determination processing unit) 205 of the server 2. You may provide the recognition part (rectangular expression part etc.) of a process target. In addition, the sign marker 4 is provided with a marker ID, information indicating the location, information indicating the direction (azimuth) of the display surface, and a three-dimensional recognition processing target recognition unit. A combination of these may also be provided.

  As described above, the sign marker 4 is arranged in a facility or in an important place such as an underground mall or a downtown area. For example, the sign marker 4 is attached to a wall facing a passage such as a store or a building, It is affixed to the front and back of billboards, guide boards, etc. that project at right angles to the passage from buildings. The number of sign markers 4 and the proximity of the positions are arbitrary, but in any case, different information is marked on each sign marker 4. That is, even with the two sign markers provided on the front and back of the same guide plate, for example, the marker ID and the information indicating the direction (orientation) of the display surface are different.

  An overview of the guidance information display method of the present embodiment, which is realized in the system configuration shown in FIGS. 1 and 2, will be described. First, the owner of the portable terminal 1 accesses the server 2 using the dedicated input terminal 3 or the portable terminal 1, and the identification information (telephone number, e-mail address) of the portable terminal 1 and destination information , The server 2 stores the identification information and destination information of the portable terminal 1 in the portable terminal identification information / destination information correspondence table 203. Next, when the holder of the portable terminal 1 takes an image including the sign marker 4 with the camera unit 104 and transmits the taken image to the server 1, the position / capturing direction recognition unit 205 of the server 2 is taken. After specifying the location of the sign marker 4 and the shooting direction in which the sign marker 4 was shot, the route search unit 207 of the server 2 searches for a route from the position of the sign marker to the destination. Then, the 3D arrow display generation unit 208 of the server 2 generates an arrow image of a three-dimensional expression to be added to the photographed image received from the mobile terminal 1, and the comment / advice generation unit 209 of the server 2 receives from the mobile terminal 1. A character image of a guidance message to be added to the received captured image is generated, and the composite image generation unit 210 of the server 2 combines the captured image received from the mobile terminal with the arrow indicating the destination direction and the guidance message. Generate an image. The generated composite image is transmitted from the server 2 to the mobile terminal 1, and the mobile terminal 1 displays the received composite image on the display unit 105, so that the owner of the mobile terminal 1 can view the real space on the spot. The direction of travel to the destination is grasped by the arrow display in the photographed image. The outline of the guidance information display method of the present embodiment has been described above. Note that the exchange of image data between the portable terminal 1 and the server 2 may be performed using a TV phone function or as an e-mail attachment file.

  Next, a procedure for setting (storing) the destination in the server 2 using the dedicated input terminal 3 will be described. FIG. 3 is a diagram showing the input terminal 3. As shown in FIG. 3, the input terminal 3 includes the touch panel 303, the display unit 302 including a thin image display panel in order from the outermost layer, and the RFID reader that performs non-contact reading. 304 is laminated. Therefore, the input terminal 3 can be the owner of the portable terminal 1 or the card by simply touching the desired item displayed on the display unit 302 with the portable terminal 1 incorporating the RFID tag 107 or the card incorporating the RFID tag. And the identification information (telephone number, e-mail address) of the mobile terminal 1 stored in the RFID tag 106 of the mobile terminal 1 or the member ID stored in the RFID tag of the card (Card ID) is recognized.

  FIG. 4 is a diagram illustrating a state in which the holder of the portable terminal 1 sets (stores) a desired destination in the server 2 using the input device 3. FIG. 4A shows the display state of the initial menu image of the input device 3. When the owner of the mobile terminal 1 touches the item “Navi Service” with a finger or the like in this state, the input device 3. Changes to the display state of the navigation service guidance image as shown in FIG. In this navigation service guidance image, for example, items such as high-level store types and product types, and operation guidance such as “Please touch the store you are looking for with a membership card or registered mobile phone.” Is displayed.

  The owner of the portable terminal 1 selects a destination candidate (selects the type of store he wants to go to, the product he is looking for, etc.) from the navigation service guidance image shown in FIG. As shown in FIG. 4, the desired item in the navigation service guidance image is touched with the membership card 7, or as shown in FIG. The desired item in the navigation service guidance image is touched. As a result, the input device 3 acquires the member ID (card ID) of the member card 7 or the identification information (telephone number, e-mail address) of the mobile terminal 1 and simultaneously recognizes the touched item. When the item selected by touching in (c) and (d) of FIG. 4 is an upper item and there are further lower items, the input device 3 is connected to the next level as shown in (e) of FIG. Transition to the display state of the navigation service guidance image. The holder of the portable terminal 1 further selects an item of a desired destination from the navigation service guidance image of FIG. 4E and touches the item (in this case, the touch is a finger, even a member) It may be the card 7 or the portable terminal 1). Then, the input terminal 3 recognizes the item touched in the lowest navigation service guidance image as the destination that the owner of the mobile terminal 1 wants to go to. After this recognition, the input terminal 3 transitions to a display image indicating that the user operation for navigation service guidance has been completed as shown in FIG. The display returns to the initial menu image display state shown in FIG.

  When the user ID of the member card 7 or the identification information of the portable terminal 1 and the destination information are recognized by the above-described user operation by the holder of the portable terminal 1, the input device 3 The identification information of the portable terminal 1 and the destination information are associated with each other and transmitted to the server 2 (communication means and route at this time are arbitrary). In response to this, when the identification information of the portable terminal 1 and the destination information are transmitted, the server 2 stores them in the portable terminal identification information / destination information correspondence table 203, and the member ID And the destination information are transmitted, the contents of the member ID / mobile terminal identification information correspondence database 202 are referred to, and the identification information of the mobile terminal 1 corresponding to the member ID is retrieved from the member ID. Then, the identification information of the portable terminal 1 and the destination information are stored in the portable terminal identification information / destination information correspondence table 203, and the destination guidance operation mode for the portable terminal 1 is set.

  When the server 2 enters the destination guidance operation mode for the mobile terminal 1, first, the server 2 transmits an email notifying that the destination input of the destination guidance service has been accepted by email to the mobile terminal 1. FIG. 4G shows an example of mail display on the display unit of the mobile terminal 1 that has received this mail. Here, the destination specified by the owner of the mobile terminal 1 and the guidance information of the present invention are shown. For a user who does not desire the destination guidance by the display method, he / she wants the address (URL) of the WEB site of the server 2 for performing the guidance by the map / route and the destination guidance by the guidance information display method of the present invention. In addition, the telephone number of the server 2 for guiding the user with the mobile terminal 1 having the TV phone function with an arrow for the captured image is displayed.

  Then, in the display state of FIG. 4G, when the cursor is moved to the location of the telephone number of the server 2 and the send button of the mobile terminal 1 is pushed, the mobile terminal 1 is connected to the location of the telephone number of the server 2 Recognize the phone number, automatically call the server 2 (automatically call), and in the display state of (g) in FIG. 4, place the cursor on the address of the WEB site, When the transmission button of the portable terminal 1 is pushed, the portable terminal 1 recognizes the address (URL) of the WEB site of the server 2 linked to the address portion, and automatically accesses the WEB site of the server 2. (Such a mechanism can be easily constructed). In the following description, the selection of the telephone number of the server 2 is referred to as the selection of “Start navigation service via TV phone”, the selection of the address of the WEB site of the server 2 is the selection of “display search results on the web” Called.

  The user who received the mail shown in (g) of FIG. 4 and desires destination guidance by the guidance information display method of the present invention and possesses the portable terminal 1 having the TV phone function is displayed by the telephone number displayed in the mail. When the telephone is connected and the line connection with the server 2 is made, the portable terminal 1 and the server 2 are automatically connected to the TV phone. Thereafter, the holder (user) of the portable terminal 1 takes a picture of the sign marker 4 ((h) in FIG. 4), and a photographed image including the sign marker 4 is transmitted to the server 2.

  Further, when a user who has received the mail shown in FIG. 4G and does not desire destination guidance in the guidance information display method of the present invention accesses the WEB site of the server 2 indicated by the mail, the map / route is used. Guidance information is transmitted to the mobile terminal 1, and the display unit 105 of the mobile terminal 1 displays the current position on a simple map, similar to a conventionally provided service, as shown in FIG. The route will be displayed.

  In this way, when the method of storing the destination in the server 2 using the dedicated input terminal 3 is used, the destination can be easily and reliably searched by the display on the large screen of the input terminal 3.

  In the procedure for storing the destination in the server 2 using the dedicated input terminal 3 described above, the identification information of the mobile terminal 1 or the membership card 7 can be obtained by touching the mobile terminal 1 or the membership card 7 having an RFID tag. The user ID is automatically read to reduce the user's trouble, but the identification information of the mobile terminal 1 may be input by the user himself / herself. In addition to the membership card 7 having an RFID tag, a magnetic card, for example, may be used as long as ID information linked to the portable terminal 1 is recorded on the card. 3 will be equipped with a magnetic card reader.

  Next, a procedure for setting (storing) the destination in the server 2 using the portable terminal 1 will be described. FIG. 5 is a flowchart showing a procedure for the owner of the portable terminal 1 to set (store) a desired destination in the server 2.

  First, the user who is the owner of the mobile terminal 1 acquires the address (URL (Uniform Resource Locator)) of the WEB site of the server 2 that provides the guidance information providing service (step S1). As a method of acquiring this URL, a blank mail is transmitted from the portable terminal 1 to the server 2 that provides the guidance information providing service, and the URL of the WEB site is returned from the server 2 by mail (in this case, the portable terminal 1 is assumed to be stored in the e-mail address book of the destination server 2), or a two-dimensional barcode representing the URL of the WEB site of the server 2 is displayed on the mobile terminal 1. The URL of the WEB site is acquired by photographing with the camera unit 104 and the mobile terminal 1 analyzing the image, or the owner (user) of the mobile terminal 1 acquires the URL of the WEB site by itself (to a person) Such as listening or obtaining from a printed material). It should be noted that the two-dimensional barcode is desirably arranged in the vicinity of the sign marker 4. In this way, when a user who notices the presence of the sign marker 4 recognizes that he / she is within the service area for providing guidance information and decides to receive the service, a two-dimensional bar will be displayed on the spot. This is because the URL of the WEB site of the server 2 that provides the guidance information providing service can be easily acquired by photographing the code. If the URL of the WEB site of the server 2 is registered in “Favorites” of the WEB function of the portable terminal 1, step S1 can be omitted.

  Next, the owner of the portable terminal 1 accesses the WEB site of the server 2 that provides the guidance information providing service from the portable terminal 1 using the URL acquired in Step S1 (Step S2). It should be noted that when a URL is photographed by photographing a two-dimensional barcode, it is possible to automatically make a mechanism for accessing the WEB site of the server 2.

  Next, display data is transmitted from the server 2 that has received access to the WEB site from the mobile terminal 1 to the mobile terminal 1 to input an appropriate telephone number and e-mail address, and the owner of the mobile terminal 1 The server 2 acquires the telephone number and the e-mail address of the portable terminal 1 by inputting the telephone number and the e-mail address of the own device interactively with the display content of the display unit 105 of the terminal 1 (step S3). Next, the server 2 transmits navigation service guidance display data for appropriate destination input (or destination selection) to the mobile terminal 1, and a navigation is displayed on the display unit 105 of the mobile terminal 1 that receives the navigation service guidance display data. A service guidance image is displayed, and a destination desired by the owner of the portable terminal 1 is input interactively with the displayed content (step S3). This destination can be entered from a hierarchical menu display, input an address, enter a postal code, enter a number on the map, enter the product or store you are looking for, functions, events, etc. Such a method can be considered. Using such a method, the owner of the mobile terminal 1 inputs and determines the final destination. As a result, the server 2 acquires the identification information and destination information of the mobile terminal 1, and stores the acquired identification information and destination information of the mobile terminal 1 in the mobile terminal identification information / destination information correspondence table 203. The destination guidance operation mode for the portable terminal 1 is taken.

  When the server 2 enters the destination guidance operation mode for the mobile terminal 1, first, the mobile terminal 1 transmits display data notifying that the destination input of the destination guidance service has been accepted by WEB data. The display content on the display unit 105 of the mobile terminal 1 that has received this display data is not shown, but is substantially the same as the display content of FIG. 4G, for example, and the owner of the mobile terminal 1 And the address (URL) of the WEB site of the server 2 for providing guidance on a map / route for a user who does not desire destination guidance in the guidance information display method of the present invention, For the user who desires the destination guidance by the guidance information display method and possesses the portable terminal 1 having the TV phone function, the telephone number of the server 2 for performing guidance with the arrow for the photographed image is displayed. The In such a display state, when the cursor is moved to the location of the telephone number of the server 2 and the send button of the mobile terminal 1 is pushed, the mobile terminal 1 recognizes the telephone number of the server 2 linked to the location of the telephone number. Then, a call is automatically made to the server 2 (step S4). In such a display state, when the cursor is placed on the address of the WEB site and the send button of the mobile terminal 1 is pushed, the mobile terminal 1 1 recognizes the address (URL) of the WEB site of the server 2 linked to the location of the address, and automatically accesses the WEB site of the server 2 (step S4). Can be easily constructed). In the following description, the selection of the telephone number of the server 2 is also referred to as the selection of “Start a navigation service with a TV phone”. The selection of the address of the WEB site of the server 2 is also referred to as “search result. This is referred to as “display on WEB”. When “display search results in WEB” is selected, guidance information based on a map / route is transmitted to the mobile terminal 1, and the display unit 105 of the mobile terminal 1 has a simple map similar to a service provided conventionally. The current position and route are displayed above (step S4). When “Start navigation service via TV phone” is selected, the line connection is established with the server 2, and the mobile terminal 1 and the server 2 are automatically connected to the TV phone (step S4).

  Thereafter, when the holder (user) of the portable terminal 1 captures the sign marker 4, a captured image including the sign marker 4 is transmitted to the server 2 (step S5).

  Next, a process in which the server 2 generates a composite image and transmits it to the mobile terminal 1 will be described with reference to FIG. FIG. 6 is an explanation mainly showing an image of a processing flow by the server.

  FIG. 6A shows an image of a real space (actual world) that the owner (user) of the mobile terminal 1 is currently viewing. In such a real space, the owner of the mobile terminal 1 Then, the surrounding area including the sign marker 4 is photographed by the camera unit 105 as shown in FIG. The photographed image including the photographed sign marker 4 is automatically transmitted from the portable terminal 1 to the server 2 by the TV phone function.

  The server 2 detects and extracts the sign marker portion from the photographed image including the sign marker 4 transmitted by the position / photographing direction recognizing unit 205. From the extracted image of the sign marker portion, for example, a marker ID is used here. Interpret and identify. At the same time, the position / photographing direction recognizing unit 205 specifies the photographing direction in which the sign marker 4 is photographed, for example, by the above-described three-dimensional recognition process ((c) in FIG. 6). Furthermore, the position / photographing direction recognition unit 205 refers to the sign marker information database 204 based on the identified marker ID and identifies the position of the sign marker 4 (the current position of the user) ((d) in FIG. 6). . As described above, other methods can be used for specifying the position information of the sign marker 4 and the shooting direction.

  Next, the mobile terminal obtained by the route search unit 207 referring to the mobile terminal identification information / destination information correspondence table 203 based on the identification information of the mobile terminal 1 that has transmitted the captured image including the sign marker 4 A predetermined search algorithm based on destination information desired by one owner, information on the location of the sign marker 4 (the current position of the user) identified by the position / photographing direction recognition unit 205, and the contents of the map information database 206 Thus, the shortest route or the easy-to-understand route from the current position of the owner (user) of the portable terminal 1 to the destination is searched. Further, the route search unit 207 determines the direction in which the user must first advance from the current position from the search result ((e) of FIG. 6).

  Next, based on the shooting direction of the sign marker 4 specified by the position / shooting direction recognition unit 205 and the search result information by the route search unit 207, the 3D arrow display generation unit 208 creates a shot image received from the mobile terminal 1. A three-dimensional representation arrow image (3D arrow image) to be added is generated ((f) in FIG. 6). If the position / shooting direction recognition unit 205 specifies the shooting direction of the sign marker 4 by the three-dimensional recognition process, the 3D arrow is created as exactly matching the shooting direction of the sign marker 4. Further, in the case where specification of the photographing direction of the sign marker 4 in the position / photographing direction recognition unit 205 is based on direction information on the display surface of the sign marker 4, the sign marker 4 is a passage such as a store or a building. 3D arrow is created as a flat arrow, and the sign marker 4 is placed on the front and back of a signboard, a guide board, etc. that protrudes at right angles to the passage from a store or a building. When affixed, the 3D arrow is created as an arrow tilted by a certain angle.

  Further, the comment / advice generating unit 209 generates a guidance message (comment, advice) significant for the owner (user) of the mobile terminal 1 based on the search result information (route information) by the route search unit 207 (FIG. 6). (G)).

  Next, the composite image generation unit 210 generates a composite image in which the 3D arrow indicating the destination direction and the guidance message are combined with the captured image including the sign marker 4 transmitted from the mobile terminal 1, and the generated composite The image is transmitted from the server 2 to the portable terminal 1 ((h) in FIG. 6). Note that image synthesis is performed so that the display position of the 3D arrow matches the position of the extracted sign marker 4.

  FIG. 7 is an explanatory diagram showing a real space (actual world) currently viewed by the owner (user) of the mobile terminal 1 and a composite image displayed on the display unit 105 of the mobile terminal 1. Just by the user finding the sign marker 4 and photographing the surrounding scenery so as to include the sign marker 4, a 3D arrow, a destination name, and a guidance message are displayed on the display unit 105 of the mobile terminal 1. And a superimposed image is displayed. Therefore, the user can easily determine which direction the user should travel with respect to the destination with an arrow display in the real space captured image similar to the real space seen on the spot. Easy to grasp. Furthermore, it is possible to accurately grasp how far the vehicle travels in the direction of the arrow by the guidance message, and to know in advance where the sign marker 4 needs to be photographed next. it can. In this example, a guidance message is displayed. However, a voice guidance message is transmitted from the server 2 to the mobile terminal 1 together with the guidance message or in place of the guidance message. Thus, the portable terminal 1 can be configured to output a guidance message by voice.

  FIG. 8 is a diagram showing the relationship of the process flow between the input terminal 2, the server 3, and the portable terminal 1, and FIG. 9 is the diagram showing the relationship of the process flow between the portable terminal 1 and the server 3, Details of processing of each device are as described above.

  In the above-described embodiment, an example in which a captured image and a composite image are exchanged between the mobile terminal 1 and the server 2 by the TV phone function is shown. However, the captured image is transferred from the mobile terminal 1 to the server 2 by E. It may be transmitted as a mail attachment file, and the composite image may be similarly transmitted from the server 2 to the mobile terminal 1 as an e-mail attachment file. In this case, the owner of the portable terminal 1 sends an e-mail attachment file every time an image including the sign marker 4 is photographed, and every time an e-mail is received from the server 2, the attachment file is attached to the e-mail. Although it is necessary to perform an opening operation, the guidance information display method according to the present invention can be realized even with the portable terminal 1 having no TV phone function.

  According to the above-described embodiment described with reference to FIG. 1 to FIG. 9, an environment in which a sign marker marked with information that can identify the location by performing image analysis of the captured image is arranged in each place. However, the owner of the mobile device analyzes the captured image including the sign marker captured by the camera unit of the mobile device on the server side, and captures the location of the captured sign marker and the sign marker. Since a composite image in which an arrow indicating the destination direction is combined with the captured image received from the mobile terminal is generated and transmitted to the mobile terminal according to the result of the route search process, this is transmitted to the mobile terminal. On the display unit of the terminal, an arrow is displayed in an image obtained by photographing the real space on the spot. Therefore, it is possible to easily and easily grasp the direction in which the user should travel with respect to the destination on the captured image in the real space with high visibility. Also, even if the mobile device does not have a positioning function, and the map information is not stored or downloaded in advance, the destination of the mobile device owner in a very easy-to-understand form that does not use a map Since the direction guidance display can be presented, the burden on the portable terminal can be reduced, and the cost of the portable terminal is not increased. It can also be used indoors where GPS positioning is difficult. Furthermore, since a guidance display can be presented with higher accuracy than when using a GPS positioning function that may cause an error of around 10 m, it is particularly effective for pedestrians.

It is explanatory drawing which shows the outline | summary of the system for implement | achieving the guidance information display method which concerns on one Embodiment (henceforth this embodiment) of this invention. It is a block diagram which shows the function structure of each component in the system of FIG. It is explanatory drawing which shows the structure outline | summary of the input device in the system of FIG. It is explanatory drawing which shows a mode that the holder of a portable terminal sets the desired destination to a server using the input device in the guidance information display method which concerns on one Embodiment of this invention. It is a flowchart which shows the procedure in which the holder of a portable terminal sets the desired destination to a server using the portable terminal in the guidance information display method which concerns on one Embodiment of this invention. It is description which mainly shows the image of the flow of a process by the server in the guidance information display method which concerns on one Embodiment of this invention. In the guidance information display method which concerns on one Embodiment of this invention, it is explanatory drawing which shows the real space (actual world) which the user is seeing now, and the synthesized image displayed on the display part of the portable terminal. It is explanatory drawing which shows the relationship of the flow of a process in an input terminal, a server, and a portable terminal in the guidance information display method which concerns on one Embodiment of this invention. It is explanatory drawing which shows the relationship of the flow of a process in a portable terminal and a server in the guidance information display method which concerns on one Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Mobile terminal 2 Server 3 Input terminal 4 Sign marker 101 Control part 102 Videophone function part 103 E-mail function part 104 Camera part 105 Display part 106 Input operation part 107 RFID tag 108 WEB connection function part 109 Storage part 201 Control part 202 Member ID / mobile terminal identification information correspondence database 203 portable terminal identification information / destination information correspondence table 204 sign marker information database 205 position / photographing direction recognition unit 206 map information database 207 route search unit 208 3D arrow display generation unit 209 comment / advice generation Unit 210 composite image generation unit 211 videophone function unit 212 e-mail function unit 213 WEB connection function unit 214 storage unit 301 control unit 302 display unit 303 touch panel 304 RFID reader 305 Connection function unit 306 storage unit 401 marker ID representation of the Ba

Claims (10)

Signs with information that can be located at various locations and that can be identified by analyzing the captured images, and the mobile terminal, a server for providing guidance information that can communicate with the mobile terminal via a network A guide information display method in a system using a marker,
The mobile terminal includes a camera unit, a transmission / reception unit capable of transmitting / receiving information to / from the outside via a network, a storage unit for storing various data, a display unit, an input unit for performing operation input, and an entire mobile terminal A control unit that controls the control,
The server includes a transmission / reception unit capable of transmitting / receiving information to / from the outside via a network, a map information database storing map information, and identification information of a mobile terminal received from the mobile terminal or from a dedicated input terminal A storage table that stores destination information in association with each other, a captured image that includes the sign marker received from the mobile terminal, and a location of the photographed sign marker and a photographing direction in which the sign marker is photographed Based on the received destination information and information on the location of the specified sign marker, the route to the destination is searched with reference to the contents of the map information database. Based on the route direction search unit and the identified shooting direction, the destination image is indicated in the shot image received from the mobile terminal. A synthetic image generation unit which synthesizes a mark has,
When the owner of the portable terminal accesses the server using the portable terminal or the dedicated input terminal, and transmits the identification information of the portable telephone and the information of the destination, the server Storing identification information and destination information of the mobile terminal in the storage table;
The user of the portable terminal takes an image including the sign marker with the camera unit, and transmits the taken image to the server, whereby the server is taken by the position / photographing direction recognition unit. After specifying the location of the sign marker and the shooting direction in which the sign marker was shot, the route search unit searches for the route to the destination, and the synthesized image generation unit sends the destination image to the shot image received from the portable terminal Generating a composite image obtained by combining the arrows indicating, and transmitting the generated composite image to the mobile terminal;
The mobile terminal displaying the composite image received from the server on the display unit;
By providing, a composite image in which an arrow indicating a destination direction to the destination is synthesized on a photographed image taken by the mobile terminal is displayed on the display unit of the mobile terminal, and guidance to the destination is performed. A guide information display method characterized in that In the guidance information display method according to claim 1,
The sign marker is marked with ID information for identifying each sign marker, and the server is associated with the ID information of each sign marker and the direction of the display surface of the sign marker and the sign information. A sign marker information database storing (direction) information is provided, and the position / photographing direction recognition unit of the server obtains ID information of the sign marker from the photographed image including the received sign marker, and the obtained ID Based on the information, refer to the information in the sign marker information database to identify the location of the sign marker and the shooting direction in which the sign marker is shot in the opposite direction to the orientation of the display surface of the sign marker Guidance information display method characterized by this. In the guidance information display method according to claim 1,
The sign marker is marked with information indicating its location and information indicating the direction (azimuth) of the display surface, and the position / photographing direction recognition unit of the server reads the received sign marker. The information on the location of the sign marker and the information on the direction of the display surface is acquired from the captured image including the sign, and the sign is in the direction opposite to the location of the sign marker and the orientation of the display surface of the sign marker A guide information display method characterized by specifying a shooting direction in which a marker is shot. In the guidance information display method according to claim 1,
The sign marker is marked with information indicating its location, or ID information for identifying individual sign markers, and the position / photographing direction recognition unit of the server recognizes the received sign marker. The information on the location marked on the sign marker is directly acquired from the captured image, or the ID information marked on the sign marker is acquired, and the position information associated with the ID information of each sign marker is stored. The location of the sign marker is specified by referring to the sign marker information database, and the position / photographing direction recognizing unit detects the sign marker by a three-dimensional recognition process from the received photographed image including the sign marker. A guide information display method characterized by specifying a shooting direction. In the guidance information display method according to claim 1,
A guide information display method characterized in that a two-dimensional bar code having information that allows access to the server from a mobile terminal is provided in the vicinity of the sign marker by photographing with the camera unit of the mobile terminal. . In the guidance information display method according to claim 1,
The mobile terminal is a mobile phone having a videophone function, the identification information of the mobile terminal is a telephone number, and the mobile phone and the server transmit and receive image information by a videophone function. Guide information display method. In the guidance information display method according to claim 1,
The portable terminal is a portable terminal having an e-mail function, the identification information of the portable terminal is an e-mail address, and the portable terminal and the server transmit and receive image information as an attached file of e-mail. Guidance information display method. In the guidance information display method according to claim 1,
A guidance information display method, wherein a guidance message is added to the composite image generated by the server in addition to the arrow. In the guidance information display method according to claim 1,
A guidance information display method, wherein guidance voice is transmitted together with the synthesized image from the server. In the guidance information display method according to claim 1,
The dedicated input terminal is associated with the RFID reader that reads the identification information of the portable terminal from a non-contact RFID (Radio Frequency Identification) tag built in the portable terminal, or the identification information of the portable terminal A reader for reading card identification information from a card having card identification information is provided, and when the card identification information is transmitted from the dedicated input terminal to the server, the server identifies the card identification information and the portable terminal. A guide information display device that identifies identification information of a mobile terminal from a database storing a correspondence relationship with the identification information.

Images