JP5728775B2 - Information processing apparatus and information processing method - Google Patents

Description

  The present invention relates to an information processing apparatus and an information processing method.

  In recent years, a technology related to augmented reality (AR) has attracted attention. For example, in Patent Document 1, a camera built in a portable device recognizes a specific object in the field of view of the camera, and the portable device creates an appropriate computer graphic based on this recognition. In addition, it is disclosed to superimpose on a real-world image displayed on a display device.

JP 2010-244575 A

  As a conventional technique related to augmented reality (AR), there is a technique in which content related to a place is displayed superimposed on a camera image. Here, FIG. 1 shows an example in which content related to a place is displayed superimposed on the video of the camera.

  As shown in FIG. 1, when content is displayed superimposed on a camera image, there is a problem in that the content is not displayed at an appropriate position due to the influence of the accuracy of GPS (Global Positioning System). In addition, there is a problem that the interrelationship of contents differs from the real space.

  In the present invention, in view of the above-described background, it is an object of the present invention to provide a technique for displaying content at an appropriate position in a technique for displaying content superimposed on a camera image.

  In order to solve the above-described problems, the present invention employs the following means. That is, the present invention accesses a camera that captures a real space, a display unit that displays a video related to the real space captured by the camera, and a storage device that stores content to be displayed over the video, and Based on position information including the position and orientation of the camera, among the objects included in the video, the position information of the object that displays the content, the acquisition unit that acquires the content, and the content acquired by the acquisition unit A control unit that superimposes the image on the video and displays the image on the display unit, the control unit sets an azimuth reference line related to an azimuth and a distance reference line related to a distance on the screen, and the position information of the camera and the Based on the position information of the object, calculate the position of the object with respect to the camera, and based on the azimuth reference line, Identify a flat position, the distance based on the reference line identifies the vertical position of the object in the screen, specifies a display position of the object in the screen.

  According to the present invention, by setting the azimuth reference line and the distance reference line on the screen, the position of the object relative to the camera can be specified in both the horizontal direction and the vertical direction on the screen. That is, the content can be displayed at an appropriate position. In the prior art, the position in the vertical direction is not particularly taken into consideration, and when content overlaps in the same direction, the content of the object close to the camera is displayed in the foreground to prevent it from being displayed overlapping. It was only. On the other hand, according to the present invention, the content can be displayed at an appropriate position including the vertical direction. The present invention can be widely applied to technologies relating to augmented reality (hereinafter also referred to as AR technology). In particular, the present invention can be suitably used for navigation technology using AR technology. In addition, the present invention can be widely applied as an apparatus including a camera, a communication function, and a computer, but can be particularly preferably used as a mobile terminal such as a smart phone or a mobile phone.

  Here, in the information processing apparatus according to the present invention, the control unit sets an azimuth reference line related to an azimuth and a distance reference line related to a distance to the screen, and sets the position information of the camera and the position information of the object Based on the angle of the object with respect to the optical axis of the camera and the distance from the camera to the object, to determine the horizontal position of the object on the screen based on the azimuth reference line, A vertical position of the object on the screen may be specified based on a distance reference line, and a display position of the object on the screen may be specified. The optical axis of the camera means an axis that passes through the center of the camera lens and is orthogonal to the plane of the lens. The distance from the camera to the object means a horizontal distance. By calculating the angle and distance as described above, the content can be displayed at an appropriate position.

  Further, in the information processing apparatus according to the present invention, the position information of the target object includes position information of three or more target points that specify the shape of the target object, and the control unit includes each of the target points. The horizontal position and the vertical position on the screen can be specified, the plane can be set by connecting the specified target points, and the text can be set on the set plane. In the prior art, the degree of freedom of content display is limited, and only balloons, pins, photos, icons, etc. can be displayed. However, according to the present invention, not only the horizontal position but also the vertical position can be displayed on the screen with high accuracy, so that the content can be displayed in various forms that are not conventional, such as a form that matches the shape of the building. Can do.

  Here, in the prior art, in addition to the above-described problem that the content is not displayed at an appropriate position, there is a concern that the content is further overlapped and difficult to see, and the content occupies the background and the video is difficult to see. In order to solve such a problem, in the information processing apparatus according to the present invention, the control unit analyzes the video and sets a boundary reference line including a scene boundary constituting the real space for the screen. Then, the content display position may be corrected based on the boundary reference line. By calculating the boundary reference line and correcting the display position of the content based on the reference line, it is possible to display the content at an appropriate position in an easy-to-see manner. For example, by providing a boundary reference line as a road boundary and arranging the content so that only the content related to the road is displayed in the road, it is possible to make it easier to see.

  Further, in the information processing apparatus according to the present invention, the control unit analyzes a boundary of a scenery to be displayed with priority over other scenery among the scenery constituting the real space by analyzing the video. A line may be set for the screen, and the content display position may be corrected based on the priority boundary reference line. As a result, it is possible to reliably display the scenery that should be prioritized as the image in the real space, and to display the content related to the image at an appropriate position in an easy-to-see manner. The scenery to be displayed with priority over other scenery can be preset according to the purpose. For example, when the present invention is used as a navigation technique using the AR technique, “road” can be set as a scenery to be displayed with priority over other scenery. As a result, the “road” that is most important for navigation can be displayed with priority and reliability. Examples of other scenery include buildings and trees.

  Further, for example, the information processing apparatus according to the present invention functions as a navigation device, and the control unit calculates the boundary line of the road by analyzing the video as the boundary reference line, and within the boundary line of the road The content other than the content related to the road can be arranged outside the boundary reference line corresponding to the boundary line of the road. Thereby, a road can be displayed preferentially, and the information processing apparatus according to the present invention can effectively function as a mobile terminal type navigation apparatus.

  In addition, the control unit may calculate a horizontal line that passes through the focal point of the boundary line of the road, and sequentially arrange content other than the content related to the road with the horizontal line as a base line. By arranging sequentially from the horizontal line, it is possible to make the display easier to see.

  Further, the control unit may analyze the video to calculate a ridge line including a contour of a natural object including a tree and a mountain, and dispose content related to the natural object above the ridge line. . Thereby, the display position of the content related to the artificial structure such as a building can be distinguished from the display position of the content related to the natural object, and the display can be made easier to see. The ridgeline may include a ridgeline as a building outline in part.

  Here, the present invention can also be specified as an information processing method executed by the information processing apparatus described above. For example, the present invention is an information processing method executed by an information processing apparatus including a camera that captures a real space, and a display unit that displays a video related to the real space captured by the camera. Based on the position information including the position and orientation of the camera, the storage device storing the content to be displayed in a superimposed manner is accessed, and the position information of the object for displaying the content among the objects included in the video, and the An acquisition step of acquiring content, and a control step of displaying the content acquired in the acquisition step on the video image on the display unit, wherein the control step includes a azimuth reference line related to a direction and a distance related to a distance. A reference line is set on the screen, and based on the position information of the camera and the position information of the object, Calculating the position of the object, specifying the horizontal position of the object on the screen based on the azimuth reference line, specifying the vertical position of the object on the screen based on the distance reference line, The display position of the object on the screen is specified.

  The present invention can also be specified as an information processing system including the information processing apparatus described above and a content management server that provides content to such an information processing apparatus. Further, the present invention may be specified as an information processing method executed in the information processing system. Further, the present invention may be specified as a program that realizes processing executed by the information processing apparatus or the information processing system or a computer-readable recording medium that records such a program. In this case, the function can be provided by causing a computer or the like to read and execute the program of the recording medium. Note that a computer-readable recording medium is a recording medium that accumulates information such as data and programs by electrical, magnetic, optical, mechanical, or chemical action and can be read from a computer or the like. Say.

  According to the present invention, it is possible to display content at an appropriate position in the technology for displaying the content superimposed on the video of the camera.

An example of the prior art in which content is superimposed on a camera video and displayed. 1 shows a configuration of an information processing system according to a first embodiment. The functional block diagram of the portable terminal which concerns on 1st embodiment is shown. The functional block diagram of a content management server is shown. An example of the content management database memorize | stored in the memory | storage part of a content management server is shown. An example of the database of the route management memorize | stored in the memory | storage part of a content management server is shown. The screen transition diagram of a portable terminal is shown. Show the home screen. Opening screen is shown. The application selection screen is shown. An operation explanation screen is shown. The region (prefecture) selection screen is shown. The district (city) selection screen is shown. The user input screen is shown. The route list screen is shown. Shows the map screen. A camera guidance screen is shown. A menu screen is shown. The language selection screen is shown. The route details screen is shown. The balloon display change screen is shown. An information details screen is shown. The radar range change screen is shown. The content management processing flow is shown. The content change processing flow is shown. A content management screen is shown. The content registration screen is shown. The content search screen is shown. The content search result screen is shown. A content change screen (basic information) is shown. A content change screen (additional information) is shown. The route registration processing flow is shown. The route change processing flow is shown. The route management screen is shown. The route registration screen (basic information) is shown. A route registration screen (content list) is shown. The route confirmation screen is shown. The route search screen is shown. The route search result list screen is shown. The route change screen (basic information) is shown. The search processing flow for root content is shown. The display processing flow of a camera guidance screen is shown. An image for determining an angle when “target point” is displayed is shown. An image for determining the distance when displaying the “target point” is shown. Sectional drawing explaining the relationship between a camera and "target point A" is shown. A state in which the content is temporarily positioned on the video is shown. An analysis of the intensity of each pixel in the image and calculation of a ridgeline, a road boundary line, and a horizontal line are shown. The manner in which the display position of the content is corrected is shown. The display example of a camera guidance screen is shown. A mode that the target point of a horizontal position is specified is shown. A mode that the target point of a height position is specified is shown. A state where an outline is specified by connecting four target points is shown. A state in which an area specified by connecting four target points is specified as a surface is shown. Shows how to set text for a specified face.

  Next, embodiments of the present invention will be described with reference to the drawings. In the present embodiment, a case where the information processing apparatus according to the present invention functions as a portable terminal having a navigation function will be described as an example. In the embodiment, an information processing system including such a portable terminal will be described.

[First embodiment]
<Configuration>
FIG. 2 shows a configuration of the information processing system 1 according to the first embodiment. An information processing system 1 according to the first embodiment includes a mobile terminal 2 and a content management server 3. The portable terminal 2 and the content management server 3 are connected to each other via a network. The network is exemplified by the Internet using a telephone line network, but the network is not particularly limited as long as the mobile terminal 2 and the content management server 3 can be electrically connected to enable communication. . In FIG. 2, only one portable terminal 2 is shown, but a plurality of portable terminals 2 can be provided.

(Mobile device)
The portable terminal 2 functions as a navigation device by installing a predetermined application in advance. The mobile terminal 2 is exemplified by a smartphone (for example, Android mobile phone (Android: registered trademark) or iphone (registered trademark)). FIG. 3 shows a functional block diagram of the mobile terminal 2 according to the first embodiment. The mobile terminal 2 includes a control unit 21, an operation unit 22, a display unit 23, a camera 24, a communication unit 25, a position information acquisition unit 26, an orientation sensor 27, and a storage unit 28.

  The control unit 21 of the mobile terminal controls each unit of the mobile terminal 2 based on information from the content management server 3 and an operation instruction of a user (user). The control unit 21 of the mobile terminal includes a CPU 211 of the mobile terminal and a memory 212 of the mobile terminal. The CPU 211 of the portable terminal controls each hardware of the portable terminal 2 and functions as a portable terminal having a navigation function using the AR technology according to a control program recorded in the memory 212 of the portable terminal. In addition, you may make it provide the function part corresponding to each function which the portable terminal 2 has. In other words, each function unit can be configured as a computer program executed on the CPU 211 of the mobile terminal, and each of these function units may be configured as a dedicated processor or configured as one processor. May be. The memory 212 of the portable terminal is a memory (storage medium) that stores various data and control programs, and is configured by a rewritable flash memory or the like.

  Each function unit is exemplified as follows. That is, the analysis unit analyzes the video and calculates a reference line including the boundary of the scenery that forms the real space. The arrangement unit arranges the content based on the reference line calculated by the analysis unit. In addition to route search, the navigation unit displays the route to the destination, and displays the content superimposed on the video captured by the camera 24 based on the analysis results of the analysis unit and the arrangement unit.

  The operation unit 22 of the portable terminal receives an instruction from the user. The operation unit 22 of the mobile terminal is configured by, for example, a touch panel, a push button, or the like. The operation unit 22 of the mobile terminal according to the present embodiment is configured by a touch panel. Signals are input to the control unit 21 of the mobile terminal according to various operations.

  The display unit 23 of the mobile terminal displays an image based on a signal from the control unit 21 of the mobile terminal. The display unit 113 of the portable terminal is configured by, for example, a liquid crystal display, an organic EL display, or the like. The display unit 23 of the mobile terminal according to the present embodiment is configured by a touch panel.

  The camera 24 images a real space. The camera 24 includes a module that has an optical system and an image receiving element and acquires a digital image. The acquired digital image is displayed on the display unit 23 of the mobile terminal. When the mobile terminal 2 functions as an AR navigation device, the acquired image is stored in the storage unit 28 of the mobile terminal, and the content is superimposed and displayed on the display unit 23 of the mobile terminal.

  The communication unit 25 of the mobile terminal is connected to the network and performs communication with the content management server 3 and other mobile terminals.

  The position information acquisition unit 26 detects the position of the mobile terminal based on GPS signals from one or a plurality of GPS (Global Positioning System) satellites. The position information regarding the detected position is input to the control unit 21 of the mobile terminal. As a result, the control unit 21 of the mobile terminal can grasp the position of the mobile terminal 2. Note that the position information acquisition unit 26 may detect the position of the mobile terminal 2 based on signals from one or a plurality of radio communication base stations, for example. The acquisition of the position information can use existing techniques as appropriate.

  The direction sensor 27 detects the direction (horizontal direction) and angle (vertical direction) of the mobile terminal 2. The detected signal corresponding to the direction / angle is input to the control unit 21 of the portable terminal. As a result, the control unit 21 of the mobile terminal can grasp the direction / angle of the mobile terminal 2.

  The storage unit 28 of the portable terminal stores various information. For example, the storage unit 28 of the mobile terminal stores video captured by the camera 24. The storage unit 28 of the mobile terminal can be configured by, for example, a semiconductor memory.

(Content management server)
A content management server (hereinafter also simply referred to as a server) 3 manages content to be provided to the mobile terminal 2. FIG. 4 shows a functional block diagram of the content management server. The content management server 3 includes a control unit 31, an operation unit 32, a display unit 33, a communication unit 34, and a storage unit 35, and the control unit 31 includes a CPU 311 and a memory 312.

  The operation unit 32 of the server includes, for example, a keyboard, a pointing device, a touch panel, operation buttons, and the like. The server display unit 33 includes, for example, a liquid crystal display device, a plasma display panel, a CRT (Cathode Ray Tube), an electroluminescence panel, and the like. Examples of the communication unit 34 of the server include a communication module (for example, a network card) that realizes connection to a network.

  The server control unit 31 includes a CPU (Central Processing Unit) 311 and a memory 312. The CPU 311 of the server controls each hardware of the content management server 3 and causes a functional unit such as a content / route management unit to function according to a control program stored in the content management server 3. The functional unit can be configured as a computer program executed on the CPU 311 of the content management server. These functional units may be configured as a dedicated processor.

  The content / route management unit manages content and routes. Specifically, the content and route posted by the user, the recording of the received content and route in the database, the update of the database storing the content and route, the provision of the content in response to the request from the mobile terminal 2 Etc.

  The storage unit 35 of the server stores contents and routes under the control of the content / route management unit. The content management server 3 can be configured by a hard disk drive (hereinafter referred to as HDD), a semiconductor memory, or the like.

  Here, FIG. 5A shows an example of a content management database stored in the storage unit 35 of the server. In the database, items of “identification number”, “content”, “latitude, longitude, height” are provided, and information corresponding to each item is stored. The “identification number” is also referred to as a content number and is unique identification information assigned to each content. The “content” includes various types of information regarding the place. For example, “content” includes names of buildings, roads, trees, mountains, etc., information about history, and the like. The “content” is not limited to text data, and may be image data or illustration data. FIG. 5B shows an example of a route management database stored in the storage unit 35 of the content management server. In the database, items of “identification number”, “root”, and “content list” are provided, and information corresponding to each item is stored. The “identification number” is also referred to as a route number, and is unique identification information assigned to each route. The “route” includes various information regarding the route. For example, the “route” includes an eating route, a history enjoying route, and a wallet-friendly route. The “content list” includes contents constituting each route.

<Operation>
Next, the operation of the information processing system 1 according to the embodiment will be described based on the drawings. In the following description, a case where the mobile terminal functions as a navigation device using the AR technology will be described.

(Mobile terminal screen transition)
In the mobile terminal 2, navigation processing using AR technology is performed. FIG. 6 shows a screen transition diagram of the mobile terminal. 7A to 7J and FIGS. 8A to 8F show the respective screens. The portable terminal 2 has a screen as shown in FIG. 6 as a related screen when functioning as a navigation device using the AR technology. Specifically, the mobile terminal 2 includes a home screen (G01), an opening screen (G02), an application selection screen (G03), an operation explanation screen (G04), a region (prefecture) selection screen (G05), and a district (city). Selection screen (G06), user input screen (G07), route list screen (G08), map screen (G09), camera guidance screen (G10), menu screen (A), language selection screen (B), route details screen (C), a balloon display change screen (D), an information detail screen (E), and a radar range change screen (F). Each screen display and screen transition is performed by the control unit 21 of the mobile terminal based on a signal corresponding to a user operation, that is, an operation signal from the operation unit 22 of the mobile terminal. Hereinafter, each screen will be described.

(Home Screen)
The home screen (G01) is displayed when the mobile terminal 2 is activated. That is, when the mobile terminal 2 is activated and power is supplied, the control unit 21 of the mobile terminal detects this and displays the home screen on the display unit 23 of the mobile terminal. On the home screen, an icon of an application installed in the mobile terminal 2 is displayed (see FIG. 7A). When an icon of a management application that manages a plurality of applications is selected on the home screen, the screen transitions to the opening screen of the selected management application.

(Opening screen)
The opening screen is the opening screen of the management application. On the opening screen (G02), “Welcome!” Is displayed (see FIG. 7B). Specifically, when a management application icon is selected on the home screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the portable terminal receives the operation signal and displays an opening screen on the display unit 23 of the portable terminal. In FIG. 7B, a “return” button and a “Menu” button are hardware keys provided on the mobile terminal 2. By selecting the “Return” button, it is possible to transition to the previous screen. In addition, transition to the menu screen (A) is possible by selecting the “Menu” button. The opening screen is displayed for a few seconds, and then automatically transitions to the app selection screen.

(App selection screen)
The application selection screen (G03) is an application selection screen. On the application selection screen, “navigation application” and “other application” are displayed (see FIG. 7C). Specifically, after the transition to the opening screen, the control unit 21 of the mobile terminal counts a preset transition time and displays the application selection screen on the display unit 23 of the mobile terminal. The “navigation application” is a selection area for starting an application that performs navigation using the AR technology, and the “other application” is a selection area for transitioning to a selection screen for other applications. When “navigation application” is selected, the operation description screen is displayed.

(Operation explanation screen)
The operation explanation screen (G04) is an operation explanation screen of “navigation application”. On the operation explanation screen, “1. Region (prefecture) selection”, “2. City selection”, “3. User information input”, and “4. Route list” are displayed (see FIG. 7D). Specifically, when “navigation application” is selected on the application selection screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the mobile terminal receives this operation signal and displays an operation explanation screen on the display unit 23 of the mobile terminal. “1. Region (prefecture) selection”, “2. City selection”, “3. User information input”, and “4. Route list” are selection areas for transition to respective explanation screens. For example, when “1. Region (prefecture) selection” is selected, an explanation of region (prefecture) selection is displayed. In the operation explanation screen, “Start (to prefecture selection)”, “Menu”, and “Language” are displayed. “To start (prefecture selection)” is a selection area for transitioning to a region (prefecture) selection screen. “Menu” is a selection area for transitioning to a menu screen. “Language” is a selection area for transition to a language selection screen. When "Start (prefecture selection)" is selected, a transition is made to the region (prefecture) selection screen.

(Region (prefecture) selection screen)
The region (prefecture) selection screen (G05) is a screen for accepting selection of a region at the prefectural level. In FIG. 7E, as an example, “Akita Prefecture”, “Aomori Prefecture”, and “Iwate Prefecture” are displayed on the region (prefecture) selection screen. When “start (prefecture selection)” is selected on the operation explanation screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the portable terminal receives this operation signal and displays a region (prefecture) selection screen on the display unit 23 of the portable terminal. “Akita Prefecture”, “Aomori Prefecture”, and “Iwate Prefecture” are selection regions for transitioning to a screen for accepting a region selection at each municipality level. For example, when “Akita Prefecture” is selected, a transition is made to a district (city) selection screen.

(District (city) selection screen)
The district (city) selection screen (G06) is a screen for accepting selection of a region at the municipal level. In FIG. 7F, as an example, “Akita City”, “Kazuno City”, and “Nikaho City” are displayed on the district (city) selection screen. When “Akita” is selected on the region (prefecture) selection screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the mobile terminal receives this operation signal and causes the display unit 23 of the mobile terminal to display a district (city) selection screen corresponding to Akita Prefecture. “Akita City”, “Kazuno City”, and “Nikaho City” are selection areas for transition to the user information input screen. For example, when “Akita City” is selected, the screen changes to a user input screen.

(User input screen)
The user input screen (G07) is a screen for accepting user information input. On the user input screen, items of “sex”, “age”, “walking time”, “journey item”, and “route list” are displayed (see FIG. 7G). When, for example, “Akita City” is selected on the district (city) selection screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the mobile terminal receives this operation signal and displays a user input screen on the display unit 23 of the mobile terminal. Options are provided for “sex”, “age”, “walking time”, and “travel item”, and the user can input user information from the options. In “gender”, selection from “male” and “female” is possible. In "age", selection from the ages such as "10's" and "20's" is possible. In “walking time”, selection from predetermined time units such as “30 minutes” and “1 hour” is possible. In the “travel purpose”, it is possible to select a travel purpose such as “sightseeing” and “meal”. After selecting each item, when “Route List” is selected, the screen transits to the route list screen.

(Route list screen)
The route list screen (G08) is a screen for displaying a route search result. On the route list screen, “eating and walking route”, “historic route”, “wallet friendly route”, “hot spring tour route”, etc. are displayed as search results (see FIG. 7H). For example, when “route list” is selected on the user input screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the portable terminal receives this operation signal, transmits the search condition to the content management server 3 via the communication unit 25 of the portable terminal, and receives the search result. And the control part 21 of a portable terminal displays the route | root list screen containing a search result on the display part 23 of a portable terminal. “Eat-walking route”, “History enjoying route”, “Wallet-friendly route”, and “Onsen tour route” are selection areas for transition to the route detail screen. For example, when “historic enjoyment route” is selected, a transition is made to the route detail screen.

(Route details screen)
The route detail screen (C) is a screen that displays the details of the route. On the route details screen, the name of the station on the route, the required time, etc. are displayed (see FIG. 8C). When, for example, “historic route” is selected on the route list screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. Upon receiving this operation signal, the control unit 21 of the mobile terminal displays a route detail screen on the display unit 23 of the mobile terminal. Information included in the route details screen may be acquired from the content management server 3 when displaying the route list screen, or may be acquired from the content management server 3 when displaying the route details screen. In the route detail screen, “route determination” and “return” are also displayed. “Route determination” is a selection area for transition to the map screen. “Return” is a selection area for transitioning to the previous screen, that is, the route list screen. When “Route determination” is selected, a transition is made to the map screen.

(Map screen)
The map screen (G09) displays a map screen (overhead information) of the selected route. On the map screen, “guidance start point”, “guidance end point”, and “balloon” are displayed together with the map (see FIG. 7I). When “Route determination” is selected on the route detail screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. Upon receiving this operation signal, the control unit 21 of the mobile terminal transmits the selected route and current location to the content management server 3 via the communication unit 25 of the mobile terminal, and receives map screen data. And the control part 21 of a portable terminal displays a map screen on the display part 23 of a portable terminal. The “guidance start point” is a place where the guidance starts, and the “guidance end point” is a place where the guidance ends. “Balloon” is provided in a recommended place, and when “balloon” is selected, an explanation for each place is displayed. The “balloon” is color-coded for each genre, for example. The genre includes facilities, eating, buying, roads, rivers, mountains, rest areas, toilets, and parking lots. Instead of “balloon”, “pin”, “photo”, “illustration”, “icon” or the like may be used. In addition, “Menu”, “Language”, “Camera”, “Current location”, and “Display change” are further displayed on the map screen. “Camera” is a selection area for transition to the camera guidance screen. “Current location” is a selection region for transition to a map screen that moves around the current location. “Display change” is a selection area for transition to a balloon display change screen. When “camera” is selected, a transition is made to the camera guidance screen.

(Camera guidance screen)
The camera guidance screen (G10) displays view information (real space integrated screen) in which content is superimposed on video captured by the camera 24. On the camera guidance screen, “radar” and “balloon” are displayed on the video imaged by the camera 24 (see FIG. 7J). When “camera” is selected on the map screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 23 of the mobile terminal. The control unit 21 of the mobile terminal stores the video captured by the camera 24 in the storage unit 28 of the mobile terminal, and superimposes the content acquired from the content management server 3 on the video so that the video captured by the camera 24 is displayed. The camera guidance screen on which the content is superimposed is displayed on the display unit 23 of the mobile terminal. The circle at the bottom right of the screen is a radar image view (hereinafter referred to as radar display), and this radar display is color-coded according to the type of tourist information around you (mobile terminal user). Displayed. When “radar display” is selected, transition to the radar range change screen is possible. The arrow at the bottom center of the screen indicates the direction to the next destination, and the distance is displayed next to the arrow. The distance displayed with this arrow indicates the distance to the next destination. As with the map screen, “balloon” is displayed on the camera guidance screen, and the distance to the location corresponding to the balloon is displayed below “balloon”. In addition, “menu”, “language”, and “display change” similar to those on the map screen are displayed on the camera guidance screen. Furthermore, a “map” is displayed on the camera guidance screen, and this “map” is a selection area for transition to the map screen.

(Menu screen)
The menu screen (A) is a screen that displays a menu of the navigation application. On the menu screen, “go to user information input screen”, “go to route list screen”, and “end navigation application” are displayed (see FIG. 8A). When “Menu” is selected on the application selection screen, operation explanation screen, region (prefecture) selection screen, district (city) selection screen, user input screen, route list screen, map screen, and camera guidance screen, the mobile terminal The operation unit 22 detects this, and a corresponding operation signal is input to the control unit 21 of the portable terminal. The control unit 21 of the mobile terminal receives this operation signal and displays a menu screen on the display unit 23 of the mobile terminal. “To user information input screen” is a selection area for transition to the user information input screen. “To route list screen” is a selection area for transition to the route list screen. “End of navigation application” is a selection area for transition to the application selection screen.

(Language selection screen)
The language selection screen (B) is a screen that accepts selection of text used in the navigation application. On the language selection screen, “Japanese”, “English”, “Chinese”, and “Korean” are displayed (see FIG. 8B). When “Menu” is selected on the application selection screen, operation explanation screen, region (prefecture) selection screen, district (city) selection screen, user input screen, route list screen, map screen, and camera guidance screen, the mobile terminal The operation unit 22 detects this, and a corresponding operation signal is input to the control unit 21 of the portable terminal. Upon receiving this operation signal, the control unit 21 of the mobile terminal displays a language selection screen on the display unit 23 of the mobile terminal. When “Japanese”, “English”, “Chinese”, or “Korean” is selected on the language selection screen, the language used by the navigation application is changed. The language displayed on the language selection screen is not limited to the above.

(Balloon display change screen)
The balloon display change screen (D) is a screen for changing settings of balloon display displayed on the map screen and the camera guide screen. In the balloon display change screen, check the corresponding genres of balloons as “facility”, “eat”, “buy”, “road / river”, “mountain”, “rest area / toilet”, “parking lot” and each. A box is displayed (see FIG. 8D). When “display change” is selected on the map screen and the camera guidance screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. Upon receiving this operation signal, the control unit 21 of the mobile terminal displays a language selection screen on the display unit 23 of the mobile terminal. On the balloon display change screen, “Facility”, “Eat”, “Buy”, “Road / River”, “Mountain”, “Restroom / Toilet”, or “Parking” is selected (the check box is checked) ), When “Determine” is selected, the screen transitions to the transition source screen (map screen or camera guidance screen).

(Information details screen)
The information detail screen (E) displays detailed information on the location corresponding to the balloon displayed on the map screen and the camera guidance screen. On the information detail screen, for example, recommended products and images of shops are displayed (see FIG. 8E). When “balloon” is selected on the map screen and the camera guidance screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. Upon receiving this operation signal, the control unit 21 of the mobile terminal displays detailed information corresponding to the selected “balloon” on the display unit 23 of the mobile terminal. When “return” is selected on the information detail screen, the screen transitions to the transition source screen (map screen or camera guidance screen).

(Radar range change screen)
The radar range change screen (F) is a screen for changing the radar display range displayed on the camera guidance screen. On the radar range change screen, “50 m”, “100 m”, “200 m”, “500 m”, “1 km”, “2 km”, and “3 km” are displayed (see FIG. 8F). When “display change” is selected on the camera guidance screen, the operation unit 22 of the mobile terminal detects this, and a corresponding operation signal is input to the control unit 21 of the mobile terminal. The control unit 21 of the portable terminal receives this operation signal and displays a radar range change screen. When any range is selected on the radar range change screen, the screen changes to the camera guidance screen.

(Content management server operation)
The content management server 3 performs content management, route management, content route search, and the like.

  First, content management will be described. Content management includes new registration of content information (hereinafter referred to as content registration processing) and change of already registered content (hereinafter referred to as content change processing). In the following description, an example in which content management is performed using the computer of the content management server 3 will be described, but it can also be performed using an external computer or a portable terminal. FIG. 9 shows a content registration processing flow. In step S01, the control unit 31 of the server activates the browser based on an instruction from the administrator or the user. When the address and password of the management screen are input via the operation unit 32 of the server, a corresponding operation signal is input to the control unit 31 of the server. Upon receiving this operation signal, the server control unit 31 causes the display unit 33 of the server to display a content management screen. FIG. 11A shows a content management screen. On the content management screen, a “content information” tab and a “route information” tab are displayed. In FIG. 11A, “content information” is selected, and “content registration” and “content search” are displayed in the content information. When the browser is activated, the process proceeds to step S02.

  In step S02, the control unit 31 of the server registers content. Specifically, the control unit 31 of the server displays a registration screen on the display unit 33 of the server and accepts content registration. FIG. 11B shows a content registration screen. In FIG. 11B, a CSV file is selected as an example. “Registration” and “Cancel” are displayed on the registration screen, and when “Register” is selected, the selected file is registered. That is, when new content is input via the operation unit 32 of the server, the control unit 31 of the server registers the input content in the database of the storage unit 35 of the server. Thus, the content registration process ends.

  Next, content change processing will be described. FIG. 10 shows a content change processing flow. Step S11 is the same as step S01, and the server control unit 31 activates the browser based on an instruction from the administrator or user, and causes the display unit 33 of the server to display a content management screen. When the browser is activated, the process proceeds to step S12.

  In step S12, the control unit 31 of the server searches for content. Specifically, the control unit 31 of the server displays a search screen on the display unit 33 of the server and accepts content search conditions. FIG. 11C shows a content search screen. On the content search screen, it is possible to input a content number and a content name. After inputting these, when “Search” is selected, the search is executed based on the input content number and content name. That is, when a content number, content name, or the like is input via the operation unit 32 of the server, the control unit 31 of the server accesses the database of the storage unit 35 of the server and searches for the content. When the content is retrieved, the process proceeds to step S13.

  In step S13, the control unit 31 of the server changes the content. Specifically, the control unit 31 of the server first displays a search result screen on the display unit 33 of the server. FIG. 11D shows a content search result screen. A list of searched contents is displayed on the contents search result screen. Further, “delete” is displayed beside each content, and when “delete” is selected, the content is deleted. Next, when the content is selected from the content search result list, the server control unit 31 displays the selected content change screen on the server display unit 33. FIG. 11E shows a content change screen (basic information). On the content change screen (basic information), the content number and the content name are displayed and can be changed. When the content number or content name is input via the operation unit 32 of the server, the server control unit 31 accesses the database in the storage unit 35 of the server and updates the database. In addition, when “additional information” is selected on the content change screen (basic information), the server control unit 31 causes the display unit 33 of the server to display the content change screen (additional information). FIG. 11F shows a content change screen (additional information). On the content change screen (additional information), additional information that is detailed content information (in FIG. 11F, entrance fee, parking lot presence / absence) is displayed, and these can be changed. When the additional information is input via the operation unit 32 of the server, the server control unit 31 accesses the database in the storage unit 35 of the server and updates the database. Thus, the content change process ends.

  Next, route management will be described. Route management includes new registration of route information (hereinafter referred to as route registration processing) and change of an already registered route (hereinafter referred to as route change processing). In the following description, an example in which route management is performed using the computer of the content management server will be described, but it can also be performed using an external computer or a portable terminal. FIG. 12 shows a route registration process flow. In step S21, the control unit 31 of the content management server 3 (hereinafter also simply referred to as a server) starts a browser based on an instruction from an administrator or a user, and causes the display unit 33 of the server to display a content management screen. FIG. 14A shows a route management screen. On the route management screen, a “content information” tab and a “route information” tab are displayed. In FIG. 14A, “route information” is selected, and “route registration” and “route search” are displayed in the route information. When the browser is activated, the process proceeds to step S22.

  In step S22, the control unit 31 of the server registers a route. Specifically, first, the control unit 31 of the server displays a registration screen (basic information) on the display unit 33 of the server, and accepts content registration. FIG. 14B shows a route registration screen (basic information). On the route registration screen (basic information), a route number and a route name are displayed, and these can be registered. When the route number or route name is input via the server operation unit 32, the server control unit 31 accesses the database in the storage unit 35 of the server and registers the route. Further, when “content list” is selected on the route registration screen (basic information), the control unit 31 of the server causes the display unit 33 of the server to display the route registration screen (content list). FIG. 14C shows a route registration screen (content list). On the route registration screen (content list), content constituting the route can be registered, and the registered content is sequentially displayed as a list. When content constituting the route is input via the operation unit 32 of the server, the control unit 31 of the server accesses the database in the storage unit 35 of the server and registers the content constituting the route in the database. In the route registration screen (content list), “map” is displayed. Transit to the route confirmation screen by selecting “Map”. FIG. 14D shows a route confirmation screen. On the route confirmation screen, the route and the contents constituting the route are displayed. “Close” is displayed on the route confirmation screen, and when “Close” is selected, the screen transits to the route registration screen (content list). This completes the route registration process.

  Next, route change processing will be described. FIG. 13 shows a route change processing flow. Step S31 is the same as step S21, and the server control unit 31 activates the browser based on an instruction from the administrator or the user, and displays a route management screen on the server display unit 33 (see FIG. 14A). When the browser is activated, the process proceeds to step S32.

  In step S32, the control unit 31 of the server searches for content. Specifically, the control unit 31 of the server displays a search screen on the display unit 33 of the server and accepts content search conditions. FIG. 14E shows a route search screen. On the route search screen, it is possible to input a route number and a route name. When these items are input and “Search” is selected, the search is executed based on the input route number and route name. That is, when a route number, a route name, or the like is input via the server operation unit 32, the server control unit 31 accesses the database in the server storage unit 35 and searches for a route. When the route is searched, the process proceeds to step S33.

  In step S33, the control unit 31 of the server changes the route. Specifically, the control unit 31 of the server first displays a route search result list screen on the display unit 33 of the server. FIG. 14F shows a route search result list screen. The route search result list screen displays a list of searched routes. Further, “delete” is displayed beside each route, and the content is deleted when “delete” is selected. Next, when a route is selected from the route search result list, the server control unit 31 causes the server display unit 33 to display a screen for changing the selected route. FIG. 14G shows a route change screen (basic information). On the route change screen (basic information), a route number and a route name are displayed, and these can be changed. When the route number or route name is input via the server operation unit 32, the server control unit 31 accesses the database in the storage unit 35 of the server and updates the database. Further, when “content list” is selected on the route change screen (basic information), the server control unit 31 displays the route change screen (content list) on the server display unit 33 (see FIG. 14C). On the route registration screen (content list), the content constituting the route can be changed, and the changed content is sequentially displayed as a list. When the content configuring the route is changed via the operation unit 32 of the server, the server control unit 31 accesses the database of the storage unit 35 of the server and changes the content configuring the route. In the route registration screen (content list), “map” is displayed. By selecting “Map”, the screen transits to the route confirmation screen (see FIG. 14D). Thus, the route change process ends.

  Next, search for route content will be described. FIG. 15 shows a route content search processing flow. In step S41, the control unit 31 of the server acquires a search condition. Specifically, the control unit 31 of the server transmits the mobile terminal 2 identification information, current location, search conditions (region (prefecture, city), gender, age, walking time, travel purpose, radar, and the like. Range). When the search condition is acquired, the process proceeds to step S42.

  In step S42, the control unit 31 of the server searches for route content. Specifically, the control unit 31 of the server accesses the database and extracts route content from the database based on the identification information of the mobile terminal, the current location, the direction, the search condition, and the like. When extracting the content, the control unit 31 of the server compares the current location with the radar range, and extracts the content within the radar range set with the current location as a reference. When the root content is retrieved, the process proceeds to step S43.

  In step S43, the control unit 31 of the server provides the search result to the mobile terminal 2. That is, the control unit 31 of the server transmits the route content extracted from the database to the mobile terminal 2. Thus, the search for the root content is completed.

(Camera guidance screen display processing)
Next, details of the display process of the camera guidance screen will be described. In the first embodiment, by executing the camera guidance screen display process described below, it is possible to display content at an appropriate position more easily than in the past.

  FIG. 16A shows a display processing flow of the camera guidance screen. In step S51, the control unit 21 of the mobile terminal acquires video and content. The video is shot by the camera 24 and stored in the storage unit 28 of the portable terminal, and the content is provided from the content management server 3 and stored in the storage unit 28 of the portable terminal. And the control part 21 of a portable terminal accesses the memory | storage part 28 of a portable terminal, and acquires an image | video and a content. More specifically, the control unit 21 of the mobile terminal acquires position information via the position information acquisition unit 26 and calculates the direction and angle of the camera from the detection value of the direction sensor 27. The position information, the camera direction, and the angle are transmitted to the content management server 3 as content search conditions. The content server 3 sends the content to be displayed on the display unit 23 to the mobile terminal 2 based on the position information of the mobile terminal 2, the camera direction (horizontal direction), and the camera angle (vertical direction). The contents to be displayed can include contents in a wider range than the field of view of the camera 24 as well as contents within the field of view of the camera 24. The control unit 21 of the portable terminal acquires content transmitted from the content server 3 via the communication unit 25. When the content sent from the content server 3 is content that exists in a wider range than the field of view of the camera 24, the control unit 21 of the mobile terminal determines from the horizontal view angle and the vertical view angle of the camera 24. Content that falls within the field of view of the camera 24 is extracted. Note that only content within a predetermined distance from the camera 24 may be extracted from the content that falls within the viewing angle of the camera 24. The predetermined distance can be preset such as 1 km or 5 km. When the video and content are acquired, the process proceeds to step S52.

  In step S52, the control unit 21 of the mobile terminal performs temporary positioning of the content. The control unit 21 of the mobile terminal performs a perspective process on the content acquired in a rectangular plane including the center point with the height of the mobile terminal 2 as the center point in the coordinate space with the current location of the mobile terminal 2 as the origin. Temporarily determine the display position. The height of the portable terminal 2 can be set to 1.5 m from the ground surface, for example. That is, a value obtained by adding 1.5 m to the altitude at the position of the mobile terminal is set as the height of the mobile terminal 2. The temporary positioning is performed in consideration of the horizontal field of view of the camera 24, the vertical viewing angle, the image size, and the like. The horizontal position on the screen is determined based on the latitude and longitude information, and the vertical position on the screen is determined according to the distance from the origin, which is the current location of the mobile terminal 2.

  First, as shown in FIG. 16B, the angle of “target point A” is determined. FIG. 16B shows an image for determining an angle when “target point” is displayed. In FIG. 16B, the left diagram is a plan view showing the positional relationship between the direction of the camera 24 and “target point A”, and the right diagram overlays “target point A” on the video imaged by the camera 24. Show the image. The “target point” is, for example, a base point of a balloon or a balloon that displays content, and corresponds to the position of the target object that displays the content. Specifically, the direction of the optical axis of the camera 24 is set as a vertical reference line passing through the center in the horizontal direction of the screen. Further, auxiliary vertical reference lines parallel to the vertical reference lines are set at predetermined intervals (in steps of 2 degrees in FIG. 16B). The vertical reference line and the auxiliary vertical reference line correspond to the azimuth reference line of the present invention. “Target point A” is plotted on the auxiliary vertical reference line on the northeast side 47 ° and on the horizontal reference line. The horizontal reference line is a horizontal reference line set at the center in the vertical direction, and corresponds to the distance reference line of the present invention.

  Next, as shown in FIGS. 16C and 16D, the distance of “target point A”, in other words, the distance in the vertical direction from the horizontal reference line is determined. The distance of the “target point A” can be represented by a distance y from the horizontal reference line, and this y is calculated based on Equation 1. FIG. 16C shows an image for determining the distance when displaying the “target point”. FIG. 16D is a cross-sectional view showing the relationship between the camera 24 and the point A. In FIG. 16C, the left diagram is a plan view showing the positional relationship between the orientation of the camera 24 and “target point A”, and the right diagram overlays “target point A” on the video imaged by the camera 24. Show the image. The angle of the camera 24 (vertical direction) is assumed to be as follows, assuming that the optical axis is parallel to the ground.

  Specifically, the visual field vertical limit length V on the vertical plane passing through the target point is expressed by Formula 2, and the altitude difference Y between the own point and the target point is expressed by Formula 3.

  On the other hand, when the vertical length of the screen screen (vertical length) is v and the distance in the vertical direction (vertical direction) from the screen center line (horizontal reference line) is y, the vertical field length limit V, the altitude difference Y, the vertical length of the screen screen The relationship shown in Equation 4 holds between v and the vertical distance y from the screen center line.

  Here, the horizontal distance from the user's point to the target point is expressed by Equation 5. As a result, the relational expression 1 is established, and the distance in the vertical direction from the screen center line can be calculated.

  As described above, the angle and distance of the “target point A” are determined. When “target points”, that is, all the contents to be displayed are processed, and the angles and distances of all the contents are determined, the temporary positioning of the contents is completed. In the above description, the target point determination process has been described as provisional positioning of the display position of the content. However, for example, when correction described later is not performed, the above-described target point determination processing can be used as positioning processing for content display positions. When the temporary positioning of the content is completed, the process proceeds to step S53.

  In step S53, the control unit 21 of the mobile terminal performs image analysis for correction. Specifically, the control unit 21 of the mobile terminal analyzes the intensity (Intensity) of each pixel of the image and calculates a ridgeline, a road boundary line, and a horizontal line. FIG. 17B shows a state in which the ridgeline, the road boundary line, and the horizontal line are calculated by analyzing the intensity (Intensity) of each pixel of the image. “Line 1” indicates a ridgeline of a building, a tree, or a mountain. “Line 2” indicates a road boundary line. “Line 3” indicates a horizontal line passing through the focal points of the two road boundary lines. When the image analysis for correcting is completed, the process proceeds to step S54.

  In step S54, the control unit 21 of the mobile terminal corrects the display position of the content. FIG. 17C shows how the content display position is corrected. Regarding the horizontal position on the screen, content other than the content related to the road among the content existing inside the road boundary line (line 2) is moved to the outside of the road boundary line. In consideration of the size of the screen, if the content does not fit within the screen when moved outside the road boundary line, the content is moved outside the road boundary line within the range that fits on the screen. The content corrected by this rule is indicated by “# 1” in FIG. 17C. As for the vertical position on the screen, the content closest to the origin is arranged in the vicinity of the horizontal line, and the other contents are sequentially arranged in the back. The content corrected by this rule is indicated by “# 2” in FIG. 17C. In addition, regarding the vertical position on the screen, content related to mountains and trees is arranged above the ridgeline. The content corrected by this rule is indicated by “# 3” in FIG. 17C. On the other hand, the content related to the road is arranged below the ridgeline. The content corrected by this rule is indicated by “# 4” in FIG. 17C. Other contents are arranged between the ridgeline and the road boundary line. The content corrected by this rule is indicated by “# 5” in FIG. 17C. When the correction of the display position of the content is completed, the process proceeds to step S55.

  In step S55, the control unit 21 of the mobile terminal displays a camera guidance screen on the display unit 23 of the mobile terminal. FIG. 17D shows a display example of the camera guidance screen. When the display position of the content is determined by the processing up to step S54, it is drawn on a transparent image layer. The radar displayed at the lower left of the screen is separately created from the current location, direction, and radar range, and is drawn on another transparent image layer. Then, the image layer including the content and the image layer including the radar are superimposed on the video captured by the camera 24. Thus, a camera guidance screen is generated. By repeating the series of processes described above as appropriate, a camera guidance screen is generated in real time.

  Note that the temporary positioning of the content may be performed by, for example, dividing the video into an upper part, a central part, and a lower part and assigning the content to each region. For example, content relating to mountains may be assigned to the upper part, content relating to rivers and roads may be assigned to the lower part, and contents relating to others (such as buildings) may be assigned to the central part.

(effect)
According to the information processing system 1 according to the first embodiment described above, a navigation service using AR technology can be provided. Since the route and content managed by the content management server 3 can be easily registered or changed, route guidance based on the latest and abundant information is possible. Further, the mobile terminal 2 can guide the route by using both the map screen and the camera guidance screen. Further, on the camera guidance screen, the “road” that is most important when guiding the route can be displayed reliably, and the user can be guided with a more easily displayed display.

[Second Embodiment]
In the prior art including the first embodiment, the degree of freedom of content display is limited, and only balloons, pins, photos, icons, etc. can be displayed. However, according to the process for determining the angle and position of the target point described in the first embodiment, not only the horizontal position but also the vertical position can be accurately displayed on the screen. Accordingly, it is possible to display an object having a high degree of design freedom, such as an object (target object) that matches the shape of the building.

  Here, FIGS. 18A to 18E show a procedure for displaying content having four target points on an object. In order to execute the following processing, the content server previously stores the longitude, latitude, and height of the target point (corresponding to the corner of the target object) for specifying the shape of one target object in advance. Must be stored. In the present embodiment, four target points will be described as an example. For example, by specifying eight target points, a three-dimensional display is also possible. Hereinafter, a case where a signboard is displayed will be described as an example.

  First, as illustrated in FIG. 18A, the control unit 21 of the mobile terminal specifies (plots) a target point in the horizontal position. That is, the target points corresponding to both ends of the lower side of the signboard are specified according to the angle and distance calculation process of the target point A described in the temporary positioning process of the first embodiment. Next, as illustrated in FIG. 18B, the control unit 21 of the mobile terminal specifies the target point of the height position. That is, the target points corresponding to both ends of the upper side of the signboard are specified according to the calculation process of the angle and distance of the target point A described in the temporary positioning process of the first embodiment. Next, as illustrated in FIG. 18C, the control unit 21 of the mobile terminal connects the four specified target points and specifies the outline. Next, as illustrated in FIG. 18D, the control unit 21 of the mobile terminal specifies an area (inside the outline) determined by connecting the four target points as a plane. Furthermore, as illustrated in FIG. 18E, the control unit 21 of the mobile terminal sets text on the specified surface. Note that a photo, an illustration, or the like may be used instead of the text. Moreover, you may set a hyperlink. Further, the control unit 21 of the mobile terminal may set the texture, brightness / brightness, color, gradation, and the like of the surface surrounded by connecting the plots.

(effect)
According to the information processing system 1 according to the second embodiment described above, objects having various shapes can be displayed on the camera guidance screen. In other words, a structure or the like freely designed by the user can be arranged in real on the augmented reality space.

  Although the embodiments of the present invention have been described above, the present invention is not limited to these, and can include combinations thereof as much as possible.

DESCRIPTION OF SYMBOLS 1 ... Information processing system 2 ... Portable terminal 3 ... Content management server

Claims (5)

A camera that captures the real space,
A display unit for displaying an image of the real space photographed by the camera;
Access to a storage device storing content to be displayed overlaid on the video, and based on position information including the position and orientation of the camera, the position of the target to display the content among the targets included in the video An acquisition unit for acquiring information and the content;
A control unit that causes the content acquired by the acquisition unit to be superimposed on the video and displayed on the display unit,
The control unit sets an azimuth reference line for azimuth and a distance reference line for distance on the screen, and calculates the position of the object relative to the camera based on the position information of the camera and the position information of the object. A horizontal position of the object on the screen based on the azimuth reference line, a vertical position of the object on the screen based on the distance reference line, and a display position of the object on the screen Further, analyzing the video and correcting the content display position,
In the correction of the content display position, the control unit analyzes the video, sets a boundary reference line as a road boundary for the screen, and moves content other than content related to the road to the outside of the road. At least one of correction and correction that places content close to the origin as the current location of the camera in the vicinity of the distance reference line and sequentially places content other than content close to the origin in the back is performed. , the information processing apparatus.   The control unit sets an azimuth reference line for azimuth and a distance reference line for distance with respect to the screen, and the object relative to the optical axis of the camera based on position information of the camera and position information of the object And the distance from the camera to the object is determined, the horizontal position of the object on the screen is specified based on the azimuth reference line, and the object on the screen is determined based on the distance reference line. The information processing apparatus according to claim 1, wherein a vertical position of the object is specified, and a display position of the object on the screen is specified. The position information of the target object includes position information of three or more target points that specify the shape of the target object,
The control unit specifies a horizontal position and a vertical position on the screen for each of the target points, sets a surface by connecting the specified target points, and sets text on the set surface,
The information processing apparatus according to claim 1 or 2. The control unit analyzes the video and sets a boundary line of a scene to be displayed with priority over other scenery among the scenes constituting the real space as a priority boundary reference line, and correcting the content display position based on the priority boundary reference line, the information processing apparatus according to any one of claims 1 to 3. An information processing method executed by an information processing apparatus comprising: a camera that captures a real space; and a display unit that displays an image related to the real space captured by the camera,
Access to a storage device storing content to be displayed overlaid on the video, and based on position information including the position and orientation of the camera, the position of the target to display the content among the targets included in the video An acquisition step of acquiring information and the content;
A control step of superimposing the content acquired in the acquisition step on the video and displaying the content on the display unit,
In the control step, an azimuth reference line for azimuth and a distance reference line for distance are set on the screen, and the position of the object relative to the camera is calculated based on the position information of the camera and the position information of the object. A horizontal position of the object on the screen based on the azimuth reference line, a vertical position of the object on the screen based on the distance reference line, and a display position of the object on the screen Further, analyzing the video and correcting the content display position,
Regarding the correction of the content display position, in the control step, a boundary reference line as a road boundary is set for the screen by analyzing the video, and content other than the content related to the road is moved out of the road. At least one of correction and correction that places content close to the origin as the current location of the camera in the vicinity of the distance reference line and sequentially places content other than content close to the origin in the back is performed. , an information processing method.

Images