JP5705793B2 - Augmented reality display device, augmented reality display system, augmented reality display method, and augmented reality display program - Google Patents

Description

  The present invention relates to an augmented reality display device that displays augmented reality, an augmented reality display system, an augmented reality display method, and an augmented reality display program.

Conventionally, augmented reality technology (AR (Augmented Reality) technology) that displays information added to a real space (real environment) perceived by a user is known (see, for example, Patent Document 1).
The terminal device described in Patent Document 1 displays video data obtained by photographing by a camera unit as a live view on a display unit, and subsequently acquires position information at the current position of the terminal device by a position information acquisition unit. Then, the acquired position information is transmitted to the server, and a list of augmented reality information around the terminal device responded by the server is received and stored in the storage unit. Thereafter, the augmented reality information in the video data is superimposed on the live view and displayed based on the posture and angle of the camera detected by the azimuth sensor and the position information.

JP 2010-238096 A

  By the way, in the apparatus described in Patent Document 1, information within a range photographed by a camera can be displayed on the display unit, but information in a remote place cannot be displayed. Moreover, there is a problem in that augmented reality is impaired when only remote information is displayed.

  An object of the present invention is to provide an augmented reality display device, an augmented reality display system, an augmented reality display method, and an augmented reality display program capable of displaying information on a remote location without impairing augmented reality.

The augmented reality display device according to the present invention is based on a captured image acquisition unit that acquires a captured image captured by a camera, an imaging direction detection unit that detects a direction in which the camera is directed as an imaging direction, and a user's input operation. From point identification means for identifying a predetermined designated point and point information storage means for storing point content information indicating contents related to the point, point content information corresponding to a point within a predetermined range including the designated point and the point content information acquiring means for acquiring, on a display unit for displaying an image, viewed contains a plurality of points, and an icon the captured image corresponding to the different remote region from the region containing the acquired current position Display means for displaying the captured image displayed in a superimposed manner, and the point content information acquisition means, when the icon is selected by a user input operation, Acquiring the location content information corresponding to the points included in the remote area corresponding to the icon, the display means to display the captured image in which the icon is displayed superimposed on the display unit, the location content information When the point content information is acquired by the acquisition unit, the point content information in which the direction from the designated point to the point corresponding to the point content information is the same direction as the imaging direction is superimposed on the captured image. It is characterized by being displayed.
In the present invention, “the direction from the designated point to the point corresponding to the point content information is the same direction as the imaging direction” means the imaging direction and the point corresponding to the point content information from the designated point. It is not limited to the case where the direction toward is exactly the same. That is, the imaging direction in the present invention indicates a predetermined angle of view (viewing angle) range according to the performance of the camera with the lens optical axis direction of the camera as the center. The point content information in which the direction toward the point corresponding to the information is within this angle of view range is displayed superimposed on the captured image.

  In the present invention, since the information on the remote location is displayed on the captured image of the real space, the information on the remote location can be displayed without impairing the augmented reality.

1 is a block diagram showing a schematic configuration of an augmented reality display system according to an embodiment of the present invention. The flowchart which shows the augmented reality display method of this embodiment. The figure which shows the area | region containing the point corresponding to point information, and the present position of a terminal device. The figure which shows an example of the augmented reality display screen displayed on the display of a terminal device. The flowchart at the time of the icon being selected in an augmented reality display screen. The figure which shows the image process when an icon is selected. The figure which shows an example of the augmented reality display screen (selected point information display screen) which displayed the point information of the selected point. The flowchart which shows the posting process of contribution information.

Hereinafter, an embodiment according to the present invention will be described in detail with reference to the drawings.
[Entire configuration of augmented reality display system]
FIG. 1 is a block diagram showing a schematic configuration of an augmented reality display system in the present embodiment.
As illustrated in FIG. 1, the augmented reality display system 1 includes a terminal device 10 (augmented reality display device) and a server device 20, and the terminal device 10 and the server device 20 communicate via a network 2. Connected as possible. As the network 2, for example, the Internet based on a general-purpose protocol such as TCP / IP can be used, but other networks may be used. For example, any configuration that transmits and receives data can be used, such as a communication network that forms a network with a plurality of base stations that can transmit and receive information using a wireless medium such as a LAN (Local Area Network).
The augmented reality display system 1 displays a captured image captured by the camera 11 on the display 12 of the terminal device 10 and displays predetermined information superimposed on the captured image. In other words, this is a system that performs augmented reality (AR) display on the display 12 of the terminal device 10. Further, in the augmented reality display system 1 of the present embodiment, the terminal device 10 accepts a request to display information on a remote location by a user operation during the augmented reality display process of the captured image captured by the camera 11. The remote location information is received from the server device 20 and displayed superimposed on the captured image.

[Configuration of server device]
As shown in FIG. 1, the server device 20 is configured by a server storage unit 21 configured by a RAM (Random Access Memory), a ROM (Read Only Memory), and the like, and a CPU (Central Processing Unit) and the like as a hardware configuration. Server control means 22. Further, the server device 20 may have a configuration in which an operation unit including a display or a keyboard, a mouse, or the like is provided. Further, the server device 20 includes server communication means (not shown) for communicating with the terminal device 10 via the network 2.

The server storage unit 21 stores an OS (Operating System) for controlling the overall operation of the server device 20, various programs, and various data.
The server storage unit 21 functions as the point information storage unit and the object information storage unit of the present invention. That is, various information such as point information for displaying augmented reality information and object information is recorded in the server storage unit 21 on the display 12 of the terminal device 10.

The point information is information related to the point. For example, as shown in Table 1 below, the region ID, the region information, the point name, the point position, the display area size, the point content information, the display position corresponding to the point content information, etc. Is remembered.
The spot ID is information for identifying the spot information, and a different ID is assigned to each spot information.
The area information is information indicating an area including the point. This area is an area obtained by dividing the map by a predetermined boundary. For this boundary, for example, a boundary that divides the map into local units, such as the Tohoku region and the Kanto region, may be used. A boundary delimited by administrative division units such as a ward level may be used. Moreover, it is good also as an area | region within the predetermined diameter range centering on a predetermined representative point. Furthermore, it is good also as a structure by which an area is set based on a user's operation.
In the area information, the name of the area and a representative point of the area (for example, the center (center of gravity) point, the prefectural office location point, the point designated by the user, etc.) are recorded.
As the spot name, for example, spot names such as “Tokyo”, “Osaka”, and “Nagoya” are stored.
The point position is the position of the point on the map. As the position of the point, for example, latitude, longitude, altitude, or the like may be stored, and a coordinate position in predetermined map information may be recorded.
The point content information is information indicating content related to the point. As this point content information, for example, information related to the point, such as posted information such as comments posted by the user and bulletin board statements, guidance information such as stores and sightseeing spots within a predetermined range including the point, is recorded. .

The display area size is information that defines a coordinate range that can be set as a display position.
Although detailed description will be described later, in the present embodiment, for example, a spherical icon is superimposed on the captured image on the display 12 to notify the user of the presence of the point information, and the user selects the icon. Then, the image is changed to a state in which the captured image is viewed from the inside of the sphere. At this time, the area displayed on the display 12 in the icon inner surface of the sphere is specified, and the point content information having the display position in the area is displayed on the display 12. The display area size is information that defines the coordinate system of the inner surface of the sphere. In this embodiment, the display area size is a sphere, for example, a sphere equation with the sphere center point as the origin, the distance r from the center point, the east direction as the X axis, the north direction as the Y axis, and the vertical direction as the Z axis. It can be represented by (r ² = x ² + y ² + z ² ). In this case, the display position (x, y, z) is recorded as an integer value x, y, z that satisfies the spherical equation.
The display area size may be set in advance depending on the location, and the display area size may be changeable according to the number of pieces of post information transmitted from the terminal device 10 or the like.

As the display position, display position coordinates for displaying the point content information are recorded. The display position is set to coordinates within a coordinate range defined by the display area size. When the point content information is displayed on the display 12 of the terminal device 10, the terminal device 10 displays the point content information based on the display position.
The display position may be set based on, for example, the direction in which the point with respect to the representative point in the region is located. As described later, the display position is set based on the direction in which the posting information is posted by the user. There may be. As the representative point, the central point of the region (for example, the position of the center of gravity of the regional mesh in the map) may be used, or the prefectural office location point of the city with the largest population may be used. For example, in Table 1 below, when the representative point of “G point” is the point g, the display position of the point information whose point ID is “ID1” is the straight line in the direction from the point g to the point A and the display area size. The coordinates of the intersection point with the inner surface of the sphere (icon) defined by is set.
Further, when displaying the point content information in the display area, the display position may be optimized as appropriate so that the information is displayed at a uniform display interval.

The object information stored in the server storage unit 21 is augmented reality information of the object included in the captured image, and for example, facility information is recorded.
As the object information, for example, as shown in Table 2, an object ID, an object name, an object type, an object position, and object guide information are recorded. In addition, as target object information, the height information etc. in which a target object is located may be recorded, for example. Moreover, a user's comment (post information) etc. with respect to a target object may be recorded.
The object ID is information for identifying the object information, and a different ID is assigned to each object information.
The object name is the name of the object such as a facility name.
The object type is a type of the object. For example, in the case of a facility, the genre of the facility such as a bus stop, a station, a school, a hotel, or the like is recorded.
The object position is detailed position information of the object, and for example, latitude, longitude, and the like are recorded. In addition, the coordinate position in predetermined map information may be recorded, and an address etc. may be recorded.
In the object guidance information, detailed contents regarding the object such as a guidance sentence and a guidance image of the object (facility) are recorded.

  In the present embodiment, as the augmented reality display information by the terminal device 10, an example is shown in which target object information for a facility existing in the captured image is displayed on the captured image, but the present invention is not limited to this. For example, information regarding an object in the captured image may be displayed as augmented reality information. For example, when an image of a new product is displayed in the captured image, the server device 20 receives the captured image from the terminal device 10, performs an image search, and stores information on the corresponding new product in the terminal device 10. May be displayed on the display 12 of the terminal device 10 as augmented reality information. In such a case, the server storage unit 21 may store, for example, a plurality of image examples of the target object as the target object information.

The server control means 22 reads and executes various programs stored in the server storage means 21, thereby, as shown in FIG. 1, the spot information search means 221, the spot information transmission means 222, the object information transmission means 223, And the post information acquisition means 224.
When the point information search unit 221 obtains a point information request for requesting transmission of the point information for the predetermined point transmitted from the terminal device 10, the point information search unit 221 searches the server storage unit 21 for the point information for the requested point.
The spot information transmitting unit 222 transmits the searched spot information to the terminal device 10.
The object information transmitting unit 223 transmits a list of object information based on the object information request transmitted from the terminal device.
The posting information acquisition unit 224 acquires the posting information transmitted from the terminal device 10 and stores it in the server storage unit 21.

[Configuration of terminal device]
As illustrated in FIG. 1, the terminal device 10 includes, as a hardware configuration, a camera 11, a display 12 (display unit), an imaging direction detection unit 13, a current position acquisition unit 14 that measures a current position, and a touch panel, for example. Operating means 15 constituted by a keyboard, mouse, operation buttons, etc., terminal storage means 16 constituted by a memory or the like, and terminal control means 17 constituted by a CPU (Central Processing Unit) or the like. . In addition, the terminal device 10 includes terminal communication means (not shown) for communicating with the server device 20 via the network 2.
As such a terminal device 10, it is preferable to use portable terminal devices, such as a mobile telephone apparatus, a smart phone, a tablet-type terminal device, a notebook-type personal computer, for example. In the present embodiment, a portable terminal device is illustrated as the terminal device 10, but the present invention is not limited to this. For example, a fixed-side terminal device such as a desktop personal computer may be used. In this case, by connecting the camera 11 to the terminal device by wire or wirelessly, it is possible to perform an extended display that adds information to the captured image captured by the camera 11.

The imaging direction detection means 13 detects the imaging direction (imaging direction and angle) in which the camera 11 is directed. The imaging direction detection means 13 includes, for example, an electronic compass that detects the azimuth by detecting geomagnetism in the triaxial direction and a triaxial acceleration sensor that detects the tilt angle of the terminal device 10 with respect to the triaxial direction. Then, based on the orientation detected by the electronic compass and the tilt angle of the terminal device 10 detected by the triaxial acceleration sensor, the imaging direction in which the camera 11 is directed is detected.
Further, in a configuration in which the camera 11 is connected to a fixed-side terminal device such as a desktop personal computer, for example, the imaging direction of the camera 11 is changed by providing the imaging direction detecting means 13 in the camera 11. Can be acquired.

The current position acquisition unit 14 receives satellite signals having time information transmitted from, for example, three position measurement satellites (for example, GPS satellites), and based on these satellite signals, the current position as a measurement target point Calculate coordinates (latitude, longitude). Note that the current position acquisition unit 14 may further calculate the altitude of the current position.
Further, in a configuration in which the camera 11 is connected to a fixed-side terminal device such as a desktop personal computer, the current position acquisition unit 14 is provided in the camera 11, for example. Current position) can be acquired.

  The terminal storage means 16 is configured by a RAM (Random Access Memory), a ROM (Read Only Memory), or the like. The terminal storage unit 16 stores an OS (Operating System), various programs, various applications, and various data. In addition, the terminal storage unit 16 includes a temporary area that can temporarily store various data transmitted from the server device 20, for example.

  The terminal control unit 17 reads and executes various programs stored in the terminal storage unit 16, thereby, as shown in FIG. 2, a captured image acquisition unit 171, an object information acquisition unit 172, a point direction determination unit 173, It functions as a spot specifying means 174, spot information acquiring means 175 (spot content information acquiring means), display area setting means 176, post information receiving means 177, and display means 178.

The captured image acquisition unit 171 acquires a captured image captured by the camera 11.
The object information acquisition unit 172 acquires a list of object information for the captured image from the server device 20 and stores it in the terminal storage unit 16.

The point direction determination unit 173 acquires a list (region list) of remote points from the server device 20 and determines whether or not a point (region) where the point content information can be displayed is located in the imaging direction.
The point specifying unit 174 specifies a predetermined designated point as the selected point based on the operation of the operation unit 15 by the user.
The spot information acquisition unit 175 acquires spot information of the same area information as the selected spot from the server device 20. That is, the spot information acquisition unit 175 acquires spot information (including spot content information) corresponding to a spot within a predetermined range (region) including the selected spot.

The display area setting unit 176 sets a display area for displaying the spot information. As described above, in the present embodiment, a spherical icon is superimposed on the captured image on the display 12 to notify the user of the presence of point information. When the icon is selected by the user, the image is changed to a state in which the captured image is viewed from inside the sphere. At this time, the display area setting unit 176 sets a display area (coordinate range) displayed on the display 12 among the inner surface of the sphere.
When the posted information is input by the operation of the user operating unit 15, the posted information receiving unit 177 receives the posted information and transmits it to the server device 20 together with the display position.
The display unit 178 causes the display 12 to display the captured image acquired by the captured image acquisition unit 171, and displays the target information acquired by the target object information acquisition unit 172 so as to be superimposed on the captured image. Further, the display unit 178 displays a spherical icon on the captured image when the point direction determination unit 173 determines that there is a point having the point information in the imaging direction. Further, when the icon is selected by the user's operation unit 15 and the point information is received from the server device 20, the display unit 178 displays the point content information of the point information superimposed on the captured image according to the display position. .

[Augmented reality display processing]
Next, augmented reality display processing (augmented reality display method) by the augmented reality display system 1 as described above will be described with reference to the drawings.
FIG. 2 is a flowchart showing the augmented reality display method of the present embodiment.
This augmented reality display process is started when the augmented reality display program is started by the user's operation of the operation means 15.
When the augmented reality display process is started, first, the camera 11, the imaging direction detection unit 13, and the current position acquisition unit 14 are activated, and the captured image acquired by the camera 11 is acquired by the captured image acquisition unit 171. . That is, a captured image acquisition step, an imaging direction detection step, and a current position acquisition step are performed. At this time, the display unit 178 displays the acquired captured image on the display 12 (S11).
And the target object information acquisition means 172 transmits the target object information request | requirement to the effect of requesting target object information to the server apparatus 20 (S12). At this time, the object information acquisition unit 172 transmits the object information request to the server device 20 together with the imaging direction detected by the imaging direction detection unit 13 and the current position calculated by the current position acquisition unit 14.

When the server device 20 receives the object information request from the terminal device 10 (S21), the object information transmitting unit 223 acquires a list of object information (S22). Specifically, the object information transmitting unit 223 extracts object information having position coordinates within a predetermined range centered on the current position as the object position from the server storage unit 21, and further, the imaging direction from the current position. The object information corresponding to the object located at is extracted, and a list of object information is generated.
And the target object information transmission means 223 transmits the list of target object information acquired by S22 to the terminal device 10 (S23).

When the object information acquisition unit 172 of the terminal device 10 receives the list of the object information from the server device 20 (S13), the object information acquisition unit 172 stores the list of the object information in the terminal storage unit 16.
Thereafter, the display unit 178 displays the object information such as the object name on the display 12 based on the object position of each object information and the current position calculated by the current position acquisition unit 14. A superimposed image is displayed on the captured image (S14).
As a result, augmented reality information for the captured image in the real space is displayed on the display 12.

Further, the point direction determination means 173 of the terminal device 10 transmits a region list request for requesting the server device 20 to transmit a remote region list (S15).
When the server apparatus 20 receives the area list request from the terminal apparatus 10 (S24), the server apparatus 20 extracts area information based on each piece of point information stored in the server storage means 21, generates an area list, and the terminal apparatus 10 (S25). In addition to the area information, the area list may include information for specifying a point such as a point ID of each point included in the area.

When the terminal apparatus 10 receives the area list from the server apparatus 20 (S16), the point direction determination unit 173 determines whether or not at least a part of the area is located in the imaging direction from the current position (S17).
FIG. 3 is a diagram illustrating an area including a point corresponding to the point information and a current position of the terminal device 10. FIG. 4 is an example of the augmented reality display screen 30 displayed on the display 12 of the terminal device 10.
In FIG. 3, a point A indicates the current position, and an arrow indicates the imaging direction. As shown in FIG. 3, in the terminal device 10 located at the point A, when the camera 11 is pointed in the direction of the arrow A-1, the area B-1 is located in the imaging direction, and the arrow A- When the camera 11 is directed in the direction of 2, the areas B-2 and B-3 are located in the imaging direction.
On the other hand, in the terminal device 10 located at the point A, when the camera 11 is pointed in the direction of the arrow A-3, the region corresponding to the point information (the region including the point where the point content information is registered) is the imaging direction. Not located.

If “No” is determined in S17, for example, in FIG. 3, when the imaging direction is the direction of the arrow A-3, the icon 33 is not displayed, and the process of S19 described later is performed.
On the other hand, when it is determined as “Yes” in S17, that is, when the point direction determining unit 173 determines that at least one area is in the imaging direction, the display unit 178 is shown in FIG. As described above, the icon 33 (icon) indicating that there is an area having the point information is superimposed and displayed on the captured image (S18).
That is, the display unit 178 displays the captured image 31 acquired in S11 on the display 12, and the object information 32 related to the object (facility) in the captured image is displayed on the captured image 31 by the process of S14. Are superimposed and displayed. Further, when there is an area having point information in the imaging direction in S17, the display unit 178 displays the icon 33 at a screen position based on the imaging direction at the current position and the position of the area in S18. In other words, the display unit 178 displays the icon 33 corresponding to an area including a plurality of points on the captured image.
In FIG. 3, when the imaging direction is the direction of the arrow A-2, areas B-2 and B-3 having point information in the imaging direction exist. In this case, icons 33 corresponding to the respective points are superimposed on the captured image 31 and displayed.
Further, the display unit 178 may display the area name associated with the area information together with the icon 33. In this case, when a plurality of icons 33 are superimposed on the captured image 31, the user can easily determine the icons 33.

  If it is determined as “No” in S17 or after the process of S18, the terminal control means 17 determines whether or not to continue the augmented reality display process (S19). In S <b> 19, when an instruction to end the process is input by the user's operation of the operation means 15 (determined as “No”), the augmented reality display process is ended. In S19, if there is no input to end the process, it is determined as “Yes” and the process returns to S11. That is, when there is no change in the imaging direction, the terminal device 10 acquires object information and point position information at regular intervals, updates the information, and when there is a change in the imaging direction, Based on the captured image newly acquired along with the change, the object information and the position position information are newly acquired.

(Process when icon is selected)
Next, processing when the icon 33 is selected by the user's operation of the operation means 15 on the augmented reality display screen 30 displayed in S18 will be described below.
FIG. 5 is a flowchart showing the display processing of the selected spot information display screen that is performed when the icon 33 is selected on the augmented reality display screen 30.
When the icon 33 is selected by the user's operation on the augmented reality display screen 30, the point specifying unit 174 specifies a region corresponding to the selected icon 33, and selects a representative point in the region. The point is specified (S101).
Next, the spot information acquisition unit 175 generates a spot information request for requesting spot information of the area corresponding to the selected spot, and transmits it to the server device 20 (S102). In addition, when the point ID is transmitted together with the region list, the point ID of the point included in the region including the selected point may be transmitted.

When receiving the point information request (S201), the point information search unit 221 of the server device 20 searches for the point information corresponding to the selected point from the point information stored in the server storage unit 21 (S202).
And the spot information transmission means 222 transmits the spot information searched by S202 to the terminal device 10 (S203).

  When the spot information acquisition unit 175 of the terminal device 10 receives the spot information corresponding to the selected spot transmitted from the server device 20 (S103), the spot information is stored in the terminal storage unit 16.

Thereafter, the display unit 178 performs a spot information display process in which spot information corresponding to the selected spot is superimposed on the captured image 31 and displayed.
FIG. 6 is a diagram illustrating image processing when the icon 33 is selected. FIG. 7 is an example of an augmented reality display screen (selected point information display screen) on which point information of a selected point is displayed.
In this point information display processing, first, as shown in FIG. 6, the display means 178 gradually enlarges the spherical image of the icon 33 selected by the user's operation of the operation means 15 so that the icon 33 is in front of the screen. Execute an animation process that is close to the side. At this time, the display unit 178 hides the object information 32.
Then, as shown in FIG. 7, the display unit 178 displays the captured image 31 in which the captured image is changed from the inside of the spherical icon 33 (S104). At this time, the display unit 178 may perform image editing for curving the captured image in accordance with the spherical shape of the icon, and display the edited captured image on the display 12.

Then, the display area setting unit 176 sets a display area to be displayed on the display 12 among the inner surfaces of the icon 33 (S105).
Here, as described above, the coordinate system of the inner surface of the icon 33 varies depending on the display area size. For example, when there is a lot of point information for a point, the display area size becomes large in order to display all the point information. Therefore, the display area setting unit 176, for example, the inner surface of the icon 33 (the inner surface of the icon image) so that the point content information displayed in one screen (in the selected spot information display screen 34) becomes a predetermined number. ), A display area to be displayed on the display 12 is set.
Thereafter, the display unit 178 extracts the spot information whose display position is within the display area set in S105 from the spot information received in S103 (S106), and performs imaging according to the display position as shown in FIG. The point content information 35 is displayed on the image 31 (S107).

In addition, on the selected point information display screen 34, the display area setting unit 176 changes the area to be displayed on the display 12 among the inner surfaces of the icon images depending on the imaging direction of the camera 11. For example, when the imaging direction is changed from east to west, the display area setting unit 176 inverts the sign of the x coordinate of the display area on the inner surface of the icon image displayed on the display 12. Thereby, the display means 178 also switches the point content information 35 to be displayed on the display 12.
Further, the display area setting unit 176 changes the display area to be displayed on the display 12 in the inner surface of the icon image when the enlargement / reduction process is performed by the zoom function of the camera 11, for example. Then, the display unit 178 displays the point content information, switches between non-display, and changes the display size corresponding to the changed display area.
For example, when the captured image is zoomed by the camera 11, the display area setting unit 176 reduces the display area on the inner surface of the icon image displayed on the display 12. As a result, the display unit 178 hides the point content information outside the display area, and displays the point content information in the display area in an enlarged manner.
Note that the display area setting unit 176 is not limited to the imaging direction of the camera 11 and the zoom function, and may change the display area according to the operation of the operation unit 15 by the user.

  In the selected spot information display screen 34, the spot content information to be displayed includes comment information, icons, etc. as shown in FIG. This icon includes, for example, information on a facility located within a predetermined range centered on the selected point, a bulletin board, and the like. Corresponding information is further displayed.

Thereafter, the terminal device 10 determines whether or not to continue displaying the selected spot information display screen 34 (S108). In S108, when it is input that the display process of the selected spot information display screen 34 is to be terminated by the user's operation of the operation means 15 (determined as “No”), the process returns to the process of S11 in FIG.
In S108, if there is no input to end the process, it is determined as “Yes”, and the display of the point content information in S107 is continued.

(Posting information posting process)
Next, posting processing when posting information is posted on the selected spot information display screen 34 as shown in FIG. 7 will be described below.
FIG. 8 is a flowchart of the posting process.
As shown in FIG. 8, when the posting information is input by the operation of the user operating means 15 while the selected spot information display screen 34 is displayed on the display 12, the posting information receiving means 177 Information is received (S111).
Then, the posted information receiving unit 177 acquires the intersection coordinates between the imaging direction detected by the imaging direction detection unit 13 and the inner surface of the icon image (S112).
Thereafter, the posting information receiving means 177 associates the posting information with the intersection coordinates and transmits them to the server device 20 (S113).

When the posting information acquisition unit 224 of the server device 20 receives the posting information and the intersection coordinates (S211), the posting information acquisition unit 224 generates the point information with the posting information as the point content information and the intersection point coordinates as the display position, and stores it in the server storage unit 21. (S212). In addition, the server device 20 transmits a response signal indicating that the posting information has been received to the terminal device 10 (S213).
As a result, post information posted by the user is accumulated in the server storage unit 21, and the spot information stored in the server storage unit 21 is updated.

  When receiving the response signal from the server device 20 (S114), the terminal device 10 returns to the process of S102. That is, the updated point information is received and displayed on the display 12. As a result, the selected point information display screen 34 reflecting the posted information is displayed.

[Effects of Embodiment]
The terminal device 10 in the augmented reality display system 1 according to the present embodiment acquires a captured image captured by the camera 11 using the captured image acquisition unit 171, detects the imaging direction using the imaging direction detection unit 13, and further detects the target object. Information acquisition means 172 acquires object information corresponding to the object in the captured image. The display unit 178 displays the augmented reality display screen 30 that displays the captured image 31 on the display 12 and displays the object information 32 such as the object name on the object in the captured image 31. Let
Further, the point specifying unit 174 specifies a predetermined designated point away from the current position as the selected point by selecting the icon 33 by the user's operation of the operating unit 15, and the point information acquiring unit 175 selects the selected point. Get local point information corresponding to the point. Then, the display unit 178 superimposes predetermined point content information on the captured image 31 of the display 12 based on the display position of the acquired spot information and the display area set by the display area setting unit 176. Display. That is, in the present embodiment, the display unit 178 displays the in-point information in which the direction from the designated point (representative point in the predetermined area) to the point corresponding to the point content information is within the angle of view of the imaging direction. Is displayed.
For this reason, in addition to the augmented reality display screen 30 displayed by superimposing the object information 32 on the captured image 31, the terminal device 10 includes point content information of a remote location that is virtual information on the captured image 31 that is a real space. The selected point information display screen 34 displayed by superimposing 35 can be displayed. That is, since the information on other regions can be displayed on the display 12 while the captured image 31 for the real space is displayed, the characteristics of the augmented reality technology such as sharing and cooperation between the real space and the virtual information are not lost. , Information on other regions can be displayed.

In this embodiment, the imaging direction detection unit 13 detects the imaging direction of the camera, and the current position acquisition unit 14 calculates the current position where the captured image is captured. Further, the point direction determination unit 173 determines whether or not there is an area having the point information in the imaging direction based on the list of point position information acquired from the server device 20, the imaging direction, and the current position. The display unit 178 superimposes and displays an icon 33 indicating that there is a point having the point information on the captured image 31 when there is an area having the point information in the imaging direction. That is, when there is a region (region) where a plurality of points having point information gather in the imaging direction, the icon 33 corresponding to the region is displayed on the captured image.
Thus, by displaying the icon 33 on the captured image 31, it is possible to easily notify the user of the presence of a selectable designated point. Further, by changing the imaging direction, it is possible to display the icon 33 corresponding to another point, and the user can easily confirm the direction of the remote point from the current position.
In the present embodiment, when the icon 33 displayed on the display 12 is selected by a user input operation, the point specifying unit 174 selects a point included in the area corresponding to the icon 33 as a selected point. Identify.
For this reason, for example, a remote point can be easily selected for the process of calling a menu screen or the like and selecting a desired point, and the selected point information display screen 34 can be displayed.

In the present embodiment, the icon 33 is a spherical image, and when the icon 33 is selected, the display unit 178 changes the captured image to a state viewed from the inside of the icon 33, and the inner surface of the sphere of the icon 33 is displayed. The point content information 35 is displayed on the inner surface of the icon image.
This allows the user to display augmented reality information (object information) with respect to the real space or to display augmented reality information (point content information) of other points with respect to the real space. Can be easily determined.

In this embodiment, the point content information is associated with a display position in the coordinate system of the inner surface of the icon image. The terminal device 10 includes a display area setting unit 176 that sets a display area of the inner surface of the icon image to be displayed on the display 12, and the display unit 178 displays the point content information of the display position corresponding to the set display area. 35 is displayed on the display 12.
Thereby, even when a plurality of pieces of point content information cannot be displayed on one screen, the other point content information can be displayed by changing the imaging direction to change the display area on the inner surface of the icon image. Further, the display area size may be changed according to the number of point information. In this case, more point content information 35 can be displayed in an appropriate size that is easy for the user to visually recognize.
The display area setting unit 176 enlarges or reduces the display area on the inner surface of the icon image based on the zoom function of the camera 11. Thereby, for example, when the display size of the spot content information 35 is small, the spot content information 35 can be displayed in a large size that can be easily recognized by the user by using the enlargement function of the camera 11.

In the present embodiment, post information receiving means 177 for receiving post information input based on the user's operation of the operation means 15 is provided. When the posting information receiving unit 177 receives the posting information, the posting information receiving unit 177 calculates intersection coordinates between the imaging direction of the camera 11 detected by the imaging direction detection unit 13 and the coordinates of the icon image inner surface. Then, the posting information acquisition means 224 of the server device 20 generates the spot information with the posting information as the spot content information and the intersection coordinates as the display position, and stores it in the server storage means 21.
Thereby, the user can post the posting information at a desired position on the inner surface of the icon image, and for example, the posting information can be displayed in the vicinity of the related spot content information.

  In the present embodiment, the display unit 178 hides the object information 32 on the augmented reality display screen 30 when switching from the augmented reality display screen 30 to the selected spot information display screen 34 on the display 12. Thereby, the object information 32 on the object in the captured image in the real space and the point content information in the remote place away from the current position are not mixed, and the information visibility can be improved.

[Other Embodiments]
In addition, this invention is not limited to embodiment mentioned above, In the range which can achieve the objective of this invention, the deformation | transformation shown below is also included.
For example, in the above-described embodiment, the example in which the augmented reality display system 1 of the present invention is configured by the terminal device 10 that is an augmented reality display device and the server device 20 is shown. It can function as a display device. In this case, the point information and the object information stored in the server storage unit 21 of the server device 20 are stored in the terminal storage unit 16. Further, the terminal control unit 17 of the terminal device 10 further functions as the point information search unit 221 by executing a program stored in the terminal storage unit 16. Even in this case, the display of the object information of the captured image corresponding to the real space and the display of the point content information corresponding to the remote point can be performed on the captured image as in the above embodiment. Moreover, the posting information transmitted from other terminal devices can be acquired via the server device, whereby the posting information can be updated sequentially.

  In the above embodiment, the display unit 178 has shown an example in which the icon 33 corresponding to a preset region is displayed on the captured image on the display 12 based on the current position and the imaging direction, but the present invention is not limited to this. For example, when there are a plurality of points having the point information within a predetermined range based on the distribution state of the points having the point information, the icon 33 corresponding to the region may be displayed on the captured image. Good.

  Further, the designated point specified by the point specifying unit 174 is not limited to a representative point in a preset region, and may be a designated point arbitrarily designated by the user. In this case, for example, the server device 20 searches for a point (a point having point information) within a predetermined distance range centered on the specified point, and transmits the point information corresponding to the searched point to the terminal device 10. That's fine.

In the above-described embodiment, the spherical icon 33 is exemplified as the icon. However, for example, an icon of another stereoscopic image such as a cube may be used.
Furthermore, it is good also as a structure which displays a point content information, without using a stereo image as an icon and without changing a captured image. Even in this case, for example, by concealing the object information and displaying the point content information, confusion between the object information and the point content information can be avoided.
Moreover, although the example which displays the icon 33 on a captured image when the area | region where the point which has point information in an imaging direction exists based on the present position and an imaging direction was shown, for example, the selectable area ( (Selectable designated points) may be displayed on a separate screen, or a list may be displayed on the captured image.

  In the above-described embodiment, the server device 20 transmits the point location information list to the terminal device 10 in the process of S25. However, for example, the point information list may be transmitted to the terminal device 10. However, if all the point information is transmitted, the transmission data size increases, and the communication time may be longer. In this case, for example, in the terminal device 10, a range for acquiring point information (for example, a distance from the current position) can be set by a user operation, and processing for acquiring point information for only the set range is performed. Also good.

  The display unit 178 displays only the icon 33 corresponding to the point within the distance range set by the user when there is a point having a plurality of pieces of point information in the imaging direction on the augmented reality display screen 30. You may superimpose on and display. In this case, for example, the display unit 178 displays an icon for switching the distance of the spot to be displayed as the icon 33 on the captured image 31, and the icon 33 is switched by the user selecting the icon. Etc. can be illustrated.

  Moreover, although the example which acquires the intersection coordinate of an imaging direction and the icon image inner surface was shown as a display position of contribution information, it is not limited to this. For example, the display position of the posted information (point content information) may be set based on the current position and the point coordinates of the selected point. For example, the coordinate position of the selected point may be the central point, and the intersection of the direction from the central point toward the current position and the inner surface of the icon image may be the display position.

Although the display means 178 showed the example in which point content information is displayed based on the display position which point information has, and the coordinate system in the icon image inner surface, it is not limited to this. For example, the point content information may be displayed regardless of the coordinate system on the inner surface of the icon image. Even in this case, the point content information in which the direction from the designated point specified by the point specifying unit 174 to the point corresponding to the point information is the same as the imaging direction can be displayed superimposed on the captured image. The same effect as the above embodiment can be obtained.
Further, when there are a large number of pieces of point content information that cannot be displayed on one screen, for example, an icon for switching the display is displayed, and the icon is selected by the operation of the user operation means 15 to be displayed on the display 12. The point content information to be switched may be switched.

  Further, the terminal control unit 17 reads and executes the program stored in the terminal storage unit 16, so that the captured image acquisition unit 171, the object information acquisition unit 172, the point direction determination unit 173, the point identification unit 174, Although the example which functions as the information acquisition means 175, the display area setting means 176, the contribution information reception means 177, and the display means 178 was shown, it is not limited to this. For example, an IC chip having various functions as described above may be arranged and each component may be configured as hardware. The same applies to the spot information search means 221, the spot information transmission means 222, the object information transmission means 223, and the posted information acquisition means 224.

  In addition, the specific structure and procedure for carrying out the present invention can be appropriately changed to other structures and the like within a range in which the object of the present invention can be achieved.

  DESCRIPTION OF SYMBOLS 1 ... Augmented reality display system, 2 ... Network, 10 ... Terminal device, 11 ... Camera, 12 ... Display (display part), 13 ... Imaging direction detection means, 14 ... Current position acquisition means, 15 ... Operation means, 16 ... Terminal Storage means, 17 ... terminal control means, 20 ... server device, 21 ... server storage means (point information storage means, object information storage means), 30 ... augmented reality display screen, 31 ... captured image, 32 ... object information, 33 ... Icon, 34 ... Selected point information display screen, 35 ... Point content information, 171 ... Captured image acquisition means, 172 ... Object information acquisition means, 173 ... Point direction determination means, 174 ... Point identification means, 175 ... Point information Acquisition means, 176 ... display area setting means, 177 ... post information receiving means, 178 ... display means, 221 ... point information search means, 224 ... post information acquisition means.

Claims (10)

Captured image acquisition means for acquiring a captured image captured by the camera;
Imaging direction detection means for detecting a direction in which the camera is directed as an imaging direction;
A point identifying means for identifying a predetermined designated point based on a user input operation;
Point content information acquisition means for acquiring point content information corresponding to a point within a predetermined range including the designated point, from point information storage means for storing point content information indicating content related to the point;
A display unit for displaying an image, viewed contains a plurality of points, and the captured image displayed by superimposing the icon corresponding to different remote region from the region in which the captured image is included by the obtained current position Display means for displaying,
When the icon is selected by a user input operation, the spot content information acquisition unit acquires the spot content information corresponding to a spot included in a remote area corresponding to the icon,
The display unit displays the captured image displayed with the icon superimposed on the display unit, and when the point content information is acquired by the point content information acquisition unit, the point content information from the designated point. The augmented reality display device, wherein the point content information whose direction toward the point corresponding to is the same direction as the imaging direction is superimposed on the captured image. The augmented reality display device according to claim 1,
The augmented reality display device, wherein when at least a part of the remote area is located in the imaging direction, the display means displays the icon corresponding to the remote area in the captured image. The augmented reality display device according to claim 1 or 2,
The icon is a three-dimensional image,
When the icon is selected by a user input operation, the display means displays the captured image in a state where the captured image is viewed from the inside of the three-dimensional object, and the content of the spot on the inner surface of the three-dimensional object An augmented reality display device characterized by displaying information. The augmented reality display device according to claim 3,
The point content information is associated with a display position of the point content information on the inner surface of the three-dimensional object,
The augmented reality display device includes:
A display area setting unit configured to set a display area to be displayed on the display unit among the inner surfaces of the three-dimensional object based on the imaging direction;
The display means displays the point content information of the display position corresponding to the area where the display position of the point content information is set by the display area setting means based on the display position. Augmented reality display device. The augmented reality display device according to claim 4,
Post information receiving means for receiving post information for the point input by the user's input operation;
The posted information received by the posted information receiving means is used as the spot content information, and an intersection position between the imaging direction of the captured image acquisition means and the inner surface of the three-dimensional object is associated with the spot content information as the display position. Posting information acquisition means stored in the information storage means;
An augmented reality display device comprising: The augmented reality display device according to any one of claims 1 to 5,
From object information storage means for storing a plurality of object information relating to the object, comprising object information acquisition means for acquiring the object information corresponding to the object existing in the captured image;
The display means displays the target information superimposed on the captured image when the spot is not specified by the spot specifying means, the spot is specified by the spot specifying means, and the spot content information is displayed. Is acquired, the object information is hidden and the acquired point content information is displayed superimposed on the captured image. An augmented reality display system comprising the augmented reality display device according to any one of claims 1 to 6 and a server device capable of communicating with the augmented reality display device,
The server device
Point information storage means for storing a plurality of pieces of point content information regarding points;
Point information searching means for searching for point content information corresponding to the point specified by the point specifying means;
Point information transmitting means for transmitting the searched point content information to the augmented reality display device,
The said point content information acquisition means acquires the said point content information transmitted by the said point information transmission means. The augmented reality display system characterized by the above-mentioned. The augmented reality display system according to claim 7,
The augmented reality display device includes:
Post information receiving means for receiving post information for the point input by the user's input operation;
Posting information transmitting means for transmitting the posted information to the server device,
The server device
An augmented reality display system comprising: spot information updating means for generating the spot content information based on the posted information transmitted by the posted information transmitting means and storing it in the spot information storage means. An augmented reality display method for performing augmented reality display by a computer,
The computer
A captured image acquisition step of acquiring a captured image captured by the camera;
An imaging direction detection step of detecting a direction in which the camera is directed as an imaging direction;
A point identification step for identifying a predetermined designated point based on a user input operation;
A point content information acquisition step for acquiring point content information corresponding to a point within a predetermined range including the designated point, from point information storage means for storing point content information indicating content related to the point;
A display unit for displaying an image, viewed contains a plurality of points, and the captured image displayed by superimposing the icon corresponding to different remote region from the region in which the captured image is included by the obtained current position A display step to display, and
In the point content information acquisition step, when the icon is selected by a user input operation, the point content information corresponding to a point included in a remote area corresponding to the icon is acquired,
The display step displays the captured image displayed with the icon superimposed on the display unit, and the point content information is acquired from the designated point when the point content information is acquired by the point content information acquisition step. The augmented reality display method, wherein the point content information whose direction toward the point corresponding to is the same direction as the imaging direction is superimposed on the captured image.   An augmented reality display program for causing a computer to function as the augmented reality display device according to any one of claims 1 to 6.