JP2014006147A - Augmented reality system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an augmented reality system that facilitates perception of information on facilities staying for only a short period within a real space range where information can be displayed overlapping each other.SOLUTION: In an augmented reality system, at a geographic point 462 balloon diagrams 421 to 422 representing facility names are shown regarding building B and building D positioned within a field of vision in the form of being blown out of the positions of building B and building D in a landscape of real space (d). If a building A and a building C (c) are not positioned in the geographic point 462 and facility names of which were displayed at a geographic point 461, and display of the facility names was started within T seconds before the time of arrival at the geographic point 462, partial balloon diagrams 423 to 424 representing the facility names of these building A and building C are displayed as if the balloons which are blown out of positions outside the display screen are displayed with only parts within the display screen (d).

Description

  The present invention relates to augmented reality technology.

  Augmented Reality (AR) technology uses a mobile phone terminal equipped with a camera and a display device to display a video representing the scenery of the real space captured by the camera on the display device. A technique is known in which facility information is superimposed and displayed at a position corresponding to the facility on the video (for example, Patent Document 1).

  As augmented reality technology, a head-up display mounted on an automobile is used to superimpose and display information on surrounding facilities at a position corresponding to the facility on the scenery of the real space viewed by the user. Is also known (for example, Patent Document 2).

JP 2010-239568 A JP 2009-020089 A

According to the augmented reality technique described above, an information display range includes a real space range photographed by a camera and a real space range in which information can be superimposed and displayed using a head-up display. Information of only the facility will be displayed.
For this reason, during the period in which the position and orientation of the camera and the vehicle are changing, the facilities included in the information display range change, so that a facility that stays within the information display range only for a short time occurs. And about such a facility, the information will be displayed only for a short time, and it will interfere with a user's recognition of the information content.

  Therefore, the present invention superimposes and displays information in an augmented reality system in which information on a facility included in the landscape is superimposed and displayed at a position corresponding to the facility on a real space landscape or a video representing the landscape. It is an object of the present invention to facilitate the recognition of information about a facility that can only stay for a short time within a real space range.

  In order to achieve the above object, the present invention provides an augmented reality system including a head-up display that superimposes and displays an image on a landscape in real space, and an extended information representing the information of the related facility related to the facility. Based on the reality information, the augmented reality information represents the information represented by the augmented reality information related to the facility included in the field of view that is the range of the real space in which the head-up display can superimpose and display the image on the landscape. Visibility range facility information display means for displaying on the head-up display in a form associated with a display reference position, which is a position overlapping with the related facilities, and the visibility range facility information display means have received information within the past predetermined period. For facilities that are related to the augmented reality information that has been displayed and that are not included in the field of view, the extended information related to the facility The information represented by the real information is out of the field of view range to be displayed on the head-up display in a form representing the portion inside the display screen of the balloon graphic whose origin is the position outside the display screen of the head-up display. And facility information display means.

  In order to achieve the above object, the present invention is related to a facility with an augmented reality system including a camera that captures a landscape in real space and a display that displays a captured image that is captured by the camera. Based on the augmented reality information representing the information of the related facility, the information represented by the augmented reality information related to the facility included in the field of view that is the range of the real space captured by the camera is Visibility range facility information display superimposed on the captured image displayed on the display in a form associated with a display reference position that is a position on the captured image corresponding to the facility with which the real information is related. And a facility information display means within the visibility range that is related to the augmented reality information that has started displaying information within a predetermined period in the past, and is not included in the visibility range. For the facility, the information represented by the augmented reality information related to the facility is represented in a graphic representing a part in the display screen of the balloon graphic whose position is outside the display screen of the display, And an out-of-view-range-facility information display unit that superimposes and displays the captured image displayed on the display.

  Here, such an augmented reality system is a facility related to the augmented reality information in which the in-view range facility information display means starts displaying information within the past predetermined period in the out-of-view range facility information display means. And the information represented by the augmented reality information related to the facility, with the left-right direction where the facility not included in the view range has deviated from the view range as the departure direction, is displayed outside the display screen. You may comprise so that it may display in the form represented in the figure showing the part in the said display screen of the balloon figure which makes the position of the said deviation direction side of the said facility the balloon origin.

In addition, such an augmented reality system may be configured to display the information in a form that is arranged in a balloon graphic with the display reference position as a balloon source, in the facility information display means within the visual field range.
Note that such an augmented reality system may be a system mounted on a vehicle, and in this case, the field of view range may be a range in front of the vehicle.
According to these augmented reality systems, even if the facility stays only in the view range, which is the range of the real space where the information can be superimposed and displayed on the landscape or its video, the information is always kept for a predetermined period. It can be guaranteed that it is displayed. In addition, for information on facilities outside the visual field range, the information is displayed outside the visual field range by displaying it in a form representing a part of the balloon graphic with the position outside the display screen as the balloon source. It is possible to make it possible for the user to intuitively understand that the information is about the facility.

  As described above, according to the present invention, in an augmented reality system that superimposes and displays information on a facility included in the scenery on a position representing the scenery in the real space or on the video representing the scenery. It is possible to facilitate the recognition of information about a facility that only stays within a real space range in which information can be superimposed and displayed.

It is a block diagram which shows the structure of the vehicle-mounted system which concerns on embodiment of this invention. It is a figure which shows the memory content of the memory | storage device of the vehicle-mounted system which concerns on embodiment of this invention. It is a flowchart which shows the AR tag display process which concerns on embodiment of this invention. It is a figure which shows the process example of the AR tag display process which concerns on embodiment of this invention. It is a figure which shows the process example of the AR tag display process which concerns on embodiment of this invention. It is a figure which shows the process example of the AR tag display process which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described.
FIG. 1a shows the configuration of an in-vehicle system according to this embodiment.
The in-vehicle system is a system mounted in an automobile. As shown in the figure, the in-vehicle system 1 includes a head-up display (HUD) 101, an operation unit 102, a camera 103 that captures a landscape in front of the automobile, and a GPS receiver 104. A vehicle state sensor 105 for measuring the vehicle speed and angular velocity of the automobile, a control unit 106, a navigation unit 107, an AR display unit 108, a storage device 109, and an external interface 110 for connecting an external device such as a smartphone wirelessly or by wire. I have.

  However, the in-vehicle system 1 may be configured using a computer having a general configuration including a microprocessor, a memory, and other peripheral devices in hardware. In this case, the in-vehicle system described above is used. Each part of the system 1 or a part of the system 1 may be realized as a process realized by a microprocessor executing a program prepared in advance. In this case, such a program may be provided to the in-vehicle system 1 via a recording medium or an appropriate communication path.

  Here, as shown in FIG. 1B, the head-up display 101 of the in-vehicle system 1 includes a transparent screen 1011 disposed in front of the driver and a projection unit 1012 that projects an image on the transparent screen 1011. The image projected from the unit 1012 is reflected by the transparent screen 1011 and is configured to be visually recognized by the driver in a form superimposed on the scenery of the real space in front of the automobile seen through the transparent screen. As the transparent screen 1011, a windshield display incorporated in the front window shield can be used.

Next, the camera 103 of the in-vehicle system 1 is a camera 103 that is mounted in the front part of the automobile and photographs a landscape in front of the automobile.
Moreover, the smart phone 2 etc. can be connected to such a vehicle-mounted system 1 using the external interface 110. FIG.
Next, the contents stored in the storage device 109 of the in-vehicle system 1 will be described.
As shown in FIG. 2, the storage device 109 stores map data, an AR database, and a display candidate AR data table.
Here, the map data is data representing a road map and is stored in the storage device 109 in advance.
The AR database stores AR data representing facility information. Each AR data includes an AR data identifier that is an identifier of AR data, an information type that indicates the type of facility for which AR data represents information, AR Object coordinates representing the position in the real space of the facility for which the data represents information, AR tag data representing the contents of the AR tag represented on the video taken by the camera 103 for augmented reality, and the facility for which the AR data represents information Detailed information which is detailed information. Here, in the present embodiment, as an example, a case where a facility name of a facility whose information is represented by AR data is used as an AR tag will be described. However, as the AR tag, an icon / landmark that is a graphic indicating the type of facility may be used.

  Here, each AR data of the AR database may be stored in the storage device 109 in advance, but the AR display unit 108 is connected to the mobile communication network via the smartphone 2 connected to the in-vehicle system 1. May be acquired from the AR server 4 on the WAN 3 including and stored in the storage device 109.

  Next, the display candidate AR data table is a table for managing AR data set in display candidate AR data to be described later, and has entries (each row in the figure) provided for each display candidate AR data. . In each entry, the AR data identifier of the corresponding display candidate AR data, the display start time, the object position, and the expected departure direction can be registered.

  In such a configuration, the navigation unit 107 of the in-vehicle system 1 uses the map data recorded in the storage device 109 and uses the GPS receiver 104 and the vehicle state sensor 105 to detect the current position of the vehicle. Calculates the current forward direction of the vehicle, accepts destination settings from the user, searches for routes to the set destination, and displays the current position and current forward direction on a map around the current position A navigation image representing the mark and the route to the destination is generated.

On the other hand, the AR display unit 108 executes an AR tag display process and displays the AR tag using the head-up display 101.
FIG. 3 shows this AR tag display processing.
As shown in the figure, in this process, first, the driver can see through the transparent screen 1011 of the head-up display 101 based on the current position calculated by the navigation unit 107 as described above and the current forward direction. The real space range is calculated as the view range (step 302).

Then, AR data in which object coordinates included in the calculated view range are registered is extracted as view range AR data (step 304).
Next, among the display candidate AR data in which the entry in which the corresponding AR data identifier is registered is the AR data existing in the display candidate AR data table, the entry is not extracted as the AR data in the visual field range in step 304 and is displayed. Display candidate AR data whose display start time registered in the corresponding entry of the candidate AR data table is within the past T seconds (for example, 4 seconds) is extracted as out-of-view range AR data (step 306). Here, there is no AR data set as display candidate AR data in the initial state.

  Next, for each AR data within the viewing range, the content of the AR tag data of the AR data within the viewing range is calculated from the current position and the current forward direction calculated by the navigation unit 107 as described above. The position on the transparent screen 1011 that overlaps the object coordinates registered in the AR data as viewed from the driver is displayed on the head-up display 101 in the form of a balloon from the display reference position as a display reference position ( Step 308).

  Here, in step 308, as shown in FIG. 4a, a balloon figure 400 including a roughly rectangular main body part 401, a beak part 402 having a triangular shape protruding from the main body part 401 in a beak shape, and a balloon is used as a balloon. As shown in FIGS. 4c411-414 and 4d421, 422, the AR tag data is displayed on the transparent screen 1011 so that the tip of the beak 402 is at the display reference position in a state where the contents of the AR tag data are expressed inside the main body 401. The entire figure 400 is displayed. 4c and 4d are a view 500, that is, a landscape 500 viewed by the driver in a real space range that the driver can see through the transparent screen 1011 of the head-up display 101, and a view 500 superimposed on the landscape 500. The display content of the transparent screen 1011 to be displayed is shown.

  Returning to FIG. 3, next, for each of the AR data outside the visual range, the contents of the AR tag data of the AR data outside the visual range are set to the AR data outside the visual range in the display candidate AR data table outside the transparent screen 1011. The position on the departure prediction direction side registered in the entry is displayed on the head-up display 101 in the form of a balloon indicating the balloon source (step 310).

  Here, in step 310, as shown in FIG. 4b, a partial balloon figure 450 having a shape lacking the tip of the beak portion 402 of the balloon figure 400 shown in FIG. 4a is used as a balloon. That is, as the partial balloon graphic 450, a partial graphic 450 including a substantially rectangular main body 451 and a missing beak portion 452 having a shape lacking a triangular tip protruding from the main body 451 in a beak shape is used.

  The partial balloon graphic 450 is displayed on the transparent screen 1011 as if only the part on the transparent screen 1011 of the balloon having the position on the deviation prediction direction side outside the transparent screen 1011 as the tip of the beak portion 452 is displayed. Display above. Note that the content of the AR tag data is represented inside the main body 451 of the partial balloon graphic 450. That is, if the departure prediction direction is the left side, the partial balloon figure 450 created so that the tip of the missing beak part 452 is located on the left side of the main body part 451 as shown in FIG. 4d423 is left-aligned with the left side of the display screen. If the deviation prediction direction is on the right side, a partial balloon graphic 450 created so that the tip of the missing beak part 452 is located on the right side of the main body part 451 as shown in FIG. 4d424 is displayed on the display screen. Align the right side of and place it on the display screen.

Now, returning to FIG. 3, next, among the AR data that is the AR data within the visual field range, AR data that is not the display candidate AR data is set as the display candidate AR data, and the display candidate is displayed together with the display start time. Register in the AR data table (step 312).
That is, here, an entry corresponding to each AR data that is not the display candidate AR data among the AR data that is the AR data within the view range is created in the display candidate AR data table and corresponds to each entry. The AR data identifier of the AR data is registered, and the current time is registered as the display start time.

Then, the object position and the deviation prediction direction of the entry of the display candidate AR data table of the display candidate AR data that is the AR data within the visual field range are updated (step 314).
In the update, if the display reference position obtained for the display candidate AR data this time is on the left side of the object position registered in the entry of the display candidate AR data table before the update, the deviation prediction direction Is updated to the left, and if it is on the right, the deviation prediction direction is updated to the right. In the update, the object position is updated to the display reference position obtained for the display candidate AR data this time.

  However, the deviation prediction direction is updated to the left if the display reference position obtained for the display candidate AR data is a position in the left half of the transparent screen 1011 and to the right if the position is in the right half. You may make it carry out by updating. The left and right are the left and right when viewed from the driver.

Then, the settings of the AR data within the visual field range and the AR data outside the visual field range are reset (step 316), and the processing returns to step 302.
The AR tag display process has been described above.
Hereinafter, a processing example of such AR tag display processing will be described.
Now, as shown in FIG. 4e, it is assumed that the automobile travels straight from point 461 to point 462.
At point 461, the buildings A and B are set as AR data located within the field of view, that is, the real space that the driver can see through the transparent screen 1011 of the head-up display 101. Assume that C Building and D Building exist.
In this case, as shown in FIG. 4c, balloon figures 411-414 blown out from the positions of the A building, the B building, the C building, and the D building in the real space scenery are superimposed and displayed on the real space scenery. Facility names (contents of AR tag data) of A building, B building, C building, and D building are displayed in graphics 411-414. Here, it is assumed that the vehicle has advanced to the point 462 within T seconds from the start of the display of the facility names of the A building and the C building.

Further, at point 462, it is assumed that Building A and Building C deviate from the view range as the vehicle passes, and that only Building B and Building D remain as facilities in which AR data located within the view range is set. .
In this case, as shown in FIG. 4d, balloon figures 421-422 that are blown out from the positions of the buildings B and D in the real space scenery are superimposed on the scenery in the real space and are displayed in the balloon figures 421-422. , B building and D building facility names (contents of AR tag data) are displayed.
Also, since the facility names of Building A and Building C that deviated from the field of view started to be displayed within the past T seconds, the partial callout graphic 423-424 is as if the position outside the transparent screen 1011 Only the part on the transparent screen 1011 of the speech balloon at the tip is displayed, and the facility names of Building A and Building C are displayed in each partial speech balloon figure 423-424. Here, since the building A deviates from the view range in the left direction, whether only the portion on the transparent screen 1011 of the balloon with the position on the deviation prediction direction side outside the transparent screen 1011 is displayed is displayed. Thus, the partial balloon figure 423 of the A building is arranged, and the building c deviates from the view range in the right direction. Therefore, the transparent screen 1011 of the balloon having the position of the deviation prediction direction side outside the transparent screen 1011 as the tip of the beak. The partial callout graphic 424 of the C building is arranged as if only the upper part is displayed.

Next, as shown in FIG. 5 a, it is assumed that the automobile has made a right turn from point 501 to point 502.
At point 501, it is assumed that there are an A building and a C building as a facility in which AR data located within the view range is set.
In this case, as shown in FIG. 5b, balloon figures 511-512 that are blown out from the positions of the buildings A and C in the real space scenery are superimposed on the scenery in the real space, and are displayed in the balloon figures 511-512. , Facility names of A building and C building (contents of AR tag data) are displayed. Here, it is assumed that the vehicle has advanced to the point 502 within T seconds from the start of the display of the facility names of the A building and the C building.

In addition, at point 502, the A building and the C building deviate from the view range as the vehicle turns to the right, and the E building and the F building are generated as facilities where AR data located within the view range is set. It shall be.
In this case, as shown in FIG. 5c, the balloon figures 513-514 blown out from the positions of the E building and the F building in the real space scenery are superimposed and displayed on the real space scenery, and in each of the balloon figures 513-514. , E building and F building facility names (contents of AR tag data) are displayed.
In addition, since the facility names of Building A and Building C that deviated from the view range have started to be displayed within the past T seconds, the partial callout graphic 515-516 is positioned as if the position outside the transparent screen 1011 is the tip of the beak. Are displayed as if only the part on the transparent screen 1011 is displayed, and the facility names of the A building and the C building are displayed in each of the partial balloon graphics 515-516. Here, since both Building A and Building C deviate from the field of view in the left direction as the car changes direction, the transparent screen of the blowout with the position on the deviation prediction direction side outside the transparent screen 1011 as the tip of the beak The partial balloon figures 515-516 for the A building and the C building are arranged as if only the part on the 1011 is displayed.

The embodiment of the present invention has been described above.
By the way, in the above embodiment, the case where the AR tag data is directly superimposed and displayed on the real space landscape using the head-up display 101 has been described. However, in the present embodiment, the real space landscape photographed by the camera 103 is described. The present invention can be similarly applied to the case where AR tag data is displayed in a tatami mat on the video.

  That is, in this case, a display such as a liquid crystal display device is provided instead of the head-up display 101, and an image captured by the camera 103 is displayed on the display. In the AR tag display process described above, the field of view is the range of the real space photographed by the camera 103, and the display reference position is the object coordinate registered in the AR data in the image photographed by the camera 103. It is applied as a reflected position. In step 310, the content of the AR tag data of the AR data outside the visual field range is displayed on the deviation prediction direction side registered in the entry of the AR data outside the visual field range in the display candidate AR data table outside the display surface of the display. The position is displayed on the display in the form of a balloon indicating the position as the balloon source.

  By doing in this way, the display similar to the case where the head-up display 101 is used is shown in FIGS. 6a and 6b, which shows the display on the display when the automobile travels as in FIGS. 5b and 5c. Can be performed in a form in which the scenery in the real space is replaced with the camera-captured image 600. 6A is a display on the point 501 in FIG. 5A, and FIG. 6B is a display on the point 502 in FIG. 5A.

  Moreover, although the above embodiment showed taking the application to the vehicle-mounted system 1 as an example, this embodiment of the facility contained in the said scenery on the scenery of real space other than the vehicle-mounted system 1 or the image | video showing the said scenery is shown. The present invention can be similarly applied to any augmented reality system in which information is arranged and displayed at a position corresponding to the facility. That is, for example, a camera 103 and a display device are provided, and the display device displays a video representing a real space landscape captured by the camera 103, and information on surrounding facilities is displayed on the video. The present invention can be similarly applied to a portable device such as the smartphone 2 that superimposes and displays at a position corresponding to the facility.

In the above embodiment, when the AR tag user is selected on the displayed AR image, the AR display unit 108 displays the detailed information registered in the AR data of the selected AR tag. Processing may also be performed.
As described above, according to the present embodiment, even in a facility where the content of the AR tag data can only be displayed for a short time in the field of view, which is the range of the real space where the content of the AR tag data can be superimposed and displayed on the landscape, It is possible to ensure that the contents of the tag data are displayed. Further, the contents of the AR tag data of the facility outside the visual field range are displayed in a form represented by a partial balloon graphic 450 representing a part in the display screen of the balloon whose position is outside the display screen. The user can intuitively understand that the content of the AR tag data is information about a facility outside the view range. Further, in this case, by setting the position outside the display screen as the position on the departure direction side, the user can intuitively grasp whether the information is the facility information on the left or right side.

  DESCRIPTION OF SYMBOLS 1 ... In-vehicle system, 2 ... Smartphone, 3 ... WAN, 4 ... AR server, 101 ... Head up display, 102 ... Operation part, 103 ... Camera, 104 ... GPS receiver, 105 ... Vehicle state sensor, 106 ... Control part, DESCRIPTION OF SYMBOLS 107 ... Navigation part, 108 ... AR display part, 109 ... Memory | storage device, 110 ... External interface, 1011 ... Transparent screen, 1012 ... Projection unit.

Claims (5)

An augmented reality system equipped with a head-up display that superimposes and displays images on a real space landscape,
Based on the augmented reality information representing the information of the related facility related to the facility, the head-up display is included in the facility included in the field of view that is the range of the real space in which the video can be superimposed and displayed on the landscape. Visibility range facility information display means for displaying information represented by the related augmented reality information on the head-up display in a form associated with a display reference position that is a position overlapping the facility with which the augmented reality information is related; ,
The facility information display means within the view range is related to the augmented reality information that has started displaying information within a predetermined period in the past, and the facility not included in the view range is related to the facility. A field of view that is displayed on the head-up display in a form in which the information represented by the augmented reality information is represented in a figure representing a part in the display screen of a balloon figure whose origin is a position outside the display screen of the head-up display An augmented reality system comprising an out-of-range facility information display means. An augmented reality system including a camera that captures a landscape in real space and a display that displays a captured image that is an image captured by the camera,
Based on the augmented reality information representing the related facility information associated with the facility, the augmented reality information associated with the facility included in the field of view that is the range of the real space photographed by the camera is represented. A field of view in which information is superimposed on the photographed image displayed on the display in a form associated with a display reference position that is a position on the photographed image corresponding to the facility to which the augmented reality information is related In-range facility information display means,
The facility information display means within the view range is related to the augmented reality information that has started displaying information within a predetermined period in the past, and the facility not included in the view range is related to the facility. The information displayed by the augmented reality information is displayed on the display in a form representing a part of the balloon graphic with a position outside the display screen of the display as a balloon source. An augmented reality system comprising: a facility information display means for viewing out of view range superimposed on an image. The augmented reality system according to claim 1 or 2,
The out-of-view range facility information display means is a facility related to the augmented reality information that the in-view range facility information display means has started displaying information within the past predetermined period, and is included in the view range. Information indicating the augmented reality information related to the facility is displayed outside the display screen on the departure direction side of the facility, with the left-right direction from which the facility has deviated from the field of view as the departure direction. An augmented reality system, characterized in that the original balloon graphic is displayed in a form represented in a graphic representing a portion of the display screen. The augmented reality system according to claim 1, 2, or 3,
The field-of-view-range facility information display means displays the information in a form arranged in a balloon graphic with the display reference position as a balloon source. The augmented reality system according to claim 1, 2, or 3,
The augmented reality system is a system installed in an automobile,
The augmented reality system, wherein the field-of-view range is a range in front of the automobile.