JP2012118882A - Information processing system, and control method and program thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To change a way of showing computer graphics image data associated with an object according to whether a spot viewed from the object is within a useful azimuth range or in an azimuth range other than it.SOLUTION: A portable terminal 10 detects its own position and attitude. Based upon detection results thereof, a tag management server 20 analyzes which azimuth range the portable terminal 10 belongs to when viewed from the object in a real space, and changes the computer graphics image data associated with the object that the portable terminal 10 represents.

Description

  The present invention relates to a technology capable of synthesizing and presenting real-space video and computer graphics image data.

  Conventionally, when “time”, “distance”, and “type of air tag” are specified, only tags that meet the specified conditions are displayed, and the tags that have been displayed complicatedly are known (for example, Non-patent document 1).

  In addition, when the user presses and holds the place where the tag is gathered, the surrounding tag is sucked in, spirals out, the tag rotates in order from the back to the front, and the user taps when finding the tag of interest A technique that can refer to details of the tag is known (for example, see Non-Patent Document 2).

  By the way, AR (Augmented Reality) is a technique for adding information to the real space, and the information related to the real space has a very significant orientation. For example, in the case of “a foreigner visiting Japan for the first time looking for Sensoji Temple with Kaminarimon as a landmark and visiting,” the information that informs the location of Kaminarimon (for example, a tag that reads “Kaminarimon”) Although it is valuable information, the value of the information is halved because it is already known information when it comes back.

Sekai Camera Support Center, Internet <http://support.sekaicamera.com/en/service> Fun Hanashi Seika Enjoying the Sekai Camera Sekai, the Internet <http://plusd.itmedia.co.jp/mobile/articles/0910/01/news114.html>

  However, in any of the techniques disclosed in Non-Patent Documents 1 and 2, if the smartphone is pointed toward Kaminarimon, the tag will be uniformly displayed as “Kaminarimon”. As described above, the information added to the image in the real space is displayed uniformly without taking into consideration the value of the information depending on the direction.

  Accordingly, an object of the present invention is to change the way in which computer graphics image data relating to an object is displayed depending on whether the point is a point of a useful azimuth range as viewed from the object or a point of a azimuth range that is not so. is there.

An information processing system according to the present invention is an information processing system including an augmented reality presentation device capable of combining and presenting a video of real space and computer graphics image data, and includes the augmented reality presentation device. Based on the detection means for detecting the position and orientation, and the detection result of the detection means, the azimuth range to which the augmented reality presentation device belongs as viewed from the position of the object in the real space corresponding to the computer graphics image data A first analysis means for analyzing, and a control means for switching the computer graphics image data to be presented in the augmented reality presentation device in accordance with an analysis result by the first analysis means. .
The control method of the information processing system of the present invention is a control method of an information processing system having an augmented reality presentation device capable of synthesizing and presenting video in real space and computer graphics image data, A detection step for detecting the position and orientation of the augmented reality presentation device, and the augmented reality presentation device as seen from the position of the object in the real space corresponding to the computer graphics image data based on the detection result of the detection step Analysis step of analyzing which azimuth range belongs to, and a control step of switching the computer graphics image data to be presented in the augmented reality presentation device according to the analysis result of the analysis step .
The program of the present invention is a program for causing a computer to execute a control method of an information processing system having an augmented reality presentation device capable of synthesizing and presenting real-space video and computer graphics image data. And detecting the position and orientation of the augmented reality presentation device, and based on the detection result of the detecting step, the augmented reality as viewed from the position of the object in the real space corresponding to the computer graphics image data. An analysis step for analyzing which azimuth range the sensation presentation device belongs to, and a control step for switching the computer graphics image data to be presented in the augmented reality presentation device according to the analysis result of the analysis step are executed by the computer It is characterized by making it.

  According to the present embodiment, it is possible to change the way the computer graphics image data related to the object is displayed depending on whether the point is in a useful azimuth range or not in the azimuth range. .

1 is a diagram schematically showing a configuration of an AR tag display system according to a first embodiment of the present invention. It is a figure which shows the hardware constitutions of the tag management server in embodiment of this invention. It is a figure which shows the hardware constitutions of the portable terminal in embodiment of this invention. It is a flowchart which shows the flow of a process of the AR tag display system which concerns on embodiment of this invention. It is a figure which shows the story for demonstrating concretely the operation | movement of the AR tag display system which concerns on this embodiment. It is a figure for demonstrating operation | movement of the AR tag display system in case the user is walking on the route | root shown to 401, 403 of FIG. It is a figure which shows the example of a screen display of a portable terminal. It is a figure for demonstrating operation | movement of the AR tag display system in case the user is walking on the path | route shown by 402, 404 of FIG. It is a figure which shows the example of a screen display of a portable terminal. It is a flowchart which shows the process started when a user taps the tag currently displayed on the screen of a portable terminal. It is a figure for demonstrating concretely the process which updates the evaluation value of a tag management table based on the promenade set by the user. It is a figure which shows schematically the structure of the AR tag display system which concerns on the 3rd Embodiment of this invention. It is a flowchart which shows operation | movement of the AR display system which concerns on the 3rd Embodiment of this invention. It is a figure for demonstrating concretely step S1004-S1008 of FIG. It is a figure for demonstrating an example of the three-dimensional tag used in the 4th Embodiment of this invention. It is a figure which shows schematically the structure of the AR tag display system which concerns on the 4th Embodiment of this invention.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  First, a first embodiment of the present invention will be described. FIG. 1 is a diagram schematically showing the configuration of an AR tag display system according to the first embodiment of the present invention. As shown in FIG. 1, the AR tag display system according to the present embodiment includes a mobile terminal 10 with a camera function possessed by a user, a tag management server 20 for managing tags displayed in combination with real-world video, Consists of. In FIG. 1, for simplicity of explanation, one mobile terminal 10 and one tag management server 20 are shown, but there may actually be an arbitrary number of each.

  The mobile terminal 10 includes a position / orientation detection unit 101, an evaluation reception unit 102, and a presentation unit 103. The presentation unit 103 combines the computer graphics image data representing the tag with the video in the real space and displays it. That is, the mobile terminal 10 displays an image of a real space photographed by the camera on a liquid crystal display or the like, and produces an expanded sense of reality by combining and displaying computer graphics image data on the displayed real space image. To do. Hereinafter, the combined display of computer graphics image data that gives such an expanded sense of reality may be referred to as AR (Augmented Reality) display.

  In the embodiment described below, a mobile terminal type AR display device is cited as an example of an AR display device that is a device that performs AR display, but is not limited thereto. For example, a glasses-type HMD (Head Mounted Display) worn on the user's head can also be applied. The HMD produces an expanded sense of reality by displaying computer graphics image data at a position that matches the video in the real space. In addition, there are various types of HMD such as non-transmission type, video transmission type, and optical transmission type. The composite display (AR display) is to display the computer graphics image data in a position matching the video with respect to the real space video displayed on the lens of the HMD by any method. If the optical transmission type is used, computer graphics image data is synthesized and displayed at a position that matches the real space image seen through, and if it is a video transmission type, the real space image taken by the video camera is displayed. The computer graphics image data is synthesized and displayed at the matching position. In addition, even a head-up display type AR display device that is installed in the line of sight of the driver's seat of a moving vehicle and displays computer graphics image data in combination with a real space image that is optically transmitted and projected. Good.

  In addition, when computer graphics image data is displayed in alignment with an image in the real space, coordinates between the object in the image in the real space and the computer graphics image data are performed. As a coordinate matching method, in the present embodiment, the position / orientation detection unit 101 detects the position (latitude, longitude) and orientation (direction, elevation angle) of the mobile terminal 10, and the presentation unit 103 is based on the position and orientation. Then, it is estimated where in the real space the user holding the mobile terminal 10 is looking, and computer graphics image data is synthesized so as to match the estimated position.

  When the tag displayed on the screen of the mobile terminal 10 is tapped by the user, the evaluation receiving unit 102 receives an input of an evaluation value corresponding to the number of times. For example, when a tag is tapped three times by the user, an input of an evaluation value of +3 points for the tag is accepted. Note that the user's voice is not necessarily limited to taps, and a user's voice may be recorded as a digital signal by a microphone and an AD converter, and a voice analysis result using a spectrum analyzer or the like may be accepted as an evaluation value input. For example, when the user speaks of “Kaminarimon best!”, An input of an evaluation value of +100 points for the tag related to Kaminarimon is accepted by processing the result of speech analysis and natural language processing. In addition, the user's position may be measured by GPS or the like to determine whether or not the user has passed by the tag, and if it passes, input of an evaluation value of +10 points for the tag may be accepted. In addition, an operation of inputting an evaluation value according to an operation pattern that the user is conscious of the tag, such as an action of the user staring at the tag, can be considered.

  The tag management server 20 includes a display tag analysis unit 201, a tag display instruction unit 202, a tag management table 203, and a display mode management table 204. The display tag analysis unit 201 receives the position and orientation from the mobile terminal 10 and analyzes which object in the real space the user is looking at. Then, the display tag analysis unit 201 analyzes in which azimuth range the user is located when viewed from the object, and acquires the tag ID of the tag of the object and an evaluation point corresponding to the analyzed azimuth range from the tag management table 203. To do. The display tag analysis unit 201 outputs the tag ID, position, and evaluation score of the object tag to the tag display instruction unit 202.

  The tag management table 203 manages azimuth ranges and evaluation points for each tag ID and position. The position here is the position of the object in the real space, and is also the position where the tag corresponding to the object is displayed in a synthesized manner. The azimuth range means an azimuth range viewed with reference to the position of the object. When the tag management server 20 receives the evaluation value from the evaluation receiving unit 102 of the mobile terminal 10, the tag management server 20 updates the evaluation value of the corresponding record in the tag management table 203.

  The display mode management table 204 manages the display mode corresponding to the evaluation points. When the tag display instruction unit 202 acquires the tag ID, the position, and the evaluation score from the display tag analysis unit 201, the tag display instruction unit 202 acquires the display mode corresponding to the evaluation point from the display mode management table 204. The tag display instruction unit 202 transmits the computer graphics image data, position, and display mode corresponding to the tag ID to the mobile terminal 10. In the display mode management table 204 shown in FIG. 1, “small size display” is associated with evaluation points from 0 to 50, and “large size display” is associated with 51 or more evaluation points. It is attached. As another embodiment, the display itself may be controlled such that “not displayed” is given to evaluation points from 0 to 50, and “displayed” is given to 51 or more evaluation points. As other display modes, the one with a higher evaluation score is displayed in a more prominent color or bold font, or the one with a higher evaluation score is displayed in front in a superimposed manner, depending on the evaluation score. The display mode may be controlled by design or layout.

  When receiving the computer graphics image data, the position, and the display mode, the presentation unit 103 of the mobile terminal 10 synthesizes and displays the computer graphics image data on the real space video in the display mode. That is, if the display mode is “small size display”, the presentation unit 103 displays the computer graphics image data in a small size. If the display mode is “large size display”, the presentation unit 103 displays the computer graphics image data. Display data in large size. The mobile terminal 10 has a cache (high-speed storage device) function, and the computer graphics image data transmitted from the tag display instruction unit 202 to the mobile terminal 10 is cached to speed up the processing. Also good. In this case, the presentation unit 103 receives the computer graphics image data, the position, the display mode, and the like from the cache, and synthesizes and displays the computer graphics image data on the real space video in the display mode.

  FIG. 2A is a diagram illustrating a hardware configuration of the tag management server 20 in the present embodiment. In FIG. 2A, the CPU 12101 comprehensively controls each device and controller connected to the system bus. The ROM 12102 or the HDD 12105 stores an operating system (OS) that is a control program for the CPU 12101 and programs for realizing various functions described below. The RAM 12103 functions as a main memory, work area, temporary save area, and the like for the CPU 12101. The communication I / F controller 12104 executes communication control processing with an external device (for example, the mobile terminal 10) via a network. As an alternative to the HDD 12105, SDD (Solid State Drive) may be used.

  FIG. 2-2 is a diagram illustrating a hardware configuration of the mobile terminal 10 in the present embodiment. In FIG. 2B, the CPU 12201 comprehensively controls each device and controller connected to the system bus. The ROM 12202 stores an OS that is a control program for the CPU 12201 and programs for realizing various functions described below. The RAM 12203 functions as a main memory, work area, temporary save area, and the like for the CPU 12201. The communication I / F controller 12204 executes communication control processing with an external device (for example, the tag management server 20) via a network. A display controller 12205 controls image display on the display 12206.

  FIG. 3 is a flowchart showing a process flow of the AR tag display system according to the present embodiment. Hereinafter, the process of the AR tag display system according to the present embodiment will be described with reference to FIGS. 1 and 3.

  The position / orientation detection unit 101 of the mobile terminal 10 detects the position and orientation of the mobile terminal 10 (step S301). Next, the mobile terminal 10 transmits the position and orientation detected by the position and orientation detection unit 101 to the tag management server 20 (step S302).

  The tag management server 20 receives the position and orientation from the mobile terminal 10 (step S303). Next, the display tag analysis unit 201 of the tag management server 20 analyzes which object the user holding the mobile terminal 10 is looking at based on the position and orientation. Then, the display tag analysis unit 201 analyzes in which azimuth range the user is located when viewed from the object, acquires the tag ID of the tag of the object and the evaluation point corresponding to the azimuth range from the tag management table 203, The tag ID, position, and evaluation score are output to the tag display instruction unit 202 (step S304).

  Next, the tag display instruction unit 202 acquires a tag ID, a position, and an evaluation score from the display tag analysis unit 201, and acquires a display mode corresponding to the evaluation score from the display mode management table 204 (step S305). The tag display instruction unit 202 transmits the computer graphics image data, the position, and the display mode corresponding to the tag ID to the mobile terminal 10 (step S306).

  The mobile terminal 10 receives the computer graphics image data, the position, and the display mode from the tag management server 20 (step S307). Next, the presentation unit 103 of the mobile terminal 10 displays the computer graphics image data in a composite manner on the real space video in the display mode (step S308).

  Hereinafter, the operation of the AR tag display system according to the present embodiment will be specifically described by taking a specific story as an example. The story given here is, for example, “Users go to Sensoji Temple from Asakusa Station with Kaminarimon as a landmark, visit them at Sensoji Temple, then buy souvenirs at“ Raikanoshi Honpo ”on Nakamise Street and return to Asakusa Station. Is. FIG. 4 is a diagram illustrating a route of the user in the story. In FIG. 4, 401 and 402 indicate a part of the outward path of the story, and 403 and 404 indicate a part of the return path of the story. 401 in FIG. 4 is a route from Asakusa Station to point A near Kaminarimon. Reference numeral 402 in FIG. 4 denotes a route from the Kaminarimon to point B near the main thunder-raising main office. 403 in FIG. 4 is a route from Sensoji Temple to the point C near the lightning wake main office. 404 of FIG. 4 is a path | route from the lightning origin main office to the point D near Kaminarimon.

First, the operation of the AR tag display system when the user is walking along the route indicated by 401 in FIG. 4 while holding the mobile terminal 10 will be described with reference to FIG. As shown in FIG. 5A, when the user walks to the point A near Kaminarimon while holding the mobile terminal 10, the display tag analysis unit 201 receives it when the mobile terminal 10 is located at the point A. The user is viewing the Kaminarimon on the screen of the portable terminal 10 based on the position and orientation of the portable terminal 10 and the user sees 1 ° to 90 ° from the Kaminarimon position (x ₁ , y ₁ ). And the evaluation score of the corresponding record is acquired from the tag management table 203.

FIG. 5-1 (b) is a diagram excerpting relevant portions of the tag management table 203. The display tag analysis unit 201 analyzes which object the user views from which azimuth range on the screen of the mobile terminal 10 based on the position and orientation of the mobile terminal 10 received from the mobile terminal 10, and the analysis result The evaluation score of the record corresponding to is acquired from the tag management table 203. Here, the object that the user is looking at is the Kaminarimon gate at the position (x ₁ , y ₁ ), and the user is located in the azimuth range 1 ° to 90 ° as seen from the tag of the Kaminarimon gate. Accordingly, the display tag analysis unit 201 acquires the evaluation score +90 of the record 501 from the tag management table 203.

The display tag analysis unit 201 outputs the computer graphics image data and the position (x ₁ , y ₁ ) corresponding to the evaluation point +90 acquired from the record 501 and the tag ID “001” to the tag display instruction unit 202. The tag display instruction unit 202 transmits the computer graphics image data corresponding to the tag ID “001” and the display mode and position (x ₁ , y ₁ ) indicating “large size display” to the mobile terminal 10. As a result, the presentation unit 103 of the mobile terminal 10 displays the computer graphics image data of the Kaminarimon tag corresponding to the tag ID “001” in the “large size display” display mode (x ₁ , y ₁ ) Composite display. That is, when the user arrives at the point A near Kaminarimon from Asakusa Station while holding the mobile terminal 10, the computer graphics image data of the Kaminarimon tag is displayed in a large size on the screen of the mobile terminal 10.

  FIG. 5A is a diagram illustrating a screen display example of the mobile terminal 10 when the user arrives at the point A near Kaminarimon from Asakusa Station while holding the mobile terminal 10. As shown in FIG. 5A, on the screen of the mobile terminal 10, computer graphics image data 522 of the Kaminarimon tag is displayed in a large size in the vicinity of the Kaminarimon video 521 in the real space.

Next, the operation of the AR tag display system when the user is walking along the route indicated by 403 in FIG. 4 while holding the mobile terminal 10 will be described with reference to FIG. As shown in FIG. 5A, when the user walks from the main office to the point D near Kaminarimon while stroking the mobile terminal 10, the user moves the position (x ₁ , y ₁₎ on the screen of the mobile terminal 10. ) And the user is located in the azimuth range of 181 ° to 270 ° from the position (x ₁ , y ₁ ) of the Kaminarimon. The display tag analysis unit 201 analyzes these matters based on the position and orientation of the mobile terminal 10 received when the mobile terminal 10 is located at the point D, and determines the evaluation score of the corresponding record from the tag analysis table 203. get. Accordingly, the display tag analysis unit 201 acquires the evaluation score +20 of the record 502 from the tag management table 203.

The display tag analysis unit 201 outputs the computer graphics image data and the position (x ₁ , y ₁ ) corresponding to the evaluation score +20 and the tag ID “001” acquired from the record 502 to the tag display instruction unit 202. The tag display instruction unit 202 transmits the computer graphics image data corresponding to the tag ID “001” and the display mode and position (x ₁ , y ₁ ) indicating “small size display” to the mobile terminal 10. Thereby, the presentation unit 103 of the mobile terminal 10 displays the computer graphics image data of the Kaminarimon tag corresponding to the tag ID “001” in the display mode of “small size” in the position (x ₁ , y ₁ of the real space). ). That is, when the user arrives at a point D near thunder from Sensoji while holding the mobile terminal 10, the computer graphics image data of the Kaminarimon tag is displayed in a small size on the screen of the mobile terminal 10.

  FIG. 5B is a diagram illustrating a screen display example of the mobile terminal 10 when the user arrives at the point D near Kaminarimon from Sensoji while holding the mobile terminal 10. As shown in FIG. 5B, on the screen of the mobile terminal 10, computer graphics image data 524 of the Kaminarimon tag is displayed in a small size near the Kaminarimon video 523 in the real space.

Next, the operation of the AR tag display system when the user is walking on the route indicated by 402 in FIG. 4 while holding the mobile terminal 10 will be described with reference to FIG. As shown in FIG. 6A, when the user walks from Kaminarimon to the point B near the main office while holding the mobile terminal 10, the user moves to a position (x ₂ , y ₂ on the screen of the mobile terminal 10. ) And the user is located in the azimuth range of 271 ° to 360 ° when viewed from the position (x ₂ , y ₂ ) of the lightning main office. The display tag analysis unit 201 analyzes these matters based on the position and orientation of the mobile terminal 10 received when the mobile terminal 10 is located at the point B, and determines the evaluation score of the corresponding record from the tag management table 203. get.

FIG. 6B is an extracted diagram of relevant portions of the tag management table 203. As described above, the user is looking at the lightning occurrence main office at the position (x ₂ , y ₂ ) on the screen of the mobile terminal 10, and the user is viewing 271 from the lightning occurrence main office (position (x ₂ , y ₂ )). Located in the azimuth range of ° to 360 °. Therefore, the display tag analysis unit 201 acquires the evaluation score +25 of the record 601 from the tag management table 203.

The display tag analysis unit 201 outputs the computer graphics image data and the position (x ₂ , y ₂ ) corresponding to the evaluation score +25 acquired from the record 601 and the tag ID “002” to the tag display instruction unit 202. The tag display instruction unit 202 transmits the computer graphics image data corresponding to the tag ID “002” and the display mode and position (x ₂ , y ₂ ) indicating “small size display” to the mobile terminal 10. As a result, the presentation unit 103 of the mobile terminal 10 synthesizes and displays the computer graphics image data of the tag of the lightning honpo at the position (x ₂ , y ₂ ) in the real space in the “small size display” display mode. That is, when the user strikes the portable terminal 10 and reaches the point B near the main office from Kaminarimon, the computer graphics image data of the tag of the main terminal is displayed in a small size on the screen of the portable terminal 10. .

  FIG. 6A is a diagram illustrating a screen display example of the mobile terminal 10 when the user reaches the point B near the main office by thundering from Kaminarimon while holding the mobile terminal 10. As shown in FIG. 6A, on the screen of the mobile terminal 10, computer graphics image data 622 of the tag of the lightning honpo is displayed in a small size in the vicinity of the image 621 of the lightning honpo in real space. .

Next, the operation of the AR tag display system when the user is walking on the route indicated by 403 in FIG. 4 while holding the mobile terminal 10 will be described with reference to FIG. As shown in FIG. 6A, when the user walks from Senso-ji Temple to the point C near the main office while stroking the mobile terminal 10, the user is positioned on the screen of the mobile terminal 10 (x ₂ , y ₂ ) And the user is located in the azimuth range of 181 ° to 270 ° when viewed from the position (x ₂ , y ₂ ) of the lightning generation main office. The display tag analysis unit 201 analyzes these items based on the position and orientation of the mobile terminal 10 received when the mobile terminal 10 is located at the point C, and determines the evaluation score of the corresponding record from the tag analysis table 203. get. Therefore, the display tag analysis unit 201 acquires the evaluation score +80 of the record 602 from the tag management table 203.

The display tag analysis unit 201 outputs the computer graphics image data and the position (x ₂ , y ₂ ) corresponding to the evaluation score +80 and the tag ID “002” referenced from the record 602 to the tag display instruction unit 202. The tag display instruction unit 202 transmits the computer graphics image data corresponding to the tag ID “002”, the display mode and the position (x ₂ , y ₂ ) indicating “large size display” to the mobile terminal 10. Thereby, the presentation unit 103 of the mobile terminal 10 displays the computer graphics image data of the tag of the lightning honpo corresponding to the tag ID “002” in the real space position (x ₂ , synthesizing displayed on the y _2). That is, when the user arrives at the point C near the main office from Senso-ji while holding the mobile terminal 10, the computer graphics image data of the tag of the main office is displayed in a large size on the screen of the mobile terminal 10. .

  FIG. 6B is a diagram illustrating a screen display example of the mobile terminal 10 when the user reaches the point C near the main office by thundering from Sensoji while holding the mobile terminal 10. As shown in FIG. 6B, on the screen of the mobile terminal 10, computer graphics image data 624 of the tag of the lightning main office is displayed in a large size in the vicinity of the lightning main office image 623 in the real space. .

  FIG. 7 is a flowchart illustrating evaluation point setting processing that is started when the user taps a tag displayed on the screen of the mobile terminal 10. The user described here does not necessarily match the user in the above description. That is, the user in the above description is a tourist who walks around the city simply relying on the tag displayed on the mobile terminal 10, whereas the user who appears in the description of FIG. Thus, it is a person who recommends an orientation range that is considered useful for displaying a tag for each object.

  Hereinafter, the evaluation point setting process will be described with reference to FIG. The user taps a tag that is considered to be useful among tags displayed on the screen while walking while holding the mobile terminal 10. When the tag is tapped, the evaluation receiving unit 102 of the mobile terminal 10 counts the number of times tapped, the tag ID of the tapped tag, and the mobile detected by the position and orientation detection unit 101 at the time of tapping. The position and orientation of the terminal 10 are transmitted to the tag management server 20 (step S701).

  The tag management server 20 receives the frequency information, the tag ID, and the position and orientation of the mobile terminal 10 from the mobile terminal 10 (step S702). Based on the position and orientation of the mobile terminal 10, the tag management server 20 analyzes in which azimuth range the mobile terminal 10 is located when viewed from the object of the tag ID, and corresponds to the tag ID and the analyzed azimuth range The points for the number of times indicated by the number of times information are added to the evaluation points of the record to be performed (step S703).

  The user performs a tap operation on a tag that is thought to be useful to the user, and thereby adds the evaluation score of the azimuth range to which the mobile terminal 10 belongs as viewed from the object corresponding to the tag at that time. In the example of FIG. 4, since the route 401 is a route that passes through Kaminarimon for the first time, the tag of Kaminarimon is tapped at point A as being useful. As a result, as shown in the record 501 in FIG. 5B, the evaluation score is set high. Further, although the route 402 in FIG. 4 passes near the lightning generator main office, the user does not tap the lightning generator main office tag at the point B because the lightning generator main office is an object that stops after visiting the Sensoji Temple. Therefore, as shown in the record 601 in FIG. 6B, the evaluation score remains low.

  In addition, the route 402 in FIG. 4 passes near Senso-ji Temple, and passes near the lightning main office. At this time, the tag for lightning main office is useful, so tap the lightning main office tag at point C. To do. Thereby, as shown in the record 502 in FIG. 5B, the evaluation score is set high. Further, the route 404 in FIG. 4 is a route that passes through the thunderhead main office and heads for Kaminarimon. At this time, Kaminarimon is not tapped at point D because Kaminarimon is the second passage. Therefore, as shown in the record in FIG. 6B, the evaluation score remains low.

  By performing the above evaluation point setting process, the tag of the object is displayed prominently at a point considered to be a useful azimuth range when viewed from each object, and at a point of the azimuth range considered to be not, the object's tag is displayed. Tags can be displayed inconspicuously. That is, according to the present embodiment, it is possible to change the way the tag of the object is displayed depending on whether the point is in a useful azimuth range as viewed from the object or not.

  In the present embodiment, only the evaluation value points are added for the number of times tapped. However, the evaluation value points may be subtracted by the user performing a predetermined operation. This makes it possible to control the tag display in a negative direction for an object that is not desired to be guided to children, for example.

  Instead of tapping a useful tag, the user may operate the mobile terminal 10 to capture image data including the tag and transmit it to the tag management server 20. The tag management server 20 analyzes the image data transmitted from the mobile terminal 10 to determine the object on which the mobile terminal 10 is deceived and in which azimuth range the mobile terminal 10 is located when viewed from the object. Then, the evaluation value of the corresponding record in the tag management table 203 may be updated. In this case, the tag management server 20 prepares image data for image matching necessary for image data analysis in advance, and collates the image data received from the mobile terminal 10 with the image data for image matching. Thus, the above-described analysis is performed.

  Next, a second embodiment of the present invention will be described. In the present embodiment, a user sets a favorite walking path, and the evaluation score is updated according to the set walking path. In the setting of the walking path here, the user traces the map information displayed on the screen of the mobile terminal 10 with his / her finger so as to connect the objects (for example, Asakusa Station → Kaminarimon → Senso-ji Temple → Raikanashi Honpo → Senso-ji Temple), the path of the promenade and the direction of travel are set. The setting method of the promenade is not limited to the above method because it is only necessary to set the route and the traveling direction. For example, the route and direction of travel of the user who has the mobile terminal 10 may be detected by GPS (global positioning system), and the detection result may be set as a walking path. Further, the configuration of the AR tag display system according to the present embodiment is the same as the configuration shown in FIG. 1, and therefore, the reference numerals in FIG. 1 are used in the following description.

  The mobile terminal 10 transmits the set walking path to the tag management server 20. The tag management server 20 adds an evaluation value to the azimuth range estimated to be useful to the user for each object from the received walking path, and does not add an evaluation value to the azimuth range estimated otherwise.

  The azimuth range estimated to be useful to the user is an azimuth range to which the user's route belongs, and the azimuth range includes an azimuth opposite to the traveling direction of the user. That is, the azimuth range estimated to be useful by the user is an azimuth range in which the user moves toward the object. On the other hand, the azimuth range estimated not to be useful to the user is an azimuth range to which the user's route does not belong, or is an azimuth range to which the user's route belongs. Is an azimuth range that does not include the opposite azimuth. That is, the azimuth range estimated to be unusable for the user is an azimuth range in which the user does not move toward the object. However, when the same object passes after the second time, the display value of the tag is low at the second time and after, and a useful azimuth range for the second time and after is not estimated.

  FIG. 8 is a diagram for specifically explaining the process of updating the evaluation value of the tag management table 203 based on the promenade set by the user. According to the definition of the azimuth range described above, the point A when heading from Asakusa Station to Kaminarimon is located in the azimuth range of 1 ° to 90 ° when viewed from Kaminarimon, and this azimuth range is the reverse of the user's traveling direction. Therefore, an azimuth range of 1 ° to 90 ° viewed from Kaminarimon is a useful azimuth range for the user.

  Further, the point B when heading from Kaminarimon to Sensoji Temple is located in the azimuth range of 181 ° to 270 ° as seen from Kaminarimon, but this azimuth range does not include the opposite direction of the user's traveling direction. Therefore, the azimuth range of 181 ° to 270 ° as viewed from Kaminarimon is not a useful azimuth range for the user. Further, the point B is located in the azimuth range of 271 ° to 360 ° as viewed from the lightning occurrence main office, but this azimuth range does not include the opposite direction of the user's traveling direction. Therefore, the azimuth range of 271 ° to 360 ° as viewed from the lightning wake main office is not a useful azimuth range for the user.

  The point C when heading for the thunderstorm main office from Senso-ji Temple is located in the azimuth range of 181 ° to 270 ° when viewed from the thunderstorm main office, and this azimuth range includes a direction opposite to the direction of travel of the user. The azimuth range of 181 ° to 270 ° as seen is a azimuth range useful for the user.

  In addition, since the Kaminarimon is the second or later passage at the point D when heading to Kaminarimon from Kaminashikoshi Honpo, a useful azimuth range is not estimated. Therefore, in FIG. 8, the evaluation values are added for the azimuth ranges indicated by the oblique lines with respect to the objects of Kaminarimon and Thunderstorm Honpo.

  As described above, the user's favorite walking path is reflected in the evaluation value in the tag management table 203. Thereby, when another user walks while walking the portable terminal 10 on the promenade, the tag of Kaminarimon tends to be displayed at a point A when heading to Kaminarimon from Asakusa Station. In addition, the tag of the thunderstorm main office tends to be displayed greatly at the point C when heading to the thunderstorm main office from Sensoji Temple. As described above, according to the present embodiment, a useful azimuth range is automatically estimated from a favorite walking path set by a certain user, and the estimated result is displayed as a tag when another user walks on a similar walking path. It is reflected in.

  Next, a third embodiment of the present invention will be described. FIG. 9 is a diagram schematically showing a configuration of an AR tag display system according to the third embodiment of the present invention. As shown in FIG. 9, the AR tag display system according to the present embodiment includes a mobile terminal 110 with a camera function possessed by a user, a tag management server 120 for managing tags displayed in combination with images in real space, Consists of. In FIG. 9, for simplicity of explanation, one mobile terminal 110 and one tag management server 120 are shown, but there may actually be an arbitrary number of each. The hardware configurations of the mobile terminal 110 and the tag management server 120 are the same as the configurations shown in FIGS.

  The mobile terminal 110 according to the present embodiment includes a position / orientation detection unit 1101 and a presentation unit 1103. The position / orientation detection unit 1101 has the same configuration as the position / orientation detection unit 101 in FIG. 1, and the presentation unit 1103 has the same configuration as the presentation unit 103 in FIG.

The tag management server 120 in this embodiment includes a display tag analysis unit 1201, an advertisement fee calculation unit 1202, and a tag management table 1203. In the tag management table 1203, the position in the real space, the tag ID, the azimuth range, the advertisement provider ID, the advertisement fee (yen / day), and the advertisement period are managed in association with each other. For example, for the position (x ₃ , y ₃ ) in the real space, the tag ID “003” and the tag ID “004” are associated with each other. That is, it is defined that there are two tags of the tag ID “003” and the tag ID “004” at the position (x ₃ , y ₃ ).

The tag ID “003” is associated with an azimuth range of 1 ° to 180 °. This is the case where the mobile terminal 110 is located in the azimuth range of 1 ° to 180 ° when viewed from the position (x ₃ , y ₃ ), and the mobile terminal 110 is placed on the position (x ₃ , y ₃ ) side. This means that the computer graphics image data of the tag with the tag ID “003” is displayed on the portable terminal 110. The tag ID “004” is associated with an azimuth range of 181 ° to 360 °. This is when the mobile terminal 110 is located in the azimuth range 181 ° to 360 ° when viewed from the position (x ₃ , y ₃ ), and the mobile terminal 110 is placed on the position (x ₃ , y ₃ ) side. This means that the computer graphics image data of the tag with the tag ID “004” is displayed on the portable terminal 110.

For example, whether or not the mobile terminal 110 is located in the azimuth range of 1 ° to 180 ° when viewed from the position (x ₃ , y ₃ ) and the mobile terminal 110 is placed on the position (x ₃ , y ₃ ) side is determined. This is performed based on the position and orientation of the mobile terminal 110. That is, it is determined whether the position of the mobile terminal 110 belongs to the azimuth range of 1 ° to 180 ° when viewed from the position (x ₃ , y ₃ ), and the mobile terminal 110 is moved to the position (x ₃ , y ₃ ) side. Whether or not the user is deceived is identified by determining whether or not the camera of the mobile terminal 110 faces the position (x ₃ , y ₃ ) in the real space from the position and orientation of the mobile terminal 110.

When the display tag analysis unit 1201 receives the position and orientation of the mobile terminal 110 from the mobile terminal 110, the mobile terminal 110 is located in which azimuth range as viewed from which position, and the mobile terminal 110 is placed on the position side. And a record that matches the analysis result is extracted from the tag management table 1203. For example, in the display tag analysis unit 1201, the mobile terminal 110 is located in an azimuth range of 1 ° to 180 ° when viewed from the position (x ₃ , y ₃ ), and the mobile terminal 110 is located at the position (x ₃ , y ₃ ). When it is analyzed that the user is deceived, the topmost record in the tag management table 1203 is extracted. When the record is extracted, the display tag analysis unit 1201 transmits the computer graphics image data of the tag with the tag ID “003” in the record to the portable terminal 110. The portable terminal 110 synthesizes and displays the computer graphics image data of the tag with the tag ID “003” at the position (x ₃ , y ₃ ) in the real space. Thereby, the user can view the tag with the tag ID “003” on the screen of the mobile terminal 10.

  The tag ID “003” is associated with the advertisement provider ID “aaa”, and the tag ID “004” is associated with the advertisement provider ID “bbb”. The tag with the tag ID “003” is an advertisement by the advertisement provider with the advertisement provider ID “aaa”, and the tag with the tag ID “004” is an advertisement by the advertisement provider with the advertisement provider ID “bbb”. It means that. That is, the tag with the tag ID “003” and the tag with the tag ID “004” are combined and displayed in the real space on the screen of the mobile terminal 110, thereby serving as an advertisement for the user who refers to the tag.

  The tag ID “003” is associated with the advertisement fee (yen / day) “10,000”, and the tag ID “004” is associated with the advertisement fee (yen / day) “5,000”. It is associated. This means that the advertising fee per day for the tag with the tag ID “003” is 10,000 yen, and the advertising fee per day for the tag with the tag ID “004” is 5,000 yen.

  The tag ID “003” is associated with the advertisement period “H22, 8, 1”, and the tag ID “004” is associated with the advertisement period “H22, 8, 5”. Yes. This means that the tag ID “003” tag has been posted as an advertisement (synthesized display) from August 1, 2008 to the present, and the tag with the tag ID “004” It means that the period formed (synthesized and displayed) as an advertisement is from August 5, 2008 to the present.

  The advertisement fee calculation unit 1202 calculates an advertisement fee for each tag or each advertisement provider. When calculating the advertisement fee for each tag, the advertisement fee calculation unit 1202 multiplies the advertisement fee (yen / day) by the advertisement period for each record in the tag management table 1203, and tabulates using the tag ID as a key. Further, when calculating the advertisement fee for each advertisement provider, the advertisement fee calculation unit 1202 multiplies the advertisement fee (yen / day) by the advertisement period for each record in the tag management table 1203, and sets the advertisement provider ID as a key. And aggregate.

  FIG. 10 is a flowchart showing the operation of the AR display system according to the third embodiment of the present invention. First, the mobile terminal 110 detects the position and orientation of the mobile terminal 110 (step S1001). Next, the mobile terminal 110 transmits the detected position and orientation to the tag management server 120 (step S1002).

  The tag management server 120 receives the position and orientation from the mobile terminal 110 (step S1003). Next, based on the position and orientation of the mobile terminal 110, the tag management server 120 determines which position the mobile terminal 110 is deceived from, and in which azimuth range when the mobile terminal 110 is viewed from the position. Analysis is performed (step S1004). Next, the tag management server 120 extracts a record that matches the analysis result of step S1004 from the tag management table 1203 (step S1005). Next, the tag management server 120 transmits the computer graphics image data and the position of the tag with the tag ID of each extracted record to the portable terminal 110 (step S1006).

  The mobile terminal 110 receives the computer graphics image data and position of the tag from the tag management server 120 (step S1007). Next, the portable terminal 110 synthesizes and displays the computer graphics image data of the tag at the position in the real space (step S1008).

FIG. 11 is a diagram for specifically explaining steps S1004 to S1008 of FIG. In FIG. 11, when the mobile terminal 110 is located at the point G and is turned toward the position (x ₃ , y ₃ ), the display tag analysis unit 1201 of the tag management server 120 displays the position and orientation of the mobile terminal 110. from the portable terminal 110 is held over the position (x _3, y _3), and determines that the mobile terminal 110 is located in the azimuth range of 1 ° to 180 ° as viewed from the position (x _3, y _3). Accordingly, the display tag analysis unit 1201 extracts the uppermost record of the tag management table 1203 and sends the computer graphics image data and position (x ₃ , y ₃ ) of the tag with the tag ID “003” to the portable terminal 110. To send. Thereby, the portable terminal 110 synthesizes and displays the computer graphics image data of the tag with the tag ID “003” at the position (x ₃ , y ₃ ) in the real space.

On the other hand, when the mobile terminal 110 is located at the point H and is deceived toward the position (x ₃ , y ₃ ), the display tag analysis unit 1201 of the tag management server 120 uses the position and orientation of the mobile terminal 110 as follows: It is determined that the mobile terminal 110 is placed at the position (x ₃ , y ₃ ) and the mobile terminal 110 is located in the azimuth range of 181 ° to 360 ° when viewed from the position (x ₃ , y ₃ ). Therefore, the display tag analysis unit 1201 extracts the second-stage record from the top of the tag management table 1203, and stores the computer graphics image data and position (x ₃ , y ₃ ) of the tag with the tag ID “004” in the portable terminal. 110 is transmitted. Thereby, the mobile terminal 110 synthesizes and displays the computer graphics image data of the tag with the tag ID “004” at the position (x ₃ , y ₃ ) in the real space.

  As described above, in the present embodiment, even if the mobile terminal 110 is deceived with respect to the same position, a different tag is assigned to the position depending on from which azimuth range the portable terminal 110 is deceived when viewed from the position. Composite display. For example, a tag with a high advertising fee is presented to the mobile terminal 110 deceived from an azimuth range with a large amount of traffic when viewed from the tag position, and an advertising fee is presented to a portable terminal 110 deceived from an azimuth range with a small traffic. A low tag may be presented. Thereby, it is possible to provide an advertisement in consideration of the advertisement effect caused by the direction.

Next, a fourth embodiment of the present invention will be described. FIG. 12 is a diagram for explaining an example of a three-dimensional tag used in the fourth embodiment of the present invention. In the present embodiment, the three-dimensional tag shown in FIG. 12 is synthesized and displayed at the position (x ₅ , y ₅ ) in the real space. FIG. 12A shows the state of the three-dimensional tag in the initial state from the side, and FIG. 12B shows the state of the three-dimensional tag in the initial state from above. The three-dimensional tag of the example shown in FIG. 12 has a cylindrical shape, and the tag of the advertisement a, the tag of the advertisement b, the tag of the advertisement c, and the tag of the advertisement d are arranged on the side surface. That is, as shown in FIG. 12B, the tag of the advertisement a is arranged at 1 ° to 90 ° when viewed from the position (x ₅ , y ₅ ) of the three-dimensional tag, and from (x ₅ , y ₅ ) found in it 91 ° to 180 ° is arranged tag ad b, from the 181 ° to 270 ° as viewed from the (x _5, y ₅₎ is arranged tag ad c, (x _5, y ₅₎ A tag of the advertisement d is arranged at 271 ° to 360 ° as viewed.

The three-dimensional tag rotates 90 ° about the position (x ₅ , y ₅ ) for each tap operation of the user. In an initial state, the tag of the advertisement a faces the front of the screen of the mobile terminal 110, and the 3D tag rotates by 90 ° every time the user performs a tap operation. The tag of the advertisement b → the tag of the advertisement c → the advertisement The front of the screen is directed in the order of d → advertisement a.

  FIG. 13 is a diagram schematically showing the configuration of an AR tag display system according to the fourth embodiment of the present invention. As shown in FIG. 13, the AR tag display system according to the present embodiment includes a mobile terminal 210 with a camera function possessed by a user, a tag management server 220 for managing tags displayed in combination with real-space video, Consists of. In FIG. 13, for simplicity of explanation, one mobile terminal 210 and one tag management server 220 are shown, but there may actually be an arbitrary number of each. The hardware configurations of the mobile terminal 210 and the tag management server 220 are the same as the configurations shown in FIGS.

  The mobile terminal 210 in this embodiment includes a position / orientation detection unit 2101 and a presentation unit 2103. The position / orientation detection unit 2101 has the same configuration as the position / orientation detection unit 101 in FIG. 1, and the presentation unit 2103 has the same configuration as the presentation unit 103 in FIG.

The tag management server 220 in this embodiment includes a display tag analysis unit 2201, an advertisement fee calculation unit 2202, and a tag management table 2203. The tag management table 2203 manages the position in the real space, the tag ID, the azimuth range, the advertisement provider ID, the advertisement fee (yen / number of times), and the number of times of reference. Regarding the position (x ₅ , y ₅ ) in the real space, tag ID “007”, tag ID “008”, tag ID “009”, and tag ID “010” are associated with each other. That is, at the position (x ₅ , y ₅ ), four tags of a tag ID “007”, a tag ID “008”, a tag ID “009”, and a tag ID “010” are combined. It is defined to be displayed. That is, as shown in FIG. 12, it means that four tags (a tag for the advertisement a, a tag for the advertisement b, a tag for the advertisement c, and a tag for the advertisement d) are arranged at the position (x ₅ , y ₅ ). is doing.

  In the tag management table 2203, as in the third embodiment, the position, tag ID, azimuth range, advertisement provider ID, and advertisement fee (yen / number of times) are managed. The significance of managing these is the same as in the third embodiment. In the tag management table 2203, the number of references is managed. The number of times of reference is the number of times that the corresponding tag is displayed facing the front of the screen and can be viewed by the user. That is, each time the tags are displayed so as to face the front of the screen sequentially by the user's tap operation, the reference count of each tag is added.

  The advertisement fee calculation unit 2202 calculates an advertisement fee for each tag or each advertisement provider in accordance with a predetermined period or according to an operation of the administrator. When calculating the advertisement fee for each tag, the advertisement fee calculation unit 2202 multiplies the advertisement fee (yen / number of times) by the reference number for each record in the tag management table 2203, and tabulates using the tag ID as a key. Further, when calculating the advertisement fee for each advertisement provider, the advertisement fee calculation unit 1202 multiplies the advertisement fee (yen / count) and the reference count for each record of the tag management table 1203, and sets the advertisement provider ID as a key. And aggregate.

  In the embodiment described above, the tag management server analyzes the tag to be synthesized and displayed based on the position and orientation of the mobile terminal. However, the analysis process may be executed on the mobile terminal side. That is, the portable terminal is provided with the above-described display tag analysis unit, tag display instruction unit, tag management table, and display mode management table, and the tag that the portable terminal displays in combination based on its own position and orientation and its display size Can be judged. Furthermore, instead of determining a tag to be synthesized and displayed based only on the position and orientation of the mobile terminal, a predetermined threshold is set for the distance from each object to the mobile terminal, and tags for objects at a distance farther than the threshold from the mobile terminal are displayed. It may not be allowed to. Since it is meaningless to display the tag of an object that is too far away from the mobile terminal, unnecessary tags can be assigned by providing this threshold value.

  In the embodiment described above, the position / orientation detection unit handles the azimuth for the sake of convenience of explanation. However, not only the azimuth but also an elevation angle may be added. For example, information on the elevation angle can be acquired by a geomagnetic sensor, a gyro sensor, image analysis, or a combination thereof, and the display of the tag can be controlled by the position coordinates, the azimuth angle, and the elevation angle. As a result, it is possible to perform display control in consideration of the case where visibility is changed depending on the angle.

  10, 110, 210: mobile terminal, 20, 120, 220: tag management server, 101, 1101, 2101: position and orientation detection unit, 102: evaluation reception unit, 103, 1103, 2103: presentation unit, 201, 1201, 2011 : Display tag analysis unit, 202: tag display instruction unit, 203, 1203, 2203: tag management table, 204: display mode management table, 1202, 2202: advertisement fee calculation unit

Claims (10)

An information processing system having an augmented reality presentation device capable of synthesizing and presenting real-space video and computer graphics image data,
Detection means for detecting the position and orientation of the augmented reality presentation device;
First analysis means for analyzing which azimuth range the augmented reality presentation device belongs to as viewed from the position of an object in the real space corresponding to the computer graphics image data, based on a detection result by the detection means;
An information processing system comprising: control means for switching the computer graphics image data to be presented in the augmented reality presentation device according to an analysis result by the first analysis means.   The information processing system according to claim 1, wherein the control unit switches a display mode of the computer graphics image data according to an analysis result by the first analysis unit. First management means for managing evaluation information for each azimuth range viewed from the position of an object in real space corresponding to the computer graphics image data;
Accepting means for accepting a predetermined operation on the computer graphics image data presented in the augmented reality presentation device;
When the predetermined operation for the computer graphics image data is received by the reception unit, the real space corresponding to the computer graphics image data is based on the position and orientation of the augmented reality presentation device at the time of the reception. Second analysis means for analyzing which azimuth range the augmented reality presentation device belongs to as viewed from the position of the object;
Update means for updating the evaluation information of the corresponding azimuth range in the first management means according to the analysis result by the second analysis means,
The control means switches the computer graphics image data to be presented in the augmented reality presentation device according to the evaluation information of the azimuth range in the first management means corresponding to the analysis result by the first analysis means. The information processing system according to claim 1 or 2, characterized in that Second management means for managing evaluation information for each azimuth range viewed from the position of an object in the real space corresponding to the computer graphics image data;
Setting means for setting a route and a traveling direction;
Estimating means for estimating a useful azimuth range when viewed from the position of the object corresponding to the computer graphics image data in the real space, based on the path and the traveling direction set by the setting means;
Updating means for updating evaluation information of the corresponding azimuth range in the second management means according to the estimation result by the estimation means;
The control means switches the computer graphics image data to be presented in the augmented reality presentation device according to the evaluation information of the azimuth range in the second management means corresponding to the analysis result by the first analysis means. The information processing system according to claim 1 or 2, characterized in that   2. The information processing according to claim 1, wherein the control unit switches a type of the computer graphics image data to be presented in the augmented reality presentation device according to an analysis result by the first analysis unit. system.   The said control means switches the kind of the said computer graphics image data shown in the said augmented reality presentation apparatus by receiving the predetermined operation in the said augmented reality presentation apparatus. Information processing system.   6. The information processing system according to claim 5, further comprising third management means for managing a charge per unit period and a presentation period for each type of the computer graphics image data.   7. The information processing system according to claim 6, further comprising fourth management means for managing a charge per unit presentation count and a presentation count for each type of the computer graphics image data. A control method for an information processing system having an augmented reality presentation device capable of combining and presenting a video of real space and computer graphics image data,
A detection step of detecting the position and orientation of the augmented reality presentation device;
Based on the detection result of the detection step, an analysis step for analyzing which azimuth range the augmented reality presentation device belongs to when viewed from the position of the object in the real space corresponding to the computer graphics image data;
And a control step of switching the computer graphics image data to be presented by the augmented reality presentation device in accordance with an analysis result of the analysis step. A program for causing a computer to execute a control method of an information processing system having an augmented reality presentation device capable of synthesizing and presenting real-space video and computer graphics image data,
A detection step of detecting the position and orientation of the augmented reality presentation device;
Based on the detection result of the detection step, an analysis step for analyzing which azimuth range the augmented reality presentation device belongs to when viewed from the position of the object in the real space corresponding to the computer graphics image data;
A program for causing a computer to execute a control step of switching the computer graphics image data to be presented in the augmented reality presentation device according to the analysis result of the analysis step.

Images