JP7125963B2 - Information processing program, information processing apparatus, and information processing method - Google Patents

Description

The present invention relates to AR technology that takes into consideration the situation of light sources in a real environment.

In recent years, software employing AR (Augmented Reality) technology is becoming widespread. It is possible to display a virtual object in a photographed image photographed by a user terminal owned by a user, thereby realizing a display as if the virtual object appeared in the real space.

For example, Patent Literature 1 discloses that users can easily handle information that they want to share with other users and information that they do not want to share with other users in the AR space for a virtual object to be displayed in AR that is stored in association with a specific position in the real space. A technique capable of doing this is disclosed.

Japanese Patent Publication No. 2012-168646

By the way, when a virtual object is displayed on an image captured by a user terminal using AR technology, the display may look unnatural if the display is performed without considering the light source conditions in the real environment. For example, if the display is performed without reflecting the influence of the light source such as the sun, it can be said that there is a sense of incongruity due to the lack of consistency with the surrounding real objects. It is also conceivable to estimate the position of a light source such as the sun in an acquired photographed image and execute shading processing based on the information of the estimated light source. If the display is made without considering the relationship with the real object that exists in the real space, for example, as a result of drawing the shadow of the virtual object superimposed on the shadow of the real object, a darker shadow will appear on top of the shadow. is drawn, which is not possible in reality.

SUMMARY OF THE INVENTION The present invention has been made in view of the above problems. An object is to provide an apparatus and an information processing method.

From a non-limiting point of view, an information processing program according to an embodiment of the present invention stores a virtual image captured by a user terminal owned by a user in association with a predetermined position in the real space. An information processing program for causing a computer to realize a process of superimposing and displaying an object, wherein the computer has a position information acquisition function for acquiring position information of the user terminal, a time acquisition function for acquiring current time information, A weather information acquisition function that acquires weather information at a location based on the location information of the user terminal; and a natural light source condition at the location of the user terminal based on the acquired location information, time information, and weather information. A function for estimating a natural light source to be estimated, and acquisition of reality-corresponding object information for acquiring information of an object in a virtual space (reality-corresponding object) pre-associated with a real object existing around the user terminal based on the position information. a photographed image acquisition function for acquiring the photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal; and a light source for the reality corresponding object in the estimated natural light source situation. a first light source effect calculation function for calculating the effect of a virtual object information acquisition function for acquiring information on the virtual object that is arranged around the position of the user terminal and may be located within the shooting range of the camera device; a second light source effect calculation function for calculating the effect of the light source on the virtual object in the estimated natural light source situation while considering the effect of the natural light source on the reality corresponding object; and grounding of the virtual object on the captured image. A superimposed display function of specifying a position and superimposing and displaying the virtual object and the calculated light source influence on the specified ground position on the captured image is realized.

From a non-limiting point of view, an information processing apparatus according to an embodiment of the present invention stores a virtual image captured by a user terminal owned by a user in association with a predetermined position in the real space. An information processing device for performing a process of superimposing and displaying an object, comprising: a position information acquisition unit for acquiring position information of the user terminal; a time acquisition unit for acquiring current time information; and based on the position information of the user terminal a weather information acquisition unit that acquires weather information for the location of the user terminal, and a natural light source estimation unit that estimates the state of natural light sources at the location of the user terminal based on the location information, the time information, and the weather information acquired. a reality-corresponding object information acquisition unit that acquires information on an object in a virtual space (reality-corresponding object) that is associated in advance with a real object existing around the user terminal based on the position information; a photographed image obtaining unit for obtaining the photographed image and photographing direction information obtained by photographing the real space with the provided camera device; a light source effect calculation unit; a virtual object information acquisition unit that acquires information on the virtual object that is arranged around the position of the user terminal and may be located within the photographing range of the camera device; a second light source effect calculation unit that calculates the effect of the light source on the virtual object in the estimated natural light source situation while taking into account the effect of the light source; and a superimposed display unit that superimposes and displays the object and the calculated light source influence on the specified ground position on the captured image.

From a non-limiting point of view, an information processing method according to an embodiment of the present invention provides a virtual image captured by a user terminal owned by a user and stored in association with a predetermined position in the real space. An information processing method for causing a computer to execute a process of displaying an object in a superimposed manner, comprising a position information acquisition procedure for acquiring position information of the user terminal, a time acquisition procedure for acquiring current time information, and a position of the user terminal. a weather information acquisition unit that acquires weather information for the location based on the information; and a natural light source that estimates the state of the natural light source at the location of the user terminal based on the acquired location information, the time information, and the weather information. an estimation step; a reality-corresponding object information obtaining procedure for obtaining information of an object in a virtual space (reality-corresponding object) previously associated with a real object existing around the user terminal based on the position information; A photographed image acquisition procedure for acquiring the photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal, and calculating the influence of the light source on the reality-corresponding object in the estimated natural light source situation. a first light source effect calculation procedure for calculating a first light source effect; a virtual object information acquisition procedure for acquiring information on the virtual object that is placed around the position of the user terminal and may be located within the shooting range of the camera device; A second light source effect calculation procedure for calculating the effect of the light source on the virtual object in the estimated natural light source situation while taking into consideration the effect of the natural light source on the object; and a superimposed display procedure for superimposing the virtual object and the calculated light source influence on the specified ground position on the captured image.

From a non-limiting point of view, an information processing program according to an embodiment of the present invention stores a real object appearing in a photographed image photographed by a user terminal owned by a user and a real object in association with the real object in advance. An information processing program for causing a computer to implement a process of identifying an object appearing in a photographed image by comparing it with an object (reality corresponding object) in a virtual space, wherein position information of the user terminal is transmitted to the computer. A location information acquisition function to acquire, a time acquisition function to acquire current time information, a weather information acquisition function to acquire weather information at the location based on the location information of the user terminal, the acquired location information, and the time information and a natural light source estimation function for estimating the state of natural light sources at the location of the user terminal based on the weather information; A reality-corresponding object information acquisition function that acquires information on an object in space (a reality-corresponding object), and acquires the photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal. A photographed image acquisition function, and a condition in which a virtual camera is installed to photograph at least one pattern in a virtual space that is estimated to be approximately the same position and approximately the same photographing direction as the photographed image, and the reality of the estimated natural light source situation. A virtual captured image generation function for generating a virtual captured image by executing rendering in consideration of the effect of a light source on a corresponding object; It is characterized by realizing a photographed object specifying function for specifying an object appearing in a photographed image based on the degree of matching.

From a non-limiting point of view, an information processing apparatus according to an embodiment of the present invention stores a real object appearing in a photographed image taken by a user terminal owned by a user and a real object in association with the real object in advance. an information processing device for performing a process of identifying an object appearing in a photographed image by matching it with an object (reality corresponding object) in a virtual space, the information processing device comprising: a position information obtaining unit for obtaining position information of the user terminal; , a time acquisition unit that acquires current time information, a weather information acquisition unit that acquires weather information at the location based on the location information of the user terminal, and based on the acquired location information, the time information, and the weather information a natural light source estimating unit for estimating a natural light source situation at the position of the user terminal; and an object in a virtual space that is pre-associated with a real object existing around the user terminal based on the position information a captured image acquisition unit that acquires the captured image and the captured direction information obtained by capturing the real space with a camera device included in the user terminal; Under the condition that a virtual camera is installed and photographed in virtual space for at least one pattern that is estimated to be approximately the same position and approximately the same photographing direction as the photographed image, the effect of the light source on the realistic object in the estimated natural light source situation is calculated. a virtual captured image generation unit that generates a virtual captured image by performing rendering considering the above; and a photographed object identifying unit that identifies an object captured in the photographed object.

From a non-limiting point of view, an information processing method according to an embodiment of the present invention provides a real object appearing in a photographed image photographed by a user terminal owned by a user and storing the real object in association with the real object in advance. An information processing method for causing a computer to execute a process of identifying an object appearing in a photographed image by comparing it with an object (reality-corresponding object) in a virtual space, wherein the positional information for acquiring the positional information of the user terminal an acquisition procedure, a time acquisition procedure for acquiring current time information, a weather information acquisition procedure for acquiring weather information at a location based on the location information of the user terminal, the acquired location information, the time information, and the weather information a natural light source estimating procedure for estimating the natural light source situation at the position of the user terminal based on the position information; A reality-corresponding object information obtaining procedure for obtaining information on a (reality-corresponding object), and a photographed image obtaining procedure for obtaining the photographed image and photographing direction information obtained by photographing the physical space with a camera device provided in the user terminal. and a light source for a realistic object in the estimated natural light source situation under the condition that a virtual camera is installed and photographed in the virtual space for at least one pattern that is estimated to be at approximately the same position and approximately the same photographing direction as the photographed image. A virtual captured image generation procedure for generating a virtual captured image by executing rendering considering the influence of the virtual captured image; and a photographed object identification procedure for identifying an object appearing in the photographed image.

Embodiments of the present application address one or more deficiencies.

1 is a block diagram showing an example of the configuration of an information processing device according to at least one embodiment of the present invention; FIG. 1 is a block diagram showing an example of a system configuration to which an information processing device according to at least one embodiment of the invention is applied; FIG. 4 is a flow chart illustrating an example of information processing in at least one embodiment of the invention; FIG. 10 is an explanatory diagram showing an example of AR display when at least one embodiment of the present invention is not applied; FIG. 11 is an explanatory diagram enlarging a part of an example of AR display when at least one embodiment of the present invention is not applied; FIG. 4 is a conceptual diagram showing an example concept of natural light source estimation processing in at least one embodiment of the present invention. FIG. 10 is an explanatory diagram showing an example of AR display when at least one embodiment of the present invention is applied; FIG. 10 is an explanatory diagram enlarging a part of an AR display example when at least one embodiment of the present invention is applied; 1 is a block diagram showing an example of the configuration of an information processing device according to at least one embodiment of the present invention; FIG. 4 is a flow chart illustrating an example of information processing in at least one embodiment of the invention; FIG. 2 is a conceptual diagram illustrating an example concept of tracking processing in at least one embodiment of the present invention; FIG. 4 is a conceptual diagram showing an example concept of camera orientation estimation processing in at least one embodiment of the present invention.

Hereinafter, examples of embodiments of the present invention will be described with reference to the drawings. It should be noted that various components in the examples of the embodiments described below can be appropriately combined within a range that does not cause contradiction. Further, the order of various processes constituting various flows described below is random as long as there is no contradiction in the processing contents. Also, the contents of operations and processes that are not related to the features of each embodiment may be omitted. Furthermore, the order of various processes constituting various flows described below is random as long as there is no contradiction in the processing contents.

[First Embodiment]
FIG. 1 is a block diagram showing an example configuration of an information processing apparatus according to at least one embodiment of the present invention. As shown in FIG. 1, the information processing apparatus 10 according to the first embodiment of the present invention includes a position information acquisition unit 11, a time information acquisition unit 12, a weather information acquisition unit 13, a natural light source estimation unit 14, A reality-corresponding object information acquisition unit 15, a captured image acquisition unit 16, a first light source effect calculation unit 17, a virtual object information acquisition unit 18, a second light source effect calculation unit 19, a superimposed display unit 20, and a storage unit. 21. Details of each configuration in FIG. 1 will be described later.

FIG. 2 is a block diagram showing an example of a system configuration to which an information processing apparatus according to at least one embodiment of the invention is applied. A user terminal owned by a user may function as the information processing apparatus 10 according to the first embodiment of the present invention. It may function as the processing device 10 . As shown in FIG. 2, the system includes a server device 30 and user terminals 401 to 40n (n is an arbitrary integer) possessed by each user. ) and

The server device 30 and the plurality of user terminals 401 to 40n are each connected to a communication network 50 such as the Internet. Although not shown, the plurality of user terminals 401 to 40n are connected to the communication network 50 by performing data communication with a base station managed by a telecommunications carrier through a wireless communication line, for example.

The system includes a server device 30 and a plurality of user terminals 401 to 40n, thereby realizing various functions for executing various processes according to user operations.

The server device 30 is managed by an administrator of the information processing device 10 and has various functions for providing information on various processes to the user terminals 401 to 40n. In this example, the server device 30 is configured by an information processing device such as a WWW server, and includes a storage medium for storing various information. Note that the server device 30 has a general configuration for performing various processes as a computer, such as a control unit and a communication unit, but description thereof will be omitted here. Moreover, in the information processing apparatus 10, it is preferable that the server apparatus 30 manages various kinds of information from the viewpoint of reducing the processing load on each of the plurality of user terminals 401 to 40n. However, the storage unit for storing various kinds of information only needs to have a storage area that can be accessed by the server device 30. For example, a dedicated storage area may be provided outside the server device 30.

Each of the plurality of user terminals 401 to 40n is managed by a user, and includes communication terminals such as mobile phone terminals and PDAs (Personal Digital Assistants). It is necessary to have a configuration in which the device is mounted and a display for displaying captured image data and the like is provided. Other examples of the configuration of the user terminal include so-called wearable devices such as smartwatches, and combinations of wearable devices and communication terminals. Moreover, the camera device may use a three-dimensional camera device in addition to the optical camera device. Also, the user terminals 401 to 40n are provided with input means such as a mouse, touch panel, and touch pen for inputting drawing information.

Further, each of the user terminals 401 to 40n is connected to the communication network 50 and has hardware and software for executing various processes by communicating with the server device 30 . Note that each of the plurality of user terminals 401 to 40n may be configured to be able to directly communicate with each other without going through the server device 30. FIG.

In the process of information processing, the server apparatus 30 appropriately transmits the processing process to the user terminals 401 to 40n, and the user terminals 401 to 40n display screen contents based on the content of the received processing process. shall be displayed on the display screen of the device. The screen contents may be generated on the server device 30 side or on the user terminals 401 to 40n side.

In the following, a case where the server device 30 shown in FIG. 2 functions as the information processing device 10 shown in FIG. 1 will be described as an example.

The location information acquisition unit 11 has a function of acquiring location information of the user terminal. Here, the location information may be any information that indicates the location of the user terminal. Examples of such information include latitude and longitude information obtained by GPS, position information obtained by position estimation based on reception strengths of multiple types of radio signals, and the like. In addition, DR (Dead Reckoning) is adopted to estimate the position information of the user terminal by estimating the direction and distance of movement of the user terminal from the position where the position information was last accurately obtained. It may be location information obtained from

The time information acquisition unit 12 has a function of acquiring current time information including date. Any method may be used as long as the current time including the date can be obtained.

The weather information acquisition unit 13 has a function of acquiring weather information for a location based on the location information of the user terminal. The weather information may be information indicating the current actual weather information, or may be weather forecast information obtained in advance. When acquiring weather forecast information as weather information, current time information is also used.

The natural light source estimation unit 14 has a function of estimating the condition of natural light sources at the position of the user terminal based on the acquired position information, time information and weather information. Here, a natural light source means a light source in the natural world. As for natural light sources, for example, the sun is the most influential direct light source, and the moon is also a direct light source. The natural light source also includes an indirect light source that is a light source by reflecting a direct light source including the sun. Also, blue skies, clouds, fog, etc. are natural light sources as ambient light. The information of the natural light source to be acquired must be configured to acquire the light source defined as the light source used for calculations during rendering. A simplified example of obtaining natural light source information is a hemispherical light source that simply obtains information about the direction of the sun, which is estimated from the date and time and the position of the user terminal, and adds diffuse ambient colors to all objects in the scene. It is conceivable to acquire the color of the sky estimated from the weather information in order to use it as a light source.

The reality-corresponding object information acquisition unit 15 has a function of acquiring information of an object in the virtual space (hereinafter referred to as a reality-corresponding object) that is associated in advance with a real object existing around the user terminal based on the position information. . The reality-corresponding objects are, for example, real outdoor objects such as buildings, street lights, traffic lights, etc., and indoor real objects such as doors, window glass, lighting, etc., which are arranged in the virtual space in the same way as they are in the real world. A three-dimensional model is generated and stored in advance with the arrangement of . The reality-corresponding object information acquisition unit 15 acquires the reality-corresponding objects around the position information of the user terminal among the reality-corresponding objects stored in advance.

The captured image acquisition unit 16 has a function of acquiring a captured image and capturing direction information obtained by capturing an image of the physical space with a camera device provided in the user terminal. The captured image may be a moving image or a still image. The shooting direction information is acquired by an orientation sensor provided in the user terminal.

The first light source effect calculation unit 17 has a function of calculating the effect of the light source on the reality corresponding object in the estimated natural light source situation. Specifically, by executing rendering processing that includes shading that considers the effects of natural light sources on the acquired reality-compatible objects, the effects of light sources such as shadows, reflections, and changes in brightness on reality-compatible objects are calculated. Perform the processing required by

The virtual object information acquisition unit 18 has a function of acquiring information on virtual objects that are arranged around the position of the user terminal and that may be located within the photographing range of the camera device. The placement of the virtual object may be stored in association with the position coordinates, or may be stored in association with the predetermined position of the object. In addition, it is possible to appropriately set which range of virtual object information is to be acquired. may be a method of acquiring information on all virtual objects arranged at a predetermined distance from the .

The second light source effect calculation unit 19 has a function of calculating the effect of the light source on the virtual object in the estimated natural light source situation while considering the effect of the natural light source on the reality corresponding object. The second light source effect calculation unit 19 calculates the effect of the natural light source on the virtual object in a state in which the virtual object is arranged in the reality corresponding object. For example, when a virtual object and a building, which is a reality-compatible object, are close to each other, depending on the direction of the sun, which is the light source, the shadow of the building may be cast on the virtual object, and the sunlight may not hit part of the virtual object directly. There may be a situation where there is no In such a case, the part of the virtual object that is shaded by the building will not cast a shadow on other objects. In this way, by calculating the effect of the light source placed in the reality corresponding object, it is possible to express the realistic effect of the light source when it is assumed that the virtual object actually exists in the real object. becomes possible. As an example of a specific calculation method, the calculation result of the first light source effect calculation unit 17 is subtracted from the result of rendering processing that takes into account the effect of the natural light source in a state in which the virtual object is placed in the reality corresponding object. By doing so, it is conceivable to extract only the influence of the natural light source on the virtual object.

The superimposed display unit 20 has a function of specifying the ground position of the virtual object on the captured image and superimposing the virtual object and the calculated light source effect on the specified ground position on the captured image.

The storage unit 21 has a function of storing data necessary for various processes performed in the information processing apparatus 10 and data obtained as a result of the processes.

Next, the flow of information processing performed in the information processing apparatus 10 of this example will be described. FIG. 3 is a flow chart illustrating an example of information processing in at least one embodiment of the invention. As shown in FIG. 3, the information processing is started by sequentially acquiring position information of the user terminal, current time information, and weather information in the information processing apparatus 10 (step S101). Next, the information processing apparatus 10 estimates the condition of the natural light source at the position of the user terminal based on the acquired position information, time information and weather information (step S102). Next, the information processing apparatus 10 acquires information on a reality corresponding object corresponding to a real object existing around the user terminal based on the position information (step S103). Next, the information processing apparatus 10 acquires the captured image and the capturing direction information by the user terminal (step S104). Next, the information processing apparatus 10 calculates the influence of the light source on the reality-corresponding object under the estimated natural light source condition (step S105). Next, the information processing apparatus 10 acquires information on virtual objects that may be displayed as ARs that are placed around the position of the user terminal and located within the photographing range of the camera device (step S106). Next, the information processing apparatus 10 calculates the influence of the light source on the virtual object in the estimated natural light source situation while considering the influence of the natural light source on the reality corresponding object (step S107). Through this processing, the effect of the light source caused by the virtual object to be drawn in the superimposed display and the virtual object such as shadows and reflections is acquired. Then, the information processing apparatus 10 specifies the ground position of the virtual object on the captured image, superimposes the virtual object and the calculated light source effect on the specified ground position on the captured image (step S106), End information processing.

Next, the result of applying the information processing apparatus 10 of this example will be described based on a specific example. FIG. 4 is an explanatory diagram showing an example of AR display when at least one embodiment of the present invention is not applied. Also, FIG. 5 is an explanatory diagram enlarging a part of an example of AR display when at least one embodiment of the present invention is not applied. FIG. 4 shows an example of AR display in which a plurality of spheres are arranged as virtual objects around a building, which is a real object, and characters are displayed on the window glass portion of the building, which is a real object. FIG. 5 is an enlarged image of a part of FIG. As shown in FIGS. 4 and 5, without applying the present invention, if only the effect of the light source on the virtual object is calculated and the result of the calculation is superimposed, the shadow of the building would originally cast the shadow of the other object. However, the shadows of the buildings and the shadows of the virtual objects overlap, creating a situation that could not have occurred in the first place. I had a problem.

Therefore, in this example, the situation of the natural light source at the position of the user terminal is estimated, the influence of the natural light source on the reality-corresponding object is calculated, and the natural light source on the virtual object placed near the reality-corresponding object is calculated. By calculating the influence, it is possible to extract only the influence of the natural light source on the virtual object and superimpose it.

FIG. 6 is a conceptual diagram showing the concept of an example of natural light source estimation processing in at least one embodiment of the present invention. Although it is possible to set the conditions of natural light sources in detail in order to faithfully reproduce phenomena in the real world, real-time calculation of such complex phenomena requires considerable processing power. Realistically, it is preferable to simplify and estimate the situation of the natural light source. Therefore, as shown in FIG. 6, the direction of the sun is estimated from the date and time and the position information of the user terminal, a hemispherical light source is set, and the hemispherical light source illuminates all the objects located inside. It shall act as an ambient light adding a diffuse ambient color to it. Specifically, the color of the sky estimated from the weather information is adopted as the ambient light color. Blue is used for clear weather, and gray is used for cloudy weather. Different colors are used depending on the time of day, such as black at night and reddish colors in the morning and evening. Also, in the case of cloudy weather, the influence of the direct light source from the sun may be reduced. By performing such a simplified natural light source estimation process, it is possible to reduce the load of subsequent calculation processing for the influence of the light source.

FIG. 7 is an explanatory diagram showing an example of AR display when at least one embodiment of the present invention is applied. Also, FIG. 8 is an explanatory diagram enlarging a part of an example of AR display when at least one embodiment of the present invention is applied. 7 and 8 show the same situation as in FIGS. 4 and 5, but when at least one embodiment of the present invention is applied as shown in FIGS. Since only the influence of the natural light source can be extracted and displayed in a superimposed manner, the shadows of the building and the shadows of the virtual objects are not overlapped, and rendering is performed in a more natural situation.

As described above, as one aspect of the above embodiment, the information processing device 10 includes the position information acquisition unit 11, the time information acquisition unit 12, the weather information acquisition unit 13, the natural light source estimation unit 14, A reality-corresponding object information acquisition unit 15, a captured image acquisition unit 16, a first light source effect calculation unit 17, a virtual object information acquisition unit 18, a second light source effect calculation unit 19, a superimposed display unit 20, and a storage unit. 21, acquires the location information of the user terminal, acquires the current time information, acquires the weather information of the location based on the location information of the user terminal, and acquires the acquired location information, time information and weather information estimating the situation of the natural light source at the position of the user terminal based on the position information, obtaining information of a reality-corresponding object in the virtual space pre-corresponding to a real object existing in the vicinity of the user terminal based on the position information; Acquire a photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal, calculate the influence of the light source on the realistic object in the estimated natural light source situation, Acquire information about virtual objects that may exist within the shooting range of the camera device, and calculate the effect of the light source on the virtual object under the estimated natural light source conditions while taking into account the effect of the natural light source on the reality-compatible object. However, since the ground position of the virtual object on the captured image is specified, and the virtual object and the calculated light source effect are superimposed on the specified ground position on the captured image, matching with the surrounding real objects is achieved. It is possible to perform AR display in consideration of the nature of the image.

[Second embodiment]
In the first embodiment, in order to perform AR display of a virtual object in consideration of consistency with surrounding real objects, the virtual object is placed in the virtual space in which the reality-corresponding object is placed, and then estimated. In the second embodiment, the effect of the natural light source is calculated, but in the second embodiment, the effect of the natural light source on the reality-corresponding object is calculated, and the situation is captured by a virtual camera to obtain a virtual captured image. Next, an example of use in tracking processing for specifying an object appearing in a photographed image actually photographed by the user will be described.

FIG. 9 is a block diagram showing an example configuration of an information processing apparatus according to at least one embodiment of the present invention. As shown in FIG. 9, the information processing apparatus 10 according to the second embodiment of the present invention includes a position information acquisition unit 11, a time information acquisition unit 12, a weather information acquisition unit 13, a natural light source estimation unit 14, It includes a reality-corresponding object information acquisition unit 15 , a captured image acquisition unit 16 , a virtual captured image generation unit 22 , a captured object identification unit 23 , and a storage unit 21 . It should be noted that the components denoted by the same reference numerals as in the first embodiment function in the same manner as in the first embodiment, and thus descriptions thereof will be omitted.

The virtual captured image generation unit 22 sets a virtual camera for at least one pattern that is estimated to be approximately the same position and approximately the same shooting direction as the captured image, and shoots in the virtual space. It has a function of generating a virtual photographed image by executing rendering considering the influence of the light source on the corresponding object. In this example, the reality-corresponding object information acquisition unit 15 acquires reality-corresponding objects around the position of the user terminal. Then, in that situation, if the position information obtained from the user terminal accurately indicates the actual position of the user terminal, then in the virtual space in which the reality-compatible object is placed, it is possible to shoot at the same position and in the same shooting direction. , it is possible to specify a reality-corresponding object that matches the object appearing in the image captured by the user terminal. In order to improve the identification accuracy, rendering processing is performed using the state of the natural light source estimated by the natural light source estimation unit 14 to obtain a virtual captured image. Note that the location information of the user terminal may not always be accurate. For example, GPS information may deviate due to the influence of surrounding buildings, etc., and position estimation based on reception strength of multiple types of radio signals and position estimation by DR may not indicate an accurate position. Therefore, the virtual captured image may be generated not only for one pattern but also for a plurality of patterns with different positions and shooting angles of the virtual camera.

The photographed object identification unit 23 has a function of identifying a location where the visual features of the photographed image and the virtual photographed image match, and identifying an object appearing in the photographed image based on the degree of matching between the two. The photographed image and the virtual photographed image are compared to extract matching portions of the visual features. For example, when the score of the matching portion exceeds a predetermined threshold, it is determined that the object shown in the photographed image and the reality-corresponding object are the same object. Identify there is. This makes it possible to identify the object appearing in the captured image.

Next, the flow of information processing performed in the information processing apparatus 10 of this example will be described. FIG. 10 is a flowchart illustrating an example of information processing in at least one embodiment of the invention. As shown in FIG. 10, information processing is started by sequentially acquiring location information of the user terminal, current time information, and weather information in the information processing apparatus 10 (step S201). Next, the information processing apparatus 10 estimates the condition of the natural light source at the position of the user terminal based on the acquired position information, time information and weather information (step S202). Next, the information processing apparatus 10 acquires information on a reality corresponding object corresponding to a real object existing around the user terminal based on the position information (step S203). Next, the information processing apparatus 10 acquires the captured image and the capturing direction information by the user terminal (step S204). Next, the information processing apparatus 10 calculates the influence of the light source on the reality-corresponding object under the estimated natural light source condition (step S205). Next, the information processing apparatus 10 acquires information on virtual objects that may be displayed as ARs that are placed around the position of the user terminal and located within the shooting range of the camera device (step S206). Next, the information processing apparatus 10 installs a virtual camera for at least one pattern that is estimated to be approximately the same position and approximately the same photographing direction as the photographed image, and performs photographing in the virtual space. A virtual photographed image is generated by executing rendering considering the influence of the light source on the reality corresponding object (step S207). Then, the information processing apparatus 10 specifies matching portions of the visual features of the captured image and the virtual captured image, and identifies objects appearing in the captured image based on the degree of matching between the two (step S106), and performs information processing. exit.

FIG. 11 is a conceptual diagram illustrating the concept of an example of tracking processing in at least one embodiment of the invention. The left side of FIG. 11 is a captured image captured by a user terminal, and the right side of FIG. 11 is a virtual captured image of a reality corresponding object in virtual space. As shown in FIG. 11, by using a virtual photographed image obtained by rendering in a more realistic situation by reflecting the situation of the natural light source, situations such as shadows, reflections, and brightness are approximated. Since matching can be performed under certain conditions, it is possible to improve the accuracy of tracking.

As described above, as one aspect of the above embodiment, the information processing device 10 includes the position information acquisition unit 11, the time information acquisition unit 12, the weather information acquisition unit 13, the natural light source estimation unit 14, A configuration including a reality-corresponding object information acquisition unit 15, a captured image acquisition unit 16, a virtual captured image generation unit 22, a captured object identification unit 23, and a storage unit 21, acquires the position information of the user terminal, Obtaining time information, obtaining weather information at the location based on the location information of the user terminal, estimating the situation of natural light sources at the location of the user terminal based on the obtained location information, time information and weather information, Obtained by acquiring information on a reality-corresponding object in the virtual space that is pre-associated with a real object that exists around the user terminal based on the position information, and photographing the real space with the camera device provided in the user terminal. Estimated natural light source conditions under the condition that a photographed image and photographing direction information are acquired, and a virtual camera is installed to photograph at least one pattern that is estimated to be approximately the same position and roughly the same photographing direction as the photographed image in the virtual space. A virtual photographed image is generated by rendering considering the effect of the light source on the reality-corresponding object, and the location where the visual characteristics of the photographed image and the virtual photographed image match is specified, and the photograph is taken based on the degree of matching between the two. Since the objects in the image are specified, the situation of shadows, reflections, brightness, etc. can be reproduced by using a virtual shot image obtained by rendering in a more realistic situation by reflecting the situation of the natural light source. Since matching can be performed under similar conditions, it is possible to improve the accuracy of tracking.

[Third Embodiment]
In the first and second embodiments, the shooting direction of the user terminal is assumed to be acquired by the orientation sensor provided in the user terminal. , the photographing direction may be specified by the principle of a solar compass.

FIG. 12 is a conceptual diagram showing the concept of an example of camera orientation estimation processing in at least one embodiment of the present invention. As shown in FIG. 12, the principle of the solar compass is that if the date and time the image was taken is known, the direction of the sun can be uniquely identified. Based on this, [1] the direction of the north is found. In this context, determine [6] camera orientation based on [1] north orientation, [2] shadow orientation, [3] shadow orientation, [4] sun azimuth, [5] relative shadow orientation. It becomes possible to

In this manner, the first and second embodiments may be implemented by adopting a method of specifying the camera orientation based on the principle of the solar compass.

As described above, the embodiments of the present application address one or more deficiencies. In addition, the effect by each embodiment is an example of a non-limiting effect or effect.

In each of the above-described embodiments, the plurality of user terminals 401 to 40n and the server device 30 perform the above-described various processes according to various control programs (for example, video game processing programs) stored in their own storage devices. to run.

Further, the configuration of the system to which the information processing apparatus 10 in each embodiment of the present application is applied is not limited to the configuration described as an example of each embodiment described above. Some or all of the processing may be executed by the server device 30, or some or all of the processing described as processing executed by the server device 30 may be executed by any of the plurality of user terminals 401 to 40n. . Further, a configuration may be adopted in which any one of the plurality of user terminals 401 to 40n has a part or all of the storage unit provided in the server device 30. FIG. That is, a configuration may be adopted in which one of the user terminal 40 and the server device 30 in the system has some or all of the functions provided by the other.

REFERENCE SIGNS LIST 10 information processing device 11 location information acquisition unit 12 time information acquisition unit 13 weather information acquisition unit 14 natural light source estimation unit 15 reality-corresponding object information acquisition unit 16 captured image acquisition unit 17 first light source effect calculation unit 18 virtual object information acquisition unit 19 Second light source effect calculation unit 20 superimposed display unit 21 storage unit 22 virtual captured image generation unit 23 captured object identification unit 30 server device 40, 401 to 40n user terminal 50 communication network

Claims (4)

An information processing program for causing a computer to realize a process of superimposing and displaying a virtual object stored in association with a predetermined position in real space on a captured image captured by a user terminal owned by a user,
to the computer;
a location information acquisition function for acquiring location information of the user terminal;
a time acquisition function for acquiring current time information;
a weather information acquisition function for acquiring weather information for a location based on the location information of the user terminal;
a natural light source estimation function for estimating the state of natural light sources at the location of the user terminal based on the acquired location information, time information, and weather information;
a reality-corresponding object information acquisition function for acquiring information on an object in a virtual space (reality-corresponding object) that is associated in advance with a real object existing around the user terminal based on the position information;
A photographed image acquisition function for acquiring the photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal;
a first light source effect calculation function for calculating the effect of a light source on the reality corresponding object in the estimated natural light source situation;
a virtual object information acquisition function for acquiring information on the virtual object that is arranged around the position of the user terminal and possibly positioned within the shooting range of the camera device;
a second light source effect calculation function for calculating the effect of a light source on the virtual object in the estimated natural light source situation while considering the effect of the natural light source on the reality corresponding object;
The ground position of the virtual object on the captured image is specified, and the light source influence calculated based on the virtual object, the first light source influence calculation function, and the second light source influence calculation function are calculated. and a superimposed display function that superimposes the specified ground contact position on the captured image , and
The natural light source estimation function obtains information on the direction of the sun estimated from the date and time and the location information of the user terminal, and also obtains a predetermined diffuse ambient color based on weather information for all objects located inside. A hemispherical light source that functions as an ambient light is defined, and the sky color estimated from the time information and the weather information is set as the ambient color for the hemispherical light source.
Information processing program. The second light source effect calculation function obtains the calculation result of the first light source effect calculation function from the result of executing the rendering processing considering the effect of the natural light source in the state where the virtual object is arranged in the reality corresponding object. 2. The information processing program according to claim 1 , wherein only the influence of a natural light source on said virtual object is extracted by subtracting . An information processing device for superimposing and displaying a virtual object stored in association with a predetermined position in real space on a captured image captured by a user terminal owned by a user, the information processing device comprising:
a location information acquisition unit that acquires location information of the user terminal;
a time acquisition unit that acquires current time information;
a weather information acquisition unit that acquires weather information for a location based on the location information of the user terminal;
a natural light source estimation unit for estimating the state of natural light sources at the location of the user terminal based on the acquired location information, time information, and weather information;
a reality-corresponding object information acquisition unit that acquires information on an object in a virtual space (reality-corresponding object) that is associated in advance with a real object existing around the user terminal based on the position information;
a captured image acquiring unit that acquires the captured image and the capturing direction information obtained by capturing the real space with a camera device included in the user terminal;
a first light source effect calculation unit that calculates the effect of a light source on the reality-corresponding object in the estimated natural light source situation;
a virtual object information acquisition unit that acquires information on the virtual object that is arranged around the position of the user terminal and that may be located within the shooting range of the camera device;
a second light source effect calculation unit that calculates the effect of a light source on the virtual object in the estimated natural light source situation while considering the effect of the natural light source on the reality corresponding object;
specifying the ground position of the virtual object on the captured image, and calculating the virtual object , the light source effect calculated by the first light source effect calculation unit, and the light source effect calculated by the second light source effect calculation unit on the captured image; A superimposed display unit that superimposes and displays the specified ground contact position ,
The natural light source estimating unit obtains information on the direction of the sun estimated from the date and time and the position information of the user terminal, and also obtains a predetermined diffuse ambient color based on weather information for all objects located inside. A hemispherical light source that functions as an ambient light is defined, and the sky color estimated from the time information and the weather information is set as the ambient color for the hemispherical light source.
Information processing equipment. An information processing method for causing a computer to execute a process of superimposing and displaying a virtual object stored in association with a predetermined position in real space on a captured image captured by a user terminal owned by a user, the method comprising:
a location information acquisition procedure for acquiring location information of the user terminal;
a time acquisition procedure for acquiring current time information;
a weather information acquisition unit that acquires weather information for a location based on the location information of the user terminal;
a natural light source estimation procedure for estimating a natural light source situation at the location of the user terminal based on the obtained location information, the time information, and the weather information;
a reality-corresponding object information obtaining procedure for obtaining information about an object (reality-corresponding object) in a virtual space pre-associated with a real object existing around the user terminal based on the position information;
a photographed image obtaining procedure for obtaining the photographed image and photographing direction information obtained by photographing the real space with a camera device provided in the user terminal;
a first light source effect calculation procedure for calculating a light source effect on the reality object in the estimated natural light source situation;
a virtual object information acquisition procedure for acquiring information on the virtual object that is arranged around the position of the user terminal and that may be located within the shooting range of the camera device;
a second light source effect calculation procedure for calculating the effect of a light source on the virtual object in the estimated natural light source situation while considering the effect of the natural light source on the reality corresponding object;
The ground position of the virtual object on the captured image is specified, and the virtual object , the light source influence calculated based on the first light source influence calculation procedure, and the light source influence calculated based on the second light source influence calculation procedure are calculated as the A superimposed display procedure for superimposing the specified ground contact position on the captured image ,
The natural light source estimation procedure obtains the information of the direction of the sun estimated from the date and time and the location information of the user terminal, and also obtains a predetermined diffuse ambient color based on the weather information for all objects located inside. A hemispherical light source that functions as an ambient light is defined, and the sky color estimated from the time information and the weather information is set as the ambient color for the hemispherical light source.
Information processing methods.

Images