JP2013149029A - Information processor, information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide AR information processing technology for improving a coincidence degree with image features to be obtained from an actually photographed image by considering an influence of a light source etc. in CG for obtaining image features or the like on the database side for evaluating the coincidence degree in matching between real and virtual.SOLUTION: An AR server creates AR information on the basis of an AR information request including position information including a photographed image or image features to be generated from the image, latitude, and longitude, and date and time information, has: an AR information request processing information creation unit which requests creation of a CG image on the basis of the position information and meteorological information acquired by a weather information acquisition unit; and a collation unit which requests creation of CG in consideration of an influence of a light source to an AR cloud CG drawing device by AR information request processing information created by the AR information request processing information creation unit, and evaluates a coincidence degree between the CG obtained as a reply, and the AR information. The AR server synthesizes the AR information obtained as a collation result by the collation unit with the image as information support to be displayed.

Description

  The present invention relates to information processing technology, and more particularly to information processing technology related to augmented reality (hereinafter referred to as AR).

  AR is a technique for extending a real “perception experience” by overlaying virtual information on a real world that is actually seen, such as a city view or a displayed product.

  With the recent expansion of the use of mobile terminals such as smartphones and tablet PCs, AR that augments the real perception experience by superimposing virtual information on the real world has become more familiar.

  For example, it has been proposed to provide services such as navigation by displaying and manipulating AR information as an image drawn in a computer graphic (CG) on an image obtained by photographing an actual space with an AR terminal.

  The following Patent Document 1 proposes a technique for evaluating the coincidence between a CG feature database (DB) generated from a captured image and a CG feature obtained from information such as an AR terminal captured image and providing AR information. Has been.

  In addition, the following Patent Document 2 collects image features according to time zone, weather (meteorology), seasons, and classifies, updates, and constructs a map database, including the case of using a 3D map, A technique for providing AR information by storing captured images or the like in a DB and evaluating matching with image characteristics of captured images acquired from a user AR terminal has been proposed.

  Patent Document 3 below proposes a technique for improving the display quality of a CG object image with respect to a CG object image that is AR information as a virtual object synthesized in AR by estimating a light source direction. .

  Further, as Non-Patent Document 1, a technique for light source estimation and shadow expression of a CG object image which is AR information is discussed.

JP-A-11-66350 JP 2002-27529 A JP 2008-16918 A

"Image recognition and understanding symposium" MIRU July 2004, Augmented reality capable of shadow representation of virtual objects by real-time estimation of lighting conditions and depth of real environment, Kanbara et al.

  The technique described in Patent Document 1 and the technique described in Patent Document 2 do not take into account the influence of the light source, but also must store all the images and the like associated with the changing light source due to the changing weather and time. There is a problem of not becoming. In general, since a three-dimensional map is information of only an appearance shape, there is a problem that a color is different from that of a captured image. In order to improve this, there is a method of adding texture to the model data, but the texture image is often based on images taken in the daytime, and if the influence of the light source is not considered, the direction of the shadow There is a problem in that the state of lighting or lighting does not match the captured image sent from the actual AR terminal, and a problem occurs when the position of the AR is specified.

  Further, in the techniques described in Patent Literature 3 and Non-Patent Literature 1, although the influence of the light source of the CG synthesized in the AR is estimated, in the image representation when the virtual (compositing CG) is superimposed on the real (real image). A virtual CG light source is considered as AR information to be combined. That is, these inventions are inventions relating to a method of creating CG as AR information for composite display based on an estimated light source.

  On the other hand, in the AR technique, matching (matching degree, similarity degree, distance calculation) for specifying a position where a captured image (actual image) of the current terminal and virtual AR information (such as CG for synthesis) are combined is specified. In addition, based on the data acquired from the DB in order to evaluate the degree of coincidence between the current captured image of the AR terminal and the evaluation CG image generated based on the data acquired from the DB and the captured image acquired from the DB. There is a problem in that AR information provision is not successful unless the influence of a light source or the like in an image (including CG) generated or acquired from a DB is considered.

More specifically, augmented reality is mainly implemented by the following steps.
1. 1. Augmented reality implementation location according to GPS, gyro, captured image, etc. by AR device 2. Specification of display position of augmented reality information in captured image by AR device Composite or multiple display of AR information (such as CG for composition) to the specified AR device for captured images

Among these, steps 1 and 2 include a method as in Patent Document 1 and Patent Document 2 and a method as in the present application. More specifically,
a. CG is generated from the captured image and stored in the DB, and AR information is provided to similar parts by comparing the contents of the DB and the CG generated from the captured image of the user AR terminal by using feature information or VQ code. Patent Document 1 for performing
b. Patent Document 2 that stores captured images in a DB, provides AR information to similar parts by comparing the contents of the DB and captured images of a user AR terminal by using feature information, VQ codes, and the like. c. CG (evaluation CG) component data is generated based on information generated or acquired from the captured image and stored in the DB. The CG (evaluation CG) generated based on the DB and the captured image of the user AR terminal And providing AR information to similar parts by comparing them with feature information or VQ code.
Several methods are being considered.

Then, in each configuration, shapes such as point group processing, line group processing, and surface group processing are compared, and image features include color features, luminance features, saturation features, hue features, shape features, contour features, and colors. There are various feature information such as spatial features.
Note that Patent Literature 3 and Non-Patent Literature 1 are techniques related to the third step described above.

  If the position and location matching method based on the captured image of the AR terminal and the DB content is Patent Document 1, the CG generated from the database based on the CG generated from the captured image and the captured image currently captured by the AR terminal. Appropriate labels are presented to the user through matching. In the case of Patent Document 2, it is appropriate to match an image feature of an image database in which a captured image is classified by position, weather, time, season, and the like with an image feature generated from a captured image currently captured by an AR terminal. Various information including labels is displayed. If it is patent document 3, CG as a virtual image for a synthesis | combination matched with the solar light source direction estimated based on the position and direction of a captured image, and time is produced | generated, and it synthesize | combines with an actual image.

  However, in any of the prior documents, in matching of database-side image features and the like and image features generated from the captured image of the AR terminal, CG ( No change in the evaluation CG) is assumed.

  The present invention relates to a CG (evaluation CG) for obtaining a database-side image feature or the like for evaluating the degree of coincidence when matching a real (actual image) and a virtual (combination CG) in the step 2 above. It is an object of the present invention to provide an AR information processing technique for improving the degree of coincidence with an image feature obtained from an actually captured image by considering the influence of a light source or the like.

  In the present invention, a real image captured by a user AR terminal and a CG (evaluation CG) generated based on data recorded in a DB are compared with images such as image information, feature information, and VQ code. The AR information is provided to similar parts of the two. In addition, based on the information extracted or acquired from the photographed image, CG component data may be generated and the DB may be updated.

  According to one aspect of the present invention, based on an AR information request including a captured real image or an image feature or VQ code extracted from the real image, position information including latitude and longitude, and date / time information, AR information is obtained. An AR server that creates (such as a CG for synthesis), an AR information request processing information creation unit that requests creation of a CG image based on the position information and the weather information acquired by the weather information acquisition unit; Using the AR information request processing information created by the AR information request processing information creation unit, the AR cloud CG drawing apparatus is requested to create a CG (evaluation CG) in consideration of the influence of the light source, and the CG (evaluation CG) and a matching unit that evaluates the degree of coincidence of the photographed image, and AR information (such as CG for synthesis) obtained as a matching result by the matching unit is stored in the image. AR server is provided, characterized in that the synthesis to be displayed as assistance.

  Further, CG (evaluation CG) component data is generated based on information extracted or acquired from the captured real image and stored in the DB. The CG (evaluation CG) generated based on the DB and the user AR terminal The captured images are compared with each other by images themselves, feature information, images such as VQ codes, and the like, and AR information (such as CG for synthesis) is provided to similar parts.

The AR information request processing information is
With the sun or moon as the light source, it has the light source angle and illuminance depending on the position information and the date and time information as light source parameters, the position of the building that is the subject of the image, model data, Referring to a database having a height, based on the height of the building where the position information and the position of the building coincide with each other and the light source parameter, the AR cloud CG drawing device stores the CG of the shadow of the building. It is preferable to make it.

  Here, when the CG (evaluation CG) is created by the AR cloud CG drawing device, the building ID, the altitude at which the building stands, model data, and the light sources around the building It is preferable to create a CG of the shadow of the building by referring to a terrain database having adjacent information that affects the shadow of the building.

  In addition, when the CG (evaluation CG) is created by the AR cloud CG drawing apparatus, the height of the feature around the building, the model data, and the area of the feature are obtained using the position information as a key. It is preferable to obtain the influence of the feature that affects the shadow of the building by the light source with reference to the feature database held.

  The position information is preferably at least one of an information group for specifying the position and line-of-sight direction of the AR terminal device including GPS information, electronic compass information, acceleration sensor information, and gyro sensor information.

  The AR server may include an accompanying information adding unit that adds additional information for information support. Here, the additional information is, for example, service information such as information on dishes provided in buildings acquired via the Internet, event information, word-of-mouth information, daily information, product information, price / stock information, and the like.

  The AR server may use image features extracted from an image, image VQ codes, or surface group information, point group information, and line group information as group information thereof as the image features.

  Moreover, the present invention is an AR terminal device that transmits the image or image feature described above, and displays the AR information received from the AR server on a display unit. .

  At this time, the AR information may be combined with the image captured by the terminal itself, or displayed by adding the AR information to the CG image drawn by the AR cloud CG drawing device 3 provided from the distribution device side. Also good. The AR information may be provided as image information, or may be provided by a polygon model (such as CG for synthesis), a character code, font information, a markup language, or voice. In addition, the designation of the display position in the AR information display provides the AR terminal device with the information used in the above-mentioned AR server collation unit, predicts and corrects based on the behavior on the AR terminal device side, and determines the display position. Also good. Alternatively, the AR terminal device may acquire the CG drawing modeling data of the AR cloud CG drawing device 3 via the AR server 1 and use it for calculating an appropriate display position according to the situation on the terminal side. You may make it display AR information on a display part with additional information.

  Moreover, this invention is CG drawing apparatus which produces CG (CG for evaluation) as described above, Comprising: The CG production | generation part which produces | generates CG, the shadow production | generation part which produces | generates the shadow of a building, a season, date and time An optical effect calculation unit that calculates the influence of sunrise and sunset by the light source, a light source effect calculation unit that calculates the effect of the type of light source such as the sun and the moon, and a latitude and longitude effect calculation unit that calculates the influence of latitude and longitude A CG drawing apparatus including a CG creation unit having

  Furthermore, it has a building DB for storing the building at the position, a terrain DB for storing the altitude at the position, an altitude difference with the surroundings, and a feature DB for storing the vegetation area at the position. The shadow generation unit generates a shadow in consideration of the influence of at least one of the building, the terrain, and the vegetation.

  In addition, it is possible to generate a CG in consideration of various influences necessary for CG drawing in the CG creation unit. In particular, since it is for drawing a CG for evaluating the degree of coincidence with a real image in consideration of the influence on the light source, a CG as close as possible to the actual image can be obtained.

  According to another aspect of the present invention, an AR server that creates AR information based on an AR information request including a captured image or an image feature generated from the image, position information including latitude and longitude, and date / time information. An AR information request processing information creation step for requesting creation of a CG image based on the position information and the weather information acquired by the weather information acquisition unit, and the AR information request processing information The AR information request processing information created in the creation step is used to request the AR cloud CG rendering device to create a CG in consideration of the influence of the light source, and collation for evaluating the degree of coincidence between the CG obtained as a reply and the AR information And AR information obtained as a collation result in the collation step is combined and displayed as information support on the image. Broadcast processing method is provided.

  The present invention may be a program for causing a computer to execute the information processing method described above, or a computer-readable recording medium for recording the program.

  According to the present invention, by performing processing based on illumination / light source environment such as latitude, longitude, time of day, and sunshine state due to weather, a real space image captured by the terminal in the AR and a DB for specifying the position and location By realizing the accuracy in matching (similarity evaluation, coincidence evaluation, distance evaluation) with the evaluation CG image formed on the side by comparing with the evaluation CG in consideration of the light source closer to the actual image, the light source change It can be improved by reducing the influence of. In addition, 3D model data and light source-independent texture data are stored in the DB, thereby storing a large number of image groups, CG groups, feature groups, etc. in a state taking into account various light sources in the DB. This eliminates the need for improvement in storage efficiency and retrieval efficiency.

It is a functional block diagram which shows the example of 1 structure of AR information processing system by one embodiment of this invention. It is a functional block diagram which shows one structural example of AR server among the AR information processing systems by this Embodiment. It is a functional block diagram which shows the example of 1 structure of AR terminal apparatus among the AR information processing systems by this Embodiment. It is a functional block diagram which shows the example of 1 structure of AR cloud CG drawing apparatus among the AR information processing systems by this Embodiment. It is a flowchart figure which shows an example of the flow of a process of the information processing method by this Embodiment. It is a figure which shows an example of AR information request process information in this Embodiment. It is a figure which shows an example of building DB in this Embodiment. It is a figure which shows an example of landform DB in this Embodiment. It is a figure which shows an example of feature DB in this Embodiment. It is a figure which shows an example of the weather dependence of the shadow of a building. It is a figure which shows an example of the time dependence of the shadow of a building. It is a figure which shows an example of the kind dependence of the light source of the shadow of a building. It is a figure which shows an example of the position dependence of the northern and southern hemispheres of the shadow of a building. It is a figure which shows an example of the latitude dependence of the shadow of a building.

  Hereinafter, an AR information processing technique according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a functional block diagram illustrating a configuration example of the AR information processing system according to the present embodiment. As illustrated in FIG. 1, the AR information processing system A according to the present embodiment includes an AR server 1, an AR cloud CG drawing device 3, and an AR terminal device 5. The AR server 1 and the AR terminal device 5 are connected by a network NT such as the Internet or a local network. The AR terminal device 5 is a mobile terminal such as a smartphone or a mobile phone.

  FIG. 2 is a functional block diagram illustrating a configuration example of the AR server 1. As shown in FIG. 2, the AR server 1 includes, for example, a control unit (CPU) 1-1, a communication unit 1-2, a collation unit (matching unit) 1-3, and a storage unit (AR database, database). 1), the accompanying information adding unit 1-5, the weather information acquiring unit 1-6, the date and time information acquiring unit 1-7, and the AR information request processing information generating unit 1 -8. The control unit (CPU) 1-1 performs overall control. The communication unit 1-2 has a function of exchanging information between the AR terminal device 5 and the AR cloud CG drawing device 3. The matching unit (matching unit) 1-3 performs matching (similarity evaluation) between a real real space image and a virtual CG image in AR. The AR database (storage unit or database access means) 1-4 stores a real image, attribute information such as position information and name of the image, and the like. The accompanying information adding unit 1-5 adds additional information accompanying a real image. The accompanying information includes realistic image attributes such as the name, height, and location of a building. The weather information acquisition unit 1-6 acquires the weather at the time of image capturing, such as the weather at a certain time, for example, the current weather, from a weather satellite, a weather radar, a rain gauge network, an automobile wiper behavior information, and the like. . The date and time information acquisition unit 1-7 acquires the date and time in consideration of the current date and time difference such as different countries. The AR information request processing information creation unit 1-8 creates information on AR information request processing for requesting creation of CG in consideration of shape information, texture information, and the like requested by the AR server 1 to the CG rendering device 3. To do.

  The AR cloud CG drawing device 3 is not a single unit, but is intended to include a mechanism for linking devices around the world, for example. Of course, this does not exclude a single device.

  FIG. 3 is a functional block diagram illustrating a configuration example of the AR terminal device 5. As illustrated in FIG. 3, the AR terminal device 5 is, for example, a smartphone or a mobile phone, and includes a control unit (CPU) 5-1, a communication unit 5-2, and a position information acquisition unit 5-3 such as a GPS. And an imaging unit 5-4 such as a camera or a movie and a display unit 5-5 such as a liquid crystal display. In addition, an input unit such as a touch pad or a button, a voice call unit, an acceleration sensor, a human sensor, or a gyro sensor may be included. You may make it reflect the sensing or search result by the audio | voice which talked, and a sensor in AR image.

  FIG. 4 is a functional block diagram showing a configuration example of the AR cloud CG drawing device 3. As illustrated in FIG. 4, the AR cloud CG drawing device 3 includes a control unit (CPU) 3-1, a communication unit 3-2, a CG creation unit 3-3, and a DB unit 3-9. ing.

  The CG creation unit 3-3 calculates the effects of the CG generation unit 3-4 that generates a CG, the shadow generation unit 3-5 that generates a shadow of a feature such as a building, and the sunrise and sunset according to the season and the date and time. Optical influence calculation unit 3-7, light source effect calculation unit 3-7 that calculates the influence of the type of light source such as the sun and moon, and latitude / longitude influence calculation unit 3-8 that calculates the influence of latitude and longitude have. The DB unit 3-9 stores, for example, a building DB 3-10 that stores a so-called facility or building such as a building, a house, a tower, or an amusement park at that position, and an altitude difference between the altitude at that position or the surroundings. It has a terrain DB 3-11 to be stored, a feature DB 3-12 to store a region of a feature (vegetation) such as a forest or a lake at the position, and the like. In addition, it is possible to generate a CG in consideration of various influences necessary for CG rendering in the CG creation unit 3-3. In the present embodiment, in particular, the CG for evaluating the degree of coincidence with the photographed image is drawn in consideration of the influence of the light source. Therefore, a CG (evaluation CG) as close as possible to the actual image is obtained. It is preferable to be able to do so.

  FIG. 5 is a flowchart showing an example of the flow of the information processing method according to this embodiment. First, in order to start the process, the AR terminal device 5 images a real image, for example, a building such as a building, by the imaging unit 5-4 (moving image shooting may be continued). Then, the position information acquisition unit 5-3 acquires the current position based on the latitude and longitude, for example, obtains the position of the building based on the relationship with the current position, the angle of view of the camera, and the like. Make an association. At this time, the AR terminal device may acquire information for obtaining the position of the building from the current position based on the angle of view of the camera capturing the building and the distance measuring function. In addition to the position information, attribute information such as date and time may be acquired.

  Further, a feature amount, a VQ code, or the like may be created together with the image or instead of the image. When the communication unit 5-2 of the AR terminal device 5 sends these pieces of information to the AR server 1, as shown in step S1, the communication unit 1-2 of the AR server 1 determines the attributes such as the captured image and the captured image. Receive information. Next, in step S2, the AR server 1 adds a date and time acquisition unit 1-7 and a weather information acquisition unit 1-6 as necessary in addition to the captured image from the AR terminal device 5 and the attribute information such as the captured image. It may be used to acquire additional information such as necessary date and time and weather information.

  The attribute information is position information by the GPS of the AR terminal device 5 as necessary, an inclination sensor or a gyro sensor for specifying the imaging direction of the imaging unit, an angle of view, a focal length, It may be an aperture value, a lens diameter, a date and the like.

  Note that, as described in Patent Document 3 described above, regarding an image of a subject such as a building such as a building or a tower in an image acquired by the AR terminal device 5, an imaging unit (camera) of the AR terminal device 5 is used. ) 5-4, based on the angle sensor output (angle of view) obtained from the tilt sensor and the focal length, the posture information (posture matrix, for example, “Takayuki Okaya,“ three-dimensional shape by fusion of mechanical sensor and image sensor) Restoration ", Information Processing Society of Japan Research Report 2005-CVIM-147, pp. 123-130, 2005") and the distance to the subject can be obtained. The position of a building or the like in the camera image can be known based on the position of the camera obtained by GPS or the like, the angle sensor output (angle of view), and the focal length.

  Next, in step S3, the AR server 1 gives the AR cloud CG drawing device 3 the location, date, time, and weather information based on the attribute information received in step S1 and the additional information acquired in step S2. Send information.

  FIG. 6 is a diagram showing a configuration example of the AR information request processing information sent from the AR server 1 to the AR cloud CG rendering device 3. This information is stored in the storage unit 1-4 of the AR server 1. You can keep it.

  The AR information request processing information shown in FIG. 6 is information on AR information request processing for requesting creation of CG in consideration of shape information, texture information, and the like requested by the AR server 1 to the CG rendering device 3. For example, the information is created based on the attribute information of the captured image sent from the AR terminal device 1 and the additional information. For example, the place name of a shooting place such as Tokyo, Sapporo, Fukuoka, etc. (more detailed place names, but here, the names of the above three places are given as examples), and the location information (latitude, longitude) of the shot AR terminal device 5 , Altitude, etc.), shooting date, shooting time (acquired from a clock, etc.), and information on the light source depending on the position information, the time from 0 to 24:00, sunrise, sunset time, south-central Light source parameters such as altitude and time, amount of sunshine per day (daytime), and illuminance (nighttime) depending on the moonlight. With this AR information request processing information including the light source parameter, the AR cloud CG rendering device 3 can create a CG image in consideration of the influence of the light source in detail.

  The AR cloud CG drawing device 3 performs CG drawing based on the AR information request processing information on the position, date, time, and weather received by the communication unit 3-2. Here, the CG creation unit 3-3 creates the building DB 3-10, the terrain DB 3-11, and the feature DB 3 of the storage unit (DB unit) 3-9 based on the AR information request processing information of the position, date, and time. -12 and the shadow generation unit 3-4 generates shadows for these buildings, terrain, and features based on the AR information request processing information of date, time, and weather.

  FIG. 7 is a diagram illustrating an example of a building DB. As shown in FIG. 7, the building DB includes a building ID that uniquely identifies the building, a name of the building, a position based on latitude and longitude, model data, and a height of the building. Yes. The model data consists of a set of (x, y, z) coordinates. For example, in the case of a cube, the model data is specified by the (x, y, z) coordinates of each vertex. Model data refers to data for performing CG drawing of an object using polygon shape data, texture data, UV map data, or the like.

  By comparing the position of the building DB with the position in the AR information request processing information, the building at the same position can be specified, and the shadow depending on the light source can be obtained from its height, model data, south-central altitude, and the like.

  FIG. 8 is a diagram illustrating an example of the landform DB. As shown in FIG. 8, the terrain DB has model data, its altitude, and adjacent (surrounding) information corresponding to the building ID. The model data consists of a set of (x, y, z) coordinates. For example, in the case of a cube, the model data is specified by the (x, y, z) coordinates of each vertex. The altitude is basically obtained from FIG. 6, but since the altitude at the shooting position is often different from the altitude of the building or the like that is the subject, it is preferable to obtain the shadow based on the altitude of the building. The adjacency information also indicates whether there are buildings in the surroundings that are covered by the shadow of the building, and if there is such a building, what height building is in which position, etc. Is stored. Thereby, the shape of a building, the shape of a shadow, etc. can be calculated | required in detail and accurately.

  FIG. 9 is a diagram illustrating an example of the feature DB. As shown in FIG. 9, the feature DB stores information on natural terrain, for example, features such as lakes and mountains. It consists of feature IDs that uniquely identify features, types such as lakes and mountains, model data, position, height, area, etc., to determine the influence of these features on the light source and shadow. it can. These model data are composed of a set of (x, y, z) coordinates. For example, in the case of a mountain close to a trapezoid, it is specified by the (x, y, z) coordinates of each vertex. For example, if there is a large mountain between the sun and the building, little shadow will be generated, and if there is a lake, estimate that the area of the lake has little effect on the building by the light source. Can do. Also, by using a physics calculation model or the like, standing cliff collapses, changes in water surface waves, and water flow may be drawn in CG simulations. You may process by approximate hue evaluation.

  As described above, based on the information from FIG. 6 to FIG. 9, it is possible to generate a highly accurate CG including the light source and the shadow based on the AR information request processing information.

  Note that the drawing data used for the CG at this time may be configured to use the data specified in FIGS. 7, 8, and 9, and is selected according to latitude, longitude, time, and line-of-sight direction, and matches according to the model data. A CG for degree evaluation is drawn. For this reason, it is possible to realize an appropriate AR by obtaining the latitude and longitude of the augmented reality image display target calculated from the latitude and longitude of the terminal and the line of sight. 7, 8, and 9, the model data is not directly described, and may be a link ID to each data or a file name of each data.

  At that time, based on the AR information request processing information, the optical influence calculation unit 3-6 determines the influence of the optical change on the building from information such as the season, the sunset by the date and time, the sunrise time, and the weather. Further, the light source influence calculation unit 3-7 calculates the influence of the light source, for example, the type of the light source such as the sun and the moon, the influence of the illuminance by the light source, and the like according to the season and time. At this time, the latitude / longitude influence calculation unit 3-8 also considers the dependency of latitude and longitude on the light source and the like. By such processing, the evaluation CG created by the CG creation unit 3-3 can be made more accurate.

  In addition, as attribute information, an evaluation CG may be generated in consideration of an angle of view, a direction, an aperture, an illuminance, a color temperature, and the like acquired in association with shooting by the imaging unit of the AR terminal.

  In addition to the evaluation CG, the present invention may be used to generate a synthesis CG adapted for shadows and image quality in consideration of the light source.

  Next, in step S4, the evaluation CG drawn by the AR cloud CG drawing device 3 is returned to the AR server 1. Here, together with the evaluation CG or instead of the evaluation CG, the characteristic amount of the evaluation CG, the VQ code, or the like may be sent. In step S5, the AR server 1 receives CG image information and the like from the AR cloud CG drawing device 3. Next, in step S6, the collation unit 1-3 of the AR server 1 collates the evaluation CG image information received in step S5 with the captured image received from the AR terminal device 5 in step S1. In step S7, the AR information stored in the AR database 1-4 is used for matching in the captured image obtained from the terminal on the basis of the matching score that is the matching result from the matching unit 1-3. A search is performed based on the positional information and the like, and the AR information obtained as a search result is transmitted to the AR terminal device 5 from the communication unit 1-2. Position information and the like refers to a group of information for specifying the position and line-of-sight direction of an AR terminal device such as GPS information, electronic compass information, acceleration sensor information, gyro sensor information, pupil detection information in HMD (head mounted display), Information necessary for specifying the AR information display mentioned in other inventions is included.

  At this time, additional information may be added for information support. The additional information is, for example, service information such as information on dishes provided in buildings acquired via the Internet, event information, word-of-mouth information, daily information, product information, price / stock information, and the like.

  As another processing method for images and the like, not only simply transmitting an image from a terminal or transmitting a CG image from an AR cloud CG rendering device, but also image features and image VQ codes generated from these images Alternatively, surface group information, point group information, and line group information as the group information may be used instead of the image.

  Next, the AR terminal device 5 can display the AR information received from the AR server 1 on the display unit 5-5 together with the additional information. At this time, the AR information may be combined with the image captured by the own terminal, or the AR information is added to the CG image (CG for evaluation) drawn by the AR cloud CG drawing device 3 provided from the distribution device side. May be displayed. The AR information may be provided as image information, or may be provided by a polygon model (composition CG), a character code, font information, a markup language, a voice, or the like. In addition, the designation of the display position in the AR information display provides the information used in the collation unit 1-3 of the AR server 1 to the AR terminal apparatus, and predicts / corrects it based on the behavior on the AR terminal apparatus side. May be determined. Alternatively, the AR terminal device may acquire the CG drawing modeling data of the AR cloud CG drawing device 3 via the AR server 1 and use it for calculating an appropriate display position according to the situation on the terminal side.

  In the above example, the AR cloud CG drawing device 3 may be distributed in plural via communication means such as the Internet. The AR cloud CG drawing device 3 may be incorporated in the AR server 1.

  According to the information processing method, the evaluation CG is drawn based on the weather, the light source state, and the like that can be estimated from the captured image and attribute information sent from the AR terminal device, and the drawn evaluation CG and the AR terminal device are drawn. Since the AR information can be acquired from the AR database based on the degree of coincidence with the image sent from the AR, and the AR information can be sent to the AR terminal device for display, the accuracy of the degree of coincidence (similarity) AR information can be obtained and AR information can be sent to the AR terminal device so that AR display with higher accuracy can be performed.

  Hereinafter, the present embodiment will be described more specifically and simply by taking the influence of the light source when drawing the evaluation CG as an example. In actual CG drawing, CG drawing for evaluation is optically corrected in consideration of the light source color, reflection, multi-light source, and the optical system of the imaging unit of the terminal device. It is preferable that

  FIG. 10 is a diagram showing the dependence of the weather on the shadow of the building (building, etc.) B. As shown in FIG. As shown in FIG. 10 (a), when the weather is clear, the shadow of building B becomes darker, whereas when it is cloudy as shown in FIG. 10 (b), almost no shadow is visible. Therefore, the shadow in CG can be estimated based on weather information. By using the CG thus obtained for evaluation of the degree of coincidence with a real image, it is possible to search and present AR information with higher accuracy.

  Similarly, FIG. 11 is a diagram showing time dependency and solar direction dependency. As shown in FIG. 11 (a), the direction of light sources such as the sun (including angles and the like) changes with time. Therefore, in the morning, since the sun is located in the east direction, the shadow of the building B is formed in the opposite direction. On the other hand, as shown in FIG. 11B, since the correspondence is located in the west direction in the evening, the shadow of the building B is formed in the opposite direction.

  FIG. 12 is a diagram showing the light source type (day and night) dependence. As shown in FIG. 12 (a), since the sun is the light source during the day, the shadow of the building B is dark, but as shown in FIG. 12 (b), the moon appears at night. Because of the degree, almost no shadow is generated. Also, the brightness of the surroundings varies greatly between night and day.

  FIG. 13 is a diagram showing position dependence. For example, as shown in FIG. 13A, in the northern hemisphere, the shadow appears on the north side of the building on a sunny day in the day, and FIG. As you can see in the southern hemisphere, the shadow appears on the south side of the building. Therefore, it is necessary to consider that the position of the shadow differs greatly between the northern and southern hemispheres.

  FIG. 14 is a diagram illustrating the influence of latitude and longitude among the position dependencies. For example, even in the same Japan, comparing Sapporo with a high latitude with Fukuoka with a low latitude, for example, the shadow at 3 pm is larger in Sapporo (because the sun's inclination is lower).

  As described above, in the CG drawing for the degree of coincidence comparison, the accuracy of the similarity evaluation between the real space image and the CG image in the AR can be improved by considering the influence of the light source or the like in detail. In addition, it is more preferable to draw CG in consideration of the fact that the sun's south-middle altitude changes depending on the season and that the declination and hour angle change depending on the time. More precisely, the altitude and azimuth of the sun at any point in time are calculated according to the astronomical chronological values.

Here, the altitude h and the azimuth angle A can be obtained by the following three formulas, for example.
[Equation 1]
cosh × sinA = −cosδ × sinΔ… (1)
cosh × cosA = cosφ × sinδ − sinφ × cosδ × cosΔ… (2)
sinh = sinφ × sinδ + cosφ × cosδ × cosΔ… (3)

Here, δ is the solar declination, Δ is the hour angle, and φ is the latitude of the place. t is the time to calculate altitude and azimuth-standard time (9h in Japan), λ is the longitude of the place (east longitude is +, west longitude is-, divided by 15 to be a unit of time), and θ ₀ is When α is the solar celestial longitude of the Greenwich star at 0h universal time, the hour angle can be calculated as Δ = θ ₀ + t × 1.0027379 + λ−α. The length of the shadow is the height of the object that makes the shadow × coth, and the azimuth angle of the shadow is obtained by adding 180 ° to the azimuth angle of the sun (if it exceeds 360 °, 360 ° is subtracted).

  It is possible to draw a CG based on the solar light source calculated by such a calculation and simultaneously calculate a shadow. The actual calculation accuracy may be a higher accuracy value.

  In addition, since the solar light source obtained by these calculations becomes a virtual surface light source at a very long distance, calculation is performed so that it becomes a scattered light source depending on the presence or absence of clouds in association with weather observation such as weather radar. It may be implemented as light source processing, CG drawing using the air perspective method, etc., or CG based on the generation probability and reflection intensity per unit area of various reflected light on the building wall surface or water area, etc. You may draw.

  In addition, it is information from a legitimate terminal by evaluating whether the latitude and longitude are approximate values from the shadow length in the captured image of the object whose height and date and time are known by calculating back these arithmetic expressions. Or the DB in the CG drawing apparatus may be updated.

(Summary)
As described above, according to the present embodiment, by performing processing based on the illumination / light source environment such as latitude / longitude / time and sunshine condition due to weather, the real space image captured by the terminal in the AR The accuracy in matching (similarity evaluation, coincidence evaluation, distance evaluation) with a CG image configured on the DB side for specifying the position and location is realized by comparison with a CG that considers a light source closer to a real image. This can be improved by reducing the influence of the light source change. In addition, 3D model data and 3D map data with light source-independent texture data are stored in the DB, so it is necessary to store images, CG groups, feature groups, etc. in a state taking into account various light sources in the DB. The storage efficiency and the search efficiency can be improved.

  In the above description, a daytime CG is formed in the daytime, and a nighttime CG is formed in the night time so that the captured image of the user's AR terminal is brought closer, and the shadow direction, length, shade, etc. are made as close as possible. It becomes easier to evaluate the degree of coincidence. Of course, processing such as changing the light source color according to the shade of the sun may be performed.

  Further, a CG shadow for comparison may be created based on the weather and the light source that can be estimated from the captured image and attribute information.

  Furthermore, it is also possible to acquire lighting conditions (neon signs, illuminations, digital signage, etc.) of buildings and the like via a network, and change the CG drawing or specify the tolerance for feature matching.

  The color and intensity of the light source may be estimated from the date / time, latitude / longitude as attribute information of the captured image of the AR terminal, and conversely, the shadow of the object whose height is known in the captured image of the AR terminal The latitude and longitude may be estimated from the date and time. In addition, if the calculation is strictly performed, correction for the precession of the ground axis may be performed.

  In addition, it is possible to preferentially evaluate the feature information of places where there are few feature errors due to the passage of people and cars, such as high-rise floors and roofs, the upper part of electric poles, traffic lights, etc., such as verandas, sidewalks and roadways Locations with changes may extend the tolerance range for feature match comparisons.

  Further, the processing may be performed by a combination of an arbitrary server, a client, and a cloud that links them.

  In addition, a three-dimensional model or texture may be generated from invisible light such as infrared rays or noctovision images.

  An image or the like includes a feature amount, feature information, and the like, which indicate information serving as an identification standard including a VQ code and the like including color, shape, luminance, hue, saturation, and the like.

  In addition, when comparing the image from the terminal with the image generated based on the DB, if the difference between the differential part and the integration part is large, or the place where the feature information etc. is changing rapidly, the variance is expanded and the feature matching is performed. Processing such as expanding an allowable error range in the comparison may be performed.

  Further, a DB having a dictionary structure in which feature information such as a three-dimensional shape based on multi-view stereoscopic vision or a VQ code based on two-dimensional information is indexed based on a position may be used.

  Further, deciduous trees, evergreen trees, etc. may be registered in the map information so that the evaluation color used for the feature evaluation can be changed according to the latitude and longitude in the lower layers close to the ground surface.

  Configure a system that classifies point group data, line group data, face group data, texture data, etc. based on position based on position as feature quantity or VQ code, and provides and changes contents appropriately from base station according to position Also good.

  Further, the shadow of the composite AR image based on the combined CG is calculated based on the light source direction such as the reflection of the sun or the building. Of course, not only sunlight, but also moonlight, solar eclipse, and lunar eclipse may be considered, and various electric lights, neon lights, digital signage, etc. may be converted into a database for use in light source calculations, or used as an index for information provided. .

  In addition, not only outdoors, but indoors as well, CG for evaluation may be drawn while estimating an outdoor light source in a time zone or the like.

  In addition, regarding the evaluation CG image for the synthesized real image, the degree of dispersion allowed in the feature comparison may be specified for each part of the image, the model base, and the map base, and may be databased.

  Also, instead of generating an evaluation CG for each access from a terminal in order to improve calculation efficiency, for example, a CG image, a feature amount, a VQ code, etc. can be generated every minute and reused as group information In such a state, the data may be appropriately stored in a storage unit such as an AR server, an AR cloud CG drawing device, or an AR terminal device.

  Also, the weather may be estimated from the characteristics of the terminal-side captured image, or the weather and time zone may be estimated from the color temperature.

  Further, dictionary information such as various group information may be appropriately cached or updated based on position information or the like.

  Alternatively, model data may be estimated from the captured image and position information, or texture data extracted by correcting light sources and correcting shadows may be extracted.

  Moreover, you may implement the social game etc. based on a position cooperation information service.

  In addition, a polygon image having a coarse particle size using Delaunay triangulation may be used as the evaluation CG.

  The display may be changed according to the time sale.

  Further, in the augmented reality performed for a person, the display position may change according to the position information output from the AR terminal of the person.

  The data used as model data in FIGS. 7, 8, and 9 is not only the actual data directly in the record but also the file name of the model data, or the ID or hash value that becomes the link information of the model data. It may be.

  In addition, for information support, additional information includes, for example, restaurant types and price information provided in buildings acquired via the Internet, event information, dealer product information, inventory / price information, etc. This is arbitrary information such as information, word-of-mouth information, congestion / vacancy information, real estate price information, and tourist information. Further, with respect to the evaluation CG image for the generated real image, the degree of dispersion may be specified for each part of the image to create a database, or the composition CG image based on the generated evaluation CG The shadow of the AR image may be calculated and synthesized based on the light source direction such as the reflection of the sun or the building.

  In the above-described embodiment, the configuration and the like illustrated in the accompanying drawings are not limited to these, and can be changed as appropriate within the scope of the effects of the present invention. In addition, the present invention can be appropriately modified and implemented without departing from the scope of the object of the present invention, and can be implemented in combination with a plurality of past patents.

  In addition, a program for realizing the functions described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute processing of each unit. May be performed. Here, the “computer system” includes various software such as an OS and hardware such as peripheral devices.

  Further, the “computer system” includes a homepage providing environment (or display environment) if a WWW system is used.

  Also, an update possibility flag is set when the characteristics of the evaluation CG and the photographed image are different, and various information of the evaluation CG that becomes map information when the distribution device receives the flag from a plurality of AR terminals is appropriately set. It may be updated. Further, the contents of the evaluation CG to be drawn may be changed according to the moon phases.

  The “computer-readable recording medium” refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, and a CD-ROM, and a storage device (various storage media) such as a hard disk built in the computer system. Say. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory in a computer system serving as a server or a client in that case, and a program that holds a program for a certain period of time are also included. The program may be a program for realizing a part of the above-described functions, or may be a program that can realize the above-described functions in combination with a program already recorded in a computer system.

  The present invention can be used as an information processing apparatus.

A ... AR information processing system, 1 ... AR server, 3 ... AR cloud CG drawing device, 5 ... AR terminal device, 1-1 ... control unit, 1-2 ... communication unit, 1-3 ... verification unit Part) 1-4 ... storage part (AR database), 1-5 ... accompanying information addition part, 1-6 ... weather information acquisition part, 1-7 ... date acquisition part, 1-8 ... AR information request processing information creation Unit, 3-1 ... control unit (CPU), 3-2 ... communication unit, 3-3 ... CG creation unit, 3-4 ... CG generation unit, 3-5 ... shadow generation unit, 3-6 ... optical influence Calculation unit, 3-7 ... Light source influence calculation unit, 3-8 ... Latitude / longitude influence calculation unit, 3-9 ... Storage unit (DB unit), 3-10 ... Building DB, 3-11 ... Topography (water area, etc.) DB), 3-12 ... feature (vegetation, etc.) DB.

Claims (12)

An AR server that creates AR information based on an AR information request including a captured image or an image feature generated from the image, position information including latitude and longitude, and date / time information,
An AR information request processing information creation unit that requests creation of a CG image based on the position information and the weather information acquired by the weather information acquisition unit;
Based on the AR information request processing information created by the AR information request processing information creation unit, the AR cloud CG drawing apparatus is requested to create a CG in consideration of the influence of the light source, and the CG obtained as a reply and the AR information A matching unit for evaluating the degree of coincidence,
An AR server characterized in that AR information obtained as a result of collation by the collation unit is combined with the image as information support and displayed. The AR information request processing information is
With the sun or moon as the light source, it has the light source angle and illuminance depending on the position information and the date and time information as light source parameters,
A database having the position of the building that is the subject of the image, model data, and height is referred to, and based on the height of the building and the light source parameter that match the position information and the position of the building The AR server according to claim 1, wherein the AR cloud CG drawing device is configured to create a CG of a shadow of the building. When creating a CG by the AR cloud CG drawing device,
Referring to the terrain database having the building ID, the altitude at which the building stands, model data, and adjacent information that affects the shadow of the light source around the building. The AR server according to claim 2, wherein a CG of a shadow of the image is created. When creating a CG by the AR cloud CG drawing device,
Using the position information as a key, refer to a feature database that holds the height of features around the building, model data, and area of the feature, and affects the shadow of the building by the light source. 4. The AR server according to claim 2, wherein an influence of the given feature is obtained. The location information is
2. From at least one of an information group for specifying a position and a line-of-sight direction of the AR terminal device including GPS information, electronic compass information, acceleration sensor information, and gyro sensor information. 5. The AR server according to any one of 4 to 4.   6. The AR server according to claim 1, further comprising an accompanying information adding unit for adding additional information for information support. As the image feature,
The image feature generated from the image, the image VQ code, or surface group information, point group information, or line group information as group information thereof is used. AR server. An AR terminal device for sending an image or image feature according to any one of claims 1 to 7,
An AR terminal apparatus that displays AR information received from the AR server on a display unit. A CG drawing apparatus for creating a CG according to any one of claims 1 to 7,
A CG generation unit that generates CG, a shadow generation unit that generates a shadow of a building, an optical influence calculation unit that calculates the influence of sunrise and sunset according to the season and date, and the type of light source such as the sun and moon A CG rendering apparatus comprising a CG creation unit having a light source influence calculation unit for calculating an influence and a latitude / longitude influence calculation unit for calculating an influence of latitude and longitude. Furthermore, it has a building DB for storing the building at the position, a terrain DB for storing the altitude at the position, an altitude difference with the surroundings, and a feature DB for storing the vegetation area at the position. ,
The CG drawing device according to claim 9, wherein the shadow generation unit generates a shadow in consideration of an influence of at least one of the building, the terrain, and the vegetation. An information processing method using an AR server that creates AR information based on an AR information request including a captured image or an image feature generated from the image, position information including latitude and longitude, and date / time information,
An AR information request processing information creation step for requesting creation of a CG image based on the position information and the weather information acquired by the weather information acquisition unit;
Based on the AR information request processing information created in the AR information request processing information creation step, the AR cloud CG drawing device is requested to create a CG in consideration of the influence of the light source, and the CG obtained as a reply and the AR information A matching step for evaluating the degree of match,
An information processing method comprising: combining and displaying AR information obtained as a result of collation in the collation step on the image as information support.   A program for causing a computer to execute the information processing method according to claim 11.

Images