JP2011022662A - Portable telephone terminal and information processing system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To superimpose and display a three-dimensional virtual object having a proper size and a proper direction, at a proper position in a real space, when a user goes to a location where an object has been photographed. <P>SOLUTION: A portable telephone terminal has a camera unit having an optical lens to acquire light passing through the optical lens as an image and a display unit for displaying the image acquired with the camera unit; receives radio waves from a GPS satellite to acquire portable telephone positional information; transmits the portable telephone terminal positional information to an information processing server having an object table storing a three-dimensional virtual object as three-dimensional computer graphics generated, on the basis of a moving image, photographing positional information indicating a location where the moving image has been photographed and the spatial information related to the moving image, in association with each other; receives the three-dimensional virtual object matching the transmitted portable telephone positional information and the space information related to the moving image from the information processing server; and uses the relation between the space information related to the moving image and the space information related to an image to composite a three-dimensional virtual object for display. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a mobile phone terminal and an information processing system that display a virtual object superimposed on a real space.

  At present, mobile phone terminals are highly functional and multifunctional, and it is possible to acquire position information by GPS (Global Positioning System) and various sensors and to capture moving images with a camera.

  In addition, a technical field called augmented reality (AR) has been developed in recent years. AR means that a virtual object (virtual object) as additional information is synthesized and presented as electronic information in a part of the actual environment.

  For example, when a user shoots a two-dimensional real moving image and displays a three-dimensional CG (Computer Graphics) of a moving object superimposed on the real moving image, the object is displayed in front of the user. (Patent Document 1).

  In addition, a technique for rendering a selected object as a three-dimensional computer graphic (three-dimensional virtual object) by selecting a specific object from the captured moving image is realized (Non-Patent Document 1). Further, a service for generating a three-dimensional virtual object from a plurality of images obtained by photographing a predetermined object over 360 degrees is performed on the web.

  It is also possible to calculate the shape of the moving image in the three-dimensional space by extracting a feature point for each frame from the moving image and detecting the plane by tracking the feature point in the time axis direction. (Non-Patent Document 2).

JP-A-11-259672

Anton van den Hengel et al., “Video Trace”, ACM Transactions on Graphics, Vol26, No.3, Article86, Publication date: July 2007. Georg Klein et al., “Parallel Tracking and Mapping for Small AR Workspaces”, International Symposium on Mixed and Augmented Reality, 2007

  By the way, it is often the case that an object photographed at a travel destination or the like is destroyed or disappears for some reason and cannot be seen by other users. In such a case, it is possible to see the object with a movie taken by another traveler, but there is a big difference between an object seen at a travel destination and an object seen in a movie taken in the past. . For example, even if a three-dimensional virtual object is generated from a moving image by the technique described in Non-Patent Document 1, it is not known where the three-dimensional virtual object is in the real space and how it looks from which direction. Therefore, there is a problem that even if an object is photographed, it cannot be shared with other people at the place where the object was photographed.

  The present invention has been made in view of such a point, and when the object is photographed, the three-dimensional virtual object is displayed in an appropriate size and direction at an appropriate position in the real space. An object is to provide a mobile phone device and an information processing server.

  In order to solve the above problem, a mobile phone terminal of the present invention includes an optical lens, a camera unit that acquires light passing through the optical lens as an image, a display unit that displays an image acquired by the camera unit, A control unit that calculates spatial information related to an image from an image acquired by a camera unit, a GPS reception unit that receives radio waves from a GPS satellite and acquires mobile phone position information, and a three-dimensional generated based on a moving image An information processing server including a three-dimensional virtual object that is computer graphics of the above, an object table in which shooting position information indicating a place where a moving image is shot and spatial information related to the moving image are stored in association with each other is provided in a mobile phone The three-dimensional virtual object transmitted from the information processing server when there is shooting position information that matches the transmitted mobile phone position information. 3D virtual objects are synthesized on the display screen by using a web communication unit that receives spatial information relating to the image and the moving image, and the relationship between the spatial information relating to the moving image and the spatial information relating to the image. A drawing unit that determines a size, a direction, and a position, and combines and displays a three-dimensional virtual object at the determined position.

  According to the above configuration, it is possible to calculate the spatial information related to the image currently being shot. Therefore, by using the relationship between the spatial information related to the moving image and the spatial information related to the image, the size, direction and position when the three-dimensional virtual object is synthesized on the display screen are determined and determined. A three-dimensional virtual object can be synthesized and displayed at the position.

  According to the mobile phone terminal and the information processing server of the present invention, for example, when visiting a place where an object was photographed at a certain past time, a three-dimensional virtual object photographed in the past is placed at an appropriate position in the real space. The images can be displayed in an appropriate size and direction. For this reason, there is an effect that an unspecified number of people can share the place where the object is photographed.

It is a schematic diagram showing an information processing system concerning one embodiment of the present invention. It is a block diagram which shows the structure of the mobile telephone terminal which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the information processing server which concerns on one Embodiment of this invention. It is a block diagram centering on the object processing part of an information processing server. It is explanatory drawing which shows an object table. It is a flowchart which shows operation | movement of the information processing system at the time of object registration. It is explanatory drawing which shows the positional relationship of the user A at the time of object imaging | photography, and an object. It is explanatory drawing which shows the positional relationship of the visual field of a camera part, and an object. It is explanatory drawing which shows an example of the three-dimensional virtual object production | generation screen displayed on the touchscreen part of an information processing server. It is a block diagram centering on the object processing part of an information processing server. It is a flowchart which shows operation | movement of the information processing system at the time of object registration. It is explanatory drawing which shows the example of a transition of the display screen of the user's B mobile telephone terminal. It is explanatory drawing which shows a virtual object (advertisement).

Hereinafter, an embodiment for carrying out the present invention (hereinafter sometimes referred to as “this example”) will be described in the order of the following (1) to (7). The embodiment described below is a preferred specific example of the present invention. Therefore, various technically preferable limitations are attached. However, the scope of the present invention is not limited to these embodiments unless otherwise specified in the following description. For example, the numerical conditions of each parameter mentioned in the following description are only preferred examples, and the arrangement relationship in each figure used for the description is also schematic.
(1) Overall configuration of information processing system (2) Configuration of mobile phone terminal of user A (3) Configuration of information processing server (4) Configuration of object registration processing unit (5) Operation of information processing system during object registration ( 6) Configuration of object distribution processing unit (7) Information processing system operation during object distribution

<Description of an embodiment of the present invention>
[1. Overall configuration of information processing system]
Hereinafter, an embodiment of the present invention will be described with reference to FIGS.
FIG. 1 is a schematic diagram of an information processing system according to an embodiment of the present invention.
The information processing system 101 includes mobile phone terminals 103 and 108 and an information processing server 106, and these elements are connected to each other via a packet network 105.

The user A 102 is a person who provides information for registration in the information processing server 106 using the mobile phone terminal 103.
The mobile phone terminal 103 transmits a moving image captured by a camera unit, which will be described later, and predetermined position information regarding the moving image to the information processing server 106 via the packet network 105. At this time, the user A 102 acquires user A position information (position information where a moving image is captured by the mobile phone terminal 103), which is current position information of the user A 102, from a GPS (Global Positioning System) satellite. This user A position information is also transmitted to the information processing server 106 via the packet network 105.

  The information processing server 106 acquires various information transmitted from the mobile phone terminal 103 via the packet network 105, and based on the information, the object ( A three-dimensional virtual object that is a three-dimensional CG of a house, a building, a tree, a signboard, etc. is generated. Various information transmitted from the mobile phone terminal 103 and the generated three-dimensional virtual object are stored in a database.

  Further, the information processing server 106 searches the database based on the operation of the user B 107 with respect to the mobile phone terminal 108, and if there is hit information, an appropriate three-dimensional virtual object is obtained via the packet network 105. Send to.

User B 107 is a person who receives distribution of predetermined information from information processing server 106 using mobile phone terminal 108.
The mobile phone terminal 108 acquires user B position information, which is the current position information of the user B 107, from the GPS satellite 104 and transmits the user B position information to the information processing server 106 via the packet network 105. Then, a three-dimensional virtual object corresponding to the transmitted user B position information is acquired from the information processing server 106. Then, a three-dimensional virtual object is synthesized and displayed at a predetermined position of the landscape displayed on the display unit through a lens of the camera unit 206 described later.

[2. Mobile phone terminals of users A and B]
FIG. 2 is a block diagram functionally showing the configuration of the mobile phone terminals 103 and 108. FIG. 2 shows only functions used in this embodiment. Therefore, descriptions of functions that are not related to the present embodiment, such as calls and electronic mail, are omitted.

  The mobile phone terminals 103 and 108 include a web communication unit 202, a storage unit 203, an operation unit 204, a display unit 205, a camera unit 206, a drawing unit 207, a GPS reception unit 208, and an attitude information acquisition unit 209. And the control unit 210, and these components are connected to each other so that data can be exchanged.

  The GPS receiving unit 208 receives radio waves received from the GPS satellite 104, and acquires user A position information or user B position information (latitude, longitude, etc., height) of the mobile phone terminals 103 and 108.

  The posture information acquisition unit 209 includes a triaxial acceleration sensor, a gyro sensor, an electronic compass, and the like, and acquires information indicating the current orientation of the mobile phone terminals 103 and 108 (hereinafter “posture information”).

  The camera unit 206 includes an optical lens, an image sensor, and the like. When activated, the camera unit 206 displays a landscape through the optical lens on the display unit 205. In addition, a moving image of the object displayed on the display unit 205 is captured, and the moving image is stored in the storage unit 203 as moving image data. The camera unit 206 calculates the distance between the object and the optical lens (inter-object lens distance) in order to realize the autofocus function.

The control unit 210 performs various arithmetic processes including control of each component of the mobile phone terminal device.
For example, in the mobile phone terminal 108 of the user A 102, when the object is captured by the camera unit 206, feature points are extracted for each frame of the object moving image being acquired. Here, the feature point means a characteristic coordinate in each frame constituting the object moving image, and a plurality of feature points exist in one frame. For example, a feature point is a coordinate such as a corner, a hole position, or a surface center in each frame. Hereinafter, all feature points extracted in each frame of the object moving image are referred to as environment feature point information. Further, the control unit 210 is based on the calculated environmental feature point information, the user A position information acquired by the GPS receiving unit 208, the posture information acquired by the posture information unit, and the object lens distance calculated by the camera unit 206. Thus, the absolute position coordinates of the specific feature point (hereinafter referred to as “feature point absolute position coordinates”) are calculated. However, details of the feature point absolute position coordinates will be described in detail with reference to FIGS. Further, the calculation processing in the control unit 210 of the mobile phone terminal 108 of the user B 107 will be described later with reference to FIG.

  The storage unit 203 includes a ROM (Read Only Memory) and a RAM (Random Access Memory) not shown. The ROM has a control program for the control unit 210 to control each unit, an application program code for performing various initial setting values, and an application program code for performing various processes on moving image data. In addition, program codes for various applications installed in the mobile phone terminals 103 and 108 are stored.

  The web communication unit 202 exchanges data with the information processing server 106 via the packet network 105.

  The display unit 205 is a liquid crystal display that displays object moving image data and scenery through the optical lens of the camera unit 206. The operation unit 204 includes a numeric keypad and a jog dial provided on a casing (not shown) of the mobile phone terminals 103 and 108, and an operation signal generation unit that generates an operation signal corresponding to the operation. .

  Further, the drawing unit 207 is based on the camera unit 206 displayed on the display unit 205 based on the three-dimensional virtual object and other information transmitted from the information processing server 106 on the packet network 105 via the packet network 105. A three-dimensional virtual object is synthesized and displayed at a predetermined position on the input video. Note that the drawing unit 207 is a component required only by the mobile phone terminal 108, but it goes without saying that the mobile phone terminal 103 may include the drawing unit 207.

[3. Configuration of information processing server]
Next, the configuration of the information processing server 106 will be described with reference to FIG.
FIG. 3 is a block diagram functionally showing the configuration of the information processing server 106.
The information processing server 106 roughly includes an object registration processing unit 302, an object distribution processing unit 303, and a nonvolatile data storage unit 304.

  The object registration processing unit 302 generates a three-dimensional virtual object based on each information transmitted from the mobile phone terminal 103 of the user A102, and records the generated three-dimensional virtual object in the nonvolatile data storage unit 304. The internal configuration of the object registration processing unit 302 will be described later with reference to FIG.

  The object distribution processing unit 303 searches the non-volatile data storage unit 304 for a three-dimensional virtual object corresponding to the user B position information transmitted from the mobile phone terminal 108 of the user B 107, and finds the hit three-dimensional virtual object in the mobile phone terminal To 108. The internal configuration of the object distribution processing unit 303 will be described later with reference to FIG.

[4. Object registration processing section]
Next, the object registration processing unit 302 will be described with reference to FIG.
FIG. 4 is a functional block diagram illustrating the configuration of the object registration processing unit 302 and the nonvolatile data storage unit 304 as functions.

The object registration processing unit 302 includes a communication unit 402, a data acquisition unit 403, and an object generation unit 404.
The communication unit 402 exchanges information with the mobile phone terminal 103 of the user A 102 via the packet network 105, and the user A position information, shooting date information, object type information, and object transmitted from the mobile phone terminal 103. The moving image data, environmental feature point information, and feature point absolute position coordinates are received, and the received information is output to the data acquisition unit 403.

  The data acquisition unit 403 prepares the user A position information, shooting date information, object type information, object moving image data, environmental feature point information, and feature point absolute position coordinates input from the communication unit 402 in the nonvolatile data storage unit 304. The object table 405 is stored in a predetermined area, and the object moving image data and the environmental feature point information are output to the object generation unit 404. Details of the object table 405 will be described later with reference to FIG.

  The object generation unit 404 extracts feature points included in the area designated by the user from the environmental feature point information based on the user's operation on the touch panel unit 406 on which the moving image is displayed. Based on the extracted feature points, a three-dimensional virtual object of the object in the designated area is generated from the object moving image data (see Non-Patent Document 1). The three-dimensional virtual object generated in this way is stored in a predetermined area of the object table 405. A specific example of generating the three-dimensional virtual object will be described later with reference to FIG.

Next, the object table 405 will be described with reference to FIG.
FIG. 5 is a diagram illustrating an example of the object table 405.
The object table includes a “position information” field, a “shooting date” field, an “object type” field, an “object movie” field, an “environment feature point information” field, and a “feature point absolute position coordinate” field. , And a “3D virtual object data” field.

The “position information” field stores user A position information indicating the latitude / longitude / height of the position where the user A 102 was at the time of shooting the object moving image.
The “shooting date” field stores shooting date information that is information indicating the date when the object moving image was shot.
The “object type” field stores object type information that is information indicating the type of object of the object moving image.
In the “object moving image data” field, object moving image data shot by the user A 102 with the mobile phone terminal 103 is stored.

  The “environment feature point information” field stores environment feature point information of a moving image. Specifically, the “environment feature point information” field stores, for example, feature points (x1, y1), (x2, y2) of a predetermined frame in order to store information of feature points for each frame of the object moving image. ), (X3, y3),... (Xn, yn) are stored for each frame. That is, if the number of frames of the object moving image is k and the average number of feature points in each frame is n, the number of feature point coordinate information is k × n. Such k × n feature points are environment feature point information.

  The “feature point absolute position coordinates” field stores the feature point absolute position coordinates of the moving image. The feature point absolute position coordinates are information such as (a frame having the target feature point, position information (x, y) of the target feature point in the frame, absolute position coordinates (latitude, longitude, altitude) of the target feature point)). It is configured information. For example, “(15, 76, 34, 35.376000, 139.45, 0.5)” in which the “object type” field of the object table 405 shown in FIG. 5 is a building record is the coordinates (x, y) on the fifteenth frame. ) = (76, 34) means that the absolute position coordinates (latitude, longitude, altitude) indicating the position of the feature point are (35.376000, 139.45, 0.5).

[5. Operation of the information processing system when registering objects]
Next, the operation of the information processing system 101 at the time of object registration will be described with reference to FIG.
FIG. 6 is a flowchart showing the operation of the information processing system 101 at the time of object registration.

  First, the user A102 with the mobile phone terminal 103 arrives at the shooting location (step S1). Then, based on an instruction to the operation unit 204 of the user A102, the GPS reception unit 208 receives radio waves from the GPS satellite 104, and acquires user A position information (latitude / longitude / height) of the user A102 (step S2). ), The user A position information is stored in the storage unit 203.

  When shooting is started by the user A 102, the camera unit 206 acquires object moving image data based on an operation on the operation unit 204 of the user A 102 (step S 3), and outputs it to the control unit 210. At this time, in the camera unit 206, the distance between the object lenses used for focusing on the object is also calculated and output to the control unit 210.

  The control unit 210 calculates the environmental feature point information described above from the object moving image data input from the camera unit 206. The calculated environmental feature point information and object moving image data are stored in the storage unit 203.

  On the other hand, the posture information acquisition unit 209 acquires the posture information of the mobile phone terminal 103 at the time of the process of step S <b> 3 and outputs it to the control unit 210. The control unit 210 calculates the feature point absolute position coordinates based on the distance between the object lenses input from the camera unit 206, the environmental feature point information read from the storage unit 203, and the posture information input from the posture information acquisition unit 209. Calculated and stored in the storage unit 203. A detailed calculation method of the feature point absolute position coordinates will be described with reference to FIGS.

  After the above processing is completed, the control unit 210 acquires the user A position information, the shooting date information acquired by the clock (not shown) during the processing in step S3 and stored in the storage unit 203, object type information, object moving image data, environment The feature point information and the feature point absolute position coordinates are read from the storage unit 203 and output to the web communication unit 202. Then, the web communication unit 202 transmits user A position information, shooting date information, object type information, object moving image data, environmental feature point information, and feature point absolute position coordinates to the information processing server 106 via the packet network 105. (Step S5).

  The user A position information, shooting date information, object type information, object moving image data, environmental feature point information, and feature point absolute position coordinates transmitted from the web communication unit 202 are communicated by the object registration processing unit 302 of the information processing server 106. The data is received by the unit 402 and output to the data acquisition unit 403.

  Then, the data acquisition unit 403 displays the user A position information, shooting date information, object type information, object moving image data, environmental feature point information, and feature point absolute position coordinates in the “position information” field of the object table 405, “shooting date”. The information is stored in the “information” field, the “object type information” field, the “object moving image data” field, the “environment feature point information” field, and the “feature point absolute position coordinate” field (step S7). Then, the object moving image data and the environmental feature point information are output from the data acquisition unit 403 to the object generation unit 404.

  Based on the user's operation on the operation unit 204, the object generation unit 404 extracts only the feature points included in the area designated by the user from the environmental feature point information. Based on the extracted feature points, a three-dimensional virtual object of the object in the designated area is generated from the object moving image data (step S8). The three-dimensional virtual object generated in this way is stored in the “three-dimensional virtual object” field of the object table 405, and the process ends.

Next, calculation processing of each parameter (see FIG. 5) of absolute position coordinates included in the feature point absolute position coordinates performed by the control unit 210 will be described with reference to FIGS.
FIG. 7A is a perspective view showing the positional relationship between the user A102 and the object.
FIG. 7B is a cross-sectional view passing through the position I and the position O of the space shown in FIG.
The position I is the viewpoint of the user A102, that is, the position of the optical lens constituting the camera unit 206 of the mobile phone terminal 103. The position O is an arbitrary point of the object 702 that can be viewed from the viewpoint of the user A102, in this example, an arbitrary one of the feature points in the environmental feature point information. The angle θ is an inclination angle of the optical lens with respect to the ground, and is information included in the posture information acquired by the posture information acquisition unit 209. The altitude H is a distance between the ground and the optical lens, and is information included in the user A position information acquired from the GPS satellite 104 by the GPS receiving unit 208. The distance D is the distance between the object lenses described above, and is information calculated by the camera unit 206.

An expression for obtaining the height h of the position O of the object 702 using these parameters is shown below.
h = H−D cos θ

FIG. 8 is an explanatory diagram showing the positional relationship between the field of view of the camera unit 206 and the object 702.
The camera field of view is a range that can be captured by the camera unit 206 of the mobile phone terminal 103 of the user A102. R in the camera field of view indicates the distance from the position I of the optical lens of the camera unit 206 of the mobile phone terminal 103 to the limit at which shooting can be performed. The angle φ is an angle formed by the straight line connecting the position I and the position O and the north direction axis 803, and is information included in the posture information acquired by the posture information acquisition unit 209.

When the position I is (x: latitude, y: longitude) and the position O is (x1: latitude, y1: longitude), the positional relationship between the position I and the position O is as shown in the following formula. . Note that the latitude x and the longitude y at the position I are information included in the user A position information acquired from the GPS satellite 104 by the GPS receiving unit 208.
x1 = x−Dsinθsinφ
y1 = y + Dsin θ cos φ

  By using the three expressions shown above, the control unit 210 calculates absolute position coordinates (x1: latitude, y1: longitude, h: height) of an arbitrary point of the object 702.

Next, the three-dimensional virtual object generation process (the process of step S8 in FIG. 6) performed by the object generation unit 404 of the information processing server 106 will be described with reference to FIG.
FIG. 9 is an explanatory diagram illustrating an example of a display screen displayed on the touch panel unit 406 during the three-dimensional virtual object generation process in the object generation unit 404. In this example, the case where the three-dimensional virtual object to be generated by the object generation unit 404 is a building 902 is taken as an example.

FIG. 9A is a diagram showing a moving image playback screen that is the basis of a three-dimensional virtual object.
First, as shown in FIG. 9B, the user on the information server side draws lines 903 and 904 on the outline of the building 902 with a finger or a pen on the touch panel unit 406. Then, the object generation unit 404 extracts the feature points of the area surrounded by the lines 903 and 904 from the environmental feature point information. A virtual plane of the part 905 is generated based on the feature points included in the line 903, and a virtual plane of the part 906 is generated based on the feature points included in the line 904.

  Further, this operation is performed on all directions of 360 degrees of the building 902, and planes of all directions of 360 degrees of the building 902 are generated. Then, a three-dimensional virtual object is generated by using all the generated planes and the relationship between the feature points used to generate these planes.

[6. Configuration of object distribution processing unit]
Next, the configuration of the object distribution processing unit 303 will be described with reference to FIG.
FIG. 10 is a block diagram illustrating functional configurations of the object distribution processing unit 303 and the nonvolatile data storage unit 304.

The object distribution processing unit 303 includes a communication unit 1002 and an object search unit 1003.
The communication unit 1002 exchanges information with the mobile phone terminal 108 of the user B 107 via the packet network 105, receives the user B location information transmitted from the mobile phone terminal 108, and receives the received user B location information. Is output to the object search unit 1003. Note that the communication unit 1002 of the object distribution processing unit 303 may be the same as the communication unit 402 of the object registration processing unit 302.

  The object search unit 1003 searches the “position information” field of the object table 405 with the user B position information input from the communication unit 1002, and “environment feature point information” and “feature point absolute value coordinates” of the corresponding record. And the information stored in each field of “3D virtual object” is acquired. The acquired information is transmitted to the mobile phone terminal 108 on the packet network 105 via the communication unit 1002.

[7. Operation of information processing system during object distribution]
Next, the operation at the time of object distribution of the information processing system 101 will be described with reference to FIG.
FIG. 11 is a flowchart showing the operation of the information processing system 101 at the time of object registration.

  First, the user B 107 having the mobile phone terminal 108 arrives at the browsing place (step S10) and activates the camera unit 206. Then, the landscape included in the camera view 802 acquired from the camera unit 206 is displayed on the display unit 205 as shown in the display screen 1202 of FIG.

  Here, based on an instruction to the operation unit 204 of the user B 107, the GPS reception unit 208 receives radio waves from the GPS satellite 104 and acquires user B position information (latitude / longitude / height) of the user B 107 ( Step S11). The acquired user B position information is output to the web communication unit 202.

  The web communication unit 202 transmits the input user B position information to the information processing server 106 via the packet network 105. Then, the communication unit 1002 constituting the object distribution processing unit 303 of the information processing server 106 receives the user B location information transmitted from the web communication unit 202 of the mobile phone terminal 108 (step S12), and the object search unit 1003 receives the information. Is output.

  The object search unit 1003 searches the “position information” field (see FIG. 5) of the object table 405 with the input user B position information (step S13). And when it determines with there being no applicable record (NO of step S13), a process is complete | finished. If it is determined in step S13 that there is a corresponding record (YES in step S13), the object search unit 1003 displays the “environment feature point information” field, “feature point absolute position coordinate” field, and “ The environmental feature point information, the feature point absolute position coordinates, and the three-dimensional virtual object stored in the “three-dimensional virtual object” field are acquired (step S14). Then, the environmental feature point information, the feature point absolute position coordinates, and the three-dimensional virtual object acquired by the object search unit 1003 are transmitted to the web communication unit 202 of the mobile phone terminal 108 via the communication unit 1002.

  Here, the control unit 210 compares the position of the three-dimensional virtual object with the camera field of view 802 (step S15). That is, the feature point of the moving image currently acquired by the camera unit 206 is extracted, and the extracted feature point is compared with the environmental feature point information transmitted from the information processing server 106.

  Then, it is confirmed whether or not there is a three-dimensional virtual object within the range of the camera field of view 802, that is, whether or not there is a feature that matches the extracted feature point in the environmental feature point information (step S16). .

  If it is determined in step S16 that there is no three-dimensional virtual object in the range of the camera field of view 802, that is, if there is no matching feature point in the environmental feature point information (step S16). (NO in S16), the process returns to step S15.

  If it is determined that a three-dimensional virtual object exists within the range of the camera field of view 802, that is, if there is a feature that matches the extracted feature point in the environmental feature point information, the extracted feature point, user Based on the B position information and the feature point absolute position coordinates, the drawing unit 207 determines the size, direction, and position of the three-dimensional virtual object to be combined with the display screen 1202 currently displayed on the display unit 205. Then, as shown in FIG. 12B, the three-dimensional virtual object 1203 is synthesized and displayed on the display unit 205 (step S17), and the process ends.

Next, the process of determining the size, direction, and position of the three-dimensional virtual object, which is the process of step S17, will be described in more detail.
First, the process for determining the size will be described.
In the real world between user B and the three-dimensional virtual object, using the user B position information (latitude, longitude, height) and the absolute position coordinates (latitude, longitude, height) included in the feature point absolute position coordinates The distance is calculated. Then, the size of the three-dimensional virtual object is determined according to the distance.

Next, the process for determining the direction will be described.
The environmental feature point information acquired from the information processing server (downloaded environmental feature point information) is compared with the environmental feature point information of the image currently captured by the camera (see FIG. 12A). Although no object exists in the current image, feature points such as the ground can be acquired. Since the downloaded environmental feature point information includes feature points such as the ground in addition to the feature points of the object, they are compared and a matching frame is calculated. Based on the calculated frame, it is possible to determine how the object is displayed in the frame. Thereby, the direction in which the three-dimensional virtual object is arranged on the display unit 205 is determined. Note that the position on the display unit 205 where the three-dimensional virtual object is arranged corresponds to the position information of the target feature point included in the feature point absolute position coordinates.

  As described above, according to the present embodiment, when the user B goes to the place where the moving image is captured by the user A, the three-dimensional virtual object is placed at an appropriate position on the display unit where the real space is displayed. Can be displayed in an appropriate size and direction. Thereby, there is an effect that an unspecified number of people can share the place where the object is photographed.

  In the above-described embodiment, an example in which the generated three-dimensional virtual object is a building is shown. However, for example, as shown in FIG. 13, a three-dimensional virtual object for an advertisement can be generated and displayed on a display screen. It is. Note that the three-dimensional object is not limited to a building or a signboard, and may be, for example, furniture, a home appliance, a car, or the like.

  In the above-described embodiment, the three-dimensional virtual object to be transmitted to the mobile phone terminal 108 of the user B is determined from all the three-dimensional virtual objects stored in the object table of the information processing server. However, the three-dimensional virtual objects stored in the object table may be transmitted to the mobile phone terminal 108 after being narrowed down by shooting date information and / or object type information. This is because when the user B position information is transmitted from the mobile phone terminal 108 to the information processing server, the shooting date information and / or the object type information is also transmitted, and the object table is stored with the shooting date information and / or the object type information. This can be achieved by narrowing down records.

  In the above-described embodiment, the three-dimensional virtual object stored in the object table in the information processing server is generated from the moving image data based on the user's operation on the touch panel unit. . However, the three-way object may be automatically generated from the moving image data by image analysis of the moving image data.

  As mentioned above, although the example of embodiment of this invention was demonstrated, this invention is not limited to the said embodiment example, Unless it deviates from the summary of this invention described in the claim, other modifications Needless to say, application examples are included.

  DESCRIPTION OF SYMBOLS 101 ... Information processing system, 102 ... User A, 103 ... Mobile phone terminal, 104 ... GPS satellite, 105 ... Packet network, 106 ... Information processing server, 107 ... User B, 108 ... Mobile phone terminal, 202 ... Web communication part, DESCRIPTION OF SYMBOLS 203 ... Memory | storage part, 204 ... Operation part, 205 ... Display part, 206 ... Camera part, 207 ... Drawing part, 208 ... GPS receiving part, 209 ... Attitude information acquisition part, 210 ... Control part, 302 ... Object registration process part, 303 ... Object distribution processing unit, 304 ... Non-volatile data storage unit, 402 ... Communication unit, 403 ... Data acquisition unit, 404 ... Object generation unit, 405 ... Object table, 702 ... Object, 802 ... Camera field of view, 803 ... North Direction axis, 1002 ... Communication unit, 1003 ... Object search unit

Claims (3)

A camera unit that includes an optical lens and acquires light passing through the optical lens as an image;
A display unit for displaying the image acquired by the camera unit;
A control unit that calculates spatial information related to the image from the image acquired by the camera unit;
A GPS receiver that receives radio waves from GPS satellites to obtain mobile phone position information;
A three-dimensional virtual object that is three-dimensional computer graphics generated based on a moving image shot in the past, shooting position information indicating a location where the moving image was shot, and spatial information related to the moving image. The mobile phone position information is transmitted to an information processing server including an object table stored in association with the image capturing position information that matches the transmitted mobile phone position information in the object table. A web communication unit that receives spatial information related to the three-dimensional virtual object and the moving image transmitted from the information processing server;
By utilizing the relationship between the spatial information relating to the moving image and the spatial information relating to the image, the size, direction and position when the three-dimensional virtual object is synthesized on the display screen are determined, A mobile phone terminal comprising: a drawing unit configured to synthesize and display the three-dimensional virtual object at the determined position. The mobile phone terminal according to claim 1, wherein the mobile phone position information and the shooting position information are information including latitude, longitude, and height. A three-dimensional virtual object that is three-dimensional computer graphics generated based on a moving image shot in the past, shooting position information indicating a location where the moving image was shot, and spatial information related to the moving image. An associated object table and
The mobile phone position information transmitted from the mobile phone terminal is received, the shooting position information matching the received mobile phone position information is searched from the object table, and the shooting position information corresponding to the mobile phone position information is retrieved. An information distribution server including: an object distribution processing unit that transmits the three-dimensional virtual object and the spatial information object related to the moving image to the mobile phone terminal;
A camera unit that includes an optical lens and acquires light passing through the optical lens as an image;
A display unit for displaying the image acquired by the camera unit;
A control unit that calculates spatial information related to the image from the image acquired by the camera unit;
A GPS receiver that receives radio waves from GPS satellites to obtain mobile phone position information;
A web communication unit that transmits the mobile phone position information to the object distribution processing unit and receives the spatial information related to the three-dimensional virtual object and the moving image transmitted from the object processing unit;
By using the relationship between the spatial information related to the moving image and the spatial information related to the image, the size, direction and position when the three-dimensional virtual object is synthesized on the display screen are determined and determined. An information processing system comprising: a mobile phone terminal including a drawing unit that synthesizes and displays the three-dimensional virtual object at the position.

Images