WO2006043319A1

WO2006043319A1 - Terminal and server

Info

Publication number: WO2006043319A1
Application number: PCT/JP2004/015497
Authority: WO
Inventors: Taizo Anan
Original assignee: Fujitsu Limited
Priority date: 2004-10-20
Filing date: 2004-10-20
Publication date: 2006-04-27

Abstract

A technique of accurately computing information on the position of a terminal which acquires a video is provided. A technique of effectively superimposing a video of a view on a video of information on the view is also provided. A video acquiring section (801) acquires a video of a label having a calibration pattern for computing the direction in which the label is imaged and the distance to the label. A calibration computing section (805) computes, from the calibration pattern, the direction in which the label is imaged and the distance to the label.

Description

Specification

Terminal and server

Technical field

[0001] The present invention relates to a technique for acquiring information in a video. The present invention also relates to a technique for realizing augmented reality.

Background art

[0002] Today, the World Wide Web is being built on the Internet. Anyone can easily obtain information. In addition, information acquisition services such as i-mode (registered trademark) are also provided in terminals such as mobile phones and personal digital assistants (hereinafter referred to as PDAs).

[0003] However, in such acquisition of information, it is difficult to directly acquire real-world information spreading in front of the user's eyes. For example, when the user wants to know the service menu of a restaurant in front of him, he performs the following process. First, the user launches a mobile phone or PDA browser that he owns. The user then accesses the Internet. The user then searches for a search engine. Then, the user searches for the name of the restaurant in front of the user using a search engine. Then, the user confirms the contents of the restaurant information of the same name that has been searched multiple times.

[0004] On the other hand, a system that allows a user to easily acquire real-world information spreading in front of the user's eyes has been developed. This system synthesizes a virtual world image and a real world image drawn with computer graphics (CG).

[0005] This technology is called mixed reality or augmented reality. A user usually wears a glasses-type display device called a head-mounted display. With this technology, information related to the landscape in front of the user is searched using the geographical position of the user, the face orientation, and information in the landscape image in front of the user as keys. And it becomes CG by the information power display device about the scenery spreading in front of the user. This CG is synthesized with the actual video. And this real image and Video power synthesized with CG Presented to the user.

[0006] Using such a system, the user views a restaurant in front of the user's eyes through a head mounted display. And the user can get the restaurant information instantly by CG.

[0007] On the other hand, other applications of the technology using mixed reality using a head mounted display include the following. For example, this technology displays a guide that points to the parts that need to be inspected for workers looking at a broken machine. In addition, when this technology is applied to the medical field, an ultrasonic image of the internal organs is superimposed on the image of the surface of the patient's body and displayed to the observer. For this reason, this technique has the effect that an observer such as a doctor is performing power surgery while looking at X-rays. The technology also displays information about the patient's blood type and the drug being administered. Therefore, this technology presents information that prevents medical errors.

[0008] A major problem of the system for realizing such a mixed reality is to superimpose the real-world video and the CG video with high accuracy. The composite reality system detects the position and orientation of the user's head so that the superimposed virtual world image overlaps the real world image in the field of view with high accuracy.

[0009] The techniques described in Patent Document 2 to Patent Document 4 describe a method of superimposing images using a magnetic sensor. On the other hand, in the technique described in Patent Document 1, a server is inquired about acquisition of information based on an image obtained by a portable camera terminal. That is, Patent Document 1 describes a means for a terminal to obtain desired information from a server.

Patent Document 1: Japanese Patent Laid-Open No. 9-330336

Patent Document 2: JP 2000-102036 A

Patent Document 3: Japanese Patent Laid-Open No. 2002-259992

Patent Document 4: Japanese Patent Laid-Open No. 2002-271691

Patent Document 5: Japanese Unexamined Patent Publication No. 2000-276613

Patent Document 6: Special Table 2003—533815

Patent Document 7: JP-A-10-267671 Non-Patent Document 1: Takashi Matsuyama, Yoshinori Kuno, Ito, “Computer Vision Technology Review and Future Prospects”, Chapter 3, New Technology Communications

Disclosure of the invention

Problems to be solved by the invention

[0010] The technique described in Patent Document 1 is a technique for obtaining information on a restaurant XXX in front of a user whose terminal is in Shibuya, for example. To that end, (1) the terminal recognizes that it is in Shibuya using the Global Positioning System (hereinafter referred to as GPS). Then, (2) the terminal takes a picture of the restaurant XXX. Then, (3) the terminal sends the information recognized in (1) and the captured information to the server. (4) The terminal receives the information about the restaurant XXX from Sanoku.

[0011] In the technique described in Patent Document 1, the terminal acquires information about a restaurant in front of the user through the steps (1) to (4). However, in the technique described in Patent Document 1, the image recognition unit generates a search key for information. Therefore, (a) the technique described in Patent Document 1 must register, for example, signboards of restaurants in Shibuya in the database. In addition, the technique described in (b) Patent Document 1 cannot be applied to a signboard or the like that has relatively easy image recognition. That is, the observed landscape varies depending on the viewing direction even at the same position. Therefore, when the technique described in Patent Document 1 is applied to landscapes, a lot of images must be taken at the same point and registered in the database. As described above, the technique described in Patent Document 1 has a problem that it can be used only for very limited purposes.

[0012] The technique described in Patent Document 2 to Patent Document 4 and the technique described in Patent Document 7 use a magnetic sensor. The technique described in Patent Document 6 uses an acceleration sensor.

[0013] With the techniques described in Patent Document 2 to Patent Document 4, Patent Document 6, and Patent Document 7 described above, the tilt and direction of the terminal are required. However, the techniques described in Patent Document 2 to Patent Document 4 and the techniques described in Patent Document 6 and Patent Document 7 cannot obtain information in the height direction.

[0014] Therefore, with these technologies, for example, in a building, different information is obtained on each floor. I can't. In addition, the accuracy of GPS that estimates the geographical location information of the observer has an error of several tens of meters even at the current level. Even if the tilt of the terminal can be determined with high accuracy using a magnetic sensor, the range of the video captured by the camera of the terminal can be estimated using only the GPS position information and the tilt of the terminal using the magnetic sensor. It is difficult to adjust the position of the actual landscape image accurately.

[0015] On the other hand, there is a method for estimating the geographical position of a terminal with high accuracy by a method such as embedding an IC tag in the ground. The accuracy of estimating the geographical location of the terminal is improved by using the IC tag. However, there are limits to the improvement in accuracy, and it is difficult to estimate the video captured by the terminal camera using this method.

[0016] The technique described in Patent Document 1 does not manage information based on time. For this reason, the technology described in Patent Document 1 provides user-friendly functions such as presenting information related to restaurants as information presented at noon, and removing information power at restaurants that do not provide lunch. Have a cunning,.

[0017] An object of the present invention is to provide a technique for accurately calculating information related to the position of a terminal that acquires video. Furthermore, an object of the present invention is to provide a technique for effectively superimposing a landscape video and a video of information related to the landscape.

Means for solving the problem

[0018] In order to achieve the above object, the terminal of the present invention provides:

Video acquisition means for acquiring a video of the shooting target having a pattern for calculating a direction of shooting the shooting target and a distance between the shooting target;

Computation means for calculating from the image of the pattern the direction in which the subject is photographed and the distance between the subject.

[0019] Further, the terminal of the present invention includes:

The pattern is the same pattern as a calibration pattern for calibrating the terminal,

The calculation means calculates a direction between the shooting target and a distance between the shooting target from the image of the pattern and the parameters obtained by the calibration. [0020] In addition, the terminal of the present invention,

Distance measuring means for measuring the distance to the object to be photographed is provided.

The terminal of the present invention

Position measuring means for measuring the position of the terminal is provided.

[0021] Further, the terminal of the present invention includes:

The imaging target includes identification information for identifying a plurality of imaging targets from each other.

[0022] Further, the terminal of the present invention provides:

Connected by a network to a server that calculates the direction in which the subject was photographed and the distance between the subject to be photographed based on the pattern that the subject has.

The image of the pattern acquired by the image acquisition means and the parameter obtained by the calibration performed in advance are transmitted to the server, and the imaging object calculated by the server based on the image and parameter of the pattern Inquiry means for receiving from the server the direction of shooting and the distance to the shooting target.

[0023] In addition, the terminal of the present invention,

The terminal power is connected by a network to a server that selects information to be transmitted to the terminal based on the received information,

Transmitting means for transmitting to the server at least one of the direction in which the subject is photographed and the distance between the subject and the image of the pattern and the parameter obtained in advance by calibration. When,

Receiving means for receiving information transmitted from the server.

[0024] Further, the terminal of the present invention includes:

The transmission means includes

At least one of the position of the terminal and identification information for identifying each of a plurality of shooting targets included in the shooting target is transmitted to the server.

[0025] Further, the terminal of the present invention includes:

Combining means for combining the video acquired by the video acquiring means and the video based on the information received by the receiving means is provided.

[0026] In addition, the terminal of the present invention, Display means for displaying the video synthesized by the synthesizing means as a computer graphics image.

[0027] In addition, the terminal of the present invention,

Display means for displaying the information received by the receiving means as a web page is provided.

[0028] Further, the terminal of the present invention includes:

A video tracking unit configured to detect a distance moved by the terminal and a direction moved by the terminal based on a change in video acquired by the terminal, which is generated in accordance with the movement of the terminal; The video acquired by the means and the information displayed on the video are combined based on the distance moved by the terminal detected by the video following means and the direction in which the terminal moved.

[0029] Further, the server of the present invention includes:

The terminal receives at least one of the direction in which the subject was photographed and the distance between the subject and the image of the subject to be photographed including the pattern and the parameters obtained by the calibration performed in advance. Receiving means;

Information management means for extracting information from data storage means for storing information based on the information received by the receiving means;

Transmitting means for transmitting the information extracted by the information management means to the terminal.

[0030] Further, the server of the present invention includes:

The receiving unit receives at least one of identification information for mutually identifying a position of the terminal and a plurality of shooting targets included in the shooting target,

The information management means is at least one of the information transmitted from the terminal received by the receiving means and the identification information for mutually identifying the position of the terminal and a plurality of photographing objects included in the photographing object To extract information from the data storage means.

[0031] Further, the server of the present invention provides:

The information stored in the data storage means is

Based on the information and time received by the receiving means,

The information management means includes information received by the receiving means, a time when the information is received, To extract information from the data storage means.

The invention's effect

[0032] The present invention calculates the distance between the shooting direction of the shooting target and the shooting target based on the pattern acquired by the terminal. Therefore, according to the present invention, the orientation of the terminal and the distance to the subject can be accurately calculated.

In the present invention, the pattern is the same pattern as the calibration pattern for calibrating the terminal. Then, the calculation means calculates the distance between the direction in which the photographing target is photographed and the photographing target from the image of the pattern and the parameters obtained by the calibration. Therefore, the present invention can accurately calculate the orientation of the terminal and the distance to the shooting target.

[0034] Further, according to the present invention, the distance between the terminal and the object to be imaged can be accurately measured by the distance measuring means. In addition, according to the present invention, the position of the terminal can be accurately measured by the position measuring means.

In addition, according to the present invention, it is possible to identify a subject to be photographed based on identification information. Further, the present invention causes the server to calculate the direction in which the photographing object is photographed and the distance between the photographing objects. Therefore, the present invention can reduce the load on the terminal.

[0036] Further, according to the present invention, the server is based on at least one of the direction in which the subject is photographed and the distance between the subject and the parameters obtained by the pattern image and the calibration. Send information to the terminal. Therefore, according to the present invention, the terminal can receive appropriate information from the server.

[0037] Further, in the present invention, the terminal transmits at least one of identification information for identifying the position of the terminal and a plurality of shooting targets included in the shooting target to the server. Then, the server selects information to be transmitted to the terminal using at least one of the identification information for mutually identifying the position of the terminal received from the terminal and the plurality of imaging targets included in the imaging target. Therefore, according to the present invention, the terminal can receive appropriate information from the server.

In the present invention, the synthesizing unit synthesizes the video acquired by the video acquiring unit and the video based on the information received by the receiving unit. For this reason, the present invention provides information received by the user. It can be recognized by the synthesized video.

[0039] Further, according to the present invention, the display means displays the video synthesized by the synthesizing means as a computer dullix image. Therefore, the present invention can improve the visibility of the synthesized video.

[0040] In the present invention, the display means displays the information received by the receiving means as a web page. Therefore, according to the present invention, the user can accurately grasp the displayed information.

[0041] Further, according to the present invention, the synthesizing unit detects the distance moved by the terminal detected by the video tracking unit and the moving of the video acquired by the video acquiring unit and the video based on the information received by the receiving unit. Based on the orientation. Therefore, according to the present invention, the synthesized video can take into account the movement of the terminal.

[0042] Further, according to the present invention, the information management means extracts information from the data storage means for storing information based on the information received by the receiving means. Then, the server transmits the information extracted by the information management means to the terminal. Therefore, according to the present invention, the server can transmit information corresponding to the information transmitted from the terminal to the terminal.

[0043] Further, according to the present invention, the information management unit includes identification information for identifying the information transmitted from the terminal received by the receiving unit and the position of the terminal and a plurality of shooting targets included in the shooting target from each other. Using at least one of them, data storage means force information in which information is stored is extracted. Therefore, according to the present invention, information transmitted from the server to the terminal can be made appropriate.

In the present invention, the information stored in the data storage means is classified based on the information received by the receiving means and the time. In the present invention, the information management means extracts information from the data storage means based on the information received by the receiving means and the time. Therefore, the present invention can consider time as information to be transmitted to the terminal. Brief Description of Drawings

FIG. 1 is a conceptual diagram of a method for measuring the orientation of a camera with respect to a label and the distance between the camera and the label in the first embodiment of the terminal of the present invention.

FIG. 2 is a schematic diagram showing an example of a label used in the first embodiment of the terminal of the present invention. The

圆 3] It is a block diagram of an internal configuration of the first exemplary embodiment of the terminal of the present invention.

圆 4] It is a block diagram of the internal configuration of the second embodiment of the terminal of the present invention.

[5] FIG. 5 is a block diagram of an internal configuration of the third exemplary embodiment of the terminal of the present invention.

[6] It is a schematic diagram of an example of a label used in the fourth embodiment of the terminal of the present invention. [7] FIG. 7 is a schematic diagram of an information providing system in which the fifth embodiment of the terminal of the present invention is used.

[8] FIG. 8 is a schematic diagram of an information providing system in which the sixth embodiment of the terminal of the present invention is used.

[9] FIG. 9 is a conceptual diagram of an image displayed in the sixth embodiment of the terminal of the present invention.

[10] It is a schematic diagram of an information providing system in which the seventh embodiment of the terminal of the present invention is used.

圆 11] It is a conceptual diagram for explaining the operation of the video follower in the seventh embodiment of the terminal of the present invention.

圆 12] It is a conceptual diagram for explaining the operation of the video follower in the seventh embodiment of the terminal of the present invention.

[13] FIG. 13 is a block diagram of the internal configuration of the first exemplary embodiment of the server of the present invention.

FIG. 14 is a block diagram of an internal configuration of a second embodiment of the server of the present invention.

15] A conceptual diagram of information stored in the data storage unit of the second embodiment of the server of the present invention.

Explanation of symbols

301 Video acquisition unit

302 Calibration calculator

401 Video acquisition unit

402 Calibration calculator

403 Distance measurement unit

501 Video acquisition unit

502 Calibration calculator 503 Distance measurement unit

504 Geographical Location Measurement Unit

601 ID

701 devices

702 network

703 server

704 Video acquisition unit

705 Calibration inquiry section

706 Geographic Location Measurement Unit

707 Network Communication Department

801 terminals

802 network

803 server

804 Video acquisition unit

805 Calibration calculator

806 Calibration inquiry section

807 Geographical Location Measurement Unit

808 Server inquiry department

809 Query result receiver

810 Video composition unit

811 Display

812 Network Communication Department

901, 902, 903 Information

904 Labenore

905 web page

1001 terminal

1002 network

1003 server 1004 Video acquisition unit

1005 Calibration calculator

1006 Calibration inquiry section

1007 Geographical location measurement unit

1008 Server inquiry department

1009 Inquiry result receiver

1010 Video composition unit

1011 Display

1012 Network communication department

1013 Video follower

1301 Calibration inquiry receiver

1302 Calibration calculator

1303 Server inquiry receiver

1304 Query result transmitter

1305 Link Information Management Department

1306 Data storage

1307 Network communication department

1401 Calibration inquiry receiver

1402 Calibration calculator

1403 Server inquiry receiver

1404 Query result transmitter

1405 Time acquisition unit

1406 Time Link Information Management Department

1407 Data storage

1408 Network Communication Department

BEST MODE FOR CARRYING OUT THE INVENTION

(First embodiment of terminal)

A first embodiment of the terminal of the present invention will be described. The terminal of this embodiment is a camera Is provided. A mobile phone is used as the terminal of this embodiment. In addition, the terminal of the embodiment of the present invention other than the first embodiment and the terminal used in each embodiment of the server of the present invention are also assumed to be mobile phones. In addition, the functions of the respective units of the embodiments of the terminal and server of the present invention described below may be realized by appropriately cooperating with, for example, a program stored in the CPU power memory and other devices. . In the terminal used in each embodiment of the present invention and the server described below, the calibration is assumed to be performed in advance at a camera manufacturing factory, for example.

[0048] Then, the terminal of the present embodiment uses a label including a calibration pattern (hereinafter also simply referred to as a label), and the orientation of a camera (hereinafter also simply referred to as a camera) mounted on the terminal, Measure the distance between the camera and the label.

[0049] Here, measurement of the above-described camera orientation and the distance between the camera and the label will be described. The method described in Non-Patent Document 1 can measure the orientation of the camera with respect to the label and the distance between the camera and the label. Also in this embodiment, the method described in Non-Patent Document 1 is used.

[0050] Hereinafter, a method of measuring the orientation of the camera with respect to the label and the distance between the camera and the label by the terminal of the present embodiment will be described with reference to FIG. FIG. 1 is a conceptual diagram of a method for measuring the orientation of a camera with respect to a label and the distance between the camera and the label in the first embodiment of the terminal of the present invention. However, the method of measuring the orientation of the camera with respect to the label and the distance between the camera and the label, which will be described below, is a terminal of an embodiment other than the first embodiment of the terminal of the present invention, and The present invention can also be applied to each embodiment of the server of the present invention.

[0051] Definition

(X, y, z): coordinates of point P in world coordinates

(X, y, z): coordinates of point P in camera coordinates

O: Camera lens center

f: Focal length

(X, Y): Image coordinates of point P

(X, Y): Actual image coordinates due to lens geometric distortion (X, Y): Image coordinates discretized with digital image

f f

s: Scale function

d, d: Distance between CCD elements in the X and Y directions

X y

N, Ν: Number of CCD elements in Χ direction, number of samples per scan line

cx fx

Let d,: d, = d (N ZN).

X x x cx fx

[0052] Preparation

As shown in Fig. 1, the coordinates of the point P in the reference world coordinate O are (X, y, z w— xw— w ~~ zw w w w

) And the coordinates of the point P in the camera coordinates O are represented as (X, y, z).

However, O represents the lens center (pinhole), and the z-axis is set to match the optical axis of the lens.

[0053] Next, consider an image plane parallel to the XY plane and having an image center of O and a coordinate axis of XY at the position of the z coordinate force ¾. In the case of an ideal pinhole camera, the coordinates (X, y, z) of the point P are expressed as (X, Y) on the image plane, but in reality the image plane is affected by the geometric distortion of the lens. (X, Y)

d It shall correspond to the position of d.

[0054] The discrete coordinates on the digital image of (X, Y) are represented by (X, Y). Point P d d f f

The relationship between the world coordinates (χ, y, z) and the image coordinates (X, Y) of the point is obtained by the following procedure: w w w f r

It can be done.

[0055] step. 1 Conversion from (x, y, z) to (x, y, z)

W W W

This is expressed using the rotation matrix R and translation vector T.

[0056] [Equation 1]

[0057] However,

[0058] [Equation 2] (2)

[0059] step. 2 Conversion from (x, y, z) to (X, Y ^) by perspective transformation:

Expressed using focal length f.

[0060] [Equation 3] r X r- y

xu = f one, Y _u = f- (3)

[0061] step. 3 Conversion from (X, Y) to (X, Y):

u u d d

[0062] [Equation 4]

^X d ^{+ D} x ^{= x} u- ^Y d ^{+ D} y ^{= Y} u (4)

[0063] where D 1 and D 2 are expressed as follows using distortion coefficients κ and κ in the radial direction of the lens as follows:

[0064] 隱 5]

D _x = Xd (c ₁ r ² + K ₂ r ^A )

/ twenty two

厂 Φά ^{+ Y} d

[0065] step .4 Conversion from (X, Y) to (X, Y):

d d f f

[0066] 隱 6] Xf = s _x d ' _x ' X _d + C

[0067] where s is the scale factor, (C _x , C) is the origin coordinate on the digital image, and ά _χ , respectively, represents the interval between the number of CCD elements in the X and Y directions. d is a correction of d using N in the X direction and N samples per scan line.

[0068] [Equation 7]

d

[0069] To organize the above relationship,

[0070] country

X 2 X _f -CY = Y f ν

Using [0071]

[0072] [Equation 9] r ⁴ )

(9)

[0073] [Equation 10] dyY + d _y Y ( ₁ r ² + / c ₂ r ⁴ )

(Ten)

[0074] is obtained. However,

[0075] [Equation 11]

[0076]

[0077] Calibration

As shown in Fig. 1, calibration is performed with the world coordinates (X, y, z) of many points and their

W W W

Image coordinates corresponding to a point (X, Y)

This is done by giving a pair to f f to the camera. Here, the world coordinates (X, y, z) of many points are known values. The calibre of this embodiment

w w w

In the simulation, the rotation matrix R, translation vector T, external parameters, focal length f, lens distortion coefficients κ1, κ2, scale factor s, and image origin (C, C) are obtained as external parameters. These external parameters and internal parameters are collectively referred to simply as parameters hereinafter. In the following, the calibration procedure when s (usually 1.0) is assumed will be described.

[0078] step.1

Find the P image coordinates (X, Y) from the image camera that captured the N calibration points on the same plane.

fi fi

[0079] step.2

N, N, d ', d are obtained from the specifications of the camera and AZD transformation.

cx fx X y

[0080] step.3

Let the origin coordinate at the center of the image be (C, C).

X y

[0081] step.4 Find (X, Y) of N points.

[0082] [Equation 12]

厂^G x

(1 2) Ydi ^{= d} _y ( ^Y fi- ^c _y )

[0083] step.5

x ^_1 rl, Τ ^_1 r2, Τ ^_1 Τ, Τ ^_1 r4, Τ— "τ5 is unknown wi wi wi di di yyyxyy

Solve linear equations with numbers.

[0084] [Equation 13]

[0085] step.6

T ^_1 rl, T ^_1 r2, T ^_1 T, T ^_1 r4, T— "Find T ² from r5.

y y y x y y y

[0086] [Equation 14]

[0087] If the row or column element of matrix C is not 0,

[0088] [Equation 15] ^Sr V ^S r " ^{4 (r} 'i ^r ' 5— ^R '4 2) ^:

2 (1 5)

^{2 (Γ} Ί r'5 '4 ^r ' 2)

[0089] is obtained. However, r 'and r' represent row or column elements that are not 0 in the matrix C.

1 J

[0090] step.7

Image origin (C, C)

A point (X, Y) well away from X

fi world coordinates of fi and its corresponding points (X

, Y, ζ) determine the sign of Τ. First, let the sign of Τ be positive,

wi wi y y

[0091] [Equation 16]

Γ _ί = (T _y η), ,, r ₂ = (T _y r ₂ ) T _y

^T x 2 ^(T y ' ^T x ^{) T} y (1 6) x = r _lXw + r ₂ y _w + T _x

y = r ₄ x _w + r ₅ y _w + T _y

[0092] If the signs of χ and X are equal and y and Y are equal, the sign of T is positive,

y

Otherwise, it is negative.

[0093] step.8

r, r, r, r force Determine the rotation matrix R.

[0094] [Equation 17]

[0095] However, s = -sgn (rr + rr). r, r, r have the property that R is an orthogonal matrix

1 4 2 5 7 8 9

Use to find. If the focal length f obtained in step 9 is negative,

[0096] [Equation 18]

(1 8)

[0097] is used.

[0098] step. 9

Solve linear equations with f and Τ as unknowns, ignoring lens distortion (κ = κ = 0)

1 2 z

[0099] [Equation 19]

[0100] However,

[0101] [Equation 20] yj = r4x _wi + r5y _wj + Ty

20) W: = r7x _wj + r8y _wj step. 10

Using f, T and K = K = 0 obtained in step.9 as initial values, Equation (10) is nonlinearly optimized z 1 2

Solve as a problem and find ί, τ, κ, κ. This completes the explanation of calibration.

[0103] Next, the first embodiment of the terminal of the present invention will be described in more detail. This embodiment The labels used in the state are shown in Figure 2. FIG. 2 is a schematic diagram showing an example of a label used in the first embodiment of the terminal of the present invention. However, the description of the label described below can be applied to the terminal of the embodiment other than the first embodiment of the terminal of the present invention and each embodiment of the server of the present invention.

[0104] As shown in Fig. 2, in order to estimate the direction of the camera as well as the image power captured by the camera, the label is a pattern having a plurality of points, for example. This pattern is a calibration pattern. That is, the label shown in FIG. 2 is the same as the label having the calibration pattern used to set the camera parameters by calibration.

In this embodiment, a label having such a calibration pattern is arranged at each point in the real world. The user then captures the label with a calibrated camera mounted on the terminal. Then, the terminal calculates the direction of the camera relative to the label and the distance between the label and the camera. Then, the terminal that recognizes the orientation of the camera with respect to the label and the distance between the label and the camera searches the information expressed in CG corresponding to the scenery in front of the user's eyes with a predetermined database power. The terminal then synthesizes the information expressed in CG with the landscape video and displays it to the user through the display.

[0106] As a method for obtaining the orientation of the camera with respect to the label and the distance between the camera and the label, for example, there is a method disclosed in Non-Patent Document 1 described above. In this embodiment, the method shown in Non-Patent Document 1 is used. By this method, the terminal of this embodiment can determine the distance between the label and the force lens and the camera orientation. That is, the terminal of the present embodiment determines in advance a meter for determining the orientation of the camera with respect to the label and the distance between the label and the camera by calibration. Then, the terminal of the present embodiment determines the orientation of the camera with respect to the label (that is, the angle of the camera with respect to the label) and the label and the camera based on the determined parameter and the image obtained by capturing the calibration label. Find the distance.

Next, the internal configuration of the first exemplary embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 3 is a block diagram of the internal configuration of the first embodiment of the terminal of the present invention. Implementation As shown in FIG. 3, the terminal in the form includes a video acquisition unit 301 and a calibration calculation unit 302.

[0108] The video acquisition unit 301 acquires data of a captured video. The video acquisition unit 301 is configured by a camera. Then, the calibration calculation unit 302 calculates the orientation of the camera with respect to the label and the distance between the label and the camera from the calibration pattern shown in FIG. 2 existing in the acquired video. This calculation is a method for obtaining the world coordinates (X, y, z) of the point P from the image coordinates (X, Y) shown in the aforementioned Non-Patent Document 1.

f f w w w

Is done. That is, the terminal according to the present embodiment executes the calibration shown in Non-Patent Document 1 described above in advance. Then, the terminal of the present embodiment stores the camera parameters calculated by the calibration in a memory (not shown). Then, the terminal according to the present embodiment uses this parameter to calculate the orientation of the camera with respect to the label and the distance between the label and the camera.

[0109] The label used in this embodiment may be a three-dimensional object! /. In this case, the calibration pattern is changed for each surface of the three-dimensional label in order to know which surface the camera captures. The camera may be an infrared camera. The label may emit light or radio waves having a special wavelength.

[0110] Thus, the first embodiment of the terminal of the present invention acquires a video of a label on which a calibration pattern is formed. Then, the terminal calculates the direction of the camera with respect to the label and the distance between the camera and the label as well as the calibration turn force of the acquired video. Therefore, in this embodiment, the direction of the terminal on which the camera is mounted and the distance from the terminal label can be accurately measured.

[0111] (Second Embodiment of Terminal)

Next, a second embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 4 is a block diagram of the internal configuration of the second embodiment of the terminal of the present invention.

[0112] The terminal of the present embodiment is characterized in that a distance measurement unit 403 is newly provided as compared with the terminal of the first embodiment described above. That is, the function of the video acquisition unit 401 is the same as the function of the video acquisition unit 301 shown in FIG. The function of the calibration calculation unit 402 is the same as the function of the calibration calculation unit 302 shown in FIG. In this embodiment, a highly reliable distance measuring unit 403 is provided. For this reason, the terminal of this embodiment can calculate the distance between the label and the camera with high accuracy by the distance measuring unit 403. In addition, this embodiment can also measure distances using only labels, like the terminal of the first embodiment described above. The distance measuring unit 403 can use an existing one, such as triangulation using millimeter radar or stereo vision.

[0114] Thus, the terminal of the present embodiment can obtain the same effects as those of the first embodiment of the terminal of the present invention described above, and accurately measures the distance between the terminal and the label. be able to.

[0115] (Third embodiment of terminal)

Next, a third embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 5 is a block diagram of the internal configuration of the third embodiment of the terminal of the present invention.

[0116] This embodiment is characterized in that the terminal is provided with a geographical position measurement unit 504, as compared with the second embodiment of the terminal of the present invention described above. That is, the function of the video acquisition unit 501 is the same as the function of the video acquisition unit 401 shown in FIG. The function of the calibration calculation unit 502 is the same as the function of the calibration calculation unit 402 shown in FIG. The function of the distance measurement unit 503 is the same as the function of the distance measurement unit 403 shown in FIG.

[0117] The geographical position measurement unit 504 is a sensor for acquiring position information such as GPS.

The geographical location measurement unit 504 is not limited to GPS, but can be implemented using other methods such as a method of measuring the geographical location using an IC tag, a wireless LAN, or a mobile phone base station.

As described above, in the terminal according to the present embodiment, the geographical position information indicating the position of the terminal on the map is measured by the geographical position measuring unit 504. For this reason, the terminal of this embodiment can obtain the same effects as those of the first embodiment or the second embodiment of the terminal of the present invention described above, and the location information ability of the geographical location measuring unit 504 can be determined in any region. You can see the label by shooting the label.

[0119] (Fourth embodiment of terminal)

Next, a fourth embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 6 is a schematic diagram of an example of a label used in the fourth embodiment of the terminal of the present invention. The configuration of the terminal according to the present embodiment is the same as the configuration of the terminal according to the first embodiment of the present invention described above. However, in this embodiment, as a terminal to be used, any one of the terminals described in this specification other than the first embodiment may be used.

[0120] The terminal of this embodiment is characterized by a calibration label photographed by the terminal.

As shown in the ID (identification) 601 of FIG. 6, the label of this embodiment describes the label ID 601 in the calibration label. This ID 601 is composed of numbers, symbols, characters, figures, or any combination thereof that can be described on the label. The ID 601 shown in FIG. This ID 601 is used to distinguish it from other labels. Therefore, in this embodiment, the calibration calculation unit of each terminal recognizes the ID 601 in the video. Then, the calibration calculation unit identifies the label that the terminal is photographing by ID. In addition, the terminal of the present embodiment stores a table for specifying a label corresponding to the ID in the memory.

In the present embodiment, the same ID 601 may exist as long as the label ID 601 is different in the region. For example, if ID601 is 1, the same ID label may exist in Shinjuku and Sapporo. In this case, a means for acquiring location information such as GPS is used to distinguish the region where the terminal is located. According to this GPS position information, in this embodiment, it is possible to distinguish whether the label ID 601 is from Sapporo or Shinjuku.

Thus, in this embodiment, ID 601 is included in the label used by the terminal. Therefore, the terminal of the present embodiment can specify the label from which the video has been acquired based on ID601. Therefore, the terminal of the present embodiment has the same effect as that of the first embodiment of the present invention, or when the terminal of any of the other embodiments other than the first embodiment is used. As long as the same effect as the terminal of any of the other embodiments other than the form can be obtained, and the correspondence between ID601 and the region is known, the label of the region where the terminal is from ID601 of the label You can see if you are acquiring video. Therefore, in this embodiment, labels having the same calibration pattern can be used.

[0123] (Fifth embodiment of terminal)

Next, a fifth embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 7 is a schematic diagram of an information providing system in which the fifth embodiment of the terminal of the present invention is used.

[0124] The information providing system shown in FIG. 7 is a terminal 7 according to a fifth embodiment of the terminal of the present invention. 01, a network 702, and a server 703. Further, even if the label from which the terminal 701 of the present embodiment acquires an image is the label shown in FIG. 2, it is shown in FIG. 6 as described above in the fourth embodiment of the terminal of the present invention. A label with ID 601 may be used.

The terminal 701 includes a video acquisition unit 704, a calibration inquiry unit 705, a geographical position measurement unit 706, and a network communication unit 707. Note that the video acquisition unit 704 shown in FIG. 7 has the same function as the video acquisition unit 301 shown in FIG. Further, the geographical position measuring unit 706 shown in FIG. 7 has the same function as the geographical position measuring unit 504 shown in FIG.

[0126] When assuming a small portable terminal 701 (for example, a mobile phone), the CPU of the terminal 701 is often inferior in performance compared to the CPU of a large terminal. Therefore, it is better to have a calculation such as calibration performed by a server having a higher computing capacity than the terminal via the network rather than the terminal.

The calibration inquiry unit 705 shown in FIG. 7 causes the server 703 to calculate the calibration. Therefore, the calibration inquiry unit 705 transmits an image including the calibration pattern to the server 703. In this case, the calibration inquiry unit 705 transmits the camera parameters obtained by the calibration to the server 703.

In addition, the network communication unit 707 controls communication between the terminal 701 and the server 703.

As described above, the terminal 701 transmits the video including the calibration pattern to the server 703 via the network 702.

Server 703 receives the video including the calibration pattern transmitted from terminal 701. The server 703 receives camera parameters obtained by calibration transmitted from the terminal 701. Then, the server 703 calculates the camera direction and the distance between the label and the camera based on the received video and the camera parameters. Then, the server 703 transmits the calculation result to the calibration inquiry unit 705 via the network 702 and the network communication unit 707. The calibration inquiry unit 705 receives the calculation result from the server 703. [0130] The network 702 may be a mobile phone network! /, Or a network such as a wireless LAN. The network communication unit 707 can include one or more available networks.

Thus, in this embodiment, the server 703 calculates information such as the camera orientation relative to the label and the distance between the camera and the label. Therefore, in this embodiment, the same effects as those of the first embodiment, the third embodiment, and the fourth embodiment described above can be obtained, and the calculation load on the terminal 701 can be reduced. .

[0132] (Sixth embodiment of terminal)

Next, a sixth embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 8 is a schematic diagram of an information providing system in which the sixth embodiment of the terminal of the present invention is used.

[0133] As shown in Fig. 8, the information providing system in which the sixth embodiment of the terminal of the present invention is used includes a terminal 801, a network 802, and a sixth embodiment of the terminal of the present invention. And Sano 03. Further, even if the label from which the terminal 801 of the present embodiment acquires the video is the label illustrated in FIG. 2, the ID 601 illustrated in FIG. 6 is used as described in the fourth embodiment of the terminal of the present invention described above. Even a label with can be added!

[0134] The video acquisition unit 804 has the same function as the video acquisition unit 301 shown in FIG. The calibration calculation unit 805 has the same function as the calibration calculation unit 302 shown in FIG. The calibration inquiry unit 806 has the same function as the calibration inquiry unit 705 shown in FIG. The geographic location measuring unit 807 has the same function as the geographic location measuring unit 504 shown in FIG. The network communication unit 812 has the same function as the network communication unit 707 shown in FIG.

The server inquiry unit 808 requests the server 803 to transmit information. The inquiry result receiving unit 809 receives information transmitted from the server 803. The server inquiry unit 808 requests the server 803 to provide information via the network communication unit 812 and the network 802. Then, the inquiry result receiving unit 809 receives information transmitted from the server 803 via the network 802 and the network communication unit 812.

When the server inquiry unit 808 requests the server 803 to transmit information, the server inquiry unit 808 transmits information on a predetermined terminal to the server 803. Information of a predetermined terminal transmitted to this server 803 The information includes at least the camera orientation relative to the label and the distance between the camera and the label, or the image of the label including the calibration pattern and the camera parameters calculated by the calibration. Further, the information transmitted to the server 803 may include at least one of the geographical location where the terminal exists and the label ID.

[0137] The information transmitted from server 803 includes information display position and information display size information in the video acquired by video acquisition section 804. Then, the video composition unit 810 and the video acquired by the video acquisition unit 804 based on the information display position and the size of the information display size in the video acquired by the video acquisition unit 804 The video corresponding to the information transmitted from the bar 803 is synthesized.

Display unit 811 displays the video synthesized by video synthesis unit 810. This display example is shown in Figure 9.

Shown in (a). The display unit 811 displays information acquired from a web server (not shown) as a web page. A display example of this web page is shown in FIG. FIG. 9 is a conceptual diagram of an image displayed on the sixth embodiment of the terminal of the present invention.

In the terminal of the present embodiment, for example, as shown in FIG. 9A, information 901, information 902, and information 903 by CG are transmitted from the server 803 to the terminal 801. In FIG. 9 (a), information 901, information 902 and information 903, and a label 904 are shown.

As shown in (a) of FIG. 9, these information 901, information 902, and information 903 display the name of a building, for example. The information 901, the information 902, and the information 903 are displayed in association with a video image of a building in the landscape spreading in front of the user's eyes. In other words, in this embodiment, as shown in (a) of FIG. 9, information related to the building in the video is displayed.

[0141] Of course, the present invention is limited to the case where the information 901, information 902, and information 903 are displayed in association with a video image of a building in the landscape spreading in front of the user. is not. In the present invention, these information 901, information 902, and information 903 need only be associated with objects such as buildings corresponding to these pieces of information.

[0142] The information 901, information 902, and information 903 are linked to information stored in a web server (not shown) on the network. And the user can get information 901 When the information 902 and the information 903 are clicked, the information linked to the information 901, the information 902, and the information 903 is transmitted from the linked websano to the terminal 801.

When the user clicks the information 901, the information 902, and the information 903, a web page 905 is displayed on the terminal as shown in FIG. 9B, for example. In the example shown in (b) of Fig. 9, service menu information is displayed for restaurant information. Of course, in the present invention, the information may not be displayed as a web page, but may be another appropriate display method.

In this embodiment, CG images such as information 901, information 902, and information 903 are created in advance by the server in order to reduce the calculation load on the terminal 801. Then, the terminal 801 of this embodiment receives CG images such as information 901, information 902, and information 903. Then, as shown in FIG. 9 (a), the terminal 801 of the present embodiment displays a CG image such as information 901, information 902, and information 903 superimposed on the actual landscape image. In this case, the positional relationship between the CG image such as information 901, information 902, and information 903 and the image of the actual landscape is calculated as the direction of the camera relative to the label and the distance force between the camera and the label. In addition, the terminal according to the present embodiment may create an image to be displayed using information received from Sano. In addition, the terminal according to the present embodiment determines the position and size for displaying CG images such as information 901, information 902, and information 903 based on the orientation of the camera with respect to the label and the distance between the camera and the label. You may do that.

In the present embodiment, as shown in FIG. 9A, information 901, information 902, and information 903 by CG are displayed so as to be superimposed on an actual landscape image. In the present embodiment, the label 904 is placed within the shooting range of the camera. In the present embodiment, the orientation of the camera with respect to the label and the distance between the label 904 and the camera are calculated from the calibration pattern of the label 904. Of course, like the terminal of the fifth embodiment described above, the server may calculate the orientation of the camera with respect to the label and between the label and the camera. The method of calculating the camera orientation and the distance between the label 904 and the camera from the calibration pattern of the label 904 according to the present embodiment is the same method as the terminal of the first embodiment described above. In this embodiment, it is possible to superimpose a CG image and a real landscape image with higher accuracy than in the conventional technology. That is, the terminal 801 of this embodiment inquires of the server 803 for information. Here, the terminal 801 of the present embodiment can access the server 803 via a wireless network, a wired network, or both. The terminal 801 of this embodiment receives information from the server 803. Then, the terminal 801 of the present embodiment synthesizes and displays a video of information such as information 901, information 902, and information 903 by CG with a landscape video. Therefore, the terminal 801 of the present embodiment can obtain the same effects as those of the first embodiment, the third embodiment, the fourth embodiment, and the fifth embodiment of the terminal of the present invention described above. Both can provide useful information to the user.

[0147] (Seventh embodiment of terminal)

Next, a seventh embodiment of the terminal of the present invention will be described with reference to FIG. FIG. 10 is a schematic diagram of an information providing system in which the seventh embodiment of the terminal of the present invention is used.

[0148] As shown in FIG. 10, the information providing system in which the seventh embodiment of the terminal of the present invention is used is a terminal 1001, which is the seventh embodiment of the terminal of the present invention, a network 1002, and And server 1003. Further, even if the label from which the terminal 1001 of the present embodiment acquires video is the label shown in FIG. 2, the ID 601 shown in FIG. 6 is used as described in the fourth embodiment of the terminal of the present invention described above. May be given a label.

[0149] Further, in the internal configuration of the terminal 1001 of the present embodiment, a video follower 1013 is added to the internal configuration of the terminal 801 shown in FIG. Therefore, the video acquisition unit 1004 has the same function as the video acquisition unit 301 shown in FIG. The calibration calculation unit 1005 has the same function as the calibration calculation unit 302 shown in FIG. The calibration inquiry unit 1006 has the same function as the calibration inquiry unit 806 shown in FIG. The geographic location measuring unit 1007 has the same function as the geographic location measuring unit 807 shown in FIG. The server inquiry unit 1008 has the same function as the server inquiry unit 808 shown in FIG. The inquiry result receiving means 1009 has the same function as the inquiry result receiving means 809 shown in FIG. The display portion 1011 has the same function as the display portion 811 shown in FIG.

[0150] The video composition unit 1010 acquires the direction and edge of the movement of the terminal acquired by the video tracking unit 1013. Based on the distance moved at the end, the video to be synthesized with the landscape video, the position of the video to be synthesized, and the size of the video to be synthesized are changed.

The configuration of this embodiment is a configuration in which a video follower 1013 is added to the configuration of the sixth embodiment described above. The video follower 1013 may be added to the terminals of the embodiments shown in FIGS. 3, 4, 5, and 7, respectively. The video follower 1013 uses motion estimation used in a compression method such as MPEG. However, the present invention can also use means for estimating video motion other than motion estimation.

[0152] The operation of the video follower 1013 shown in Fig. 10 will be described with reference to Figs. 11 and 12 are conceptual diagrams for explaining the operation of the video follower in the seventh embodiment of the terminal of the present invention.

[0153] The calibration calculation unit 1005 detects the orientation of the camera and the distance between the label and the camera from the label copied on the screen of Fig. 11 (a). After that, as shown in Fig. 11 (b), the user moves the terminal to the area where the label is off the screen.

In this case, as shown in FIGS. 12A, 12B, and 12C, the video follower 1013 divides a part or the whole of the image with rectangular blocks. Then, the video follower 1013 detects how the rectangle having the same luminance pattern moves as the camera moves. Based on this detection result, the video follower 1013 of the terminal according to the present embodiment detects a change in the direction of the camera and a change in the distance between the label and the camera.

[0155] Then, the video composition unit 1010 obtains information on the orientation of the camera relative to the label, the distance between the label and the camera, and the orientation of the power camera obtained by photographing the label before the terminal moves. Based on the change information and the change information of the distance between the label and the camera, the CG image is updated. In this manner, the terminal according to the present embodiment maintains a state in which the CG video and the real-world video are appropriately superimposed.

Here, the video follower 1013 shown in FIG. 10 measures, for example, a change in the orientation of the terminal and a moving distance using an optical flow. However, the terminal of the present invention may measure a change in the orientation of the terminal and a moving distance using a magnetic sensor.

Thus, the terminal of the present embodiment can obtain the same effects as those of the terminal of the sixth embodiment described above, and can update the CG video in accordance with the movement of the terminal. The terminal according to the present embodiment can maintain a state in which the CG video and the real-world video are appropriately superimposed and displayed.

[0158] (First embodiment of server)

Next, a first embodiment of the server of the present invention will be described with reference to FIG. FIG. 13 is a block diagram of the internal configuration of the first exemplary embodiment of the server of the present invention.

This embodiment is a server that receives a request for information transmission from the sixth embodiment or the seventh embodiment of the terminal of the present invention described above. As shown in FIG. 13, the server according to the present embodiment includes a calibration inquiry reception unit 1301, a calibration calculation unit 1302, a server inquiry reception unit 1303, an inquiry result transmission unit 1304, a link information management unit 1305, and a data storage unit. 1306 and a network communication unit 1307.

[0160] The calibration inquiry receiving unit 1301 receives video information including a calibration pattern from the terminal. Further, the calibration inquiry receiving unit 1301 receives the camera parameters obtained by the calibration in which the terminal force is also transmitted.

[0161] When the calibration inquiry reception unit 1301 receives video information and parameters, the calibration calculation unit 1302 uses the received calibration pattern video information and parameters to determine the camera direction and camera Calculate the distance to the label.

[0162] The server inquiry reception unit 1303 receives from the terminal at least the orientation of the camera with respect to the label, the distance between the camera and the label, the video information of the label including the calibration pattern, and the parameters calculated by the calibration. Either one is received.

[0163] Further, this server inquiry reception unit 1303 may further receive at least one of the geographical location where the terminal exists and the label ID. Then, the link information management unit 1305 extracts information from the data storage unit 1306 based on the received information.

That is, the information stored in the data storage unit 1306 is classified based on at least the orientation of the camera with respect to the label and the distance between the label and the camera. Then, the link information management unit 1305 extracts information on the direction of the camera relative to the label and the distance between the camera and the label from the information received by the server inquiry reception unit 1303. And Thus, the link information management unit 1305 extracts information from the data storage unit 1306 based on information on the direction of the camera relative to the label and the distance between the camera and the label.

[0165] Then, the inquiry result transmission unit 1304 transmits the information extracted from the data storage unit 1 306 by the link information management unit 1305 to the terminal. Here, the inquiry result transmission unit 1304 calculates information on the position and size at which the information extracted from the data storage unit 1306 is displayed based on the orientation of the camera with respect to the label and the distance between the camera and the label. To do. Then, the inquiry result transmission unit 1304 transmits information regarding a position and a size at which the information is displayed to the terminal. The information transmitted from the inquiry result transmission unit 1304 to the terminal is information such as information 901, information 902, and information 903 shown in (a) of FIG. 9, for example.

[0166] When the camera orientation with respect to the label and the distance between the label and the camera are not transmitted to the server, the calibration calculation unit 1302 calculates the camera orientation with respect to the label and the distance between the label and the camera. .

[0167] For example, if the terminal does not have a calibration calculation unit, the calibration inquiry reception unit 1301 also receives the video and parameters of the calibration label with the terminal power.

[0168] Then, the calibration calculation unit 1302 calculates the orientation of the camera with respect to the label and the distance between the camera and the label from the calibration pattern and the parameters. This calculation is the same as the calculation described in the first embodiment of the terminal of the present invention described above. Then, the link information management unit 1305 extracts information from the data storage unit 1306 using the calculation result of the calibration calculation unit 1302.

[0169] The data storage unit 1306 stores the orientation of the camera relative to the label, the distance between the camera and the label, and information displayed on the terminal in association with each other. In addition, the information stored in the data storage unit 1306 may be stored in association with at least one of position information such as geographical position information obtained by GPS or the like and a label ID.

[0170] As described above, the server according to the present embodiment also receives an information acquisition request for the terminal power. The server of the present embodiment transmits information from the terminal, such as the camera orientation relative to the label, the distance between the camera and the label, the geographical position obtained by GPS, and the label ID. Can be sent to the terminal.

[0171] (Second Embodiment of Server)

Next, a second embodiment of the server of the present invention will be described with reference to FIG. FIG. 14 is a block diagram of the internal configuration of the second exemplary embodiment of the server of the present invention.

As shown in FIG. 14, the server of the present embodiment includes a calibration inquiry reception unit 1 401, a calibration calculation unit 1402, a server inquiry reception unit 1403, an inquiry result transmission unit 1404, a time It has an acquisition unit 1405, a time link information management unit 1406, a data storage unit 1407, and a network communication unit 1408.

The calibration inquiry receiving unit 1401 has the same function as the calibration inquiry receiving unit 1301 shown in FIG. The calibration calculation unit 1402 has the same function as the calibration calculation unit 1302 shown in FIG. The server inquiry reception unit 1403 has the same function as the server inquiry reception unit 1303 shown in FIG. The inquiry result transmission unit 1404 has the same function as the inquiry result transmission unit 1304 shown in FIG.

In the present embodiment, the concept of time is incorporated into the information stored in the data storage unit 1407. In the first embodiment of the server of the present invention described above, the information stored in the data storage unit 1306 is constant regardless of the time. On the other hand, in the present embodiment, information stored in the data storage unit 1407 is classified based on time.

[0175] The time acquisition unit 1405 acquires the time when the server inquiry reception unit 1403 receives the information. The time acquisition unit 1405 transmits the acquired time to the time link information management unit 1406. The time link information management unit 1406 manages information stored in the data storage unit 1407. Then, the time link information management unit 1406 extracts information corresponding to a certain time from the data storage unit 1407.

Also, the time link information management unit 1406 extracts information from the data storage unit 1407 based on the information received by the server inquiry reception unit 1403 and the time acquired by the time acquisition unit 1405. Here, the server inquiry reception unit 1403 includes at least one of the camera direction with respect to the label, the distance between the camera and the label, the image of the label including the calibration pattern, and the camera parameters calculated by the calibration. one Receive. The server inquiry reception unit 1403 may further receive at least one of a geographical location where the terminal exists and a label ID.

Here, information stored in data storage section 1407 shown in FIG. 14 will be described with reference to FIG. FIG. 15 is a conceptual diagram of information stored in the data storage unit of the second embodiment of the server of the present invention.

As shown in (a) of FIG. 15, information A corresponds to 0 and information B corresponds to 0 at a time corresponding to a predetermined label ID, camera position, and orientation. Further, as shown in FIG. 15 (b), information X corresponds to 0 and information Y corresponds to 1 in correspondence with a predetermined label ID, the position and orientation of the terminal. Of course, the correspondence between time and information in the present invention is not limited to the example shown in FIG. For example, the time range may be changed in units of minutes other than that shown in FIG. 15, for example. Note that the positions shown in FIG. 15 include at least the orientation of the camera with respect to the label and the distance between the camera and the label.

As described above, the server according to the present embodiment changes information to be transmitted to the terminal at each time.

For example, the server of the present embodiment changes the menu of the restaurant at noon and at night, so that the menu provided to the terminal also changes depending on the time. In addition, the server of the present embodiment changes the price information provided to the terminal because the grocery of the department store may become cheaper depending on the time of day.

[0180] As described above, the server according to the present embodiment can obtain the same effects as those of the first embodiment of the server according to the present invention described above, and can change the information transmitted to the terminal according to the time. Can do.

Claims

The scope of the claims

[1] Image acquisition means for acquiring an image of a shooting target having a pattern for calculating a direction of shooting the shooting target and a distance between the shooting target;

A terminal comprising: calculation means for calculating a direction between the shooting target and a distance between the shooting target and the image of the pattern.

[2] The pattern is the same pattern as a calibration pattern for calibrating the terminal,

2. The terminal according to claim 1, wherein the calculating unit calculates a direction between the shooting target and a distance between the shooting target and the image of the pattern and the parameters obtained by the calibration.

3. The terminal according to claim 1, further comprising distance measuring means for measuring a distance between the object to be imaged.

4. The terminal according to claim 1, further comprising position measuring means for measuring the position of the terminal.

5. The terminal according to claim 1, wherein the shooting target includes identification information for identifying a plurality of shooting targets from each other.

[6] Connected by a network to a server that calculates a direction in which the subject is photographed and a distance between the subject to be photographed based on a pattern included in the subject.

The image of the pattern acquired by the image acquisition means and the parameter obtained by the calibration performed in advance are transmitted to the server, and the imaging target calculated by the server based on the image of the pattern and the parameter is transmitted. The terminal according to claim 1, further comprising inquiry means for receiving a direction from which the image is taken and a distance to the object to be photographed from the server.

[7] The terminal power is connected by a network to a server that selects information to be transmitted to the terminal based on the received information,

The terminal according to claim 1, further comprising receiving means for receiving information transmitted from the server. The end.

[8] The transmission means includes

8. The terminal according to claim 7, wherein the terminal transmits at least one of identification information for identifying the position of the terminal and a plurality of shooting targets included in the shooting target to the server.

9. The terminal according to claim 7, further comprising a combining unit that combines the video acquired by the video acquiring unit and the video based on the information received by the receiving unit.

10. The terminal according to claim 9, further comprising display means for displaying the video synthesized by the synthesis means as a computer graphic image.

[11] The terminal according to claim 7, further comprising display means for displaying the information received by the receiving means as a web page.

[12] The image following means for detecting the distance moved by the terminal and the direction moved by the terminal based on the change in the image acquired by the terminal, which occurs with the movement of the terminal, and the combining means, 10. The video acquired by the video acquisition unit and the information displayed on the video are combined based on the distance moved by the terminal detected by the video tracking unit and the direction in which the terminal has moved. Terminal.

[13] At least one of a direction in which the photographing target is photographed and a distance between the photographing target and an image of the photographing target including the pattern and a parameter obtained by a calibration performed in advance. Receiving means for receiving terminal power;

A server comprising transmission means for transmitting the information extracted by the information management means to the terminal.

[14] The receiving unit receives at least one of identification information for mutually identifying a position of the terminal and a plurality of shooting targets included in the shooting target,

The information management means is at least one of the information transmitted from the terminal received by the receiving means and the identification information for mutually identifying the position of the terminal and a plurality of photographing objects included in the photographing object And information from the data storage means 14. The server according to claim 13, wherein the server is extracted.

The information stored in the data storage means is

Based on the information and time received by the receiving means,

14. The server according to claim 13, wherein the information management means extracts information from the data storage means based on information received by the receiving means and a time when the information is received.