JP3114862B2 - An interactive landscape labeling system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention superimposes and displays, on an image display device, geographical information about each partial area in an image taken by a user using a landscape image input device such as a camera. The present invention relates to a device that teaches a user by voice guidance or the like.

[0002]

2. Description of the Related Art Conventionally, there have been various navigation systems as systems for instructing users on geographical information about the area where the users are located.

FIG. 12 is a structural diagram of a navigation device disclosed in Japanese Patent Application Laid-Open No. 8-273000. This device receives a vehicle position data and a movement data, receives a road map data, updates a vehicle position with reference to a position updating unit 71, and displays display road data and display background data based on map data and the like. Display data generator 7 to be generated
2, a three-dimensional moving image data creating unit 73 for creating three-dimensional moving image data based on these display data, and a storage unit 74.
When a travel route including a waypoint is set in advance, the device has a function of setting a route while viewing a real moving image display screen along a road that actually exists instead of a map screen.

According to this device, when a user travels along a route that actually exists, the user can see a moving image display (for example, FIG. 13) along the route.

[0005]

However, in the case of using the same device, the human finally associates the real scene with the geographical information in the computer world with the naked eye, so that the person in the real scene can be used. You have to recognize what is. In other words, humans rely on the naked eye to work the human brain unconsciously, based on the symbols in the map displayed as a moving image, based on the actual buildings, roads, and mountains in front of the user. It must be understood by performing the work of association. On street corners, etc., comparing the map with the computer and the actual scenery, grasping the direction, finding the landmarks, gazing at the direction, understanding the characteristics of the building in that direction, and looking at the map again Understand what the building is.

[0006] For this reason, there is a problem that it is not possible to omit the trouble of comparing the map on the computer with the actual scenery many times and associating it with a human. Especially in the dark or at night, it is difficult to see the actual scenery and to take measures.

There have been computer systems that allow an unspecified number of people or specific people to know or work together by chance. However, such a computer system can be used only in the world inside a computer. There was only a system. An example is a so-called cyber space. In such a system, in a virtual city space created inside a computer, a three-dimensional figure expressing CG the people registered in the system is displayed.
By clicking and moving a dimensional CG figure, information acquisition and retrieval have been made possible in a virtual world.

However, it is a function in a virtual invisible world, and does not enable information acquisition and retrieval between system registrants in a real visible world.

[0009] An object of the present invention is to provide the user with geographical information on a computer and each part in an image of a real scene (hereinafter referred to as a landscape image) in association with each other and to make the image visible to the real world. It is an object of the present invention to provide an interactive landscape labeling system that enables information acquisition and search between system registrants in the world.

[0010]

According to the present invention, there is provided a camera in which map data on a computer is created in advance as three-dimensional data, and an image (hereinafter referred to as a landscape image to distinguish it from a CG image) is input. The computer graphics (hereinafter, referred to as the position, camera angle, focal length, and image size) are acquired at the time of shooting, and viewed from the position, camera angle, and focal length at the time of shooting the actual scene in the three-dimensional map space on the computer. This is to realize association by acquiring geographical information in an image and superimposing and displaying the geographical information on a landscape image which is a real scene. The geographic information is the name of a structure or the like or its attribute information in an image, and the attribute information means information on all attributes (for example, contour, color, etc.) of the structure.
In this specification, the term structure is used to mean all data having a certain geographical structure in the map DB, including natural topography such as mountains, rivers, and the sea, in addition to artificial structures. . When acquiring geographic information, a landscape image is obtained based on a camera position, a camera angle, a focal length, and an image size, and a structure of a plurality of images is obtained. The position of the landscape image where the structure should be shown (hereinafter, referred to as the assigned position)
Is obtained, and the name or attribute information of the structure is superimposed and displayed.

Further, in order to further increase the accuracy of associating the structure in the landscape image with the structure in the CG image,
The previously acquired structure is associated with each partial region of the landscape image by pattern matching. A CG image is created on the basis of the acquired structure, a partial region in the CG image is associated with the partial region of the landscape image by pattern matching, and the structure as a source of the associated partial region Ask for.

Here, an example of a method of creating a CG image will be described. By accessing the three-dimensional map DB based on the camera position, camera angle, focal length, and image size obtained earlier,
Obtain the visual field space in the three-dimensional map space. A structure in the visual field space is obtained, and three-dimensional data of each structure is three-dimensionally projected and converted onto the projection plane using the camera screen as a projection plane. Further, of the line data constituting the projected figure of each structure, line data hidden by other structures and invisible is erased using a normal vector method or the like. The CG image is divided into regions based on the line data remaining after the elimination of the hidden lines. 3D map DB
Is used, the name of the structure that is the basis of the area can be associated with each area.

Then, the structure names of the partial areas of the CG image corresponding to the respective partial areas of the landscape image are extracted by pattern matching. The position coordinates of the actual scenery image on which the extracted structure name is to be superimposed are obtained by three-dimensionally projecting the position coordinates of the structure in the three-dimensional map space onto the projection plane. Label information is created from the position coordinates of the actual scenery image on which the extracted structure name is to be superimposed. Based on the label information, the structure name is superimposed on a landscape image as a real scene and displayed on a visual device.

Further, in the present invention, position information (with time) and attribute information on the user are sent from the user terminal to the center at regular intervals. Here, the attribute information may be any information as long as it is information on each user. Examples of the attribute information include acoustic information around the user including the user's remark, a document file created by the user, an image file photographed by the user, and the like. In addition, from the user terminal, information relating to another user, which is desired to be acquired, can be specified. The communication control means on the center side passes position information and attribute information on the user to the user information management means via the center control means. The user information management means manages position information and attribute information of each user, and when another user requests acquisition of information about the user, the information acquisition request is made according to the access right level of each information of the user. It is determined whether information can be provided to the user, and if the information can be provided, the information is provided to the user who issued the acquisition request. Examples of the information include history information of the position of the user and user attribute information.

The user information management means passes the position information of the user to the center control means, and the center control means passes the position information of the user to the label information creation means. In the label information creating means, based on the position information of each user connected to the system, each user is present or the nearest structure is obtained, and each user is present inside or near the structure. Is created, and this is added to the previously created label information and passed to the center control means. Here, the user position information includes the name of the structure where each user is present or the nearest structure, attribute information, and its assigned position information. The user position information is passed to the label information output unit via the communication control unit, and is displayed on, for example, an HMD (a visual device such as a head-mounted display).

The mutual use type landscape labeling system according to the present invention comprises a plurality of user terminals and a center, wherein each of the user terminals includes an image obtaining means for obtaining an image, and a position information obtaining apparatus for obtaining a camera position at the time of obtaining the image. Means, camera attribute information acquisition means for acquiring the camera angle, focal length, and image size at the time of image acquisition, image processing means for dividing the acquired image into a plurality of partial areas, and information about other users, A user information acquisition request unit for outputting a user information acquisition request specifying information to be acquired, transmitting time-added position information and attribute information relating to the user to the center at regular intervals, and transmitting the user information acquisition request to the center. And transmits the information on the area division of the image, the camera position, the camera angle, the focal length, and the image size via a communication network. Communication control means for transmitting to the center, receiving information of another user who issued a user information acquisition request from the center, and receiving label information from the center, and acquiring user information for acquiring the information of the other user. Means, label information output means for superimposing the name of the structure in the label information or its attribute at a corresponding position in the image, and outputting the superimposed image to a visual device, and terminal control for controlling each of the means Means for receiving, from the user terminal via the communication network, information relating to area division of the image, the camera position, the camera angle, the focal length, and the image size; From the user terminal, and sends information on other users and the label information to the user terminal. Communication control means for managing position information and attribute information of each user, and when a user information acquisition request is sent, determines whether information can be provided according to the access right level of the user who issued the user information acquisition request. Determine, if the information can be provided, user information management means for providing information to the user, and manage map information,
Obtain a visual space in the map information space based on the received camera position, camera angle, focal length, and image size, obtain a structure existing in the visual space, and connect each user connected to the center. Based on the information of the terminal, the map information management means in which each user is present, or seeks the nearest structure, and the obtained structure is associated with the partial region of the image, and the associated Label information for creating the label information including the name of the structure or the attribute information and the coordinates of the assigned position, and the name of the structure where each user is present or the nearest, or the user's position information including the attribute information and the assigned position coordinates It includes a creating means and a center control means for controlling each of the means.

According to an embodiment of the present invention, the label information creating means creates a CG image based on the acquired structure, and performs pattern matching on the partial area of the image to obtain a portion in the CG image. The structure of the partial area associated with the area is obtained, and label information including the name or attribute information of the structure and the assigned position is created.

According to the embodiment of the present invention, the label information creating means performs three-dimensional projection conversion of the acquired structure on a camera screen, erases the structure that cannot be seen from the viewpoint, creates a CG image, and creates a CG image. The CG image is divided into partial regions by the contour line of the middle partial region, and the partial region of the image is
The partial area of the G image is associated with the partial area of the image by pattern matching, and the partial area of the image is determined. The structure that is the basis of the partial area of the CG image is obtained. Create label information to include.

[0019]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram of an interactive scenery labeling system according to an embodiment of the present invention.

The landscape labeling system of the present embodiment comprises a user terminal 1, a center 2, and a communication network 3 connecting the user terminal 1 and the center 2.
Although only one user terminal 1 is shown in FIG. 1, there are actually a plurality of user terminals.

The user terminal 1 includes an image acquisition unit 11 for acquiring an image, a position information acquisition unit 12 for acquiring a camera position at the time of image acquisition, and a camera for acquiring a camera angle, a focal length, and an image size at the time of image acquisition. An attribute information acquisition unit 13, an image processing unit 14 for dividing the acquired image into a plurality of partial areas,
A user information acquisition request unit 15 that outputs a user information acquisition request specifying information to be acquired, which is information about another user, and transmits time-added position information and attribute information about the user to the center 2 at regular intervals. A communication control unit 16 that transmits a user information acquisition request to the center 2 and receives information and label information of another user who has issued the acquisition request from the center 2, and a user information acquisition unit 17 that acquires information of another user. And superimpose the structure in the label information or its attribute information at a position corresponding to the application position in the image,
It includes a label information output unit 18 that outputs the superimposed image to the visual device, and a terminal control unit 19 that controls each unit.

The center 2 receives information on the area division of the image, the camera position, the camera angle, the focal length, and the image size from the user terminal 1 via the communication network 3, and receives the time-stamped position from each user terminal 1. A communication control unit 24 for receiving information, attribute information, and a user information acquisition request, and transmitting information and label information on other users to the user terminal 1;
And manages the position information and attribute information of each user, and when a certain user receives a request to obtain information on another user, determines whether to provide information according to the access right level of the user, and If it can be provided, the user information management unit 21 that provides information to the user, manages the map information, obtains a visual field space in the map information space based on the received camera position, camera angle, and image size. A map information management unit 22 that acquires a structure existing in the visual field space, and obtains a structure where each user is present or the nearest structure based on position information of each user terminal 1 connected to the center 2. And the acquired structure is associated with the partial region of the image, and the name or attribute information of the associated structure and assigned position coordinates, and each user is present or nearby Structure name of the attribute information and the label information preparation section 23 for creating a label information comprising the assigned position coordinates, the central control unit 2 for controlling the respective means
5 is included.

Next, the operation of this embodiment will be described.

When the user terminal 1 is started, first, the terminal control unit 19 starts processing the position information acquisition unit 12, the camera attribute information acquisition unit 13, and the image acquisition unit 11 in order to acquire information on the landscape image. Send a command. The position information acquisition unit 12 receives a command from the control unit 19, collects position information with a GPS receiver or the like every second, and transfers the collected position information to the control unit 19 (step 31). Here, the time interval is not limited to the unit of seconds, but may be any value. The image acquisition unit 11 receives a command from the control unit 19, acquires a landscape image every second, and passes it to the control unit 19 (step 32). The camera attribute information acquisition unit 13
In response to a command from the control unit 19, the camera angle of a landscape image recording device such as a camera at the time of image capturing is acquired as a set of a horizontal angle and an elevation angle (step 33). Acquire (Step 34). Since the image size is fixed for each landscape image device, the control unit 19 holds the image size information. The control unit 18 holds the collected information as a landscape image file.

FIG. 3 shows the file format of the data structure of the landscape image file. The landscape image file has header information and image data. The header information includes position information, camera angle information, focal length, time information, image size, type, and size of an image file. As location information,
East longitude, north latitude, and elevation data (for example, 137 degrees east 5 degrees
5 minutes 10 seconds, latitude 34 degrees 34 minutes 30 seconds north, altitude 101m3
3 cm). The camera angle includes horizontal angle data and elevation angle data (for example, horizontal angle 254 degrees clockwise, elevation angle 15 degrees, etc.). The focal length data is the focal length (for example, 28 mm) of the camera lens at the time of capturing an image. The time information includes the time at the time of shooting (for example, 15:06:17, January 31, 1997, Japan time). As the image size of the image file, the vertical and horizontal pixel sizes (for example, 64
0 × 480). File type (TIF
E format, 8-bit color, etc.). It also has the number of bytes of the file (307.2 KB, etc.). The image data itself has, for example, a binary format.

After storing the landscape image file, the control unit 19 instructs the image processing unit 14 to extract the outline from the landscape image and divide the landscape image into a plurality of regions. The image processing unit 14 extracts a contour line by performing differentiation processing based on the density difference in the landscape image, roughly speaking (step 35), and divides the region by performing labeling using the contour line as a boundary (step 35). Step 36). Note that the technical term of labeling used here is a technical term used in image area division, and is different from landscape labeling, which is the name of the present invention. As a procedure, first, the image is converted into a monochrome grayscale image. Since the contour is a portion where the brightness changes suddenly, the contour is extracted by performing a differentiation process to find a portion where the differential value is larger than a threshold value. At this time, the line width of the outline is one pixel, and the outlines are connected. For this purpose, a thinning process is performed to obtain a connected line having a line width of one pixel. Where the differentiation process,
It is sufficient to use a conventional method for the thinning processing.

The obtained contour is regarded as the contour of the area, and an operation of numbering the area constituted by the contour is performed.
The largest number among the numbers is the number of areas, and the number of pixels in the area indicates the area of the area. FIG. 8 shows an example in which a landscape image is divided into a plurality of partial regions. Note that a measure of similarity (closeness) between regions may be introduced, and a clustering process may be performed to combine a plurality of regions having similar properties into one region. Any existing clustering method may be used.

The information on the area division of the image, the camera position, the camera angle, the focal length, and the image size obtained above are sent from the communication control unit 16 to the center 2 via the communication network 3 (step 37). Also, the time-based position information and attribute information of the user terminal 1 (audio information around the user including the user's remark, a document file created by the user, an image file created by the user, etc.) are transmitted from the communication control unit 16 at regular intervals. It is sent to the center 2 (step 38). Further, when the user issues an acquisition request from the user information acquisition unit 15 for user information that specifies information to be acquired, which is information about another user, this is sent from the communication control 16 to the center 2 (step 39).

When the map information management unit 22 of the center 2 receives the header information of the landscape image file from the user terminal 1, it accesses the map DB (step 40) and performs a visual space calculation process (step 41). Map information management unit 2
A second example is a map database program.
The map information management unit 22 manages three-dimensional map data. Although two-dimensional map data may be used, in this case, since there is no height, the accuracy of the position at which the labeling is applied to the actual scenery is inferior. When the two-dimensional map data is used as the basis, the processing is performed by supplementing the height information. For example, in the case of two-dimensional data of a house, if there is floor information indicating the number of floors of the house,
The height of the house is estimated by multiplying the number of floors by a certain number, and three-dimensional data is created based on the height information obtained by estimating the two-dimensional data. Even when there is no floor information, the height information can be estimated by assigning a certain number of heights according to the area of the house figure, and similarly, three-dimensional data is created based on the estimated height information. . In this way, three-dimensional data is created and the process proceeds.

FIG. 4 shows an example of three-dimensional map data. FIG.
(1) shows a map information space expressed in two dimensions, and FIG.
(2) shows a map information space expressed in three dimensions. This 3
With respect to the three-dimensional map information space, the map information management unit 22 calculates the visual field space based on the header information of the landscape image file in response to an instruction from the control unit 25 (step 41). FIG. 5 shows a calculation example of the visual field space. First, it is assumed that the XY axes extend in the horizontal direction and the Z axis extends in the vertical direction. The position of the viewpoint E is set in the three-dimensional map information space from the position information in the header information of the landscape image file. For example, 137 degrees 55 minutes 19 seconds east longitude, 34 degrees 34 minutes 30 seconds north latitude, 101 m elevation
If it is 33 cm, the corresponding coordinates in the corresponding map mesh number are set. Similarly, the camera angle direction is set based on the horizontal angle and the elevation angle in the same camera angle information in the header information. The focal point F is set at a point advanced from the viewpoint E by the focal length on a straight line representing the camera angle direction. The line-of-sight direction vector is a unit vector having a length 1 coming out of the viewpoint E on the straight line.
In the image size of the landscape image file, the width x on the X-axis of the camera screen is set from the horizontal size, and the width y on the Y-axis is set from the vertical size. The horizontal x vertical y plane is set so as to be perpendicular to the camera angle direction with respect to the line of sight vector and to include the focal point F. From the coordinates of the viewpoint E,
Straight lines connecting the corner points are obtained, and a three-dimensional space formed by four half lines extending from the viewpoint E is defined as a visual field space. In FIG.
4 shows an example of a visual field space in a three-dimensional map space. The three-dimensional map space is viewed from the XZ plane. In FIG. 6, a portion surrounded by oblique lines is a cross-sectional view of the space belonging to the viewing space on the XZ plane. In the example of FIG. 6, buildings and mountains in the visual field space are included.

Further, the map information management unit 22 obtains a structure existing in the obtained visual field space. For each structure,
It is calculated whether or not each vertex constituting the solid representing the structure exists in the internal region of the viewing space. Usually, a two-dimensional map space is divided by a two-dimensional mesh of a fixed size. As a method of cutting a mesh in the three-dimensional map space, a mesh is cut at regular intervals in the height direction in addition to the two-dimensional mesh in the vertical and horizontal directions. The space is divided by a rectangular parallelepiped unit space. First, the presence or absence of an overlapping portion between the rectangular parallelepiped unit space and the visual field space is checked, and the number of the three-dimensional unit map space having the overlapping portion is obtained. The number of the three-dimensional unit map space here is the same as a so-called mesh number. A check is made as to whether or not there is an overlapping portion between the visual field space and the structure in the overlapping three-dimensional unit map space. A straight line connecting the coordinates of the vertices constituting the structure and the coordinates of the viewpoint is obtained. If the straight line has an intersection with the camera screen of FIG. 7, it is within the visual field space. Of the vertices that make up the structure,
If at least one vertex satisfies this condition, the structure has an overlapping part with the viewing space.

When a structure is included in the view space or a part of the structure, the camera screen is used as a projection plane, and a process for three-dimensionally projecting each structure onto this projection plane is started (step 42). ). Here, as shown in FIG. 7, after the point P is re-expressed in the coordinate system based on the viewpoint E based on the following equation (1), the point P is projected on the camera screen to obtain the intersection Q. .

[0034]

(Equation 1) Here, point P = (x, y, z): coordinates of the vertices constituting the structure point E = (ex, ey, ez): coordinates of the viewpoint vector L = (lx, ly, lz): gaze direction vector (Unit vector) Point P ′ = (x ′, y ′, z ′): coordinates in a coordinate system based on the viewpoint E of the point P r = (lx ² + ly ² ) ^1/2 intersection Q = (X, Y): The projection point t of the point P on the camera screen is the focal length. In the three-dimensional projection conversion, first, for each structure, the surface on which the vertex extends is obtained. For example, in the case of a structure represented by a rectangular parallelepiped, six surfaces are obtained. When each plane is projected onto the camera screen, the distance between the viewpoint and the corresponding point on the plane is calculated for each pixel on the camera screen included in the projection area, and stored as a depth value (Z value) in the memory. Store. For each surface of each structure, the depth value (Z
Value) and store it in memory. Note that z ′ in (Equation 1) represents a depth value (Z value) from the viewpoint.

Among the structures three-dimensionally transformed on the camera screen, there are structures visible from the viewpoint and structures not visible. Among them, it is necessary to find only the structure that can be seen from the viewpoint, and to find the surface on the opposite side from the viewpoint or the surface that is blocked by other structures. Therefore, hidden surface processing is performed (step 4).
3). There are various methods for processing the hidden surface.
Use the buffer method. Other scan line methods and ray tracing methods may be used.

A pixel on the camera screen is arbitrarily determined, and a plane having the smallest depth value for the pixel is obtained. As described above, when processing is sequentially performed on each surface of each structure, the surface closest to the viewpoint is left for each pixel on the camera screen. The plane closest to the viewpoint is determined for each pixel on the camera screen, and pixels on the camera screen that share the plane closest to the viewpoint generally form an area.
There can be a plurality of regions consisting of pixels with the common surface being the closest surface. The region obtained in this way is a region obtained by performing a three-dimensional projection conversion of the partial region of the structure seen from the viewpoint. Surfaces on the other side of the view and those obstructed by other structures have been erased.

The area thus formed forms a CG image area (step 44).

With respect to the coordinates of the vertices of the two-dimensional figure constituting the CG image area, three-dimensional coordinates before projection transformation are obtained, and the correspondence between the two is stored in the memory as link information. It is used to determine which structure the two-dimensional area is a projection of based on the link information.

Based on the line data remaining after erasing hidden lines,
The CG image is divided into regions. Since the three-dimensional map DB is used, the name of the structure serving as the basis of each area can be associated with each area. The divided areas of the CG image are numbered sequentially. FIG. 9 shows an example in which a CG image is divided into a plurality of partial areas.

When a user information acquisition request is sent to the center 2, the user information management unit 21 receives the request via the center control unit 25, and acquires the information according to the access right level for each information of the user. It is determined whether or not the information can be provided to the request. If the information can be provided, the information is sent to the user terminal 1 that issued the acquisition request via the center control unit 25 and the communication control unit 24 (step 45). User terminal 1
Receives the other requested user information, displays it on the screen, or prints it out. When the information management unit 22 of the center 2 receives the time-added position information of the user terminal 1 via the center control unit 25, the information management unit 22 accesses the map DB to determine the structure where the user terminal 1 exists or is located closest to the center. The data is passed to the label information creating unit 23 via the control unit 25 (step 46).

Next, the center control unit 25 instructs the label information creating unit 23 to associate the CG image divided area with the landscape image divided area. The label information creating unit 23 associates the divided area of the CG image with the divided area of the labeling by template matching (step 47, see FIG. 10).

In the divided areas of the landscape image, the areas are associated with the partial areas of the CG image in ascending order of the number (for example, No. 1). For matching, any of the conventional matching methods may be used, but here, a simple template matching method is used. That is, two regions to be compared are superimposed, and when the ratio of overlapping portions is equal to or greater than a certain ratio determined as a threshold, the regions are associated as regions relating to the same structure. For example, the coordinate value of each pixel in the first divided region R1 of the landscape image is set to (A, B). The value of the pixel at coordinates (A, B) is 1 because it is inside the area. In the first divided area S1 of the CG image,
If the coordinates (A, B) are within the area S1, the pixel value is 1 and overlaps, but if the coordinates (A, B) are outside the area S1, the pixel value is 0 and does not overlap. In this way, the overlap coefficient K (A, B) at the coordinates (A, B) is determined to be 1 when they overlap and 0 when they do not overlap. The coordinates (A, B) are moved within the region R1, and the overlap coefficient K
(A, B) is obtained. Then, with respect to the number N1 of coordinates (A, B) moved in the region R1, the overlap coefficient K (A,
The number N2 of coordinates where B) was 1 is obtained, and when N1 / N2 is equal to or larger than the threshold value, it is determined that the partial region R1 of the landscape image and the partial region S1 of the CG image correspond to each other. This association is performed from the first to the last divided area of the landscape image. In addition, as the matching method, an evaluation function that has the same value even if there is a slight displacement in the XY directions may be used.

After associating the partial area of the landscape image with the partial area of the CG image, the label information creating unit 23 further obtains information to be superimposed on each partial area of the landscape image, The process for creating information (steps 48 and 49) is entered. First, a partial area of the CG image corresponding to the partial area of the landscape image is extracted. The partial region of the extracted CG image is originally obtained by three-dimensionally projecting and converting a certain surface of the three-dimensional structure in the three-dimensional map space onto the camera screen. So 3
The surface of the three-dimensional structure that is the basis of the three-dimensional projection transformation is determined using the depth value (Z value) of the partial region of the CG image as a key. The link information created at the time of the three-dimensional projection conversion may be used as a key. Based on the surface of the original structure, the three-dimensional map DB is accessed to acquire the name or attribute information of the structure. Here, the attribute information means information accompanying the structure, and may be any information as long as the information is related to the structure. Then, the position coordinates where the name or attribute information is to be superimposed are determined for the partial area of the landscape image. How to decide may be decided any way. For example,
The center of gravity of the figure extending the partial area may be used. Label information including name or attribute information of the structure and assigned position coordinates, and user position information including the structure or attribute information and the assigned position information where the user exists or is closest is created. Table 1 shows an example of label information.

[0044]

[Table 1] The label creating unit 23 passes the label information to the center control unit 25 after creating the label information.

Upon receiving the label information, the center control unit 25 sends the label information to each user terminal 1 via the communication control unit 24.
The terminal control unit 19 of each user terminal 1 instructs the label information output unit 18 to display and output the label information to the visual device. Here, the visual device includes a video display device such as a display or a head-mounted display. Structures in the label information (where each user exists,
Or, the name or attribute information (including the nearest structure) is superimposed on the position in the landscape image (step 50), and the superimposed landscape image is output to the video display device (step 5).
1). FIG. 12 shows an example of a landscape image on which label information is superimposed.

When the label information output section 18 outputs the label information, it notifies the terminal control section 19 of the completion of the output. When receiving the output completion notification, the terminal control unit 19 executes the above-described series of processing procedures again in the case where the landscape labeling process is continuously performed.

[0047]

As described above, according to the present invention, it is possible to obtain information and search for information among system registrants in a real visible world.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of an interactive use landscape labeling system according to an embodiment of the present invention.

FIG. 2 is a flowchart of a process of the interactive use landscape labeling system of FIG. 1;

FIG. 3 is a diagram showing a data structure of a landscape image file.

FIG. 4 is a diagram showing an example of a two-dimensional map (FIG. 1 (1)) and its three-dimensional map (FIG. 2 (2)).

FIG. 5 is a diagram illustrating a calculation method of a visual field space.

FIG. 6 is a diagram illustrating an example of a visual field space in a three-dimensional map space.

FIG. 7 is a diagram showing an example of a projection view.

FIG. 8 is a diagram showing an example of area division of a landscape image.

FIG. 9 is a diagram illustrating an example of area division of a CG image.

FIG. 10 is an explanatory diagram of pattern etching of a partial region of a landscape image and a partial region of a CG image.

FIG. 11 is a diagram illustrating an example of superimposition of label information on a landscape image.

FIG. 12 is a configuration diagram of a navigation device disclosed in JP-A-8-273000.

FIG. 13 is a diagram illustrating a display example of a moving image.

[Explanation of symbols]

 Reference Signs List 1 user terminal 2 center 3 communication network 11 image acquisition unit 12 position information acquisition unit 13 camera attribute information acquisition unit 14 image processing unit 15 user information acquisition request unit 16 communication control unit 17 user information acquisition unit 18 label information output unit 19 terminal control Unit 21 user information management unit 22 map information management unit 23 label information creation unit 24 communication control unit 25 center control unit 31 to 51 step

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Shoji Takano Nippon Telegraph and Telephone Corporation 3-9-1-2 Nishishinjuku, Shinjuku-ku, Tokyo (72) Inventor Takeshi Ikeda 3-19 Nishishinjuku, Shinjuku-ku, Tokyo No. 2 Within Nippon Telegraph and Telephone Corporation (56) References JP-A-9-180098 (JP, A) JP-A-5-119699 (JP, A) JP-A-9-33271 (JP, A) JP-A-5-181411 (JP, A) JP-A-5-79848 (JP, A) JP-A-8-273000 (JP, A) JP-A-9-33277 (JP, A) JP-A-8-201076 (JP , A) JP-A-10-339649 (JP, A) (58) Fields investigated (Int. Cl. ⁷ , DB name) G01C 21/00-21/36 G06T 15/00 G08G 1/0969-1/137 G09B 29/00-29/10

Claims (3)

(57) [Claims] 1. A user terminal comprising: a plurality of user terminals; and a center, wherein each of the user terminals includes an image obtaining unit for obtaining an image,
Position information acquisition means for acquiring a camera position at the time of image acquisition, camera attribute information acquisition means for acquiring a camera angle, a focal length, and an image size at the time of image acquisition, and image processing for dividing the acquired image into a plurality of partial areas Means, a user information acquisition request unit that outputs a user information acquisition request that specifies information to be acquired, which is information about another user, and sends time-added position information and attribute information about the user to the center at regular intervals. Transmitting the user information acquisition request to the center, and transmitting the information regarding the area division of the image, the camera position, the camera angle, the focal length, and the image size to the center via a communication network. A communication control unit that receives information of another user who has issued a user information acquisition request from the center, and receives label information from the center. User information acquisition means for acquiring the information of the other user; and superimposing the name of the structure in the label information or its attribute information at a position corresponding to the application position in the image, and displaying the superimposed image on a visual device. And a terminal control unit for controlling each of the units, wherein the center is configured to output information on area division of the image, the camera position, and the camera angle from the user terminal via the communication network. And the focal length and the image size, and also receives time-added position information, attribute information, and a user information acquisition request from each of the user terminals, and sends information about other users and the label information to the user terminal. Communication control means to transmit, and manages the position information and attribute information of each user, and when a user information acquisition request is sent, the user information acquisition request is issued. User information management means for providing information to the user if the information can be provided, managing the map information, and determining the received camera position and camera angle. Obtain the visual field space in the map information space based on the focal length and the image size, acquire the structure existing in the visual space, and based on the position information of each user terminal connected to the center. Map information management means for finding a structure where each user is present or closest, and associating the acquired structure with the partial region of the image, and associating the name or attribute information of the structure And a label for generating label information including the coordinates of the assigned position and the name of the structure where each user is present or the nearest, or its attribute information, and user position information including the assigned position coordinates. Mutual use type scene labeling system including a Le information creating means, a central control means for controlling said respective means. 2. The label information creating means creates a CG image, which is a computer graphics image, based on the acquired structure, and a part in the CG image by pattern matching with respect to the partial area of the image. The system according to claim 1, wherein a structure of the partial area associated with the area is obtained, and label information including the name or attribute information of the structure and the assigned position is created. 3. The label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, creates a CG image by erasing a structure that cannot be seen from a viewpoint, and creates a contour of a partial area in the CG image. Dividing the CG image into partial areas by lines,
The partial area of the image and the partial area of the CG image are associated with each other by pattern matching, and the structure that is the basis of the partial area of the CG image is determined for the partial area of the image. The system according to claim 1, wherein label information including name or attribute information and an assigned position is created.