JPH1166350A - Retrieval type scene labeling device and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To associate geographical information on a computer with each part of in a scene image of real scenery to present to a user and also to display information the user wants to retrieve. SOLUTION: When a user inputs retrieval information that shows information he wants to retrieve in an image, a map information managing part 6 extracts a structure which has attribute information that fits the retrieval information, creates retrieval result information that consists of the retrieval information and the attribute information and transfers it to a controlling part 9A. The part 9A gives the retrieval result information to a label information creating part 7A. After the part 7A seeks the name of the structure or its attribute information and its assigned position in a normal procedure, it confirms whether or not each structure is included in the retrieval result information. When a certain structure exists in the retrieval result information, a label intra- information retrieval result flag is made '1'. A label information outputting part 8A receives label information and about an item where a retrieval result flag is '1', shows it on a visual device so that the user can understand that it is a retrieval result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention superimposes and displays, on an image display device, geographical information about each partial area in an image taken by a user using a landscape image input device such as a camera. The present invention relates to a device that teaches a user by voice guidance or the like.

[0002]

2. Description of the Related Art Conventionally, there have been various navigation systems as systems for instructing users on geographical information about the area where the users are located.

FIG. 14 is a block diagram of a navigation device disclosed in Japanese Patent Application Laid-Open No. 8-273000. This device receives a vehicle position data and a motion data, and receives a position update unit 71 that updates the vehicle position with reference to the road map data, and displays display road data and display background data based on the map data and the like. Display data generator 7 to be generated
2, a three-dimensional moving image data creating unit 73 for creating three-dimensional moving image data based on these display data, and a storage unit 74.
When a travel route including a waypoint is set in advance, the device has a function of setting a route while viewing a real moving image display screen along a road that actually exists instead of a map screen.

According to this device, when a user travels along an actual route, the user can see a moving image display (for example, FIG. 15) along the route.

[0005]

However, in the case of using the same device, the human finally associates the real scene with the geographical information in the computer world with the naked eye, so that the person in the real scene can be used. You have to recognize what is. In other words, what is the actual building, road or mountain in front of the user is displayed on a moving image, and based on the symbols etc. in the map, humans rely on the naked eye to work the human brain unconsciously. You have to understand the work of mapping. At street corners, etc., comparing the map on the computer with the actual scenery, grasping the direction and finding landmarks, gazing at the direction, understanding the characteristics of the building in that direction, and then looking at the map again Understand what the building is.

[0006] For this reason, there is a problem that the trouble of comparing the map on the computer with the actual scenery many times and associating with the human cannot be omitted. Especially in the dark or at night, it is difficult to see the actual scenery and to take measures.

An object of the present invention is to provide a user with information on geographical information on a computer and each part in an image of a real scene (hereinafter referred to as a landscape image) in association with each other, and to search for information that the user wants to search. Is to provide a search-type landscape label device and a system for displaying a display.

[0008]

According to the present invention, map data on a computer is created in advance as three-dimensional data, and a position at which an image (hereinafter referred to as a landscape image to distinguish it from a CG image) is input. , Camera angle, focal length, and image size at the time of shooting, and computer graphics (hereinafter referred to as CG) when viewed from the position at the time of shooting the actual scenery, the camera angle, and the focal length in the three-dimensional map space on the computer. This is to achieve the association by acquiring geographical information in the image and superimposing and displaying the geographical information on a landscape image which is a real scene. The geographic information is the name of a structure or the like or its attribute information in an image, and the attribute information means information on all attributes (for example, contour, color, etc.) of the structure. In this specification, the term structure is used to mean all data having a certain geographical structure in the map DB, including natural topography such as mountains, rivers, and the sea, in addition to artificial structures. In obtaining geographic information, a landscape image is obtained based on a camera position, a camera angle, a focal length, and an image size, and a structure of a plurality of images is obtained. The position of the landscape image where the structure is to be taken (hereinafter referred to as the assigned position) is obtained, and the name or attribute information of the structure is superimposed and displayed.

Further, in order to further improve the accuracy of associating the structure in the landscape image with the structure in the CG image,
The previously acquired structure is associated with each partial region of the landscape image by pattern matching. A CG image is created on the basis of the acquired structure, a partial region in the CG image is associated with the partial region of the landscape image by pattern matching, and the structure as a source of the associated partial region Ask for.

Here, an example of a method for creating a CG image will be described. By accessing the three-dimensional map DB based on the camera position, camera angle, focal length, and image size obtained earlier,
Obtain the visual field space in the three-dimensional map space. A structure in the visual field space is obtained, and three-dimensional data of each structure is three-dimensionally projected and converted onto the projection plane using the camera screen as a projection plane. Further, of the line data constituting the projected figure of each structure, line data hidden by other structures and invisible is erased using a normal vector method or the like. The CG image is divided into regions based on the line data remaining after the elimination of the hidden lines. 3D map DB
Is used, the name of the structure that is the basis of the area can be associated with each area.

Then, the structure name of the partial area of the CG image associated with each partial area of the landscape image is extracted by pattern matching. The position coordinates of the actual scenery image on which the extracted structure name is to be superimposed are obtained by three-dimensionally projecting the position coordinates of the structure in the three-dimensional map space onto the projection plane. Label information is created from the position coordinates of the actual scenery image on which the extracted structure name is to be superimposed. Based on the label information, the structure name is superimposed on a landscape image as a real scene and displayed on a visual device.

In the present invention, further, a search information input means is provided, and when the user inputs information that the user desires to search in the landscape image, the information is received as search information and passed to the control means. Examples of search information include the type of business of the owner of a structure such as a restaurant or a bank, zoning information on land such as XX town or what number of streets, or height such as how many meters or more objects are high. There can be information etc. The search information may be any information as long as the information is related to map information. The control means passes the search information to the map information management means. The map information management unit extracts a structure having attribute information that matches the received search information, creates search result information including the search information and the attribute information that matches the search information, and passes it to the control unit.

The control means also passes search result information when passing various information to the label information creating means in a normal procedure. The label information creating means obtains the name of the structure or its attribute information and its assigned position in a normal procedure, and then checks whether or not each structure is included in the search result information. If a certain structure exists in the search result information, information indicating that the structure is a search result is added to the label information. For example, a search result flag is added following the structure name or its attribute information and its assigned position. The generated label information is passed to the control means. When the control unit passes the label information to the label information output unit, the label information output unit allows the visual device to recognize that the item whose search result flag is “1” in the label information is the search result. (For example, change the drawing color to a specific color) and display it on the visual device.

[0014] Further, the search information input means may be constituted so as to be capable of designating detection of search information invisible in the landscape image (data hidden behind other structures). In this case, the label information is constituted by the search result identification / hidden surface flag of the name and outline of the structure, which is the search result on the hidden surface when the label information is created, and the position where the structure is provided. The label information output means displays the outline of the search result structure on the hidden surface in a skeleton with a specific drawing color.

[0015] The search-type landscape labeling apparatus of the present invention comprises: an image obtaining means for obtaining an image; a position information obtaining means for obtaining a camera position at the time of obtaining the image; a camera angle, a focal length, and an image at the time of obtaining the image. Camera attribute information acquisition means for acquiring size, search information input means for entering search information indicating information that the user wants to search in an image, map information is managed, and the acquired camera position, camera angle, focal length, and image are acquired. Obtain a visual space in the map information space based on the size, obtain a structure existing in the visual space, and, when receiving the search information, obtain a structure having attribute information matching the search information. Map information management means for extracting and creating search result information including search information and the attribute information, and creating label information including the name of the structure or its attribute information and the assigned position; For a structure in the search result information, label information creating means for adding search result identification information indicating that the structure is a search result in the label information, and information on an assigned position in the acquired label information The name of the structure or its attribute information is superimposed at a position in the image corresponding to the above, and the superimposed image is displayed on a visual device. At this time, for items to which search result identification information in the label information is added, And an information output means for displaying the search result so that the user can recognize the search result, and a control means for controlling each of the above means.

According to another aspect of the present invention, there is provided a retrieval type landscape labeling apparatus, comprising: an image acquisition unit for acquiring an image; a position information acquisition unit for acquiring a camera position at the time of image acquisition; a camera angle, a focal length, and an image at the time of image acquisition. Camera attribute information acquiring means for acquiring a size, image processing means for dividing the acquired image into a plurality of partial areas, search information input means for inputting search information indicating information that the user wants to search in the image, and map information And calculate the visual field space in the map information space based on the acquired camera position, camera angle, focal length, and image size.
Acquiring a structure existing in the visual field space, and upon receiving the search information, extracting a structure having attribute information matching the search information, and retrieving search result information including search information and the attribute information. Map information management means to be created, the acquired structure is associated with the partial region of the image, and label information including the name or attribute information and the assigned position of the associated structure is created; For a structure in the search result information, label information creating means for adding search result identification information indicating that the structure is a search result in the label information, and the name of the structure in the label information or its The attribute information is superimposed on the position corresponding to the assigned position in the image, and the superimposed image is displayed on a visual device, and at this time, the item to which the search result identification information in the label information is added For having a label information output means for displaying such it can be seen that the user is a search result, a control means for controlling said respective means.

A search-type landscape labeling system according to the present invention comprises a landscape labeling terminal and a landscape labeling center, wherein the landscape labeling terminal includes an image acquisition unit for acquiring an image and a position information acquisition for acquiring a camera position at the time of image acquisition. Means, camera attribute information acquiring means for acquiring a camera angle, a focal length, and an image size at the time of image acquisition; image processing means for dividing the acquired image into a plurality of partial areas; and search information indicating information that the user wants to search for A search information input unit for inputting the information, area information division of the image, and transmitting the camera position, the camera angle, the focal length, and the image size to the landscape labeling center via a communication network, Communication control means for receiving label information from a labeling center, and names of structures in the label information or attributes thereof Information is superimposed on a position corresponding to the application position in the image, and the superimposed image is displayed on a visual device. At this time, the user searches for items added from the search result identification information in the label information. A label information output unit for displaying the result so that it can be understood as a result, and a terminal control unit for controlling each of the above units, wherein the landscape labeling center receives the image area from the landscape labeling terminal via the communication network. Communication control means for receiving information regarding division, the camera position, the camera angle, the focal length, and the image size, transmitting the label information to the landscape labeling terminal, managing map information, and receiving the camera position The visual field space is obtained in the map information space on the basis of the camera angle, the focal length, and the image size, and the structure existing in the visual field space is obtained. Receiving the information, extracting a structure having attribute information that matches the search information, creating map result management information including search information and the attribute information, and a map information management unit, Correlate the acquired structure, create the label information including the name or attribute information and the assigned position of the associated structure, for the structure in the search result information, in the label information It has a label information creating means for adding search result identification information indicating that the structure is a search result, and a center control means for controlling each of the means.

According to the embodiment of the present invention, the label information creating means includes, for the structure in which the information specified by the user in the search information in the search result information is not visible in the image, the name or attribute information of the structure; The coordinates of the outline of the structure and the assigned position and the search result identification / hidden surface information are added to the label information, and the label information output means outputs the structure of the structure to which the search result identification / hidden surface information in the label information is added. Displays outlines in a skeleton with a specific drawing color.

According to an embodiment of the present invention, the label information creating means creates a CG image which is a computer graphics image based on the acquired structure, and the partial area of the image is subjected to the CG image by pattern matching. It associates with the partial area in the image, finds the structure of the associated partial area, and creates label information including the name or attribute information of the structure and the assigned position.

According to an embodiment of the present invention, the label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, erases a structure that cannot be seen from the viewpoint, creates a CG image, and creates a CG image. The CG image is divided into partial regions by the contour line of the middle partial region, and the partial region of the image is
The partial region of the G image is associated with the partial region of the CG image by pattern matching, the structure that is the basis of the partial region of the CG image associated with the partial region of the image is obtained, and the name or attribute information of the structure and the assignment Create label information including location.

[0021]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of a search-type landscape labeling apparatus according to a first embodiment of the present invention.

The search type landscape labeling apparatus according to the present embodiment acquires an image, for example, an image acquisition unit 1 which is a digital camera, and acquires a camera position when acquiring an image, for example, a position information which is a GPS receiver. An acquisition unit 2, a camera attribute information acquisition unit 3, which is, for example, a three-dimensional electronic compass attached to a digital camera, for acquiring a camera angle, a focal length, and an image size when acquiring an image; A search information input unit 4 for inputting desired information (search information), and managing the map information, obtaining a visual field space in the map information space based on the acquired camera position, camera angle, focal length, and image size. When a structure existing in the visual field space is acquired and the search information is received, a structure having attribute information matching the search information is extracted, and the structure including the structure and the attribute information is extracted. For example, a map information management unit 6 which is a map DB management program for creating search result information, and label information including a name or an outline of a structure and an assigned position are created, and label information is included for a structure in search result information. In the search result information, a search result flag indicating that the structure is a search result is set to “1”. Further, in the search result information, for a structure hidden by other structures and not visible in the image, the name of the structure Alternatively, a label information creating unit 7A that creates the label information including the attribute information, the outline of the structure, and the coordinates of the assigned position, and both the search result flag and the hidden surface flag are “1”
And superimpose the name or outline of the map information on the position in the image corresponding to the information of the assigned position in the acquired label information, and display the superimposed image on a visual device (not shown). For items with a search result flag of “1”, the drawing color is changed to a specific color and displayed on a visual device so that the user can recognize that the item is a search result, and for a structure with a hidden surface flag of “1”. Is a label information output unit 8A that displays the outline in a skeleton with a specific drawing color,
The control unit 9 controls A.

FIG. 2 is a configuration diagram of a search type landscape labeling apparatus according to a second embodiment of the present invention, and FIG. 3 is a flowchart of processing of the landscape labeling apparatus of FIG.

The landscape labeling apparatus according to this embodiment acquires a landscape image, for example, an image acquisition unit 1 which is a digital camera, and acquires a camera position at the time of acquiring an image.
For example, a position information acquisition unit 2 that is a GPS receiver, and a camera angle, a focal length, and an image size that are also acquired when acquiring an image.
A camera attribute information acquisition unit 3 which is a three-dimensional electronic compass, a search information input unit 4 for inputting search information indicating information (search information) that the user wants to search in an image, and the obtained image is divided into a plurality of partial areas The image processing unit 5 manages geographic information, obtains a visual space in the map information space based on the acquired camera position, camera angle, focal length, and image size, and acquires a structure existing in the visual space. Upon receiving the search information, the map information management unit 6 extracts a structure having attribute information matching the search information, and creates search result information including the search information and the attribute information; The acquired structures are associated with each other by pattern matching, label information including the name or attribute information of the associated structures and the assigned position is created, and the structures in the search result information are identified. Indicates a label information creation unit 7B that sets a search result flag indicating that the structure is a search result in the label information to “1”, and displays the name or attribute information of the structure in the generated label information in the image. Superimposed on the position corresponding to the application position of the, the superimposed image is displayed on a visual device,
At this time, for the item whose search result flag is “1” in the label information, the drawing color is changed to a specific color and displayed so that the user can recognize the search result, and the hidden surface flag is “1”. And a label information output unit 8B for displaying the outline of the structure in a specific drawing color with a skeleton.
To 8B.

Next, the operation of this embodiment will be described in detail with reference to FIG.

First, a user inputs search information indicating information to be searched in an image (step 20). Next, in order for the control unit 9B to acquire information on the landscape image, the position information acquisition unit 2, the camera attribute information acquisition unit 3, the image acquisition unit 1
Send a processing start command to Location information acquisition unit 2
Receives a command from the control unit 9B, collects position information (camera position) every second by a GPS receiver or the like, and passes it to the control unit 9B (step 21). Here, the time interval is not limited to the unit of seconds, but may be any value. Camera attribute information acquisition unit 3
Obtains the camera angle of a landscape image recording device such as a camera at the time of image capturing as a set of a horizontal angle and an elevation angle in response to a command from the control unit 9B (step 22), and focuses on a landscape image device having a zoom function at the same time. The distance is obtained (step 23). The image acquisition unit 1 receives a command from the control unit 9B, acquires a landscape image every second, and passes it to the control unit 9B (step 24). Since the image size is fixed for each landscape image device, the control unit 9B holds the image size information. The control unit 9B holds the collected information as a landscape image file.

FIG. 4 shows the file format of the data structure of the landscape image file. The landscape image file has header information and image data. The header information includes position information, camera angle information, focal length, time information, image size, type, and size of an image file. As location information,
East longitude, north latitude, and elevation data (for example, 137 degrees east 5 degrees
5 minutes 10 seconds, latitude 34 degrees 34 minutes 30 seconds north, altitude 101m3
3 cm). The camera angle includes horizontal angle data and elevation angle data (for example, horizontal angle 254 degrees clockwise, elevation angle 15 degrees, etc.). The focal length data is the focal length (for example, 28 mm) of the camera lens at the time of capturing an image. The time information includes the time at the time of shooting (for example, 15:06:17, January 31, 1997, Japan time). As the image size of the image file, the vertical and horizontal pixel sizes (for example, 64
0 × 480). File type (TIF
E format, 8-bit color, etc.). It also has the number of bytes of the file (307.2 KB, etc.). The image data itself has, for example, a binary format.

After storing the landscape image file, the control unit 9B instructs the image processing unit 5 to extract the outline from the landscape image and divide the landscape image into a plurality of regions.
The image processing unit 5 performs a differentiation process on the basis of the density difference in the landscape image to extract a contour line (step 2).
5) The region is divided by performing labeling with the outline as a boundary (step 26). Note that the technical term of labeling used here is a technical term used in image area division, and is different from landscape labeling, which is the name of the present invention. As a procedure, first, the image is converted into a monochrome grayscale image. Since the contour is a portion where the brightness changes suddenly, the contour is extracted by performing a differentiation process to find a portion where the differential value is larger than a threshold value. At this time, the line width of the outline is one pixel, and the outlines are connected. For this purpose, a thinning process is performed to obtain a connected line having a line width of one pixel. Where the differentiation process,
It is sufficient to use a conventional method for the thinning processing.

The obtained contour is regarded as the contour of the area, and an operation of numbering the area constituted by the contour is performed.
The largest number among the numbers is the number of areas, and the number of pixels in the area indicates the area of the area. FIG. 9 shows an example in which a landscape image is divided into a plurality of partial areas. Note that a measure of similarity (closeness) between regions may be introduced, and a clustering process may be performed to combine a plurality of regions having similar properties into one region. Any existing clustering method may be used.

When the control section 9B completes the area division processing of the landscape image, it issues a processing request for calculating the view space by passing the header information of the landscape image file to the map information management section 5 and issues a map information management. The unit 6 accesses the map DB (step 27). Examples of the map information management unit 6 include:
There is a map database program. Map information management unit 6
Manages three-dimensional map data. Although two-dimensional map data may be used, in this case, since there is no height information, the accuracy of the position at which the labeling is applied to the actual scenery is inferior. When the two-dimensional map data is used as the basis, the processing is performed by supplementing the height information. For example, in the case of two-dimensional data of a house, if there is floor information indicating the number of floors of the house, the number of floors is multiplied by a certain number to estimate the height of the house, and the two-dimensional data is obtained. 3D data is created based on the height information. Even when there is no floor information, the height information can be estimated by assigning a certain number of heights according to the area of the house figure, and similarly, three-dimensional data is created based on the estimated height information. . In this way, three-dimensional data is created and the process proceeds.

FIG. 5 shows an example of three-dimensional map data. FIG.
(1) shows a map information space expressed in two dimensions, and FIG.
(2) shows a map information space expressed in three dimensions. This 3
With respect to the three-dimensional map information space, the map information management unit 6 calculates a visual field space based on the header information of the landscape image file in response to a command from the control unit 9B (step 28). FIG. 6 shows a calculation example of the visual field space. First, the XY axes stretch in the horizontal direction,
The Z axis extends in the vertical direction. The position of the viewpoint E is set in the three-dimensional map information space from the position information in the header information of the landscape image file. For example, 137 degrees east longitude 55
Minute 19 seconds, latitude 34 degrees 34 minutes 30 seconds north, altitude 101m33
If it is cm, the corresponding coordinates in the map mesh number corresponding to the cm are set. Similarly, the camera angle direction is set based on the horizontal angle and the elevation angle in the camera angle information in the header information.
The focal point F is set at a point advanced from the viewpoint E by the focal length on a straight line representing the camera angle method. The line-of-sight direction vector is a unit vector having a length of 1 from the viewpoint E on the straight line. In the image size of the landscape image file, the width x on the X-axis of the camera screen is set from the horizontal size, and the width y on the Y-axis is set from the vertical size. The horizontal x vertical y plane is set so as to be perpendicular to the camera angle direction with respect to the line of sight vector and to include the focal point F. Straight lines connecting the four corner points of the camera screen are obtained from the coordinates of the viewpoint E, and a three-dimensional space formed by four half lines extending from the viewpoint E is defined as a visual field space. FIG. 7 shows an example of the visual field space in the three-dimensional map space. XZ 3D map space
It is viewed from a plane. FIG. 8 is a cross-sectional view taken along the XY plane of a space that belongs to the viewing space in FIG. In the example of FIG. 7, buildings and mountains in the viewing space are included.

Further, the map information management section 6 determines a structure existing in the determined visual space. For each structure, it is calculated whether or not each vertex constituting the solid representing the structure exists in the internal region of the viewing space. Usually, a two-dimensional map space is divided by a two-dimensional mesh of a fixed size. As a method of cutting a mesh in the three-dimensional map space, a mesh is cut at regular intervals in the height direction in addition to the two-dimensional mesh in the vertical and horizontal directions. The space is divided by a rectangular parallelepiped unit space. First, the presence or absence of an overlapping portion between the rectangular parallelepiped unit space and the visual field space is checked, and the number of the three-dimensional unit map space having the overlapping portion is obtained. The number of the three-dimensional unit map space here is the same as a so-called mesh number. For structures in the overlapping 3D unit map space,
Investigate whether there is any overlap with the visual field space. A straight line connecting the coordinates of the vertices constituting the structure and the coordinates of the viewpoint is obtained. If the straight line has an intersection with the camera screen of FIG. 8, it is within the visual field space. If at least one vertex of the plurality of vertices constituting the structure satisfies this condition, the structure has an overlapping portion with the viewing space.

When the map information management unit 6 receives the search information from the control unit 9A, it extracts structures having attribute information matching the search information and creates search result information including the search information and the attribute information. (Step 29).

If the structure is included in the view space or a part of the structure, the process enters a process of three-dimensionally projecting each structure onto the projection surface using the camera screen as a projection surface (step 30). ). Here, as shown in FIG. 8, after the point P is re-expressed in the coordinate system based on the viewpoint E based on the following equation (1), the point P is projected on the camera screen to obtain the intersection Q. .

[0036]

(Equation 1) Here, point P = (x, y, z): coordinates of the vertices constituting the structure point E = (ex, ey, ez): coordinates of the viewpoint vector L = (lx, ly, lz): branch line direction vector (Unit vector) Point P ′ = (x ′, y ′, z ′): coordinates r = (lx ² + ly ² ) ^1/2 when the point P is expressed based on the viewpoint E of the point P = (x, y): The projection point t of the point P on the camera screen is the focal length. In the three-dimensional projection conversion, first, for each structure, the surface on which the vertex extends is obtained. For example, in the case of a structure represented by a rectangular parallelepiped, six surfaces are obtained. When each plane is projected onto the camera screen, the distance between the viewpoint and the corresponding point on the plane is calculated for each pixel on the camera screen included in the projection area, and stored as a depth value (Z value) in the memory. Store. For each surface of each structure, the depth value (Z
Value) and store it in memory. Note that z ′ in (Equation 1) represents a depth value (Z value) from the viewpoint.

Among the structures three-dimensionally transformed on the camera screen, there are structures that can be seen from the viewpoint and structures that cannot be seen. Among them, it is necessary to find only the structure that can be seen from the viewpoint, and to find the surface on the opposite side from the viewpoint or the surface that is blocked by other structures. Therefore, hidden surface processing is performed (step 3
1). There are various methods for processing the hidden surface.
Use the buffer method. Other scan line methods and ray tracing methods may be used.

A pixel on the camera screen is arbitrarily determined, and a plane having the smallest depth value for the pixel is determined. As described above, when processing is sequentially performed on each surface of each structure, the surface closest to the viewpoint is left for each pixel on the camera screen. The plane closest to the viewpoint is determined for each pixel on the camera screen, and pixels on the camera screen that share the plane closest to the viewpoint generally form an area.
There can be a plurality of regions consisting of pixels with the common surface being the closest surface. The region obtained in this way is a region obtained by performing a three-dimensional projection conversion of the partial region of the structure seen from the viewpoint. Surfaces on the other side of the view and those obstructed by other structures have been erased.

The area thus formed forms a CG image area (step 32).

With respect to the vertex coordinates of the two-dimensional figure constituting the CG image area, three-dimensional coordinates before projection transformation are obtained, and the correspondence between the two is stored in the memory as link information. It is used to determine which structure the two-dimensional area is a projection of based on the link information.

Based on the line data remaining after erasing hidden lines,
The CG image is divided into regions. Since the three-dimensional map DB is used, the name of the structure serving as the basis of each area can be associated with each area. The divided areas of the CG image are numbered sequentially. FIG. 10 shows an example in which a CG image is divided into a plurality of partial regions.

When the CG image area division processing is completed, the control section 9B instructs the label information creation section 7B to associate the CG image division area with the landscape image division area. The label information creating unit 7B associates the divided regions of the CG image with the divided regions of the landscape image by template matching (step 33, see FIG. 11).

Among the divided regions of the landscape image, the regions are sequentially associated with the divided regions of the CG image in ascending order of the number (for example, No. 1). For matching, any of the conventional matching methods may be used, but here, a simple template matching method is used. That is, two regions to be compared are superimposed, and when the ratio of overlapping portions is equal to or greater than a certain ratio determined as a threshold, the regions are associated as regions relating to the same structure. For example, the coordinate value of each pixel in the first divided region R1 of the landscape image is set to (A, B). The value of the pixel at coordinates (A, B) is 1 because it is inside the area. In the first divided area S1 of the CG image,
If the coordinates (A, B) are within the area S1, the pixel value is 1 and overlaps, but if the coordinates (A, B) are outside the area S1, the pixel value is 0 and does not overlap. In this way, the overlap coefficient K (A, B) at the coordinates (A, B) is determined to be 1 when they overlap and 0 when they do not overlap. The coordinates (A, B) are moved within the region R1, and the overlap coefficient K
(A, B) is obtained. Then, with respect to the number N1 of coordinates (A, B) moved in the region R1, the overlap coefficient K (A,
The number N2 of coordinates where B) was 1 is obtained, and when N1 / N2 is equal to or larger than the threshold value, it is determined that the divided region R1 of the landscape image and the divided region S1 of the CG image correspond to each other. This association is performed from the first to the last divided area of the landscape image. In addition, as the matching method, an evaluation function that has the same value even if there is a slight displacement in the XY directions may be used.

After associating the partial area of the landscape image with the partial area of the CG image, the label information creating section 7B extracts the structure of the associated partial area (step 3).
4) obtaining information to be superimposed for each partial area of the landscape image,
The process (step 35) of creating label information together with the position to be superimposed is entered. First, a partial area of the CG image corresponding to the partial area of the landscape image is extracted. The partial region of the extracted CG image is originally obtained by three-dimensionally projecting and converting a certain surface of the three-dimensional structure in the three-dimensional map space onto the camera screen. Therefore, the surface of the three-dimensional structure on which the three-dimensional projection conversion is based is obtained using the depth value (Z value) of the partial region of the CG image as a key. The link information created at the time of the three-dimensional projection conversion may be used as a key. Based on the surface of the original structure, 3
Access the dimensional map DB to obtain the name or attribute information of the structure. Here, the attribute information means information accompanying the structure, and may be any information as long as the information is related to the structure. Then, the position coordinates where the name or attribute information is to be superimposed are determined for the partial area of the landscape image.
How to decide may be decided any way. For example, the center of gravity of the figure extending the partial area may be used. Label information is created from the name or attribute information of the structure and the assigned position coordinates.
The associated structure is compared with the search result information, and the search result flag in the label information is set to “1” for the structure in the search result information. For a structure that is hidden behind other structures and cannot be seen, the name or attribute information of the structure and the outline of the structure are added to the label information, and the search result flag and the hidden surface flag are set to “1”. Table 1 shows an example of label information.

[0045]

[Table 1] After completing the creation of the label information, the label information creating unit 7B passes the label information to the control unit 9B.

Upon receiving the label information, the control section 9B
It instructs the label information output unit 8B to display the label information on the visual device. Here the visual equipment is
Includes video display devices such as displays and head mounted displays. The name or attribute information of the structure in the label information is superimposed on the position in the landscape image (step 36), and the superimposed landscape image is displayed on the video display device (step 37). At this time, the drawing color is changed to a specific color for the structure whose search result flag is “1”, and the outline of the structure whose hidden surface flag is “1” is displayed in the specific drawing color. indicate. FIG. 12 shows an example of a landscape image on which label information is superimposed.

When the label information output section 8B outputs the label information, it notifies the control section 9B of the completion of the output. Control unit 9B
Upon receiving the output completion notification, if the landscape labeling process is to be performed continuously, the series of processing procedures described above are executed again.

FIG. 13 is a configuration diagram of a landscape labeling system in which the search-type landscape labeling device of FIG. 2 is applied to a communication system.

The landscape labeling system includes a landscape labeling terminal 40, a landscape labeling center 50, and a communication network 60.
It consists of.

The landscape labeling terminal 40 acquires an image acquiring section 41 for acquiring an image, a position information acquiring section 42 for acquiring a camera position when acquiring an image, and acquires a camera angle, a focal length, and an image size when acquiring an image. Camera attribute information acquisition unit 43
A search information input unit 44 for inputting search information indicating information that the user wants to search, an image processing unit 45 for dividing an acquired image into a plurality of partial regions, information on image region division, camera position and camera angle , The focal length, and the image size via the communication network 60 to the landscape labeling center 50 and receive the label information from the landscape labeling center 50, and the name or attribute information of the structure in the label information. The superimposed image is superimposed on the position corresponding to the assigned position in the image, and the superimposed image is output to the visual device. At this time, for the item of which the search result identification flag in the label information is “1”, the user is the search result The drawing information is displayed in such a manner that the drawing color is changed so that the user can see the label, and the outline of the structure is displayed in a skeleton with a specific drawing color for the structure whose hidden surface flag is “1”. And parts 47, and a terminal control unit 48 for controlling the respective units.

The landscape labeling center 50 has a communication network 60.
A communication control unit 53 that receives information related to the area division of the image, a camera position, a camera angle, a focal length, and an image size from the landscape labeling terminal 40 via the interface and transmits label information to the landscape labeling terminal 40; Manage and obtain the view space in the map information space based on the received camera position, camera angle, focal length and image size, acquire the structure existing in the view space, and receive the search information,
A map information management unit 51 for extracting a structure having attribute information matching the search information and creating search result information including the search information and the attribute information; Label information including the name or attribute information of the associated structure and the assigned position is created by pattern matching, and the search result flag in the label information is set to “1” for the structure in the search result information. And, for structures that are hidden in other images in the search result information and are not visible in the image, include the name or attribute information of the structure, the outline of the structure and the assigned position,
It is composed of a label information creation unit 52 for creating label information in which both the search result flag and the hidden surface flag are “1”, and a center control unit 54 for controlling the above-mentioned units.

The operation of this system is the same as the operation of the apparatus shown in FIG. .

[0053]

As described above, according to the present invention, the geographical information on the computer and each part in the scenery image of the actual scene can be presented to the user in association with each other.
The information that the user wants to search can be obtained without the human being comparing the map on the computer with the actual scenery and associating with the human.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a search-type landscape labeling device according to a first embodiment of the present invention.

FIG. 2 is a configuration diagram of a search-type landscape labeling device according to a second embodiment of the present invention.

FIG. 3 is a flowchart of processing of a search-type landscape labeling device according to a second embodiment.

FIG. 4 is a diagram showing a data structure of a landscape image file.

FIG. 5 is a diagram showing an example of a two-dimensional map (FIG. 1 (1)) and its three-dimensional map (FIG. 2 (2)).

FIG. 6 is a diagram illustrating a calculation method of a visual field space.

FIG. 7 is a diagram showing an example of a visual field space in a three-dimensional map space.

FIG. 8 is a diagram showing an example of a projection view.

FIG. 9 is a diagram illustrating an example of area division of a landscape image.

FIG. 10 is a diagram illustrating an example of area division of a CG image.

FIG. 11 is an explanatory diagram of pattern matching between a partial area of a landscape image and a partial area of a CG image.

FIG. 12 is a diagram illustrating an example of superimposition of label information on a landscape image.

FIG. 13 is a configuration diagram of a landscape labeling system of the present invention.

FIG. 14 is a configuration diagram of a navigation device disclosed in Japanese Patent Application Laid-Open No. 8-273000.

FIG. 15 is a diagram illustrating a display example of a moving image.

[Explanation of symbols]

 Reference Signs List 1 image acquisition unit 2 position information acquisition unit 3 camera attribute information acquisition unit 4 search information input unit 5 image processing unit 6 map information management unit 7A, 7B label information creation unit 8A, 8B label information output unit 9A, 9B control unit 20 to 37 Step 40 Scenery labeling terminal 41 Image acquisition unit 42 Position information acquisition unit 43 Camera attribute information acquisition unit 44 Search information input unit 45 Image processing unit 46 Communication control unit 47 Label information output unit 48 Terminal control unit 50 Landscape labeling center 51 Map information Management unit 52 Label information creation unit 53 Communication control unit 54 Center control unit 60 Communication network

Continuation of the front page (51) Int.Cl. ⁶ Identification symbol FI G06T 1/00 G09G 5/36 510C G08G 1/0969 520N G09G 5/36 510 G06F 15/40 370C 520 15/62 380 (72) Inventor Takano Shoji Nippon Telegraph and Telephone Corporation, 3-19-2 Nishi Shinjuku, Shinjuku-ku, Tokyo (72) Inventor Takeshi Ikeda 192-1 Nishi Shinjuku, Shinjuku-ku, Tokyo Nippon Telegraph and Telephone Corporation

Claims (9)

[Claims] An image acquisition unit for acquiring an image; a position information acquisition unit for acquiring a camera position at the time of image acquisition; and a camera attribute information acquisition for acquiring a camera angle, a focal length, and an image size when the image is acquired. Means, a search information input means for inputting search information indicating information that the user wants to search in the image, and map information, and managing the map information, and generating a map information space based on the acquired camera position, camera angle, focal length, and image size. In the visual field space in the, to obtain a structure existing in the visual space, and, when receiving the search information, extract a structure having attribute information that matches the search information, the search information and the Map information management means for creating search result information comprising attribute information; and label information including the name of the structure or its attribute information and the assigned position, and a structure included in the search result information. A label information creating means for adding search result identification information indicating that the structure is a search result in the label information, and a structure at a position in the image corresponding to the information of the assigned position in the acquired label information. Superimpose the name of the object or its attribute information,
An information output unit that displays the superimposed image on a visual device, and displays an item to which the search result identification information in the label information is added so that the user can recognize the search result; A search-type landscape labeling device having control means for controlling each means. 2. An image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position at the time of image acquisition, a camera attribute information acquisition unit for acquiring a camera angle, a focal length, and an image size at the time of image acquisition. An image processing means for dividing the acquired image into a plurality of partial areas; a search information input means for inputting search information indicating information that the user wants to search in the image; Obtain a visual space in the map information space based on the angle, the focal length, and the image size, acquire a structure existing in the visual space, and, when receiving the search information, obtain an attribute matching the search information. Map information management means for extracting a structure having information and creating search result information including the search information and the attribute information, and associating the acquired structure with the partial region of the image In addition, label information including the name or attribute information of the associated structure and the assigned position is created, and for the structure in the search result information, the structure is a search result in the label information. Label information creating means for adding search result identification information indicating the name of a structure in the label information or attribute information thereof is superimposed on a position corresponding to a given position in the image, and the superimposed image is displayed on a visual device. Label information output means for displaying and, at this time, for the item to which the search result identification information in the label information is added, so that the user can recognize that the search result is a search result; and control means for controlling each of the means Search-type landscape labeling device with 3. The structure according to claim 3, wherein the label information creating means includes a structure name or attribute information, a contour of the structure, and a structure in which information specified by the user in the search information is not visible in the image. The coordinates of the application position and the search result identification / hidden surface information are added to the label information, and the label information output means draws a specific outline of the structure to which the search result identification / hidden surface information in the label information is added. The device according to claim 1, wherein the skeleton display is performed in color. 4. The label information creating means creates a CG image, which is a computer graphics image, based on the acquired structure, and assigns the partial area of the image to a partial area in the CG image by pattern matching. 4. The apparatus according to claim 2, wherein the association is determined, a structure of the associated partial area is obtained, and label information including the name or attribute information of the structure and the assigned position is created. 5. 5. The label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, creates a CG image by erasing a structure that cannot be seen from a viewpoint, and generates a contour of a partial area in the CG image. Dividing the CG image into partial areas by lines,
The partial area of the image and the partial area of the CG image are associated with each other by pattern matching, and a structure based on the partial area of the CG image associated with the partial area of the image is obtained. 4. The apparatus according to claim 2, wherein label information including the name or attribute information of the object and the assigned position is created. 6. A landscape labeling terminal and a landscape labeling center, wherein the landscape labeling terminal is an image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position at the time of image acquisition, and a camera at the time of image acquisition. Camera attribute information acquisition means for acquiring an angle, a focal length, and an image size; image processing means for dividing an acquired image into a plurality of partial regions; and search information input means for inputting search information indicating information that a user wants to search. Transmitting the information on the area division of the image, the camera position, the camera angle, the focal length, and the image size to the landscape labeling center via a communication network,
Communication control means for receiving label information from the landscape labeling center, and superimposing the name of the structure in the label information or its attribute information at a position corresponding to the application position in the image, and the superimposed image to a visual device A label information output means for displaying, at this time, for the item to which the search result identification information in the label information is added, so that the user can recognize the search result as a search result, and a terminal for controlling each of the above means Having control means, the landscape labeling center receives information related to area division of the image, the camera position, the camera angle, the focal length, and the image size from the landscape labeling terminal via the communication network. Communication control means for transmitting the label information to the landscape labeling terminal; managing map information; and receiving the camera position, the camera angle and the focus. Obtaining a visual space in the map information space based on the distance and the image size, acquiring a structure existing in the visual space, and receiving the search information, a structure having attribute information matching the search information Map information management means for extracting an object and creating search result information including the search information and the attribute information, and associating the acquired structure with the partial region of the image, and the associated structure The label information including the name or attribute information of the object and the assigned position is created, and for the structure in the search result information, search result identification information indicating that the structure is a search result is included in the label information. A search-type landscape labeling system including a label information creating unit to be added and a center control unit for controlling each of the above units. 7. The structure according to claim 1, wherein said label information means comprises: for a structure in which the information specified by the user in the search result information in the search information is not visible in the image, the name or attribute information of the structure; And the search result identification / hidden surface information are added to the label information, and the label information output means displays the outline of the structure to which the search result identification / hidden surface information is added in the label information in a specific drawing color. 7. The system according to claim 6, wherein a skeleton display is performed. 8. The label information creating means creates a CG image, which is a computer graphics image, based on the acquired structure, and converts the partial area of the image into a partial area in the CG image by pattern matching. 8. The system according to claim 6, wherein the association is determined, a structure of the associated partial area is obtained, and label information including the name or attribute information of the structure and the assigned position is created. 9. 9. The label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, creates a CG image by erasing a structure that cannot be seen from a viewpoint, and generates a contour of a partial region in the CG image. Dividing the CG image into partial areas by lines,
The partial area of the image and the partial area of the CG image are associated with each other by pattern matching, and a structure based on the partial area of the CG image associated with the partial area of the image is obtained. The system according to claim 6, wherein label information including a name or attribute information of an object and an assigned position is created.