JPH1186035A - Distance reference type scenery labeling device and system therefor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To show geographic information on a computer and respective parts in a scenery image of an actual scene to a user while making them correspond to each other. SOLUTION: An image acquisition part 1 acquires the scenery image, a position information acquisition part 2 acquires the position at the time of the image acquisition by the image acquisition part 1, and a camera attribute information acquisition part 3 obtains the camera angle, focal length, and scenery image size. A map information management part 5 finds a visual field space in a map information space from the obtained camera position, camera angle, focal length, and image size to obtain a structure present in the visual field space. A label information generation part 7 makes the structure correspond to partial areas of the scenery image by pattern matching by using the ratio of depth values at respective points by the partial areas of the scenery image and depth values at respective points by partial areas of a CG scenery image to generate the name or its attribute information of the structure and label information including its imparting position. A label information output part 8 superimposes the name or attribute information of the map information on an image corresponding to the information of the imparting position in the label information and displays the superimposed image on a visual equipment.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention superimposes and displays, on an image display device, geographical information about each partial area in an image taken by a user using a landscape image input device such as a camera. The present invention relates to a device that teaches a user by voice guidance or the like.

[0002]

2. Description of the Related Art Conventionally, there have been various navigation systems as systems for instructing users on geographical information about the area where the users are located.

FIG. 15 is a block diagram of a navigation device disclosed in Japanese Patent Application Laid-Open No. 8-273000. This device, when inputting vehicle position data and motion data, generates a position update unit 71 that updates the position of the vehicle with reference to road map data, and generates display road data and display background data based on map data and the like. Display data generator 72
A three-dimensional moving image data creating unit 73 for creating three-dimensional moving image data based on the display data; and a storage unit 74. The user of the navigation device can determine a travel route including a destination and a waypoint. When setting in advance, it has a function of setting a route while watching a real moving image display screen along a road that actually exists instead of a map screen.

According to this device, when a user travels along an actual route, the user can view a moving image display (for example, FIG. 16) along the route.

[0005]

However, in the case of using the same device, the human finally associates the real scene with the geographical information in the computer world with the naked eye, so that the person in the real scene can be used. You have to recognize what is. In other words, humans rely on the naked eye to work the human brain unconsciously, based on the symbols in the map displayed as a moving image, to determine what the actual buildings, roads, and mountains are in front of the user. It must be understood by performing the work of association. For street corners, etc., compare the map on the computer with the actual scenery, grasp the direction, find landmarks, gaze at the direction, understand the characteristics of the building in that direction, look at the map again, Understand what a building is.

[0006] For this reason, there is a problem that the trouble of comparing the map on the computer with the actual scenery many times and associating with the human cannot be omitted. Especially in the dark or at night, it is difficult to see the actual scenery and to take measures.

An object of the present invention is to provide a landscape labeling apparatus and system for associating a user with geographical information on a computer and each part in an image of a real landscape (hereinafter referred to as a landscape image). It is to be.

[0008]

According to the present invention, map data on a computer is created in advance as three-dimensional data, and a position at which an image (hereinafter referred to as a landscape image to distinguish it from a CG image) is input. , Camera angle, focal length, and image size at the time of shooting, and computer graphics (hereinafter referred to as CG) when viewed from the position at the time of shooting the actual scenery, the camera angle, and the focal length in the three-dimensional map space on the computer. This is to achieve the association by acquiring geographical information in the image and superimposing and displaying the geographical information on a landscape image which is a real scene. The geographical information is a name of a structure or the like or its attribute information in an image, and the attribute information means information on all attributes (for example, contour, color, etc.) of the structure. In this specification, the term structure is used to mean all data having a certain geographical structure in the map DB, including natural topography such as mountains, rivers, and the sea, in addition to artificial structures. . In obtaining geographic information, a landscape image is obtained based on a camera position, a camera angle, a focal length, and an image size, and a structure of a plurality of images is obtained. The position of the landscape image where the structure is to be taken (hereinafter referred to as the assigned position) is obtained, and the name or attribute information of the structure is superimposed and displayed.

Further, in order to further improve the accuracy of associating the structure in the landscape image with the structure in the CG image,
The previously acquired structure is associated with each partial region of the landscape image by pattern matching. The present invention utilizes the depth value at each point of the image to perform pattern matching. The depth value at each point of each partial region of the landscape image, which is the distance from the camera position at the time of image acquisition to each point of the landscape image, is determined. Similarly, a depth value that is a distance from the camera position at the time of image acquisition to each point of each partial region of the CG image is obtained. Then, for example, the ratio of the average of the depth value set for each partial region of the landscape image to the average of the depth value set for each partial region of the CG image is obtained. From the ratio, it is determined whether or not the partial area of the landscape image is associated with the partial area of the CG image. Alternatively, the overlapping ratio between the partial region of the landscape image and the partial region of the CG image is obtained, and whether the partial region of the landscape image and the partial region of the CG image are associated with each other based on the ratio of the average depth values and the overlapping ratio of the partial regions. May be determined. A CG image is created based on the acquired structure, a partial region in the CG image is associated with the partial region of the landscape image by pattern matching, and the structure on which the associated partial region is based is defined. Ask. The statistical value is not limited to the average depth value and may be any statistical value obtained from a set of depths.

Here, an example of a method for creating a CG image will be described. By accessing the three-dimensional map DB based on the camera position, camera angle, focal length, and image size obtained earlier,
Obtain the visual field space in the three-dimensional map space. A structure in the visual field space is obtained, and three-dimensional data of each structure is three-dimensionally projected and converted onto the projection plane using the camera screen as a projection plane. Further, of the line data constituting the projected figure of each structure, line data hidden by other structures and invisible is erased using a normal vector method or the like. The CG image is divided into regions based on the line data remaining after the elimination of the hidden lines. 3D map DB
Is used, the name of the structure serving as the basis of the area can be associated with each area.

Then, the structure names of the partial areas of the CG image corresponding to the respective partial areas of the landscape image are extracted by pattern matching. The position coordinates of the actual scenery image on which the extracted structure name is to be superimposed are obtained by three-dimensionally projecting the position coordinates of the structure in the three-dimensional map space onto the projection plane. Label information is created from the position coordinates of the actual scenery image on which the extracted structure name is to be superimposed. Based on the label information, the structure name is superimposed on a landscape image as a real scene and displayed on a visual device.

[0012] A distance reference type landscape labeling apparatus according to the present invention comprises: an image obtaining means for obtaining an image; a position information obtaining means for obtaining a camera position at the time of obtaining an image; a camera angle and a focal length when the image is obtained; Camera attribute information acquiring means for acquiring an image size; image processing means for dividing the acquired image into a plurality of partial areas; and a partial area of the image, which is a distance from a camera position at the time of the image acquisition to each point of the image. Distance information obtaining means for obtaining a depth value at each point for each,
Map information management means for managing map information, obtaining a view space in the map information space based on the acquired camera position, camera angle, focal length and image size, and acquiring a structure existing in the view space; After creating a CG image, which is a computer graphics image, based on the structure acquired by the map information management means, the distance from the camera position at the time of image acquisition to each point in each partial area of the CG image Is obtained by comparing a depth value set at each point for each partial area of the CG image with a depth set at each point for each partial area of the image. Label information creating means for associating with the partial area in the image, obtaining the structure of the associated partial area, and creating label information including the name or attribute information of the structure and the assigned position; A label information output unit that superimposes the name of the structure or its attribute information at a position in the image corresponding to the information of the assigned position in the bell information and displays the superimposed image on a visual device, and controls the above units. It has control means.

According to another aspect of the present invention, there is provided a distance reference type landscape labeling apparatus, comprising: an image acquiring unit for acquiring an image; a position information acquiring unit for acquiring a camera position at the time of acquiring the image; a camera angle and a focus when the image is acquired. A camera attribute information acquisition unit that acquires a distance and an image size, an image processing unit that divides the acquired image into a plurality of partial areas, and a distance from a camera position at the time of the image acquisition to each point of the image. A distance information obtaining means for obtaining a depth value at each point of each partial region of the image, and managing the map information, and a view space within the map information space based on the obtained camera position, camera angle, focal length, and image size. Map information management means for obtaining a structure existing in the visual field space, and CG which is a computer graphics image based on the structure obtained by the map information management means. After creating the image, a depth value that is a distance to each point of each partial region of the CG image with respect to the camera position at the time of acquiring the image is obtained, and a set of depth values at each point of each partial region of the CG image is obtained. From the ratio of the set of depth values at each point for each partial region of the image and the overlap ratio of the partial region of the CG image and the partial region of the image,
Label information creating means for associating with the partial area in the G image, obtaining the structure of the associated partial area, and creating label information including the name or attribute information of the structure and the assigned position; Label information output means for superimposing the name of the structure or its attribute information at a position in the image corresponding to the information of the assigned position in the information, and displaying the superimposed image on a visual device, and control for controlling each of the above means Having means.

According to an embodiment of the present invention, the label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, erases a structure that cannot be seen from the viewpoint, creates a CG image, and creates a CG image. The CG image is divided into partial areas by the outline of the middle partial area.

According to an embodiment of the present invention, there are provided a plurality of image obtaining means, position information obtaining means, and camera attribute information obtaining means.

The distance reference type landscape labeling system of the present invention comprises a landscape labeling terminal and a landscape labeling center. The landscape labeling terminal has an image obtaining means for obtaining an image and a position information obtaining means for obtaining a camera position at the time of obtaining the image. Means, camera attribute information acquiring means for acquiring a camera angle, a focal length, and an image size at the time of image acquisition; image processing means for dividing the acquired image into a plurality of partial regions; and A distance information obtaining unit that obtains a depth value at each point for each partial area of the image, which is a distance to each point of the image, and information regarding area division of the image, the camera position, the camera angle, the focal length, and the like. Transmitting the image size to the landscape labeling center via a communication network and receiving label information from the landscape labeling center Signal control means, label information output means for superimposing the name of the structure in the label information or its attribute information at a position corresponding to the application position in the image, and displaying the superimposed image on a visual device; Terminal control means for controlling means, the landscape labeling center, information on the area division of the image from the landscape labeling terminal via the communication network, the camera position, the camera angle, the focal length, and the image Communication control means for receiving the size and the depth value and transmitting the label information to the landscape labeling terminal; managing the map information; and a map information space based on the received camera position, camera angle, focal length and image size. A map information management means for obtaining a visual space in the visual space and acquiring a structure existing in the visual space, and a structure obtained by the map information management means. After creating a CG image is a computer graphic image Te, obtains a depth value is the distance to each point of each partial area of the CG image to the camera position at the time of the image acquired for each partial region, C
By comparing the set of depth values for each partial region of the G image with the set of depth values at each point for each partial region of the image, the partial region of the image is associated with the partial region in the CG image, and It has a label information creating means for finding a structure in the partial area and creating label information including the name or attribute information of the structure and the assigned position, and a center control means for controlling each of the above means.

According to another aspect of the present invention, there is provided a distance reference type landscape labeling system including a landscape labeling terminal and a landscape labeling center, wherein the landscape labeling terminal obtains an image obtaining means for obtaining an image and a camera position at the time of obtaining the image. Position information acquiring means, camera attribute information acquiring means for acquiring a camera angle, a focal length, and an image size at the time of image acquisition; image processing means for dividing the acquired image into a plurality of partial regions; A distance information obtaining unit that obtains a depth value at each point for each partial region of the image, which is a distance to each point of the image with respect to a position, information on region division of the image, the camera position, the camera angle, and The focal length, the image size, and the depth value are transmitted to the landscape labeling center via a communication network, and from the landscape labeling center. Communication control means for receiving the bell information, and label information output for superimposing the name of the structure in the label information or its attribute information at a position corresponding to the application position in the image, and displaying the superimposed image on a visual device Means,
Having a terminal control means for controlling each of the above means, the landscape labeling center, information on the area division of the image from the landscape labeling terminal via the communication network, the camera position, the camera angle, the focal length and Communication control means for receiving the image size and the depth value, transmitting the label information to the landscape labeling terminal, managing the map information, and generating a map based on the received camera position, camera angle, focal length, and image size. Find the viewing space in the information space,
A map information management means for acquiring a structure existing in the visual field space, and a CG image which is a computer graphics image is created based on the structure acquired by the map information management means. A depth value which is a distance from the camera position to each point of each partial area of the CG image is obtained for each partial area, and a set of depth values at each point of each partial area of the CG image, The partial area of the image is associated with the partial area of the CG image based on the ratio of the set of depth values at each point in each area and the overlapping ratio of the partial area of the CG image and the partial area of the image. It has a label information creating means for finding a structure of the attached partial area and creating label information including the name or attribute information of the structure and an assigned position, and a center control means for controlling each of the above means.

[0018]

Next, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of a landscape labeling apparatus according to a first embodiment of the present invention, and FIG. 2 is a flowchart of processing of the landscape labeling apparatus of FIG.

The landscape labeling apparatus according to the present embodiment is, for example, an image acquisition unit 1 which is an image acquisition unit, for example, a digital camera, and a GPS receiver, for example, which acquires the position of the image acquisition unit 1 when acquiring an image. A position information acquisition unit 2 and a camera attribute information acquisition unit 3 that acquires a camera angle, a focal length, and an image size when the image acquisition unit 1 acquires an image, for example, a three-dimensional electronic compass attached to a digital camera. And an image processing unit 4 for dividing the acquired image into a plurality of partial regions; and distance information for obtaining a distance (depth value) to each point of the image with respect to a camera position at the time of acquiring the image for each partial region of the image The map information is managed by the acquisition unit 5 and a visual field space is obtained in the map information space based on the obtained position (camera position), camera angle, focal length, and image size. A map information management unit 6 for acquiring an existing structure, and a CG image is created based on the structure acquired by the map information management unit 6, and then a CG image for each partial area with respect to a camera position at the time of image acquisition is created. The depth, which is the distance to each point, is determined, and the ratio of the set of the average depth value at each point for each partial region of the CG image to the set of the depth value at each point for each partial region of the image is calculated. A label information creating unit 7 for associating the partial area with the partial area of the CG image by pattern matching and creating label information including the name or attribute information of the associated structure and the assigned position; The label information output unit 8 that superimposes the name or attribute information of the structure on the position corresponding to the application position in the image, and outputs the superimposed image to the visual device, and the control unit 9 that controls the above units 1 to 8 Composed There.

The distance information acquisition unit 5 uses each point of each partial area in the landscape image obtained by the image acquisition unit 1 using the conventional technique relating to the three-dimensional measurement method and the image acquisition unit 1 (camera) when handling images. Position). Here, as a reference related to the conventional technique of the three-dimensional measurement method, the Journal of the Institute of Image Electronics Engineers of Japan, Vol.
In this document, the optical radar method (pulse light projection method or modulated light projection method), monocular vision method, stereo image method, active stereo method (slit light projection method, pattern light projection method, etc.) are described. I have. For example, the pulsed light projection method
The modulated light projection method projects a light beam whose intensity is time-modulated by a sine wave or a rectangular wave, by measuring the time until the light pulse is projected and reflected and returns.
In this method, the distance is obtained from the phase difference with the reflected wave. For example, the stereo image method is a method of obtaining a three-dimensional position by applying triangulation between images obtained from a plurality of cameras, and the slit light projection method is based on a position of an image formed by projecting slit light. The pattern light projection method is a method of obtaining a distance image from a small number of images by coding a measurement space with a light pattern.

In the present embodiment, since there is one image acquisition unit 1, the distance information acquisition unit 5 obtains the distance by using a three-dimensional measurement method such as an optical radar method that requires only one acquired image.

The control unit 9 handles the image obtained from the image acquisition unit 1 as a landscape image using the image acquisition unit 1 as a reference camera. The position / camera attribute / image information is passed to the distance information acquiring unit 5 via the control unit 9 by the image acquiring unit 1.

When the landscape labeling device is started, first, the control unit 9 sends a processing start command to the image acquisition unit 1, the position information acquisition unit 2, and the camera attribute information acquisition unit 3 to acquire information on the landscape image. Send. Location information acquisition unit 1
Receives a command from the control unit 9, collects position information every second by a GPS receiver or the like, and passes it to the control unit 9 (step 2).
0). Here, the time interval is not limited to the unit of seconds, but may be any value. The image acquisition unit 1 receives a command from the control unit 9, acquires a landscape image every second, and passes it to the control unit 9 (step 21). The camera attribute information acquisition unit 3 receives a command from the control unit 9 to acquire a camera angle of a landscape image recording device such as a camera at the time of image capturing as a set of a horizontal angle and an elevation angle (Step 22), and at the same time, a landscape having a zoom function. If it is an image device, the focal length is obtained (step 23). Since the image size is fixed for each landscape image device, the control unit 9 holds the image size information. The distance information obtaining unit 5 obtains a depth value at each point of each partial area, which is a distance from the camera position at the time of obtaining the image to each point of the image (step 24). The control unit 9 holds the collected information as a landscape image file.

FIG. 3 shows the file format of the data structure of the landscape image file. The landscape image file has header information and image data. The header information includes position information, camera angle information, focal length, time information, image size, type, and size of an image file. As location information,
East longitude, north latitude, and elevation data (for example, 137 degrees east 5 degrees
5 minutes 10 seconds, latitude 34 degrees 34 minutes 30 seconds north, altitude 101m3
3 cm). The camera angle includes horizontal angle data and elevation angle data (for example, horizontal angle 254 degrees clockwise, elevation angle 15 degrees, etc.). The focal length data is the focal length (for example, 28 mm) of the camera lens at the time of capturing an image. The time information includes the time at the time of shooting (for example, 15:06:17, January 31, 1997, Japan time). As the image size of the image file, the vertical and horizontal pixel sizes (for example, 64
0 × 480). File type (TIF
E format, 8-bit color, etc.). It also has the number of bytes of the file (307.2 KB, etc.). The image data itself has, for example, a binary format.

After storing the landscape image file, the control unit 9 instructs the image processing unit 4 to extract the outline from the landscape image and divide the landscape image into a plurality of regions.
The image processing unit 4 performs a differentiation process on the basis of the density difference in the landscape image to extract a contour line (step 2).
5) The region is divided by performing labeling with the outline as a boundary (step 26). Note that the technical term of labeling used here is a technical term used in image area division, and is different from landscape labeling, which is the name of the present invention. As a procedure, first, the image is converted into a monochrome grayscale image. Since the contour is a portion where the brightness changes suddenly, the contour is extracted by performing a differentiation process to find a portion where the differential value is larger than a threshold value. At this time, the line width of the contour is 1 pixel, Are connected. For this purpose, a thinning process is performed to obtain a connected line having a line width of one pixel. Here, it is sufficient to use a conventional method for the differentiation processing and the thinning processing.

The obtained contour is regarded as the contour of the area, and an operation of numbering the area constituted by the contour is performed.
The largest number among the numbers is the number of areas, and the number of pixels in the area indicates the area of the area. FIG. 8 shows an example in which a landscape image is divided into a plurality of partial regions. Note that a measure of similarity (closeness) between regions may be introduced, and a clustering process may be performed to combine a plurality of regions having similar properties into one region. Any existing clustering method may be used.

When the control section 9 completes the area division processing of the landscape image, it issues a processing request to the map information management section 6 to calculate the view space by passing the header information of the landscape image file (step 27). . An example of the map information management unit 6 is a map database program. The map information management unit 6 manages three-dimensional map data. Although two-dimensional map data may be used, in this case, since there is no height information, the accuracy of the position at which the labeling is applied to the actual scenery is inferior. In addition, 2
When it is based on the dimensional map data, the processing is performed by supplementing the height information. For example, in the case of two-dimensional data of a house, if there is floor information indicating the number of floors of the house, the number of floors is multiplied by a certain number to estimate the height of the house, and the two-dimensional data is obtained. 3D data is created based on the height information. Even when there is no floor information, the height information can be estimated by allocating a certain number of heights according to the area of the house figure, and similarly, three-dimensional data is created based on the estimated height information. In this way, three-dimensional data is created and the process proceeds.

FIG. 4 shows an example of three-dimensional map data. FIG.
(1) shows a map information space expressed in two dimensions, and FIG.
(2) shows a map information space expressed in three dimensions. This 3
With respect to the three-dimensional map information space, the map information management unit 6 receives a command from the control unit 9 and calculates a visual field space based on the header information of the landscape image file (step 28). FIG. 5 shows a calculation example of the visual field space. First, it is assumed that the XY axes extend in the horizontal direction and the Z axis extends in the vertical direction. The position of the viewpoint E is set in the three-dimensional map information space from the position information in the header information of the landscape image file. For example, 137 degrees 55 minutes 19 seconds east longitude, 34 degrees 34 minutes 30 seconds north latitude, 101 m33c altitude
If m, the corresponding coordinates in the corresponding map mesh number are set. Similarly, the camera angle direction is set based on the horizontal angle and the elevation angle in the camera angle information in the header information. The focal point F is set at a point advanced from the viewpoint E by the focal length on a straight line representing the camera angle direction. The line-of-sight direction vector is a unit vector having a length of 1 from the viewpoint E on the straight line. In the image size of the landscape image file, the width x on the X-axis of the camera screen is set from the horizontal size, and the width y on the Y-axis is set from the vertical size. The horizontal x vertical y plane is set so as to be perpendicular to the camera angle direction with respect to the line of sight vector and to include the focal point F. Lines connecting the four corner points of the camera screen are obtained from the coordinates of the viewpoint E, and a three-dimensional space formed by four half-lines extending from the viewpoint E is defined as a visual field space. FIG. 6 shows an example of the visual field space in the three-dimensional map space. The three-dimensional map space is viewed from the XZ plane. In FIG. 6, a portion surrounded by oblique lines is a cross-sectional view of the space belonging to the viewing space on the XZ plane. In the example of FIG. 6, buildings and mountains in the visual field space are included.

Further, the map information management unit 6 obtains a structure existing in the obtained visual field space. For each structure, it is calculated whether or not each vertex constituting the solid representing the structure exists in the internal region of the viewing space. Usually, a two-dimensional map space is divided by a two-dimensional mesh of a fixed size. As a method of cutting a mesh in the three-dimensional map space, a mesh is cut at regular intervals in the height direction in addition to the two-dimensional mesh in the vertical and horizontal directions. The space is divided by a rectangular parallelepiped unit space. First, the presence or absence of an overlapping portion between the rectangular parallelepiped unit space and the visual field space is checked, and the number of the three-dimensional unit map space having the overlapping portion is obtained. The number of the three-dimensional unit map space here is the same as a so-called mesh number. For structures in the overlapping 3D unit map space,
Investigate whether there is any overlap with the visual field space. A straight line connecting the coordinates of the vertices constituting the structure and the coordinates of the viewpoint is obtained. If the straight line has an intersection with the camera screen of FIG. 7, it is within the visual field space. If at least one vertex of the plurality of vertices constituting the structure satisfies this condition, the structure has an overlapping portion with the viewing space.

When a structure is included in the view space or a part of the structure, the process enters a process of three-dimensionally projecting each structure onto the projection surface using the camera screen as a projection surface (step 29). ). Here, as shown in FIG. 7, after the point P is re-expressed in the coordinate system based on the viewpoint E based on the following equation (1), the point P is projected on the camera screen to obtain the intersection Q. .

[0032]

(Equation 1) Here, point P = (x, y, z): coordinates of the vertices constituting the structure point E = (ex, ey, ez): coordinates of the viewpoint vector L = (lx, ly, lz): gaze direction vector (Unit vector) Point P ′ = (x ′, y ′, x ′): coordinates in a coordinate system based on the viewpoint E of the point P r = (lx ² + ly ² ) ^1/2 intersection Q = (X, Y): The projection point t of the point P on the camera screen is the focal length. In the three-dimensional projection conversion, first, for each of the structures, the surface on which the vertex extends is determined. For example, in the case of a structure represented by a rectangular parallelepiped, six surfaces are obtained. When each plane is projected onto the camera screen, the distance between the viewpoint and the corresponding point on the plane is calculated for each pixel on the camera screen included in the projection area, and stored as a depth value (Z value) in the memory. Store. For each surface of each structure, the depth value (Z
Value) and store it in memory. Note that z ′ in Expression (1) represents a depth value (Z value) from the viewpoint.

The structures three-dimensionally transformed on the camera screen include structures that can be seen from the viewpoint and structures that cannot be seen. Among them, it is necessary to find only the structure that can be seen from the viewpoint, and to find the surface on the opposite side from the viewpoint or the surface that is blocked by other structures. Therefore, hidden surface processing is performed (step 3
0). There are various methods for processing the hidden surface.
Use the buffer method. Other scan line methods and ray tracing methods may be used.

A pixel on the camera screen is arbitrarily determined, and a plane having the smallest depth value for the pixel is determined. As described above, when processing is sequentially performed on each surface of each structure, the surface closest to the viewpoint is left for each pixel on the camera screen. The plane closest to the viewpoint is determined for each pixel on the camera screen, and pixels on the camera screen that share the plane closest to the viewpoint generally form an area.
There can be a plurality of regions consisting of pixels with the common surface being the closest surface. The region obtained in this way is a region obtained by performing a three-dimensional projection conversion of the partial region of the structure seen from the viewpoint. Surfaces on the other side of the view and those obstructed by other structures have been erased.

The area thus formed forms a CG image area (step 31).

With respect to the coordinates of the vertices of the two-dimensional figure constituting the CG image area, three-dimensional coordinates before projection transformation are obtained, and the correspondence between the two is stored in the memory as link information. It is used to determine which structure the two-dimensional area is a projection of based on the link information.

Based on the line data remaining after erasing hidden lines,
The CG image is divided into regions. Since the three-dimensional map DB is used, the name of the structure serving as the basis of each area can be associated with each area. The divided areas of the CG image are numbered sequentially. FIG. 9 shows an example in which a CG image is divided into a plurality of partial areas.

When the CG image region division processing is completed, the control unit 9 instructs the label information creation unit 7 to associate the CG image division region with the landscape image division region.

After creating the CG image, the label information creating unit 7 obtains a depth value which is the distance between each point of the partial area of the CG image and the camera position at the time of image acquisition. next,
An average depth value for each partial region of the CG image is determined based on the depth value at each point of the partial region of the CG image (step 3).
2). For example, it is assumed that the m-th partial region of the CG image is a region occupied by an image of a certain structure Z. At that time, it is determined whether or not the structure Z is actually shown in the landscape image.
It is desired to estimate by comparing the distance (depth value) between the viewpoint and a point in the image. Therefore, a plurality of sample coordinates X ₁ , X ₂ , X ₃ are placed in the m-th partial area in the CG image.
Taken to calculate the distance of the viewpoint and (camera position) distance between the point X _1, the distance between the viewpoint and the point X _2, the viewpoint and the point X ₃ (Fig. 1
5 (1)). Corresponding to the coordinates of points X ₁ , X ₂ , X ₃ ,
Points having coordinates in the landscape image are represented by X ₁ ′, X ₂ ′, respectively.
X ₃ '. Distance between viewpoint and point X ₁ ′, viewpoint and point X ₂ ′
, And the distance between the viewpoint and the point X ₃ ′ are measured using the distance information acquiring unit 5. If the distances between the viewpoint and points X ₁ , X ₂ , and X ₃ are 101 m, 103 m, and 102 m, respectively, the average depth value is 102 m. The distance between the viewpoint and the points X ₁ ′, X ₂ ′, X ₃ ′ is 99 m, respectively.
If they were 3m and 101m, the average depth value would be 101
m. At this time, the distance ratio is 0.99019. If the value were set to 0.90, then 0.99
Since 019> 0.90, it is considered that the structure Z exists in the landscape image. Next, a ratio (distance ratio) of the average depth value of each partial region of the landscape image to the average depth value of each partial region of the CG image is determined. On the other hand, two areas to be compared between the landscape image and the CG landscape image are overlapped, and the ratio of overlapping portions (overlap ratio) is determined. Then, the distance ratio of the two regions to be compared is multiplied by the overlap ratio, and the result is compared with a set threshold value. If the result is equal to or larger than the threshold value, the regions are associated with the same structure. This association is made with an area with a small number (for example, 1
(No.) in order from the CG image divided area (step 3)
3). The calculation of the overlap ratio is performed as follows. For example, the coordinate value of each pixel in the first divided region R1 of the landscape image is set to (A, B). Coordinates (A, B)
Is 1 because of the interior of the region. In the first divided area S1 of the CG image, if the coordinates (A, B) are within the area S1, the pixel value is 1 and overlaps, but if the coordinates (A, B) are outside of S1, the pixel value is 0 and does not overlap. Thus the coordinates (A, B)
The overlap coefficient K (A, B) is determined to be 1 if they overlap and 0 if they do not overlap. The coordinates (A, B) are moved in the region R1, and the overlap coefficient K (A, B) is obtained. Then, with respect to the number N1 of coordinates (A, B) moved in the region R1, the number N of coordinates where the overlap coefficient K (A, B) is 1
2, N1 / N2 is the overlap ratio.

In addition, as the matching method, XY
It is also possible to use an evaluation function that has the same value even if there is some displacement in the direction.

After associating the partial area of the landscape image with the partial area of the CG image, the label information creating section 7 extracts the structure of the associated partial area (step 3).
4) Further, processing to obtain information to be superimposed for each partial area of the landscape image and create label information together with the position to be superimposed (step 35) is entered. First, a partial area of the CG image corresponding to the partial area of the landscape image is extracted. The partial region of the extracted CG image is originally obtained by three-dimensionally projecting and converting a certain surface of the three-dimensional structure in the three-dimensional map space onto the camera screen. So 3
The surface of the three-dimensional structure that is the basis of the three-dimensional projection transformation is determined using the depth value (Z value) of the partial region of the CG image as a key. The link information created at the time of the three-dimensional projection conversion may be used as a key. Based on the surface of the original structure, the three-dimensional map DB is accessed to acquire the name or attribute information of the structure. Here, the attribute information means information accompanying the structure, and may be any information as long as the information is related to the structure. Then, the position coordinates where the name or attribute information is to be superimposed are determined for the partial area of the landscape image. How to decide may be decided any way. For example, the center of gravity of the figure extending the partial area may be used. Label information is created from the name or attribute information of the structure and the assigned position coordinates. Table 1 shows an example of label information.

[0042]

[Table 1] When the label information creating unit 7 finishes creating the label information,
The label information is passed to the control unit 9.

When receiving the label information, the control section 9 instructs the label information output section 8 to display and output the label information to the visual device. Here, the visual device includes a video display device such as a display or a head-mounted display. The name or attribute information of the structure in the label information is superimposed on the position in the landscape image (step 3).
6) Output the superimposed landscape image to the video display device (step 37). FIG. 12 shows an example of a landscape image on which label information is superimposed.

When the label information output section 8 outputs the label information, it notifies the control section 9 of the completion of the output. When receiving the output completion notification, the control unit 9 executes the above-described series of processing procedures again when performing the landscape labeling process continuously.

FIG. 13 is a configuration diagram of a landscape labeling apparatus according to a second embodiment of the present invention.

The landscape labeling apparatus according to the present embodiment acquires first and second images, respectively, for example, first and second image acquisition units 1A and 1B, which are digital cameras, and second and third image acquisition units 1A and 1B, respectively. 1. First and second position information obtaining units 2A and 2B, which are, for example, GPS receivers, which obtain the positions of the first and second image obtaining units 1A and 1B, respectively, and first and second image obtaining units 1 and 1B. A first and a second camera attribute information acquisition units 3A and 3B, which are three-dimensional electronic compasses attached to a digital camera, for example, acquire a camera angle, a focal length, and an image size when acquiring an image; An image processing unit 4 that divides the acquired image into a plurality of partial areas, and distance information acquisition that obtains a distance (depth value) from the second image to each point of the first image for each partial area of the first image Part 5,
Map information that manages map information, obtains a visual space in the map information space based on the acquired position (camera position), camera angle, focal length, and image size, and acquires structures existing in the visual space. The management unit 6 associates the structure acquired for the partial region of the image with the ratio of the average depth value for each partial region of the image and the CG image by pattern matching,
Label information creating section 7 for creating label information including the name or attribute information of the associated structure and the assigned position
And a label information output unit 8 that superimposes the name or attribute information of the structure in the created label information at a position corresponding to the application position in the image, and outputs the superimposed image to the visual device.
And a control unit 9 for controlling the above units 1 to 8.

In the present embodiment, since a plurality of image acquisition units, position information acquisition units, and camera attribute information acquisition units are provided, the distance information acquisition unit 5 needs not only the optical radar method but also a plurality of acquired images. Each point of the partial region in the landscape image and the camera position at the time of image acquisition are obtained using a three-dimensional measurement method such as the stereo image method and the active stereo image method described above.

The control section 9 treats the image (first image) obtained from the first image acquisition section 1A as a landscape image using the first image acquisition section 1A as a reference camera. Second image acquisition unit 1
B is a camera installed to obtain distance information.
Image (second image) by the second image acquisition unit 1B that acquires the position / camera attribute / image in the same manner as the image acquisition unit 1A
Is not treated as a landscape image. The position / camera attribute / image information from the first and second image acquisition units 1A and 1B is passed to the distance information acquisition unit 5 via the control unit 9.

When the landscape labeling device is activated, first, the control unit 9 obtains information on the landscape image, so that the image acquisition units 1A and 1B, the position information acquisition units 2A and 2B, and the camera attribute information acquisition units 3A and 3B. Send a processing start command to The position information acquisition units 1A and 1B receive a command from the control unit 9, collect the position information with a GPS receiver or the like every second, and pass it to the control unit 9 (step 20). Here, the time interval is not limited to the unit of seconds, but may be any value. The image acquisition units 1A and 1B receive a command from the control unit 9, acquire a landscape image every second, and pass it to the control unit 9 (step 21). The camera attribute information acquisition units 3A and 3B acquire a camera angle of a landscape image recording device such as a camera at the time of image capture in a set of a horizontal angle and an elevation angle in response to an instruction from the control unit 9 (step 22), and simultaneously perform a zoom function. If it is a landscape image device having the same, the focal length is acquired (step 23). Since the image size is fixed for each landscape image device, the control unit 9 holds the image size information. The distance information acquisition unit 5 obtains the distance (depth value) from the second image to each point of the first image for each partial area (step 24). The control unit 9 holds the collected information as a landscape image file. The following operation is the same as in the first embodiment.

[0050] Three or more image acquisition units, position information acquisition units, and camera attribute information acquisition units may be provided.

FIG. 14 is a configuration diagram of a landscape labeling system in which the landscape labeling device of FIG. 1 is applied to a communication system.

The landscape labeling system includes a landscape labeling terminal 40, a landscape labeling center 50, and a communication network 60.

The landscape labeling terminal 40 acquires an image acquiring section 41 for acquiring an image, a position information acquiring section 42 for acquiring a camera position when acquiring an image, and acquires a camera angle, a focal length, and an image size when acquiring an image. Camera attribute information acquisition unit 43
And an image processing unit 44 that divides the acquired image into a plurality of partial regions, and obtains a depth value at each point of each partial region of the image, which is a distance from the camera position at the time of image acquisition to each point of the image. A distance information acquisition unit 45, a scene labeling center 50 via a communication network 60, and information on image area division, a camera position, a camera angle, a focal length, and an image size.
And the communication control unit 46 that receives the label information from the landscape labeling center 50, and superimposes the name or attribute information of the structure in the label information at a position corresponding to the assigned position in the image, and displays the superimposed image. A label information output unit 47 for outputting to a visual device, and a terminal control unit 48 for controlling the above units
It consists of.

The landscape labeling center 50 has a communication network 60.
A communication control unit 51 that receives information related to area division of the image, a camera position, a camera angle, a focal length, an image size, and an average depth value from the landscape labeling terminal 40 via the landscape labeling terminal 40, and transmits label information to the landscape labeling terminal 40; A map information management unit that manages map information, obtains a view space in the map information space based on the received camera position, camera angle, focal length, and image size, and obtains structures existing in the view space. 52, and a computer graphics image C based on the structure acquired by the map information management unit 52.
After creating the G image, a depth value which is a distance from the camera position at the time of acquiring the image image to each point of each partial region of the CG image is obtained, and the depth value at each point of each partial region of the CG image is calculated. By comparing the set and the set of depth values at each point for each partial area of the image, the partial area of the image is associated with the partial area of the CG image, and the structure of the associated partial area is determined. A label information creating unit 53 for creating label information including the name or attribute information and the assigned position, and a center control unit 54 for controlling the above units.

The operation of this system is the same as the operation of the apparatus shown in FIG. Further, an image acquisition unit, a position information acquisition unit,
Two or more camera attribute information acquisition units may be provided.

[0056]

As described above, according to the present invention, the geographical information on the computer and each part in the scenery image of the actual scene can be presented to the user in association with each other.
This eliminates the need for a human to compare the map on the computer with the actual scenery and associate it with the human.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a distance reference type landscape labeling device according to a first embodiment of the present invention.

FIG. 2 is a flowchart of processing of the landscape labeling device of the embodiment of FIG. 1;

FIG. 3 is a diagram showing a data structure of a landscape image file.

FIG. 4 is a diagram showing an example of a two-dimensional map (FIG. 1 (1)) and its three-dimensional map (FIG. 2 (2)).

FIG. 5 is a diagram illustrating a calculation method of a visual field space.

FIG. 6 is a diagram illustrating an example of a visual field space in a three-dimensional map space.

FIG. 7 is a diagram showing an example of a projection view.

FIG. 8 is a diagram showing an example of area division of a landscape image.

FIG. 9 is a diagram illustrating an example of area division of a CG image.

FIG. 10 is an explanatory diagram of pattern matching between a partial region of a landscape image and a partial region of a CG image.

FIG. 11 is a diagram showing points in the m-th partial region of the CG image (FIG. 11A) and corresponding points in the landscape image.

FIG. 12 is a diagram illustrating an example of superimposition of label information on a landscape image.

FIG. 13 is a configuration diagram of a distance reference type landscape labeling device according to a second embodiment of the present invention.

FIG. 14 is a configuration diagram of a distance reference type landscape labeling system according to an embodiment of the present invention.

FIG. 15 is a configuration diagram of a navigation device disclosed in JP-A-8-273000.

FIG. 16 is a diagram illustrating a display example of a moving image.

[Explanation of symbols]

 1, 1A, 1B Image acquisition unit 2, 2A, 2B Position information acquisition unit 3, 3A, 3B Camera attribute information acquisition unit 4 Image processing unit 5 Distance information acquisition unit 6 Map information management unit 7 Label information creation unit 8 Label information output Unit 9 Control unit 20 to 37 Step 40 Scenery labeling terminal 41 Image acquisition unit 42 Position information acquisition unit 43 Camera attribute information acquisition unit 44 Image processing unit 45 Distance information acquisition unit 46 Communication control unit 48 Label information output unit 47 Terminal control unit 50 Landscape labeling center 51 Communication control unit 52 Map information management unit 53 Label information creation unit 54 Center control unit 60 Communication network

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Akira Suzuki 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Japan Telegraph and Telephone Corporation (72) Inventor Takeshi Ikeda 3- 192-1 Nishi-Shinjuku, Shinjuku-ku, Tokyo No. Japan Telegraph and Telephone Corporation

Claims (8)

[Claims] An image acquisition unit for acquiring an image; a position information acquisition unit for acquiring a camera position at the time of image acquisition; and a camera attribute information acquisition for acquiring a camera angle, a focal length, and an image size when the image is acquired. Means, image processing means for dividing the acquired image into a plurality of partial areas, and depth at each point for each partial area of the image, which is a distance from the camera position at the time of image acquisition to each point of the image. A distance information obtaining means for obtaining a value; managing map information; obtaining a visual field space in the map information space based on the obtained camera position, camera angle, focal length, and image size; Map information management means for acquiring a structure to be obtained, and after creating a CG image which is a computer graphics image based on the structure obtained by the map information management means, A depth value which is a distance from the camera position to each point of each partial area of the CG image is obtained, and a set of depth values at each point of each partial area of the CG image and each point of each partial area of the image are obtained. By comparing the set of depth values in the above, the partial area of the image is associated with the partial area in the CG image, and the structure of the associated partial area is obtained.
Label information creating means for creating label information including the name or attribute information of the structure and the assigned position, and the name of the structure or its attribute at a position in the image corresponding to the information of the assigned position in the created label information A distance reference type landscape labeling device, comprising: label information output means for superimposing information and displaying the superimposed image on a visual device; and control means for controlling the respective means. 2. An image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position at the time of image acquisition, and a camera attribute information acquisition for acquiring a camera angle, a focal length, and an image size when an image is acquired. Means, an image processing means for dividing the acquired image into a plurality of partial areas, and a part of the image, which is a distance from a camera position at the time of image acquisition to each point of the partial area of the camera position at the time of image acquisition. Distance information obtaining means for obtaining a depth value at each point in each area; managing map information; and setting a visual field space in a map information space based on the obtained camera position, camera angle, focal length, and image size. Map information management means for obtaining a structure existing in the visual space, and a CG image which is a computer graphics image based on the structure obtained by the map information management means. After the creation, a depth value which is a distance from the camera position at the time of the image acquisition to each point of the partial area of the CG image is obtained,
The ratio of the set of depth values at each point for each partial area of the CG image, the ratio of the set of depth values at each point for each partial area of the image, and the difference between the partial area of the CG image and the partial area of the image From the overlap ratio, the partial area of the image is associated with the partial area in the CG image, a structure of the associated partial area is obtained, and label information including the name or attribute information of the structure and the assigned position is obtained. Label information creating means to be created, and a label for superimposing the name of the structure or its attribute information at a position in the image corresponding to the information of the assigned position in the created label information, and displaying the superimposed image on a visual device A distance reference type landscape labeling device, comprising: information output means; and control means for controlling each of the means. 3. The label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, creates a CG image by erasing a structure that cannot be seen from a viewpoint, and creates a contour of a partial area in the CG image. 3. The landscape labeling device according to claim 1, wherein the CG image is divided into partial regions by lines. 4. The apparatus according to claim 1, further comprising a plurality of each of the image acquiring unit, the position information acquiring unit, and the camera attribute information acquiring unit. 5. A landscape labeling terminal and a landscape labeling center, wherein the landscape labeling terminal is an image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position at the time of image acquisition, and a camera at the time of image acquisition. Camera attribute information acquiring means for acquiring an angle, a focal length, and an image size; image processing means for dividing an acquired image into a plurality of partial regions; and a distance from a camera position at the time of the image acquisition to each point of the image. Distance information obtaining means for obtaining a depth value at each point for each partial region of the image; information on area division of the image; the camera position; the camera angle; the focal length; the image size; and the depth value Communication control means for transmitting the label information to the landscape labeling center via a communication network, and receiving label information from the landscape labeling center; Label information output means for superimposing the name of the structure in the file information or its attribute information at a position corresponding to the application position in the image, and displaying the superimposed image on a visual device, and a terminal for controlling each of the above means Control means, the landscape labeling center, from the landscape labeling terminal via the communication network, information on area division of the image, the camera position, the camera angle, the focal length, the image size, and the depth value And communication control means for transmitting the label information to the landscape labeling terminal, managing the map information, and viewing the map in the map information space based on the received camera position, camera angle, focal length, and image size. Map information management means for obtaining a space and obtaining a structure existing in the visual field space; and a computer based on the structure obtained by the map information management means. After creating a CG image which is a raster image, a depth value which is a distance from a camera position at the time of image acquisition to each point of each partial region of the CG image is obtained for each partial region, and a depth value is obtained for each partial region of the CG image. By comparing the set of depth values at each point with the set of depth values at each point for each partial region of the image, the partial region of the image was associated with the partial region in the CG image, A distance reference type landscape labeling system comprising: a label information creating unit that obtains a structure in a partial area and creates label information including the name or attribute information of the structure and an assigned position; and a center control unit that controls each of the units. 6. A landscape labeling terminal and a landscape labeling center, wherein the landscape labeling terminal is an image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position at the time of image acquisition, and a camera at the time of image acquisition. Camera attribute information acquiring means for acquiring an angle, a focal length, and an image size; image processing means for dividing an acquired image into a plurality of partial areas; and distances to respective points of the image with respect to a camera position at the time of the image acquisition. A distance information obtaining unit for obtaining a depth value at each point for each partial region of the image, and communicating information on region division of the image, the camera position, the camera angle, the focal length, the image size, and the depth value. Communication control means for transmitting to the landscape labeling center via a network and receiving label information from the landscape labeling center; Label information output means for superimposing the name of the structure or its attribute information at a position corresponding to the application position in the image, and displaying the superimposed image on a visual device, and a terminal control means for controlling each of the above means Having the landscape labeling center, receives information about the area division of the image, the camera position, the camera angle, the focal length, the image size, and the depth value from the landscape labeling terminal via the communication network. Communication control means for transmitting the label information to the landscape labeling terminal, and manages the map information, to determine the view space in the map information space based on the received camera position, camera angle, focal length and image size, Map information management means for acquiring structures existing in the visual field space; and computer graphics based on the structures acquired by the map information management means. After creating a CG image which is a source image, a depth value which is a distance from the camera position at the time of image acquisition to each point of each partial region of the CG image is obtained for each partial region, and for each partial region of the CG image, From the set of depth values at each point, the ratio of the set of depth values at each point for each partial area of the image, and the overlap ratio between the partial area of the CG image and the partial area of the image. Label information creating means for associating the partial area with the partial area in the CG image, finding the structure of the associated partial area, and creating label information including the name or attribute information of the structure and the assigned position; A distance reference type landscape labeling system having a center control unit for controlling each of the above units. 7. The label information creating means performs a three-dimensional projection conversion of the acquired structure on a camera screen, creates a CG image by erasing a structure that cannot be seen from a viewpoint, and creates a contour of a partial region in the CG image. 7. The landscape labeling system according to claim 4, wherein the CG image is divided into partial regions by lines. 8. The system according to claim 5, comprising a plurality of each of said image acquisition unit, said position information unit, and said camera attribute information.