JPH1166349A - Movement amount predictive scene labeling device and system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent the assigning position of a label from shifting when a scene labeling device is mounted on an automobile or the like, and a label is applied to an image that is seen from the inside of the automobile. SOLUTION: A movement amount predicting part 4 predicts a position P at a time E based on a position at a time S from geographical information and a traveling speed of an automobile on the assumption that the automobile proceeds along a road with a camera angle and focal distance in a constant state from the time S when an labeling instruction is received till the time E when time difference T at which label information can be shown passes. A map information managing part 5 accesses a map DB, calculates visual field space, projects and transforms each structure in the visual field space into a camera surface based on the camera angle and focal distance at the position P at the time S and further performs hidden-surface processing and extracts a structure that must be seen at the time E. A label information creating part 6 creates a label based on a structure that does not exist on the hidden-surface, and a label information outputting part 7 superimposes the label information on an unworked image that is photographed at the time E and shows it.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention superimposes and displays, on an image display device, geographical information about each partial area in an image taken by a user using a landscape image input device such as a camera. The present invention relates to a device that teaches a user by voice guidance or the like.

[0002]

2. Description of the Related Art Conventionally, there have been various navigation systems as systems for instructing users on geographical information about the area where the users are located.

FIG. 10 is a block diagram of a navigation device disclosed in Japanese Patent Application Laid-Open No. 8-273000. This device receives a vehicle position data and a movement data, receives a road map data, updates a vehicle position with reference to a position updating unit 71, and displays display road data and display background data based on map data and the like. Display data generator 7 to be generated
2, a three-dimensional moving image data creating unit 73 for creating three-dimensional moving image data based on these display data, and a storage unit 74.
When a travel route including a waypoint is set in advance, the device has a function of setting a route while viewing a real moving image display screen along a road that actually exists instead of a map screen.

According to this device, when a user travels along a route, the user can view a moving image display (for example, FIG. 11) along the route.

[0005]

However, in the case of using the same device, the human finally associates the real scene with the geographical information in the computer world with the naked eye, so that the person in the real scene can be used. You have to recognize what is. In other words, humans rely on the naked eye to work the human brain unconsciously, based on the symbols in the map displayed as a moving image, to determine what the actual buildings, roads, and mountains are in front of the user. It must be understood by performing the work of association. At street corners, etc., comparing the map on the computer with the actual scenery, grasping the direction and finding landmarks, gazing at the direction, understanding the characteristics of the building in that direction, and then looking at the map again Understand what the building is.

[0006] For this reason, there is a problem that the trouble of comparing the map on the computer with the actual scenery many times and associating with the human cannot be omitted. Especially in the dark or at night, it is difficult to see the actual scenery and to take measures.

SUMMARY OF THE INVENTION An object of the present invention is to provide a user with geographical information on a computer, which is mounted on a vehicle such as an automobile, and associates each part in an image of a real scene (hereinafter, referred to as a landscape image) with each other. The teaching is to provide a landscape labeling device and system.

[0008]

When a landscape labeling device is mounted on an automobile or the like and a label is attached to a landscape viewed from the interior of the automobile, the processing speed of the computer cannot keep up with the moving speed of the automobile or the like. There is a problem that the label application position shifts.

Therefore, according to the present invention, when an object such as an automobile moves, a moving speed is obtained, and a position which is likely to be moving at a label display processing time after a predetermined time is obtained, and the position is determined. Create and display label information for.

According to the present invention, there is provided a moving amount estimating means.
Assuming that the camera angle and the focal length remain constant and the vehicle is traveling along the road until the time E when the time difference T until the time when the label information can be displayed elapses, the position P at the time E is Based on the position at S, it is estimated from the geographical information on the road and the moving speed of the object.

Hereinafter, the camera angle at the position P and the time S
The map DB is accessed based on the focal length, the visual field space is calculated, each structure in the visual field space is projected and converted to the camera surface, and the hidden surface processing is performed. Extract. Label information is created based on the structure not on the hidden surface, and the label information is superimposed and displayed on an unprocessed video taken by the camera at time E.

The geographic information is the name of a structure or the like or its attribute information in an image, and the attribute information means information on all attributes (for example, outline, color, etc.) of the structure. In this specification, the term structure is used to mean all data having a certain geographical structure in the map DB, including natural topography such as mountains, rivers, and the sea, in addition to artificial structures. . When acquiring geographic information, a landscape image is obtained based on a camera position, a camera angle, a focal length, and an image size, and a structure of a plurality of images is obtained. The position of the landscape image where the structure is to be taken (hereinafter referred to as the assigned position) is obtained, and the name or attribute information of the structure is superimposed and displayed.

[0013] A movement amount prediction type landscape labeling apparatus according to the present invention comprises: an image acquisition unit for acquiring an image; a position information acquisition unit for acquiring a camera position; and a camera attribute information acquisition for acquiring a camera angle, a focal length, and an image size. Means, and assuming that the camera angle and the focal length remain constant from a first time at which a labeling command is received to a second time at which label information can be displayed, and position the camera at the second time as a second time. A movement amount predicting means for estimating from the geographical information and the moving speed of the vehicle based on the first time, and managing the map information, based on the position at the second time and the camera angle and the focal length at the first time. Map information for obtaining a view space in the map information space, projecting and transforming each structure in the view space to a camera surface, and further performing a hidden surface process to extract a structure that will be seen at a second time. Management means and names of extracted structures Or label information creating means for creating label information including the attribute information and the assigned position, and the structure information at the position in the image acquired at the second time corresponding to the information of the assigned position in the created label information. It has label information output means for superimposing the name or its attribute information and outputting the superimposed image to the visual device, and control means for controlling each of the above means.

Further, the movement amount prediction type landscape labeling system of the present invention comprises a landscape labeling terminal and a landscape labeling center mounted on a vehicle, and the landscape labeling terminal obtains an image and obtains an image. Position information acquiring means for acquiring a camera angle, a focal length, and an image size, and a camera attribute information acquiring means for acquiring a labeling command from a first time to a second time at which label information can be displayed. Assuming that the angle and the focal length remain constant, a movement amount estimating means for estimating the position at the second time from the geographical information and the moving speed of the vehicle based on the first time, The focal length, the image size, and the position at the second time are transmitted to the landscape labeling center via a communication network, and label information is received from the landscape labeling center. Communication control means for superimposing the name of the structure in the label information or its attribute information at a position corresponding to the application position in the unprocessed image acquired at the second time, and visually superimposing the superimposed image. Label information output means to be displayed on a device, and terminal control means for controlling each of the above means, the landscape labeling center, from the landscape labeling terminal via the communication network, the camera angle, the focal length and the image A communication control means for receiving the size and the position at the second time and transmitting the label information to the landscape labeling terminal; and managing the map information, and controlling the position at the second time and the position at the first time. The visual field space in the map information space is obtained based on the camera angle and the focal length, and each structure in the visual field space is projected and converted to a camera plane, hidden surface processing is performed, and visible at a second time. The wax structure A map information management means for output, and the label information creation means for creating the label information including the extracted name or attribute information and the assigned position of the structure, the central control means for controlling the respective means.

[0015]

Next, embodiments of the present invention will be described with reference to the drawings.

Referring to FIG. 1, a movement amount prediction type landscape labeling apparatus according to an embodiment of the present invention acquires an image.
For example, an image acquisition unit 1 which is a digital camera, a camera position is acquired, for example, a position information acquisition unit 2 which is a GPS receiver, and a camera angle, a focal length, and an image size are acquired. A camera attribute information acquisition unit 3 that is a three-dimensional electronic compass, and the camera angle and the focal length are kept constant from a first time S receiving a labeling instruction to a second time E at which label information can be displayed. Assuming that the position at the second time is estimated from the geographical information and the moving speed of the vehicle based on the first time S, the movement amount prediction unit 4 manages the map information, and the map information is managed at the second time E. The visual field space is obtained in the map information space based on the position, the camera angle and the focal length at the first time S, and each structure in the visual field space is projected and transformed on the camera surface, and further hidden surface processing is performed. Visible at second time E A map information management unit 5 for extracting a structure to be extracted, a label information creation unit 6 for creating label information including the name of the extracted structure or its attribute information and an assigned position, and a label information creating unit 6 for the assigned position in the acquired label information. A label information output unit 7 that superimposes the name of the structure or its attribute information at a position in the unprocessed image acquired at the second time E corresponding to the information, and outputs the superimposed image to the visual device;
The control section 8 controls the above sections.

FIG. 2 is a flow chart of the processing of the landscape labeling device of FIG.

Next, the operation of this embodiment will be described in detail with reference to FIG.

First, upon receiving a labeling command for creating label information at time S, the control unit 8 sends a processing start command to the position information acquisition unit 2 and the camera attribute information acquisition unit 3. The position information obtaining unit 2 receives a command from the control unit 8, obtains position information using a GPS receiver or the like, and passes the position information to the control unit 8 (step 21). The camera attribute information acquisition unit 3 receives a command from the control unit 8, acquires a camera angle of a landscape image recording device such as a camera as a set of a horizontal angle and an elevation angle (Step 22), and acquires a focal length (Step 23). Since the image size is fixed for each landscape image device, the control unit 8 holds the image size information. The control unit 8 holds the collected information as a landscape image file.

FIG. 3 shows the file format of the data structure of the landscape image file. The landscape image file has header information and image data. The header information includes position information, camera angle information, focal length, time information, image size, type, and size of an image file. As location information,
East longitude, north latitude, and elevation data (for example, 137 degrees east 5 degrees
5 minutes 10 seconds, latitude 34 degrees 34 minutes 30 seconds north, altitude 101m3
3 cm). The camera angle includes horizontal angle data and elevation angle data (for example, horizontal angle 254 degrees clockwise, elevation angle 15 degrees, etc.). The focal length data is the focal length (for example, 28 mm) of the camera lens at the time of capturing an image. The time information includes the time at the time of shooting (for example, 15:06:17, January 31, 1997, Japan time). As the image size of the image file, the vertical and horizontal pixel sizes (for example, 64
0 × 480). File type (TIF
E format, 8-bit color, etc.). It also has the number of bytes of the file (307.2 KB, etc.). The image data itself has, for example, a binary format.

After storing the landscape image file, the control unit 8 acquires the moving speed of the automobile and the label display interval T (steps 24 and 25). Next, the movement amount prediction unit 4 determines that the camera angle and the focal length remain constant from the first time S after the time T to the second time E, and the car is traveling along the road. And the position P of the car at time E
Is estimated from the geographical information on the road and the moving speed of the vehicle based on the position of the vehicle at time S (step 26).
For example, if the road is straight, the position P can be obtained by calculating the distance over the traveling speed of the automobile over time.

Next, the control section 8 sends the header information of the landscape image file to the map information management section 5 and issues a processing request for calculating the view space. An example of the map information management unit 5 is a map database program. The map information management unit 5 manages three-dimensional map data. Although two-dimensional map data may be used, in this case, since there is no height information, the accuracy of the position at which the labeling is applied to the actual scenery is inferior. When the two-dimensional map data is used as the basis, the processing is performed by supplementing the height information. For example, in the case of two-dimensional data of a house, if there is floor information indicating the number of floors of the house, the height of the house is estimated by multiplying the number of floors by a certain number, and the two-dimensional data is estimated. Three-dimensional data is created based on the obtained height information. Even when there is no floor information, the height information can be estimated by assigning a certain number of heights according to the area of the house figure, and similarly, three-dimensional data is created based on the estimated height information. . In this way, three-dimensional data is created and the process proceeds.

FIG. 4 shows an example of three-dimensional map data. FIG.
(1) shows a map information space expressed in two dimensions, and FIG.
(2) shows a map information space expressed in three dimensions. This 3
The map information management unit 5 accesses the map DB in response to an instruction from the control unit 8 (step 2).
7) A visual field space is calculated based on the header information of the landscape image file (step 28). FIG. 5 shows a calculation example of the visual field space. First, it is assumed that the XY axes extend in the horizontal direction and the Z axis extends in the vertical direction. The position of the viewpoint E is set in the three-dimensional map information space from the position information in the header information of the landscape image file. For example, if 137 degrees 55 minutes 19 seconds east longitude, 34 degrees 34 minutes 30 seconds north latitude, and 101 m 33 cm above sea level,
The corresponding coordinates in the corresponding map mesh number are set. Similarly, the camera angle direction is set based on the horizontal angle and the elevation angle in the camera angle information in the header information. Focus F at a point advanced by focal length from viewpoint E on a straight line representing the camera angle direction
Take. The line-of-sight direction vector is a unit vector having a length 1 coming out of the viewpoint E on the straight line. In the image size of the landscape image file, the width x on the X-axis of the camera screen is set from the horizontal size, and the width y on the Y-axis is set from the vertical size. The horizontal x vertical y plane is set so as to be perpendicular to the camera angle direction with respect to the line of sight vector and to include the focal point F.
Straight lines connecting the four corner points of the camera screen are obtained from the coordinates of the viewpoint E, and a three-dimensional space formed by four half lines extending from the viewpoint E is defined as a visual field space. FIG. 6 shows an example of the visual field space in the three-dimensional map space. The three-dimensional map space is viewed from the XZ plane. In FIG. 6, a portion surrounded by oblique lines is a cross-sectional view of the space belonging to the viewing space on the XZ plane. In the example of FIG. 6, buildings and mountains in the visual field space are included.

Further, the map information management unit 5 obtains a structure existing in the obtained visual field space. For each structure, it is calculated whether or not each vertex constituting the solid representing the structure exists in the internal region of the viewing space. Usually, a two-dimensional map space is divided by a two-dimensional mesh of a fixed size. As a method of cutting a mesh in the three-dimensional map space, a mesh is cut at regular intervals in the height direction in addition to the two-dimensional mesh in the vertical and horizontal directions. The space is divided by a rectangular parallelepiped unit space. First, the presence or absence of an overlapping portion between the rectangular parallelepiped unit space and the visual field space is checked, and the number of the three-dimensional unit map space having the overlapping portion is obtained. The number of the three-dimensional unit map space here is the same as a so-called mesh number. For structures in the overlapping 3D unit map space,
Investigate whether there is any overlap with the visual field space. A straight line connecting the coordinates of the vertices constituting the structure and the coordinates of the viewpoint is obtained. If the straight line has an intersection with the camera screen of FIG. 7, it is within the visual field space. If at least one of the vertices constituting the structure satisfies this condition, the structure has an overlapping portion with the viewing space.

If the structure is included in the visual field space or a part of the structure is included, a process for three-dimensionally projecting each structure onto the projection plane as a camera screen and a projection plane is started (step 29). ). Here, as shown in FIG. 7, after the point P is re-expressed in the coordinate system based on the viewpoint E based on the following equation (1), the point P is projected on the camera screen to obtain the intersection Q. .

[0026]

(Equation 1) Here, point P = (x, y, z): coordinates of the vertices constituting the structure point E = (ex, ey, ez): coordinates of the viewpoint vector L = (lx, ly, lz): gaze direction vector (Unit vector) Point P ′ = (x ′, y ′, z ′): coordinates in a coordinate system based on the viewpoint E of the point P r = (lx ² + ly ² ) ^1/2 intersection Q = (X, Y): Projection point t of point P on camera screen t is focal length

In three-dimensional projection conversion, first, a surface on which the apex extends is determined for each structure. For example, in the case of a structure represented by a rectangular parallelepiped, six surfaces are obtained. When each plane is projected onto the camera screen, the distance between the viewpoint and the corresponding point on the plane is calculated for each pixel on the camera screen included in the projection area, and stored as a depth value (Z value) in the memory. Store. For each surface of each structure, a depth value (Z value) for each pixel on the camera screen is calculated and stored in a memory. Note that z ′ in (Equation 1) represents a depth value (Z value) from the viewpoint.

Among the structures three-dimensionally projected and transformed onto the camera screen, there are structures that can be seen from the viewpoint and structures that cannot be seen. Among them, it is necessary to find only the structure that can be seen from the viewpoint, and to find the surface on the opposite side from the viewpoint or the surface that is blocked by other structures. Therefore, hidden surface processing is performed (step 3
0). There are various methods for processing the hidden surface.
Use the buffer method. Other scan line methods and ray tracing methods may be used.

A pixel on the camera screen is arbitrarily determined, and a plane having the smallest depth value for the pixel is determined. As described above, when processing is sequentially performed on each surface of each structure, the surface closest to the viewpoint is left for each pixel on the camera screen. The plane closest to the viewpoint is determined for each pixel on the camera screen, and pixels on the camera screen that share the plane closest to the viewpoint generally form an area.
There can be a plurality of regions consisting of pixels with the common surface being the closest surface. The region obtained in this way is a region obtained by performing a three-dimensional projection conversion of the partial region of the structure seen from the viewpoint. Surfaces on the other side of the view and those obstructed by other structures have been erased. In this way, a structure that is not on a hidden surface is extracted (step 31).

The label information creating section 6 enters a process of creating label information (step 32). First, 3D map D
B is accessed to acquire the name or attribute information of the structure. Here, the attribute information means information accompanying the structure, and may be any information as long as the information is related to the structure. Then, the position coordinates where the name or attribute information is to be superimposed are determined. How to decide may be decided any way. Label information is created from the name or attribute information of the structure and the assigned position coordinates. Table 1 shows an example of label information.

[0031]

[Table 1]

When the label information creation unit 6 has completed the creation of the label information, it passes the label information to the control unit 8.

When receiving the label information, the control unit 8 instructs the label information output unit 7 to display the label information on the visual device. Here, the visual device includes a video display device such as a display or a head-mounted display. The name or attribute information of the structure in the label information is superimposed on the position in the landscape image (step 34), and the superimposed landscape image is output to the video display device (step 3).
5). FIG. 8 shows an example of a landscape image on which label information is superimposed.

When the operation of adding the label information is repeated, the processing of FIG. 2 may be repeated.

FIG. 9 is a configuration diagram of a movement amount prediction type landscape labeling system in which the landscape labeling device of FIG. 1 is applied to a communication system.

The landscape labeling system includes a landscape labeling terminal 40 mounted on a vehicle, a landscape labeling center 50, and a communication network 60.

The landscape labeling terminal 40 includes an image acquisition unit 41 for acquiring an image and a position information acquisition unit 4 for acquiring a position.
2, a camera attribute information acquisition unit 43 for acquiring a camera angle, a focal length, and an image size;
It is assumed that the camera angle and the focal length remain constant during the time difference from the time S to the second time E at which the label information can be displayed, and the position P of the vehicle at the second time E is set to the first time. A movement amount prediction unit 44 for estimating from the geographical information and the moving speed of the vehicle based on the time S, and a camera angle, a focal length, an image size, and a position P of the vehicle at the second time are landscaped via the communication network 60. Communication control unit 45 that transmits to labeling center 50 and receives label information from landscape labeling center 50
And a label that superimposes the name or attribute information of the structure in the label information at a position corresponding to the application position in the unprocessed image acquired at the second time E, and displays the superimposed image on a visual device. It comprises an information output unit 47 and a terminal control unit 46 for controlling each of the above units.

The landscape labeling center 50 has a communication network 60.
A communication control unit 53 that receives the camera angle, the focal length, the image size, and the position of the vehicle at the second time E from the landscape labeling terminal 40 via the GPS controller and transmits label information to the landscape labeling terminal 40; Manages, obtains a view space in the map information space based on the position at the second time E and the camera angle and the focal length at the first time S, and places each structure in the view space on the camera plane. Projection transformation, hidden surface processing and second
A map information management unit 51 for extracting a structure that may be seen at time E, a label information creation unit 52 for creating label information including the extracted name or attribute information of the structure and the assigned position, It comprises a center control unit 54 for controlling.

The operation of the present system is the same as the operation of the apparatus shown in FIG.

[0040]

As described above, according to the present invention, when a scene labeling device is mounted on an automobile or the like and label information is added to an image viewed from the automobile, the position where the label information is added may be shifted. Disappears.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a movement amount prediction type landscape labeling device according to an embodiment of the present invention.

FIG. 2 is a flowchart of processing of the landscape labeling device of FIG. 1;

FIG. 3 is a diagram showing a data structure of a landscape image file.

FIG. 4 is a diagram showing an example of a two-dimensional map (FIG. 1 (1)) and its three-dimensional map (FIG. 2 (2)).

FIG. 5 is a diagram illustrating a calculation method of a visual field space.

FIG. 6 is a diagram illustrating an example of a visual field space in a three-dimensional map space.

FIG. 7 is a diagram showing an example of a projection view.

FIG. 8 is a diagram illustrating an example of superimposition of label information on a landscape image.

FIG. 9 is a configuration diagram of a movement prediction type landscape labeling system according to an embodiment of the present invention.

FIG. 10 is a configuration diagram of a navigation device disclosed in JP-A-8-273000.

FIG. 11 is a diagram illustrating a display example of a moving image.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 Image acquisition part 2 Position information acquisition part 3 Camera attribute information acquisition part 4 Movement amount prediction part 5 Map information management part 6 Label information creation part 7 Label information output part 8 Control part 21-35 Step 40 Labeling terminal 41 Image acquisition part 42 Position information acquisition unit 43 Camera attribute information acquisition unit 44 Movement amount estimation unit 45 Communication control unit 46 Terminal control unit 47 Label information output unit 50 Scenery labeling center 51 Map information management unit 52 Label information creation unit 53 Communication control unit 54 Center control unit 60 communication network

 ──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Shoji Takano 3-19-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo Nippon Telegraph and Telephone Corporation (72) Takeshi Ikeda 3--19, Nishishinjuku, Shinjuku-ku, Tokyo No. 2 Nippon Telegraph and Telephone Corporation

Claims (2)

[Claims] 1. A movement amount prediction type landscape labeling device mounted on a vehicle and attaching a label to an image viewed from the vehicle, wherein the image acquisition unit acquires an image, and the position information acquisition unit acquires a camera position. Camera attribute information obtaining means for obtaining a camera angle, a focal length, and an image size; and a camera angle and the focal length from a first time at which a labeling instruction is received to a second time at which label information can be displayed. Is assumed to be constant, a movement amount prediction means for estimating the position at the second time from the geographical information and the moving speed of the vehicle based on the first time, and managing the map information, the second time Calculates the visual field space in the map information space based on the position at, the camera angle and the focal length at the first time, projects each structure in the visual field space onto the camera plane, and further performs hidden surface processing. At the second time Map information management means for extracting a structure to be used, label information creation means for creating label information including the name of the extracted structure or its attribute information and the assigned position, and information on the assigned position in the created label information. Corresponding,
A label information output unit that superimposes the name of the structure or its attribute information at a position in the image acquired at the second time and displays the superimposed image on a visual device; and a control unit that controls the above units. A landscape labeling device that predicts the amount of movement. 2. A landscape labeling terminal and a landscape labeling center mounted on a vehicle, wherein the landscape labeling terminal is an image acquisition unit for acquiring an image, a position information acquisition unit for acquiring a camera position, a camera angle and a focus. A camera attribute information acquiring unit for acquiring a distance and an image size; and assuming that the camera angle and the focal length remain constant from a first time at which a labeling instruction is received to a second time at which label information can be displayed. A moving amount estimating means for estimating a position at a second time from geographical information and a moving speed of the vehicle based on the first time; a camera angle, the focal length, the image size, and a second time Communication control means for transmitting the position at the landscape labeling center via the communication network, and receiving the label information from the landscape labeling center, the structure of the structure in the label information Label information output means for superimposing the name or its attribute information on the position corresponding to the application position in the unprocessed image obtained at the second time, and displaying the superimposed image on a visual device; Has a terminal control means to control, the landscape labeling center receives the camera angle, the focal length, the image size and the position at the second time from the landscape labeling terminal via the communication network,
Communication control means for transmitting the label information to the landscape labeling terminal, and managing the map information, in the map information space based on the position at the second time and the camera angle and the focal length at the first time Map information management means for obtaining the visual field space of the object, projecting and transforming each structure in the visual field space onto the camera surface, further performing hidden surface processing, and extracting the structure that will be visible at the second time. A movement amount prediction type landscape labeling system, comprising: label information creating means for creating the label information including the name or attribute information of the structure and the assigned position; and a center control means for controlling each of the means.