JP2003337947A - Method and device for image display, and storage medium recorded with image display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a mark, etc., of an object correspond to a map of the circumference in an image, to display a moving picture, etc., of a reproduced image of a three-dimensional shape of the object changing as a camera moves, and to display changes in camera direction and camera position with marks, etc., in the image. <P>SOLUTION: Objects 22,..., etc., are specified on an image display and while the objects 22,..., etc., are traced, features of the objects 22,... including changes in relative angle, direction, and distance between the camera and objects 22,..., etc., are detected; and image frames having correspondence relation with the objects 22,..., etc., and image data of the features are retrieved from databases 13 to 33, etc., to constitute reproduced images 26 (26a,...) including two-dimensional and three-dimensional shapes by time-series changes. Then the reproduced images 16,..., etc., are displayed in a necessary area on the image display 16, etc., constratively in sequence as a moving picture or a group of successive still pictures based upon a necessary size standard and name and property data on the reproduced images 26, etc., are also displayed. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention is, for example, for searching the status of roads, buildings, rivers, etc., searching the status of the ground, etc. from an aircraft in the sky, or searching for the model or status of other flying objects. A system related to image display used for, for example, the position of a camera (vehicle-mounted video camera) mounted on a vehicle traveling on the road or near a river, search for the position, shape, etc. of an arbitrary object in a video image, Or the position of a camera (aerial video camera) mounted on an aircraft, the position of any object in the video image such as the ground, the shape, the surrounding condition search, etc. in normal times, emergency, or other safety The present invention relates to an image display method, an image display device, and a storage medium recording the image display method, which are suitable for use in monitoring and maintaining the damage situation.

[0002]

2. Description of the Related Art In recent years, many vehicles are equipped with a car navigation system for convenience such that a destination can be easily reached by following a map search guide displayed by the navigation system when driving a car. It has become. In addition, in a general navigation system, it is possible to plot its own position almost accurately on a map on which images are displayed by GPS position search, and various equipment, traffic lights installed on the map, etc. Is displayed, and depending on the system, a bird's-eye view can be displayed.

On the other hand, at a road / river management office that manages roads / rivers, etc., the present applicant has proposed, for example, to grasp the actual conditions of roads in the jurisdiction and to manage them appropriately. For example, Japanese Patent No. 3099103. The current video search device for roads, rivers, etc. is used. With this search device, by actually traveling on a road, moving images and continuous still images related to the surrounding conditions of the road, which are sequentially captured,
By displaying it as that at any point, you can immediately search for the situation in normal times and display an image, and at the same time, check the status of various facilities at that point,
It is also possible to display related facility maps, records, etc. for management.

Further, when the search or image display is performed, not only the actual image of the road and the building are displayed but also together with them, a specific object or the like is displayed by computer graphics in another area in the image. A system in which the function of displaying an image is added and the function is greatly expanded to improve convenience is proposed by the applicant in Japanese Patent Application No. 2002-017363.

[0005]

However, in the above-mentioned conventional navigation system or so-called present-day video search apparatus, the camera itself moves, so that a building or an object in a certain point in the image is displayed in the video image. Since the object changes its position and shape from moment to moment, it is difficult to observe the process of changing the shape by changing the direction at the same time or under the same condition with time transition. In addition to not being able to display the camera position and camera direction, the video image changes from moment to moment depending on the direction of the camera. , Area and volume cannot be measured, and 3 of any object in the video image can be measured.
It was not possible to calculate the 3D shape and display it as a 3D image.

Therefore, according to the present invention, even if the video image changes moment by moment, a map of the surrounding area is displayed as an image on the screen where the image is displayed, and a mark or the like corresponding to an arbitrary or specially designated object is displayed. It always corresponds to the map in the image, and it corresponds to the object that changes in time series with the movement of the camera, and continues as a group of continuous moving images or a moving image of a three-dimensional reproduced image that continues. 3 that corresponds to the display area of the map that is capable of being displayed, and further includes the mark corresponding to the object or the image display of the reproduced image, and the object.
It is also possible to separately display or overlap the display areas of the three-dimensional shape reproduction image, and even if the camera direction and camera position change, the change situation (distance recognition and change Image display method, an image display device, and a storage medium storing the image display method for solving the above problems. The purpose is to provide.

[0007]

In order to achieve the above object, the image processing method of the present invention provides one or a plurality of objects at arbitrary positions on the image display of a video image taken by one or a plurality of cameras. Things 22, 51, 72, 73,
74, 75, 76, etc. are specified by name or attribute, or surrounded by a square with a mouse, click a one-point position, or specify with a light pen or touch panel, and the surroundings of the target object 22 ... While excluding the existence of each of the image frames of the target object 22 ... While tracking the target object 22 ... Including the change in the relative angle and direction between the camera and the target object 22 ... The object 22, etc. are detected from the databases 13 to 33, etc. in which the characteristics or the characteristics of each component such as the object 22 are sequentially detected in time series, and the image data relating to the plurality of various characteristics are abundantly stored. Of the image frames, or the image data having the correspondence relationship with the continuous change of the characteristics of the respective component parts, or the image data of the features are sequentially searched, and the search is performed according to the search result. 2D to reproduce image 26a including a three-dimensional shape corresponding to the object 22 ... and the like well-over down matching, 26b, 26c, etc. with sequentially constituting each time-series change of 26 d, the reproduced image 26a
Each image frame including ..., or image data of the feature is required on the image display 16, 36, 67 or the like, or on a required area on another image display (not shown) via a communication line or the like. A name or attribute data that is displayed in accordance with the setting of the size standard and is sequentially displayed as a moving image or a group of continuous still images in contrast with each other, and if necessary, also associated with or generated in the reproduced image 26 (26a ...) The point is to display in the required area. Further, according to the method of the present invention, the image data of various characteristics stored in the databases 13 to 33 and the like are stored in advance in the respective image frames of various reproduction objects.
Alternatively, if each attribute data is associated with the image data of the characteristic corresponding to each component, and the above-mentioned pattern matching is performed on the image data of at least one characteristic, the attribute data is immediately changed based on the attribute data. The main point is to construct a reproduced image 26 including a two-dimensional or three-dimensional shape corresponding to the target object 22 and the like and display the above-described image of the reproduced image 26 and the like. In addition, according to the method of the present invention, the image data of various characteristics stored in the databases 13 to 33 and the like includes various reproduction objects, various straight lines, various curves, various angles, various edges, various planes, various curved surfaces, etc. in advance. In addition to the combination of the elements, the attribute data that clarify the combination, the assembly, and the connection relationship between the image data of each feature are associated with each other, and the combination of the image data of at least one set or more, etc. On the other hand, when the above-mentioned pattern matching is obtained, the reproduction image 26 including a two-dimensional or three-dimensional shape corresponding to one or a plurality of objects 22 ... The main point is to perform the above-described image display of the reproduced image 26 ... Further, the method of the present invention corresponds to the object 22 ...
The reproduced image 26 including a three-dimensional to three-dimensional shape is viewed from various viewpoints such as the target object 22 which is associated with the calculation of the two-dimensional to three-dimensional shape even if the line-of-sight direction changes due to the movement of the camera. The main point is to perform the above-mentioned image display of the reproduced image 26 (26a ...) And the like including the two-dimensional to three-dimensional shape. Further, the method of the present invention obtains image data having a spatial distribution and a temporal distribution of the object using a plurality of cameras, and at least three-dimensional corresponding to the object 22 ... etc. by parallax information and time difference information. A reproduced image 26 including a shape is generated, and thus the reproduced image 26 is generated.
The main point is to generate a three-dimensional model such as the above and display the above-mentioned image. In addition, the method of the present invention uses the reproduced image 26 described above.
, Etc. are sequentially and sequentially displayed as a group of the above-mentioned moving images or continuous still images with time-series changes, and are displayed on the image display 16, 36, 67, etc. or on the other image display. Displays an image of the surrounding maps 23 to 71 etc. of the above-mentioned objects 22 ... etc. as necessary and displays the objects 22 ... etc. at the corresponding positions in the maps 23 to 71 etc. with the required marks 24 etc. Alternatively, the reproduced image 26 is displayed on the basis of the pattern matching, and the display areas of the maps 23 to 67 and the reproduced image 26 are compared with each other, if necessary. , The point is to overlap both. In the method of the present invention, the positional relationship between the known point and the camera is specified by designating a plurality of points on the object 22 ... Etc. designated on the image display or the other image display. When the above calculation algorithm is also stored in the database and other points are specified, the calculation algorithm is used to calculate the mutual points among arbitrary objects 22 ... The point is to calculate the distance and to calculate and measure the area or volume of the entire surface of the target object 22 ... In addition, the method of the present invention uses the reproduced image 26 accompanying the movement of the camera.
With regard to moving images such as ... or those sequentially displayed as a group of continuous still images, the camera position, camera direction, and camera moving speed are obtained by the required calculations, and the relevant items are given marks or numeric characters. The point is to display an image with. In addition, the method of the present invention uses the above-described reproduced image 26 on the image display or the other image display.
Areas to be sequentially displayed as a moving image such as ... Or a group of continuous still images, or the reproduced image 26 ...
The point is that the area for displaying the name and attribute data of the above, or the area for displaying the camera position, the camera direction, and the camera moving speed can be arbitrarily set according to a required operation or designation. Further, according to the method of the present invention, when the target object 22 ... Is designated, the target object 22 ...
The main point is that even if an area range that is wider or narrower than, for example, is selected and specified, only the target objects 22 ... Can be extracted and recognized. In addition, the method of the present invention, when the above-mentioned objects 22 ... Or the like or a part of them is designated, the shape formed by connecting them, the position of the shape, the distance,
Attribute data including area or volume, and these 2
The main point is to display the reproduced images 26, etc., which are obtained by calculating the required correlations for the three-dimensional to three-dimensional shapes, as described above. Further, the method of the present invention constructs a CG image of a three-dimensional object such as a target object 22 such as a building on the above image, and sequentially updates the CG image in time series as the camera moves as a landscape simulation. The main point is to display an image on the video image on the image display or the other image display, on the maps 23 to 71, or in a required area.
On the other hand, the image display device of the present invention includes one or a plurality of cameras for photographing the outside world, an input unit 11 for converting a time-sequentially changing video image photographed by the camera into an electric signal and inputting it at any time, and an image. Any one or more objects 22,
A designation unit (not shown) for designating 51, 72, etc., and an image clipping unit 12 for clipping the existence other than the target objects 22, 51, 72, etc., and extracting only the target objects 22, 51, 72, etc.
Correlation with each image frame of the object 22 or the like, or a plurality of image frames in which the image data of the characteristics of each component is abundantly stored in the image storage unit 13 (that is, the database) or the image data of the characteristics. And a pattern matching image frame or feature image data are calculated, and a new image frame or feature image data is additionally stored in the image storage unit 13 if necessary, and the correlation calculation unit A three-dimensional image forming unit 15 for generating a reproduced image 26 or the like including a two-dimensional or three-dimensional shape corresponding to the target object 22 or the like based on the image frame or the image data of the characteristic obtained in 14, and the video image or map. Two
3 to 71 and the like are displayed as an image, and the output display unit 16 that sequentially outputs the reproduced image as a moving image or a group of continuous still images, and the reproduced image 26 and the like in the output display unit 16. A setting unit (not shown) for setting the display area 25 and the like, and a required display in the output display unit 16 in which the reproduction image 26 and the like generated by the three-dimensional image forming unit 15 is set by the setting unit. Images are sequentially displayed in the area on the basis of a required size as described above, and the reproduced image 2 is displayed if necessary.
The main point is to display the names and attribute data associated with or generated in 6 and the like at the required positions in the same or other required display areas set by the setting unit. Further, the image display device of the present invention includes one or a plurality of cameras for photographing the outside world, an input unit 31 for converting time-sequentially changing video images photographed by the cameras into electric signals and inputting them at any time, and the video. Any one or more objects 2 from the video image
A designation unit (not shown) that designates 2, 51, 72, etc. is compared with the features of the entire subject 22, 51, etc. or each component excluding the existence other than the subject 22, 51, 72, etc. A comparison image generation unit 32 that generates a reference image, a database 33 that abundantly stores image data of various reproduction objects and characteristics of various components, and the database 3
A comparison component generation unit 34 that generates an image frame of the comparison component or image data of characteristics by performing a comparative search of 3;
The comparison image data of the object 51 or the like and the image data of the characteristic of the comparison part generated by the comparison part generating unit 34 are compared and searched to obtain two-dimensional to three-dimensional images corresponding to the objects 22, 51 or the like. A reproduction part image including a three-dimensional shape is specified, and a name attached to the reproduction part image, if necessary,
And the component specifying unit 35 that searches or generates attribute data and finds a position to be fixed for the image display of the reproduced component image, displays the video image and the maps 23 to 71, and the reproduced component image. An output display unit 36 for displaying an image at a position to be fixed on the maps 22 to 71 and the like is provided, and the tissue of the reproduced component image is required at a position to be fixed on the maps 23 to 71 and the like in the output display unit 36. Mark or number character, or as a group of moving images or continuous still images in two-dimensional or three-dimensional shape, the images are sequentially and sequentially displayed in time series, and if necessary, the moving images or continuous still images are displayed. The main point is to display the associated or generated names and the attribute data at the required positions in the output display unit 36. Further, the image display device of the present invention,
One or a plurality of cameras for photographing the outside world, a video input device 61 for inputting an electric signal of the time-sequentially changing video image, and a time-sequentially changing video image captured by the camera to an electric signal. A video recording / reproducing output device 62 for recording and reproducing, and one or a plurality of arbitrary objects 22, 51, 72 from the time-sequentially changing video images.
A video map for displaying the target video image and the maps 23 to 71, etc., on the target display screen 63 or the like, which is capable of designating ... etc. by excluding unnecessary surroundings, or at the same time relatively dividedly and displayed on a required display screen. A display device 64, a name of the reproduced image, and an attribute recording / reproducing device 65 that retrieves or generates, records and reproduces attribute data, and various image frames corresponding to the target object 72, etc., or features of various components. 2 corresponding to the target object 72, etc. by comparing and searching corresponding image frames or image data of features from a database (not shown) in which abundant image data of 2 are stored.
A target object search and designation device 66 for generating a reproduced image including a three-dimensional to three-dimensional shape, and a group of moving images or continuous still images of a reproduced image including the two-dimensional to three-dimensional shape corresponding to the target object 72. And a lock-on display unit 67 for sequentially performing lock-on display as a comparison, and the objects 22, 51, on the display screen on which the video image and the map are displayed in an overlapping manner or are simultaneously displayed separately. 72 ...
Required marks are displayed at positions corresponding to existing positions, etc., and the reproduced images corresponding to the objects 22, 51, 72, etc. are sequentially displayed on the lock-on display section 67 as described above. The main point is to display and, if necessary, also display a name attached to or generated in the reproduction image and attribute data at a required position on the display screen or a required position in the lock-on display unit 67. . Furthermore, the method and apparatus of the present invention can be applied to the image sizes and line-of-sight angles of the objects 22, 51, 72 and the like, and various reproducible objects or images of various characteristics that are abundantly stored in the image storage unit or the database. A two-dimensional or three-dimensional object corresponding to the objects 22, 51, 72, etc. having the same size and line-of-sight angle, provided with a search adjustment unit (not shown) that matches the size and line-of-sight angle with data The main point is that the reproduced images including the shapes are comparatively searched and generated, and the reproduced images are sequentially displayed as described above. Furthermore, the method and apparatus of the present invention make several points 72 etc. in the above video image correspond to the map, and track these with time-series changes to read the coordinates on the map, and read the several points 72 etc. Converting means (not shown) for calculating the camera position, the camera direction, and the camera speed from the coordinates of, and converting the video image and the map into the same coordinate system or a required coordinate system having a required line-of-sight angle. And a reproduced image including a two-dimensional or three-dimensional shape corresponding to the mark or the line-of-sight angle indicating the object 22, 51, 72 or the like at a corresponding position on the map having any coordinate system as described above. The main point is that the names and the attribute data that are sequentially displayed and, if necessary, associated or generated with the reproduced image are displayed at required positions. Further, the method and apparatus of the present invention comprises a calculating means (not shown) for calculating and calculating the one or more or a representative camera position, the camera direction, and the camera moving speed in relation to the outside world, and the maps 23 to 23. The point is to display a mark or numeral character indicating the camera position and the camera direction at the corresponding position on 71. Furthermore, the method and apparatus of the present invention calculates v = f · y · sin (tan ^{−1 −1)} by calculating the relationship between the one or more or each of the camera positions, the ground surface, and the object at any position. (H / y)) / sin θ / {h · cos θ (tan ⁻¹ (h / y) −θ)} (1) or v = f {tan ((π / 2) −θ) −tan ((π / 2) -θ-tan ^-1 (y / h))} (2), where y is the distance from directly under the camera to the object, v
Is the length of the object on the CCD surface in the camera, and θ is the dip angle of the camera. Furthermore, the method and apparatus of the present invention can calculate and clarify the correspondence between the above-mentioned one or more camera positions or a video image captured by the camera and an object on the map. f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) −θ)} (3) or v = f · {tan ((π / 2) -θ) -tan ((π / 2) -θ-tan ⁻¹ (y / h))} (4) or v = {(f / sin θ) · y · sin β) / (h · cos ( β-θ))} (5) u = x · (f / y) · (cos β / cos (β-θ)) (6) where β = arcTan (h / y) and y is the camera Distance from beneath to the object, v
Is the length of the object on the CCD surface in the camera, x is the length in the direction perpendicular to the line segment connecting the object and the camera, u is the lateral length of the CCD surface in the camera, and θ is the camera The main point is that it is a dip. Furthermore, the method and apparatus of the present invention calculate and clarify the relationship between the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface. (F / sin (θ + (π / 2)) · Z · sinγ)} / {y · cos (γ− (θ + (π / 2)))} (7) where γ = arcTan (y / z) F is the focal length of the lens, θ is the dip angle of the camera, and φ = θ + (π / 2), φ is the perpendicular of the connecting position of the line segment connecting the lens and the object on the camera intersecting axis. Angle, ε corresponds to β + (π / 2), y corresponds to Z, and the relationship between the height dimension of the target object and the height dimension i of the image of the target object on the CCD surface. I = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (β + (π / 2)-(θ + (Π / )))} (8) where, beta = determined by arcTan (y / z), h is the height of the position of the camera, other
f, θ, etc. are the same as in the case of the above formula (7),
Further, in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above Wi-{= [(f / sin (θ + (π / 2)) · Z · sinγ )] / [Y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h · sin (β + (π / 2))] / [Y · cos (β−θ)]} (9) Further, i′−W ′ = (i−W) / cos δ (10) where δ = arcTan {(t−u) / (i−W) }, I'is the height dimension of the other target object at the other part of i or the same row position, and W'is the height apex of the other part of W or the height of the other object at the same row position. The above-mentioned target object is represented by using any one or a plurality or all of the above-mentioned formula (7), formula (8), formula (9), or formula (10), as necessary. To the object It is also essential to be able to perform a comparative search of a reproduced image including a corresponding two-dimensional to three-dimensional shape. In order to calculate and clarify the position of the image t of the extending portion, t = S (f / Z) · {cosγ / cos (γ-θ- (π / 2))} (11) This is a portion that extends laterally from the target object, and by using this formula (11), it is possible to perform a comparative search between the target object and a reproduced image including a two-dimensional to three-dimensional shape corresponding to the target object. What you have done is the main point.
Furthermore, the method and apparatus of the present invention use any one or more mathematical expressions, or all mathematical expressions (1) to (11) from the above expressions (1) to (11) as the attribute. The point is that it is added to the data. Further, the method and apparatus of the present invention, the object is a road, a sign, a river, a canal, a building, various facilities, various equipment, a vehicle, a motorcycle, an aircraft, a hull, or a moving vehicle, a motorcycle, an aircraft, The main point is that it is a hull or a structure with other operating functions. On the other hand, the method and apparatus according to the present invention itself display the reproduced image corresponding to the one or more target objects 22, 51, 72, ... For the required maps 23 to 71, etc., and if necessary, Web distribution including name and attribute data, camera position, camera direction, and camera moving speed is possible, and the above-mentioned display status of the above-mentioned reproduced image on the maps 23 to 71 and the like is also requested by the client. , And the map 2
Arbitrary objects 22 and 5 instructed to 3 to 71 etc.
The main point is to enable distribution of the display status such as marks or numerical characters indicating 1, 72, ..., The name associated with or generated in the reproduced image, and the attribute data. On the other hand, the recording medium of the present invention has at least the above-mentioned calculation algorithm, equations (1) to (11), and one or more objects 22, 51, 7 photographed by one or more cameras.
A step of calling a comparative part which is a component such as 2; a step of retrieving the image data of the feature relating to the corresponding part of the comparative part from a database in which abundant image data of a plurality of types of features are stored; A step of determining whether or not the comparison part and the image data of the feature correspond, a step of determining whether or not the calling of the predetermined comparison part is completed when the both do not correspond, and the call Both of the above steps correspond to the step of determining whether or not the processing for the predetermined component has ended if not ended, the step of designating the next comparison part if the processing for the predetermined part has not ended, In the case of doing, the step of recognizing the specific object 22, 51, or 72, etc., the step of specifying the next part, and the object 2
And a reproduction image including a two-dimensional or three-dimensional shape of the target object 22 or the like which is a recognition result when the processing of a predetermined portion of the comparison image in the second or the like is completed, And the step of sequentially outputting the moving images or a series of continuous still images as a group.

In the image display method of the present invention as described above, one or a plurality of objects 22, 51, 72, etc., at arbitrary positions on the image display of the video image are designated and the camera and the image display device are selected. Alternatively, the characteristics of each image frame of the target object 22 or the like, or the characteristics of the target object 22 or the like, including changes in the relative angle and direction with respect to the plurality of target objects 22 and the like, and changes in the distance, and tracking the target object 22 or the like. Etc., the characteristics of each component, etc. are sequentially detected in time series. The characteristics of each image frame of the target object 22 or the like from the databases 13 to 33,
Alternatively, the image data of the feature having a correspondence relationship with the continuous change of the feature of each component is sequentially searched. Two-dimensional to three-dimensional corresponding to the target object 22 or the like corresponding to this search result
Each image frame or the like of the reproduced image including the dimensional shape is sequentially configured for each time series change, and each image frame or the like including the reproduced image is displayed on the image displays 16 to 33 or 67 or on the communication line or the like. Through the other image display (not shown), it is sequentially displayed as a group of moving images of a required size or a series of continuous still images on a required area, and is attached to the reproduced image as needed. It is also possible to display the generated name and attribute data in a required area. Further, in the method of the present invention, the image data of various characteristics stored in the databases 13 to 33, etc., includes the image data of the characteristics corresponding to each image frame of each reproduction object or each component in advance. Since the name and each attribute data are associated with each other, when the above-described search processing is performed, the target object 22, 5 is recognized based on the recognition of each attribute data attached to or generated in the image data of the feature.
It is possible to configure an image frame or the like of a reproduced image including a two-dimensional or three-dimensional shape corresponding to 1, 72 or the like and perform the above-described image display of the reproduced image. Further, in the method of the present invention, the image data of various characteristics stored in the databases 13 to 33 and the like includes various reproduction objects, various straight lines, various curves, various angles, various edges, various planes, in advance. Elements such as various curved surfaces, or in particular, each attribute data including a combination relation between image data of each feature are associated with each other. Therefore, by referring to each attribute data, one or a plurality of objects 22, 51, An image frame or the like of a reproduced image including a two-dimensional or three-dimensional shape corresponding to 72 or the like can be configured and the above-described image display of the reproduced image can be performed. Further, in the method of the present invention, the one or more objects 22, 51,
Even if the reproduced image including the two-dimensional or three-dimensional shape corresponding to 72 or the like changes in the line-of-sight direction due to the camera movement,
2D to 3 from various viewpoints of the object 22
The above-mentioned image display of the reproduced image of the three-dimensional shape can be performed. Also,
The method of the present invention comprises one or more objects 22, 51, 7
Image data having a spatial distribution and a temporal distribution such as 2 is obtained, and image data of a reproduced image including a three-dimensional shape corresponding to the target object 22 and the like is generated based on at least parallax information and time difference information, and thus the reproduced image. It is possible to generate a three-dimensional model of the above and display the above-mentioned image. In addition, in the method of the present invention, one or a plurality of the specified objects 22, 51, 72, etc. specified above may be displayed on the image display 16 to 36 or 67 or the other image display as needed. The surrounding maps 23 to 71 and the like are displayed as images, and the target 22 and the like are reproduced at the corresponding positions in the maps 23 to 71 and the like with a required mark or the two-dimensional to three-dimensional shape based on the above pattern matching. It can be displayed as an image, and if necessary, the display area of the map 23 or the like and the display area 67 of the reproduced image or the like can be contrasted or both can be overlapped. Further, in the method of the present invention,
Of the objects 22, 51, 72, etc., the positional relationship between the known point and the camera is calculated by designating a plurality of points, and the calculation algorithm is also stored in the database, regarding other points. Even when specified, the respective calculation points are used to calculate the mutual points and distances of the arbitrary target object 22 and the like, and the area of the entire surface of the target object 22 or the like or the surface of some component parts And volume calculation can be performed. Further, in the method of the present invention, the camera position, camera direction,
The camera moving speed is obtained by a required calculation, and the relevant item can be displayed as an image with a required mark or numeral character. Further, in the method of the present invention, a region to be sequentially displayed as a moving image or a group of continuous still images, or the name or attribute data of a reproduced image corresponding to the object 22, 51, 72 or the like. Can be arbitrarily set according to a required operation or designation of the area for displaying, or the area for displaying the camera position, camera direction, and camera moving speed. Further, in the method of the present invention, when the target objects 22, 51, 72, etc. are designated, even if the area range wider or narrower than the target object 22, etc. is selected and designated, the target object 22, 51, 72, etc. Two
Only 2 etc. can be extracted and recognized. Further, in the method of the present invention, the above-mentioned arbitrary objects 22, 51, 72
Etc. or a shape formed by connecting a part of them, and attribute data including the position, distance, area, or volume of the shape,
Also, the reproduced image including the two-dimensional or three-dimensional shape can be displayed. Further, in the method of the present invention,
A CG image of a three-dimensional object such as the target object 22, 51, 72 is constructed, and the CG image is sequentially updated in time series as the camera moves as a landscape simulation, and the video image or maps 23 to 71, or a required area. Image can be displayed. On the other hand, in the image display device of the present invention, in particular, the designation unit (not shown) designates an arbitrary object 22, 51, 72 or the like from the image in the output display unit 16, and the image cutting unit 12 Existences other than the specified target objects 22, 51, 72, etc. are cut out and only the target object 22, etc. are extracted, and a plurality of image frames stored in the image storage unit 33 in the correlation calculation unit 14 or image data of features Of the two-dimensional to three-dimensional image corresponding to the object 22 or the like in the three-dimensional image construction unit 15
In addition to generating a three-dimensional shape reproduced image and displaying the video image and the map 21 on the output display unit 16, the reproduced image is sequentially output as a moving image or a group of continuous still images in comparison. At this time, the display area of the reproduced image is set in the output display unit 16 via the setting unit (not shown). Therefore, the reproduced image is sequentially displayed on the required display area in the output display unit 16 set by the setting unit with the required size as described above, and is attached or generated to the reproduced image as necessary. The name and the attribute data to be displayed can also be displayed at the required position in the same or another required display area set by the setting unit. Further, in the image display device of the present invention, in particular, the designation unit (not shown) designates an arbitrary object 22, 51, 72 or the like from the image in the output unit 36, and the comparison image generation unit 32. Then, the features of the entire object 22, 51 or the like or each component is generated as a comparison image, and the comparison component generation unit 34 performs a comparative search on the database 33 to generate the image data of the feature of the comparison component. In addition, the part specifying unit 35 performs a comparative search between the comparison image data of the objects 22, 51 and the like and the image data of the characteristics of the comparison part generated by the comparison part generating unit 34 to perform the search. , 51, etc., the reproduced part image including a two-dimensional to three-dimensional shape is specified, and if necessary, the name and attribute data associated with the reproduced part image are also retrieved or generated, and then output to the output unit 36. Find the position to fix That. Therefore, the tissue of the reproduced part image including the two-dimensional or three-dimensional shape is placed at the position to be fixed on the map in the output unit 36, and the required mark or numeral character, or the two-dimensional or three-dimensional moving image or continuous stillness is obtained. As a group of images, the images can be sequentially displayed in time series in contrast, and if necessary, the name and attribute data associated with or generated in the moving image or the continuous still image can be displayed. Further, in the image display device of the present invention, in particular, the video recording / reproducing output device 62 converts the video image into an electric signal for recording and reproduction, and the video input device 61 inputs the electric signal of the video image, In addition, the video map and the map are superposed on the video map designating device 64, or simultaneously displayed in a relatively divided manner on a required display screen,
Then, the object specifying device 63 specifies an arbitrary object 22, 51, 72, etc. from the video image, and the object search specifying device 66 specifies the corresponding feature from the database (not shown). The image frame or the image data of the feature is compared and searched to generate a reproduced image including a two-dimensional or three-dimensional shape corresponding to the object 22, 51 or the like.
The attribute recording / reproducing device 65 retrieves or generates the name and attribute data of the reproduced image and records and reproduces the reproduced image, and at the same time, the lock-on display section 67 displays a moving image or a continuous image of the reproduced image corresponding to the object 22, 51 or the like. Since the lock-on display is performed sequentially as a group of typical still images, the object 2 is displayed on the display screen on which the video image and the map are displayed in an overlapping manner or are simultaneously displayed separately.
Required marks are displayed at positions corresponding to existing positions of 2, 51, etc., and reproduced images corresponding to the objects 22, 51, etc. are sequentially displayed on the lock-on display section 67 as described above. In addition, an image of the name or attribute data attached to or generated in the reproduced image can be displayed at a required position on the display screen or a required position in the lock-on display section 67. Further, in the method and apparatus of the present invention, the search adjusting unit (not shown) and the size of the object 22, 51, 72 or the like on the image or the line-of-sight angle and the databases 13 to 33 are used.
In order to match the size and line-of-sight angle with the image data of various reproduced objects or various characteristics that are abundantly stored in the two-dimensional corresponding to the object 22 and the like having the same size and line-of-sight angle. A reproduced image including a three-dimensional shape can be comparatively searched and generated, and the reproduced image can be sequentially displayed as described above. Further, in the method and apparatus of the present invention, several points 72, 73, 74, 7 in the above video image are included.
5, 76, etc. are made to correspond to the map 71 etc., and this is tracked along with changes in time series to read the coordinates on the map 71 etc., and calculate the camera position, camera direction, and camera speed from the coordinates of several points read. Since the video image and the map 71 or the like can be converted into the same coordinate system or the required coordinate system having the required line-of-sight angle, the correspondence on the map 71 or the like having any coordinate system can be obtained. The reproduced images including the two-dimensional or three-dimensional shapes corresponding to the marks or the line-of-sight angles indicating the target objects 22, 51, 72, etc. are sequentially displayed at the positions to be displayed, and the reproduced images are attached to the reproduced images as necessary. The generated name and the attribute data can also be displayed at desired positions. Furthermore, in the method and apparatus of the present invention, one or more or a representative camera position and camera direction are calculated and obtained in relation to the external environment,
At the corresponding positions on the maps 23, 71, etc., it is possible to display a mark A or the like indicating the camera position and the camera direction, or numeric characters. Furthermore, in the method and apparatus of the present invention, the above-mentioned equation (1) or equation (2) is used to calculate the above-mentioned one or more or representative camera positions, so that the processing time is short. Moreover, it is possible to obtain an accurate camera position. Furthermore, in the method and apparatus of the present invention, it is necessary to calculate and clarify the correspondence relationship between one or more or a representative camera position or a video image taken by the camera and an object on a map. Since the calculation is performed using the above formulas (3) to (4), or the formulas (5) and (6), the processing time is short and the camera position is accurate, or the map and each target are similar to the above. It is possible to find the exact correspondence of objects. Furthermore,
In the method and apparatus of the present invention, the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface.
The above equation (7) is used to calculate and clarify the relationship with and the relationship between the height dimension of the target object and the height dimension i of the image of the target object on the CCD surface is calculated. In order to clarify, the above formula (8) is used, and in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above formula (9) or the above formula is used. (10) is used, and since the above equation (11) is used to calculate and clarify the position of the image t of the portion that extends laterally from the object on the CCD surface, each calculation can be performed in a short time. Can be done,
By appropriately using the above formulas (1) to (11) as necessary, it is possible to perform a comparative search between the object and a reproduced image including a two-dimensional to three-dimensional shape corresponding to the object. Also enables. Further, in the method and apparatus of the present invention, any one or a plurality of formulas (1) to (11) among the formulas (1) to (11), or all the formulas (1) to (11), as required. Is added to the attribute data, the above-mentioned comparison search can be performed efficiently and easily by referring to and decoding the attribute data.
Further, in the method and apparatus of the present invention, the object is a road, a sign, a river, a canal, a building, various facilities, various equipment, a vehicle, a motorcycle, an aircraft, a hull, or a moving vehicle, a motorcycle. , An aircraft, a hull, or any other structure that has operational functions, it can handle many objects. 2
It is possible to construct a three-dimensional to three-dimensional reproduced image and perform the above-mentioned image display. On the other hand, in the method and apparatus of the present invention, the display status of the above-mentioned reproduced image corresponding to the one or more target objects 22, 51, 72, etc. with respect to the required maps 23, 71, etc. It is possible to perform Web distribution including the name and attribute data, the camera position, the camera direction, and the camera speed according to the above, and conversely, even if the client requests the map 23,
The above-mentioned display status of the reproduced image for 71 etc.,
And a mark or numeral character indicating an arbitrary object 22, 51, 72, 73, 74, 75, 76, etc. designated for the map 23, 71 or the like, or the name attached to or generated in the reproduced image, and It is possible to distribute the display status of attribute data and the like. Furthermore, on the other hand, in the storage medium of the present invention, at least one or a plurality of objects recorded with at least the above-mentioned arithmetic algorithm, Expressions (1) to (11), and taken by one or a plurality of cameras. 22, 51, 72
The comparison parts which are the constituent parts of the comparison parts are called, and the image data of the characteristics related to the corresponding parts of the comparison parts are searched from the databases 13 to 33 etc. in which the image data of abundant plural kinds of characteristics are stored, and the comparison is performed. It is determined whether or not the parts for image and the image data of the feature correspond, and if both do not correspond, it is determined whether or not the calling of the predetermined comparison part is completed. Further, when the call is not completed, it is determined whether or not the process for the predetermined part is completed, and when the process for the predetermined part is not completed, the next comparison part is designated, Is recognized as a specific object image 22, 51, 72, etc., and the next part is specified. However, it is determined whether or not the processing of the predetermined portion of the comparison image in the objects 22, 51, etc. is completed, and if the processing is completed, the two-dimensional or This makes it possible to sequentially output a moving image of a reproduced image including a three-dimensional shape or a group of continuous still images in contrast to each other.

[0009]

BEST MODE FOR CARRYING OUT THE INVENTION [Outline of the Invention]
Input time-series and real-time video images, specify one or more objects from the video images, and record the names and attribute data in association with each other. The images are displayed separately in contrast, and the objects in the video image are made to correspond so that the positional relationship with the corresponding points on the map on the image display matches, while the objects are displayed on the image display of the video image. Two-dimensional to three-dimensional images are sequentially displayed as a group of moving images or continuous still images that change in time series, and on the other hand, for example, a map and a video image are superimposed,
By designating each point with a mouse or the like, the area, volume, and distance of the target object within the enclosed range are obtained by the required calculation, and the position of the camera, the camera direction, the camera moving speed, etc. are also obtained. As a display feature of a three-dimensional image, it can be moved to another place in the image at the same time, simultaneously with the movement of the camera, in parallel as in a decomposed photograph at various angles (that is, for example, a perspective image), or continuously superimposed on one another in real time. The image is displayed as if it is updated (that is, in a moving image) or locked on like a flip-over picture.

A first embodiment of the present invention will be described below with reference to FIGS. 1 to 3. FIG. 1 shows the configuration of the main part of the image display device according to the present embodiment. The image display device includes one or a plurality of cameras (not shown), an input unit 11,
A designation unit (not shown), an image cutout unit 12, an image storage unit (database) 13, a correlation calculation unit 14, a three-dimensional image construction unit 15, and an output display unit 16 are electrically connected as shown in the figure. Has been done.

The one or a plurality of cameras are installed in a predetermined arrangement on a roof of a predetermined vehicle or on a roof of a cargo container container or a box, for example, and are required to operate in a vehicle or to operate a base. Depending on the required operation of the station, it is possible to arbitrarily change the height and the shooting direction by electric drive, for example, the surrounding external buildings, the situation of rivers, various facilities, various signs, various traffic signals and power transmission. Photograph equipment (including tracking photography) of equipment such as steel towers, general houses, roads, etc. In addition, the camera measures the distance to the object by detecting the focus, measures the camera position of itself (or a representative camera position) by a triangulation method for photographing, and the two-dimensional object of the two-dimensional object or the three-dimensional object. It is used for the purpose of measuring shapes and three-dimensional shapes.

The input unit 11 converts a video image captured by a camera, which changes in time series, into an electric signal and inputs the electrical signal at any time. In the video image, all the object situations in the external image range are luminance, Video signals of color images such as density and contrast are included.

The designation unit (not shown) is located at an arbitrary position on the output display unit 16 of the image processing apparatus including a processor in the vehicle or on the output display unit (not shown) of the image processing apparatus of the base station. The line of sight of one or more objects can be specified by specifying the name or attribute of one or more objects, enclosing with a mouse in a square, clicking a single point position, or specifying with a light pen or touch panel. The target can be specified by determining the angle and performing corresponding point determination (matching). Also, regarding the designation,
It is possible to specify only the target object by selecting and specifying a wider range range or a narrower range range than the target object with a mouse or the like.

The image clipping unit 12 is a video image designated by the designating unit by, for example, the triangulation method using the two cameras described above, the twin-lens stereo image method using two or more ITV cameras, or the three-lens stereo image method. By stereoscopically viewing one or more required objects in the object and extracting, for example, the edge of the object, for example, the edge of the contour, the electric signal related to the existence other than the object is cut (disappeared), and thus the object Only the image is detected, and the entire target object,
Alternatively, each element such as various straight lines, various curves, various angles, various edges, various planes, various curved surfaces of each component is also detected. Further, the image cropping unit 12 or a separate tracking function activated in relation to the central processing unit and the image cropping unit 12 tracks the target object including the movement of the camera and the change of the camera direction, and captures the target object. It is possible to continue.

The image storage unit (database) 13 has various structures in the outside world, various rivers, various signs, various signals, equipment such as various towers related to power transmission, and various objects such as general houses and roads. Features of various components (that is, various straight lines, various curves, various angles, various edges, various flat surfaces, various curved surfaces, etc. of those various components, and various straight lines, various curves, etc. in the state of being rotated at an arbitrary angle, Various angles, various edges, various planes, various curved surfaces, etc., that is, image data of images or image configuration data of images, names of image data of various characteristics, attribute data, and the like are abundantly stored. When the name of the object or the attribute data is input, it is possible to increase the search speed of the correlation calculation unit 14, which will be described later, and the attribute data includes the nominal data related to the classification of the feature and other information such as the manufacturing date. Date, age, owner, manager, past inspection history, repair history, schedule of replacement / repair, replacement schedule, etc., or generation of a combination of image data of each feature in the image storage unit 13 (organization ), Or the shape formed by connecting the objects, and the position, distance, area, or volume of the shape are included.

The correlation calculation unit 14 is a feature of each image frame of the object or a feature of each component and the image storage unit 13.
Correlation with the above-mentioned plurality of image frames or the image data of the feature stored in abundance (including correlation after processing such as enlargement, reduction, rotation angle change (gaze direction change) of the comparison target by the search adjustment unit) (Including) and searching for the image data of the image frame or feature that has been subjected to the pattern matching, and if necessary, a new image frame or feature of the new image frame or feature that reflects the feature of the component The image data is also stored in the image storage unit 13.
Further, when the correlation calculation unit 14 retrieves and obtains one or more new objects or a part of them, for example,
The shape formed by connecting them and the position, distance, area, or volume of the shape are also calculated and stored in the image storage unit 13 as the obtained attribute data.

Further, for example, a camera position calculation function associated with the correlation calculation unit 14 or activated by a central processing unit or the like is separately used, for example, by using a triangulation method or a binocular or trinocular stereo image method. , The camera direction, the camera moving speed, and the like can be calculated and obtained. In addition, for example, the calculation function of the correlation calculation unit 14 calculates a correlation with the next image frame or an image frame after a fixed time, and The object is also tracked by examining the relationship between the part having the highest correlation coefficient from the image frame and the part close to the previous image frame, and sequentially changing the reference image (that is, the reproduced image) to a new image (new reproduced image). Therefore, it is possible to contribute to the control of changing the camera direction for the purpose of tracking the object accurately.

When the above-described pattern matching is performed on the image data of at least one feature, the correlation calculation unit 14 adds each attribute data (that is, the target object) attached or generated to the image data of the feature. It is possible to instantly construct an image frame of a reproduced image including a two-dimensional or three-dimensional shape corresponding to one or a plurality of objects based on a combination of each image data of all features, assembly, or connection relation data). Image data having a spatial distribution and a temporal distribution of one or a plurality of objects is obtained using a plurality of cameras, and a three-dimensional shape corresponding to the one or the plurality of objects is obtained by at least parallax information and time difference information. It is also possible to generate image data of a reproduced image including the specified image, and further, to specify a plurality of points on the object and to calculate the positional relationship between the known points and the camera by a predetermined calculation algorithm. Determined by the rhythm, thereby each cross of any object point, and it is also possible to determine the distance.

Further, the correlation calculation section 14 is also provided with a coordinate system calculation means (not shown), and several points in the video image are made to correspond to those on the map, and these are tracked along with changes in time series. In addition to finding the coordinate system on the map corresponding to the result and calculating the camera position, camera direction, and camera speed from the coordinates of several points, the video image and the map must have the same coordinate system or various line-of-sight angles. It is possible to convert to a coordinate system and display an image.

Furthermore, a calculation function (not shown) of the correlation calculation unit 14 calculates one or a plurality of or representative camera positions, y is the distance from directly under the camera to the object, and v is the inside of the camera. When the angle of the object on the CCD surface, θ is the dip angle of the camera, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan−1 (h / y ) −θ)} (1) or by v = f {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (2) It is possible to ask. Further, in order to calculate and clarify the correspondence between one or a plurality of or representative camera positions, or the video image taken by the camera and the target object on the map, y is the distance from directly below the camera to the target object, v is the length of the object on the CCD surface in the camera,
x is the length in the direction perpendicular to the line segment connecting the object and the camera, u is the lateral length on the CCD surface in the camera, and θ is the dip angle of the camera: v = f · y · sin (Tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) −θ)} (3) or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (4) or v = {(f / sin θ) · y · sin β) / (h · cos (β−θ)) } (5) u = x (f / y)  (cos β / cos (β-θ)) (6) However, β = arcTan (h / y) can be obtained. In addition, the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface.
In order to calculate and clarify the relationship with, f is the focal length of the lens, θ is φ = θ + (π / 2), φ is the connection of the line segment connecting the lens and the object on the camera intersection axis. When the angle formed by the perpendicular of the position, ε corresponds to β + (π / 2), and y corresponds to Z, W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {Y · cos (γ− (θ + (π / 2)))} (7) However, it is possible to obtain by γ = arcTan (y / z). Further, in order to calculate and clarify the relationship between the height dimension of the object and the height dimension i of the image of the object on the CCD surface, h is the height dimension of the position of the camera, and , Θ, etc. are the same as in the case of the above formula (7), i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (β + (π / 2)-(θ + (π / 2)))} (8) However, β = arcTan (y / z) can be obtained. Further, in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above W and i described above are used, and the above Wi = {[(f / sin (Θ + (π / 2)) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h * Sin ((beta) + ((pi) / 2))) / [y * cos ((beta)-(theta))]} ... (9) Furthermore, i'-W '= (iW) / cos (delta) ... (10) However, (delta) = arcTan. It can be obtained by {(t−u) / (i−W)}. Furthermore, in order to calculate and clarify the position of the image t of the part extending laterally from the object on the CCD surface, when S is the part extending laterally from the object, t = S (f / Z) -{Cosγ / (cosγ-θ- (π / 2))} (11) can be obtained.

If necessary, the above-mentioned object and two-dimensional to three-dimensional corresponding to the object are expressed by using the above-mentioned expression (7), expression (8), expression (9), expression (10) or expression (11). It becomes possible to perform a comparative search of reproduced images including a three-dimensional shape. Then, one or a plurality of mathematical expressions, or all mathematical expressions (1) to (11) among the expressions (1) to (11) are added in addition to the attribute data, and a comparative search of the object is performed. It can also be used at the time of.

The three-dimensional image construction unit 15 includes a correlation calculation unit 14
Based on the image data of the various image frames or the various characteristics obtained by the calculation and the search and the attribute data, the reconstructed image of the two-dimensional or three-dimensional shape corresponding to the object is assembled and generated. Then, when the three-dimensional image construction unit 15 assembles and generates a two-dimensional to three-dimensional shape reproduction image corresponding to the object, the three-dimensional image construction unit 15 requires a predetermined area within a region set by a setting unit described later or within a preset region. A reproduced image of a two-dimensional or three-dimensional shape having a size corresponding to the size reference is assembled. When displaying the image of the target object at a corresponding position on the map described later, a process of generating or selecting a required mark or numeral character indicating the existence of the target object is also performed.

The output display unit 16 displays an image of the surrounding structures, rivers, etc. taken by the camera, various signs, equipment such as various signals and power transmission towers, general houses, roads, etc. In addition, a predetermined area set by a setting unit (not shown: a structure basically the same as that of the designating unit may be adopted) is also set for the surrounding map, or the entire region or a preset region. , And the reproduced image including the two-dimensional or three-dimensional shape generated by the three-dimensional image forming unit 15 is also displayed at the corresponding position on the map after being converted into, for example, a predetermined mark or numeral character. The reproduced image of the two-dimensional to three-dimensional shape is a moving image or a group of continuous still images of a two-dimensional to three-dimensional shape in a predetermined area set by the setting unit or a preset area. Sequentially in contradistinction, or sequentially displaying images as picture turning, and also the name and the attribute data associated to generate the reproduced image as necessary to display the required area. The output display unit 16 contrastively divides the display region of the map and the display region of the reproduced image of the object based on the function of a setting unit (not shown) or based on a preset setting as necessary. Alternatively, it is possible to display both images by superimposing them.

On the other hand, the above-mentioned image displayed on the output display section 16 can be displayed on an output display section (not shown) of an image display device such as a base station via a communication line. The designation of the target object and various operations and various processes are performed on the image display device side of the base station,
It is also possible to cause the image display device of this example to perform the processing or the like by an instruction from the image display device side of the base station.

Next, an example of use of the image display device according to this embodiment will be described. First, the camera is activated to cause the output display section 16 to display an image of a video image, a predetermined object in the video image is designated through the designation section, and a peripheral area is displayed in a required area in the image of the output display section 16. A map is displayed as an image, and the map is made to correspond to this map, and the correspondence between the object and the corresponding point on the map is designated or set. When designating the target object, for example, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan−1 (h / y) −θ)} ... (1) or v = f {tan ((π / 2) -θ) -tan ((π / 2) -θ-tan ^-1 (y / h))} ... (2) It is possible to ask. Then, once the correspondence between the map and the object is specified, the image cutout unit 12 or the automatic recognition by the above-mentioned tracking function
Automatically track the few points and continue the correspondence. By tracking the object in the video image with the momentary changes, a trajectory for several points is created. Next, the camera position calculation function or the like is used to calculate the camera position that changes from moment to moment and the distance from the object to the camera by calculation. If the camera position is calculated, the map and the object are calculated. Images can be expressed in the same coordinate system. That is, it is possible to display a map corresponding to an image that changes every moment. Further, the moving speed of the camera is measured from the moving distance of the moving camera by the camera position calculating function or the like, and the image is displayed with a specific number (Km / h). Further, by knowing the position and moving speed of the camera, if the coordinates corresponding to three or more points of the map and the image are known even with one camera, for example, the correlation calculation unit 14, the camera position calculation function, etc. The mutual distance, area, volume, and the like between other arbitrary points can also be calculated by.

Next, as shown in FIG. 2, as an example of image display of the output display section 16 of the image display apparatus, an area 21 of the entire screen 20 is an area for displaying a video image from a camera. The area 23 is an area in which a map around the vehicle (not shown) used in this example including the position corresponding to the place displayed in the area 21 is displayed, and here, the correlation calculation unit 14 calculates The obtained vehicle position is indicated by arrow A in the map. The area 21 also serves as a screen on which an actual video image and a map are superimposed, and the target object 22A calculated by the correlation calculation unit 14 is calculated.
The position of 22B is an asterisk (generated or selected by a partial function of the three-dimensional image construction unit 15) 24 in the region 23.
It is represented by a, 24b and the like. Two objects 22A,
By specifying at least three points by combining the positions of two points such as 22B and the third point (not shown), the positional relationship between the map and the video image is associated.
As a result, that is, the camera position on the map (region 23) and the positions of the objects 22A and 22B and the video image (region 21)
It is possible to associate the positional relationship with the top, and the result is displayed as an image.

Here, for example, the object 22 in the area 21
Around A or 22B in a square area with a pointing device such as a mouse, or this object 2
2A or 22B may be simply clicked with the mouse, but if the object 22A or 22B is designated by these, for example, enlarged images (3) of the objects 22A and 22B surrounded by a rectangle in the area 21 in the areas 25a and 25b (3
Reproduced image generated by the three-dimensional image construction unit 15) 26A, 26
B is displayed. When constructing a reproduced image, that is, with the calculation function of the correlation counter 14, for example, W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {y · cos (γ− (θ + (Π / 2)))} (7) where γ = arcTan (y / z) i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / { y · cos (β + (π / 2) − (θ + (π / 2)))} (8) where β = arcTan (y / z) where Wi = {[(f / sin (θ + (π / 2)) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h · sin (β + (Π / 2))) / [y · cos (β + (π / 2)-(θ + (π / 2)))]} (9) or i′−W ′ = (i−W) / cosδ ... (10) However, δ = arcTan {( −u) / (i−W)} t = S (f / Z) · {cosγ / (cosγ−θ− (π / 2))} (11) The contour of the image of the object reflected on the CCD surface, etc. Is numerically clarified, and it is possible to perform comparative search or generation of a reproduced image including a two-dimensional or three-dimensional shape corresponding to this result.

Based on the above calculation, the objects 22A, 2
The size, shape, inclination, etc. of the 2B image are measured, but the enlarged images 26A, 26B corresponding to the objects 22A, 22B are measured.
The image display of is standardized according to a preset size standard. Further, the positions and orientations of the objects 22A and 22B change as the vehicle moves, that is, the camera moves. However, since the state is tracked by the tracking function described above, even if the video image changes in time series. , An enlarged image (that is, a reproduced image) 26A that continuously responds to the time-series change until the relative positional relationship is outside the image range of the camera whose direction is changed.
The image display of 26B is performed. Incidentally, the objects 22A, 22
When B is surrounded by a rectangle, the rectangle does not necessarily have to be large enough to cover the objects 22A and 22B as described above, that is, the object 22A in the region 21 is represented by a broken line. In addition, a region smaller than the object 22A (22B) may be surrounded by a rectangle, or may be simply a mouse click.

When the objects 22A, 22B are designated by clicking, for example, the objects 22A, 2B are calculated by the correlation calculation unit 14 or by the information contained in the attribute data.
The surface area and volume of 2B can be obtained as described above, or by applying the camera position calculation function, for example, two star marks 24a and 2 in the area 23.
4b or the like can be clicked with a mouse to measure the distance between them. Further, in the area 21, unlike a fixed object such as a general house, which is not a moving object, for example, by clicking a moving object such as another vehicle traveling in an oncoming lane with a mouse or the like, the moving object is also moved. It is also possible to track and display, for example, an enlarged image (that is, a reproduced image) of a moving object in the area 25 along with a time series change on the basis of a predetermined size and display the image.

In the map of the area 23, the object 2
The positions of 2A and 22B can also be obtained from the dip angle of the camera, the height of the camera, and the focal length of the lens when the map is a plane. , The number of pixels of the enlarged image 26 (200 × 15) based on the calculation function of the correlation calculation unit 14.
It is also possible to display 0), the inclination (61.5 °), or the like, or to display the actual size of the object 22A or the like with numbers or the like.

On the other hand, as shown in FIG. 3, when displaying an image while tracking the object 22A as the camera moves,
On the entire screen 20, a state after a certain time has elapsed from the state shown in FIG. 2 is displayed as an image, and the display size of the target object 22A becomes relatively large in the area 21 due to the approach of the vehicle running. There is. For example, when the object 22A is surrounded by a rectangular area again in a proper state, the area (map area) 23 is located at a corresponding position 27 on the map.
A star 24a indicating the object 22A is displayed,
Since the other object 22B is located outside the map, it disappears from the map. With the movement of the camera position, the position of the arrow A indicating the vehicle also changes to the position on the road ahead. For example, the view angle of 22A is changed. For example, enlarged images (that is, reproduced images) 26a, 26b, 26c, 2 in which the line-of-sight angle is continuously changed in four stages.
The image is displayed as 6d. Note that the equations (1) to (11) described above are also used for lock-on display, that is, image display of enlarged images (that is, reproduced images) with different gaze directions.

On the other hand, as shown in FIG. 4, when displaying an image while tracking the object 22B as the camera moves,
In the entire screen 20, for example, on the side of the area 28 corresponding to the above-mentioned area 25 or near the area 25, the required shooting direction of the camera which changes in time series is also indicated by the arrow 28.
The area 23 for displaying a map on the entire screen 20 while sequentially displaying a, 28b, 28c, 28d, 28e.
Is moved to a desired position on the lower side, for example, and the area 2 after movement is moved.
3 on the right side of the upper side, for example, the arrow 28a
A required area 29A for displaying an enlarged image (that is, a reproduced image) having a three-dimensional shape corresponding to the object 22B reflected from the line-of-sight direction is provided, and an enlarged image (that is, a reproduced image) of the three-dimensional shape is provided in this area 29A. Image) 29a is displayed as an image,
Further, on the left side in parallel with the area 29A, the arrow 28 is chronologically arranged.
Areas 29B, 29C, 29 for displaying an enlarged image (that is, a reproduced image) having a three-dimensional shape corresponding to the target object 22B that is projected in the forward direction from the line-of-sight direction of b, 28c, 28d, 28e.
While sequentially setting D and 29E with a predetermined interval, the time series of the target object 22B that is photographed in the directions of the arrows 28b, 28c, 28d, and 28e in the regions 29B, 29C, 29D, and 29E and has different directions. Images (that is, reproduced images) 29b, 29c, which have a three-dimensional shape that changes dynamically
29d and 29e are sequentially displayed as images.

Next, a second embodiment of the present invention will be described with reference to FIGS. FIG. 5 shows a configuration of a main part of the image display device according to the present embodiment. The image display device includes one or a plurality of cameras (not shown), an input unit 31,
A designation unit (not shown), a comparison image generation unit 32, a database 33, a comparison component generation unit 34, a component identification unit 35, and an output unit 36 are electrically connected as illustrated. Note that the configurations of the camera, the input unit 31, the designation unit, the database 33, and the output unit (that is, the output display unit) 36 are basically the same as those described in the first embodiment, so a detailed description will be given. Omit it.

The comparison image generating unit 32 excludes the existence of objects other than the object designated by the designating unit, and features of the entire object or each component (in terms of technical idea in the first embodiment). Image data is generated as a comparison image, and the comparison image includes an image having a two-dimensional or three-dimensional shape. Forward.

The comparative part generator 34 has abundantly stored image data of the characteristics (at least about the same technical concept as in the case of the first embodiment) of each object to be reproduced or each component. Image data of a characteristic of the comparative part is generated by searching the database 33, and the comparative part includes a reproduced image of a two-dimensional or three-dimensional shape. It is transferred to the component specifying unit 35 described later.

The component specifying unit 35 is provided with a search 35A having a search processing function, a recognition 35B having a recognition processing function, a specification 35C having a specific processing function, and a fixed 35D having a fixed function. It is what
That is, in the search 35A, the image data of the characteristics of the comparison image of the object generated by the comparison image generation unit 32 is input, while the images of the characteristics of various comparison components generated by the comparison component generation unit 34 are input. The data is sequentially input, and a comparative search of both image data is performed. The recognition 35B monitors the state of the comparative search of the search 35A, recognizes the result of the comparative search, and, when the image data of both match, the information indicating the match and the image data of both the match or the match. The image data of the characteristic of the matching comparison part is transferred to the specification 35C together with the information indicating. The identification 35C corresponds to the object based on the information indicating the match from the recognition 35B and at least the image data of the matching comparison parts (image data of the characteristics of the comparison parts).
A reproduced part image including a three-dimensional shape or a three-dimensional shape is specified, and information indicating the specification is output to the fixed 35D. The fixing 35D obtains an area and a position to be fixed with respect to the image display of the reproduction component image, and the area and position to be fixed (that is, image display) to the output unit 36 together with the image data (image frame) of the reproduction component image. Is output to the output unit 36, so that the reproduced component image includes a two-dimensional or three-dimensional shape at a corresponding position of the output unit 36, that is, specifically, a three-dimensional CG image (that is, a reproduced image). As a group of moving images or continuous still images, the images are sequentially displayed in contrast to each other or sequentially like a flipping picture.

Further, the parts specifying section 35 is, for example, specified 3
The name and attribute data attached to or generated in the reproduced part image specified by 5C are also transferred to the fixed 35D. In this case, the name and attribute are set at a required position of the output unit 36 (position set by the fixed 35D). Display data as well. When the part specifying unit 35 specifies a search for one or a plurality of objects or a part of them, it also calculates the shape formed by connecting them and the position, distance, area, or volume of the shape, and obtains the attribute data. As database 33
Also save to. In addition, for example, when a new one or a plurality of objects or a new part of them are searched for and obtained, the shape formed by connecting the new objects and the position, distance, area, or volume of the shape are also calculated. It is also possible to execute the process of including it in the requested attribute data.

The component specifying unit 35 also has a separate tracking function (not shown) which is activated including the relationship with the central processing unit and the comparison image generating unit 32. Also, it is possible to track the target object including the change of the camera direction and continue shooting the target object. On the other hand, the component specifying unit 35 also has the function of the search adjusting unit (not shown) shown in the first embodiment to improve the efficiency of comparison search. The correlation with the image frame or the image frame after a certain period of time is specified, and from the next image frame, the part having the highest correlation coefficient is examined in the part close to the previous image frame, and the reference image (that is, the reproduced image) is newly added in sequence. Changing to an image (newly reproduced image) can also contribute to changing the camera direction, which also makes it possible to track an object.

Further, the calculation function (not shown) of the component specifying unit 35 calculates one or a plurality of or representative camera positions, y is the distance from directly under the camera to the object, and v is the inside of the camera. When the angle of the object on the CCD surface, θ is the dip angle of the camera, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan−1 (h / y ) −θ)} (1) or by v = f {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (2) It is possible to ask. Further, in order to calculate and clarify the correspondence between one or a plurality of or representative camera positions, or the video image taken by the camera and the target object on the map, y is the distance from directly below the camera to the target object, v is the length of the object on the CCD surface in the camera,
x is the length in the direction perpendicular to the line segment connecting the object and the camera, u is the lateral length on the CCD surface in the camera, and θ is the dip angle of the camera: v = f · y · sin (Tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) −θ)} (3) or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (4) or v = {(f / sin θ) · y · sin β) / (h · cos (β−θ)) } (5) u = x (f / y)  (cos β / cos (β-θ)) (6) However, β = arcTan (h / y) can be obtained. In addition, the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface.
In order to calculate and clarify the relationship with, f is the focal length of the lens, θ is φ = θ + (π / 2), φ is the connection of the line segment connecting the lens and the object on the camera intersection axis. An angle formed by the perpendicular of the position, ε is β + (π / 2), and y is Z. W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {Y · cos (γ− (θ + (π / 2)))} (7) However, it is possible to obtain by γ = arcTan (y / z). Moreover, in order to calculate and clarify the relationship between the height dimension of the target object and the height dimension i of the image of the target object on the CCD surface, h is the actual height dimension of the position of the camera, and others. , F, θ, etc. are the same as in the case of the above equation (7), i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / { y · cos (β + (π / 2) − (θ + (π / 2)))} (8) However, β = arcTan (y / z) can be obtained. Further, in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above W and i described above are used, and the above Wi = {[(f / sin (Θ + (π / 2)) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h * Sin ((beta) + ((pi) / 2))) / [y * cos ((beta)-(theta))]} ... (9) Furthermore, i'-W '= (iW) / cos (delta) ... (10) However, (delta) = arcTan. It can be obtained by {(t−u) / (i−W)}. Furthermore, in order to calculate and clarify the position of the image t of the part extending laterally from the object on the CCD surface, when S is the part extending laterally from the object, t = S (f / Z) -{Cosγ / (cosγ-θ- (π / 2))} (11) can be obtained.

Also in the present embodiment, if necessary, the above-mentioned object and the object can be obtained by using the above-mentioned formula (7), formula (8), formula (9), formula (10) or formula (11). It is possible to perform a comparative search of a reproduced image including a two-dimensional to three-dimensional shape corresponding to the. Then, of the above formulas (1) to (12), it is used when detecting an object, that is, any one or a plurality of formulas, or all the formulas (1) to (11) as required. In addition to the above-mentioned attribute data, it can also be used at the time of comparative search of objects.

Further, as shown in FIG. 6, a specific configuration for identifying the comparison part corresponding to the object is as follows.
The input image of the object is once stored in the memory Im1 of the comparison image generation unit 32, and then the image conversion unit Im2 converts the input image into image data including a two-dimensional shape, and the component identification unit 3
5, the image data of the component, that is, the comparison component (three-dimensional shape) is temporarily stored in the memory M1 of the comparison component generation unit 34, and then the projection unit Ma
Then, image data is generated by projecting the part from 3D to 2D (including projection in various multi-directions), and recording is performed in a required storage area of each storage area M2 of the database 33. When the image data of the comparative parts are generated and recorded in each storage area M2 of the database 33, when the image data of various kinds of comparative parts are recorded in advance for each classification, for example. Is also included. Thus, when the input image converted into the image data including the two-dimensional shape is transferred to the component identifying device 41, the component identifying device 41 selects a required one of the storage areas M2 ... Of the database 33. The image data of the comparative parts are sequentially fetched from the storage area, a comparison search for matching with the input image is performed, and the image data of the matching comparative parts are identified 35B to 35C in the above-described part specifying unit 35. Will be transferred to.

Next, an example of use of the image display device according to this embodiment will be described. In the case of the present example, a video image including a state in which the left side of a vehicle (not shown) equipped with a camera and traveling in parallel forward at a speed higher than that of the vehicle, for example, the traveling movement of a general vehicle is also included. A case of designating an object from inside will be described.

First, the camera is activated to cause the output section 36 to display an image of a video image, and as shown in FIG. 7A, an area (object 51, tree 52 and general vehicle 53 is included in the video image. It is desired to specify (area) 54 as a portion to be locked-on displayed by enclosing it in a rectangle with a pointing device such as a mouse, and to particularly search, identify, and fix within area 54 to perform observation (enlarged image, that is, reproduction image). When selecting an object as the road sign 51, for example,
The road sign 51 is surrounded by a rectangle as shown by the dotted line in FIG.

When designating an object, for example, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) − θ)} (3) or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (4) or v = {(F / sin θ) · y · sin β) / (h · cos (β−θ))} (5) u = x · (f / y) · (cos β / cos (β−θ))… ( 6) However, β = arcTan (h / y) can be used to detect the contour or the like of the object.

On the other hand, a video image that changes with the passage of time is input to the input unit 31, and at least, for example, the output unit 36 is instantaneously input to the output unit 36 with the passage of time as shown in FIGS. ), The tree 52 in the background of the road sign 51 in the area 54 shifts to the left side in the drawing over time, and the general vehicle 53 shifts to the right side in the drawing over time, only the road sign 51 that is the object. Is tracked along with the tracking change in the camera direction, and is displayed as a lock-on image in a fixed state (the line-of-sight angle is different) in the center of the area 54.

However, in actuality, by the designation of the above-mentioned object (road sign) 51, that is, the comparison image generation unit 32,
As shown in FIG. 7 (D), the tree 52 and the general vehicle 53 are detected as an optical flow having a predetermined vector that is directed in the opposite direction, the images of the tree 52 and the general vehicle 53 are removed, and the image of the road sign 51 is removed. Only the vector is 0, and the image is transferred to the component identifying unit 35 as a comparison image in which at least the contour and the plane are detected. As a result, the component identifying unit 35 searches 35A, recognizes 35B, and identifies 3
After each processing of 5C and fixed 35D, as shown in FIG. 7E, a 3DCG image corresponding to the road sign 51 is displayed as an image in a required area of the output unit 36.

When obtaining the contour of the road sign 51, W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {y · cos (γ− (θ + (π / 2)))} (7) where γ = arcTan (y / z) i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (β + (π / 2)-(θ + (π / 2)))} (8) where β = arcTan (y / z) where W−i = {[(f / sin (θ + (π / 2 )) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h · sin (β + (π / 2))) / [y · cos (β−θ)]} (9) Furthermore, i′−W ′ = (i−W) / cos δ (10) where δ = arcTan {(t−u) / (I-W)} t = S (f / Z) cosγ / (cosγ-θ- (π / 2))} (11) The numerical outline of the image of the object on the CCD surface is clarified by (11), and a two-dimensional or three-dimensional shape is included corresponding to this result. It can be used as one reference when performing comparative search or generation of reproduced images.

Further, by changing the various variables used in the above equations (3) to (12), as shown in FIG. 7F, for example, the fixed 35D can be operated by operating an input device (not shown). It is also possible to change the angle of the road sign 51 through activation and display the state observed from the front as a 3DCG image.

Next, the processing operation of the image display device of the present embodiment will be described with reference to the flowchart shown in FIG. First, the camera is activated to cause the output unit 36 to display a video image as an image, and as shown in FIG. 7A, a specific area (area including the target object 51, the tree 52 and the general vehicle 53) 54 is designated. In addition, the road sign 51 is designated in the area 54 as an object to be searched, specified, fixed and observed (enlarged image, that is, a reproduced image) to be observed. In a suitable state, in step S1, the comparison image generation unit 34 is activated to call the comparison component that is a component of the road sign 51, that is, the image data of the comparison image, and in step S2,
The comparison component, that is, the comparison image, which is transferred from the database 33 in which the search 35A of the component identification unit 35 is activated and the image data of abundant plural types of features is stored and the comparison component generation unit 34 is activated. The image data of the feature related to the corresponding portion of the image data of is searched. After that, step S
3, it is determined whether or not the image data of the comparison image (comparison component) and the image data of the required feature retrieved from the database 33 correspond to each other through the activation of the component identification unit 35.

If YES in step S3, that is, if it is determined that they correspond to each other, in step S4,
The recognition 35B is activated and recognizes that the image data of the feature this time is image data corresponding to the target road sign 51, and in step S5, the specification 35C is activated and the content of the image data of the feature this time is road. The component corresponding to the marker 51 is specified, and then, in step S6, it is determined whether or not the processing of the predetermined portion of the comparison image is completed. If NO, that is, if it is determined not to end, the process returns to step S1, but if YES, that is, to end, in step S7, the fixed value 35 is set.
Through the activation of D, a predetermined area of the output unit 36 is displayed in FIG.
As shown in, the present recognition result, that is, the image of the 3DCG (three-dimensional reproduction image) corresponding to the road sign 51 is displayed as an image, and the present flow ends.

However, if it is determined NO in step S3, that is, if the two image data do not correspond to each other, the process proceeds to step S8, and a predetermined part to be retrieved from the database 33 by activating the comparative part generating part 34 or the part specifying part 35. It is determined whether or not the comparison part is finished. If NO is determined here, that is, the process is not finished, the step S
Returning to step 1, if YES, that is, if it is determined that the process has ended, then the procedure advances to step S9, and it is determined whether or not all processing of each component of the road sign 51, which is the object, has been completed. If the result of the determination is NO, that is, if it is determined that all the processes have not been completed, the process proceeds to step S10, the next part is designated, and then the process returns to step S1, but if YES, that is, all the processes are completed. If determined, step S11
In, the flow ends after it is determined that there is no corresponding component.

The storage medium such as a flexible disk, a CD-ROM or the like according to the present invention, or a database such as a cylinder or the like is stored in the above-mentioned arithmetic expressions (1) to (1).
1) and other arithmetic algorithms (including the arithmetic algorithm used in the first embodiment), and the above-described programs from step S1 to step S11 that embody the image processing method of this example are recorded. It is a thing.

Next, a third embodiment of the present invention will be described with reference to FIGS. FIG. 9 shows the configuration of the main part of the image display device according to the present embodiment. The image display device includes one or a plurality of cameras (not shown), a video input device 61, a video recording / reproducing output device 62, and a target. Object designation device 6
3, a video map display device 64, an attribute recording / reproducing device 65, an object search designation device 66, and an output display unit (not shown) including an image lock-on display unit 67 (also installed in the aircraft and the base station) As described above, they are electrically connected. In the present embodiment, the case where the present invention is applied to, for example, a helicopter or an airplane as an aircraft will be described as an example, and detailed description of the same parts as those in the first embodiment will be omitted.

The one or more cameras are arranged, for example, at a front side position near the cockpit on the bottom side of the fuselage of the aircraft in a required arrangement relationship, for example, for a required operation in the aircraft or a required base station. Depending on the operation, for example, the height and shooting direction are electrically driven to at least the horizontal direction, or an arbitrary position including the direction from the ground to the sea surface from the periphery of the direction at a predetermined angle above the horizontal direction, and the camera direction is arbitrary. It is possible to change to.

The video input device 61 converts a video image captured by the camera into an electrical signal and sequentially generates a plurality of image data in time series. The image data includes color images such as brightness, density and contrast. Video signal is included.

The video recording / reproducing device 62 temporarily records the video signal in the video memory and scans it to output the video image to the output display device (not shown). The video recording / reproducing device 62 is provided with a mask processing unit, a window display unit, and the like.

The object designating device 63 has the same structure as the designating unit described in the first embodiment, and is also provided in the image display device of the base station, for example, and is based on any operation designation. It is possible to arbitrarily specify multiple objects (may be flying objects in the air or hulls on the sea) from the video image, and once the objects are specified, it is necessary to track the objects. It also contributes to the generation of data.

The video map display device 64 displays a map corresponding to the line-of-sight direction of the camera on the output display section when the object specifying device 63 specifies the object, and also calculates the coordinate system (not shown). () Is also provided, and several points in the video image are made to correspond to the map, and the coordinate system on the map is obtained based on the result of tracking these with time-series changes. Alternatively, it is possible to perform processing such as converting into a required coordinate system having a line-of-sight angle and displaying an image, and processing such as enlarged display by zooming a map.

The attribute recording / reproducing apparatus 65 stores nominal data relating to the classification of the characteristics of the object, and other information such as manufacturing date, age, owner, manager, past inspection history, repair history, and revision / repair. Schedules, exchange schedules, etc., or data relating to combination generation of image data of each feature in the image storage unit 13, assembly formation, or connection relation (that is, structure of tissue), or shape formed by connecting objects, and The attribute data including the position, distance, area, or volume of the shape is recorded, and when the object is designated, the name and the attribute data regarding the object are requested in response to a request from the object search designating device 66. Is read out and transferred to the object search and designation device 66, while the content of the attribute data is displayed at a required position in the display area of the output display unit or a lock-on display, if necessary. And displays the required position in the 67.

The object retrieval designating device 66 determines the characteristics of each image frame of the object or the characteristics of each component,
Correlation (search adjustment unit) corresponding to the target from a database (not shown: may be the same as the image storage unit 13) that abundantly stores image data of a plurality of types of image frames or features relating to the plurality of types of targets. Expansion of comparison target,
If necessary, the image data of the image frame or the characteristic image that has been subjected to the pattern matching is searched for, and the image data, that is, the two-dimensional image Reproduced images including a three-dimensional shape are displayed on the lock-on display unit 67 as a moving image or a group of continuous still images in a sequential manner in contrast or in a time-sequential manner like a flipping picture. Accordingly, the image data of the new image frame relating to the new object or the image data of the new feature reflecting the feature of the new component is also stored in the database. Further, the target object search designating device 66 also includes a calculation unit (not shown) for calculating various correlations, and in addition to the above-described processing, for example, searches for one or more new target objects or a part thereof. When it is obtained, it is also possible to perform a process of calculating and obtaining the shape newly formed by connecting each and the position, distance, area, or volume of the shape, including this in the corresponding attribute data, and storing it in the database.

Further, the object retrieval designating device 66 uses, for example, a triangulation method or a binocular or trinocular stereo image method by a separately-configured camera position operation function (not shown) activated by a central processing unit or the like. It is possible to calculate and obtain the camera position, the camera direction, the camera moving speed, etc., for example, the vertical position of the corresponding point on the CCD surface of each point on the ground that appears on the actual image taken by the camera. ,
Not only can they be associated with each other in the form of trigonometric functions, but they can also be expressed in the lateral direction by the proportional relationship with the distance to the camera, which allows the camera to calculate where the height is and where it is on the map. It is also possible to obtain by. In addition, the correlation with the next image frame or the image frame after a fixed time is calculated, and the relation between the portion having the highest correlation coefficient from the next image frame and the portion close to the previous image frame is checked, and the reference image is sequentially displayed. (That is, reproduced image)
It is also possible to contribute to the control of changing the camera direction by changing to a new image (new reproduced image), which allows the object to be accurately tracked.

Further, the object retrieval designating device 66, similar to the correlation calculation unit 14 described in the first embodiment, performs the above-mentioned pattern matching on at least image data of one feature of the object on the ground. If it is obtained, one or a plurality of data are immediately generated based on the attribute data associated with or generated by the image data of the feature (that is, the data of the combination, the assembly, or the connection relation of the image data of all the features of the object). It is possible to construct an image frame of a reproduced image including a two-dimensional or three-dimensional shape corresponding to an object, and a spatial distribution and a temporal distribution of one or a plurality of objects such as the ground using a plurality of cameras. And generating image data of a reproduced image including a three-dimensional shape corresponding to one or a plurality of objects such as the ground based on at least parallax information and time difference information. It is also possible to specify a plurality of points among the objects on the ground, etc., and obtain the positional relationship between the known points and the camera by a predetermined calculation algorithm, thereby It is also possible to find mutual points and distances.

Further, the object retrieval designating device 66, in particular,
Coordinate system calculation means (not shown) is also provided, and several points in the video image are made to correspond to the map, and the coordinate system on the map is obtained by tracking this along with changes in time series. In addition to calculating the camera position from the coordinates (for example, the camera position that has the same positional relationship as the aerial image can be obtained by, for example, the partial differential asymptotic method), the camera direction, and the camera speed, the video image and the map have the same coordinate system, or It is possible to convert the image into a required coordinate system having a required line-of-sight angle and display the image. Therefore, for example, the output display unit including the lock-on display unit 67 does not always change even when the attitude of the aircraft changes, and the image of the reproduced image including the mark and the three-dimensional shape corresponding to the object such as the map and the ground is constantly changed. It is possible to continue the display.

Further, an arithmetic function (not shown) of the object retrieval designating device 66 calculates one or a plurality of or representative camera positions, y is the distance from directly under the camera to the object, and v is in the camera. The length of the object on the CCD surface at
When θ is the dip angle of the camera, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan−1 (h / y) −θ)} (1 ) Or v = f {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (2) where β = arcTan (h / y ). Further, in order to calculate and clarify the correspondence between one or a plurality of or representative camera positions, or the video image taken by the camera and the target object on the map, y is the distance from directly below the camera to the target object, v is the length of the object on the film surface in the camera, x is the length in the direction perpendicular to the line segment connecting the object and the camera,
Where u is the lateral length of the CCD surface in the camera and θ is the dip angle of the camera, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (Tan ⁻¹ (h / y) −θ)} (3) or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (4) or v = {(f / sin θ) · y · sin β) / (h · cos (β−θ))} (5) u = x · (f / y) · (cos β / Cos (β−θ)) (6) However, β = arcTan (h / y) can be obtained. In addition, the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface.
In order to calculate and clarify the relationship with, f is the focal length of the lens, θ is φ = θ + (π / 2), φ is the connection of the line segment connecting the lens and the object on the camera intersection axis. An angle formed by the perpendicular of the position, ε is β + (π / 2), and y is Z. W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {y · cos (γ− (θ + (π / 2)))} (7) However, γ = arcTan (y / z) can be obtained. Further, in order to calculate and clarify the relationship between the height dimension of the object or the like and the height dimension i of the image of the object imaged on the CCD surface, h is the height dimension of the camera position, and f, When θ and the like are the same as those in the above formula (7), i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (Β + (π / 2)-(θ + (π / 2)))} (8) However, β = arcTan (y / z) can be obtained. Further, in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above W and i described above are used, and the above Wi = {[(f / sin (Θ + (π / 2)) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h * Sin ((beta) + ((pi) / 2))) / [y * cos ((beta)-(theta))]} ... (9) Furthermore, i'-W '= (iW) / cos (delta) ... (10) However, (delta) = arcTan. It can be obtained by {(t−u) / (i−W)}. Furthermore, in order to calculate and clarify the position of the image t of the part extending laterally from the object on the CCD surface, when S is the part extending laterally from the object, t = S (f / Z) -{Cosγ / (cosγ-θ- (π / 2))} (11) can be obtained.

Also in this embodiment, if necessary, the above-mentioned object and the object can be obtained by using the above-mentioned formula (7), formula (8), formula (9), formula (10) or formula (11). It is also possible to perform a comparative search of reproduced images including two-dimensional to three-dimensional shapes corresponding to. Then, it is used in the detection of the object among the above equations (1) to (11),
That is, any one or a plurality of mathematical expressions or all mathematical expressions (1) to (11) may be added to the attribute data as needed, and the attribute data may be compared to perform a comparative search of the object.

Next, an example of use of the image display device according to this embodiment will be described. First, aerial image and map 71
Regarding the correspondence of No. 1, the front view whole image of the map 71 is, for example, on the video map display device 64, as shown in FIG. Image is displayed) and the image is displayed without aligning the object. Therefore, the optical axis direction of the camera direction is next located at the center of the screen shown in FIG. 10A, and the correspondence with the map 71 is first made by the video recording / reproducing device 62 in the real screen (video image; That is, the correspondence between the map 71 and the video image having the actual line-of-sight angle taken by the camera corresponding to the attitude of the aircraft is calculated by the calculation of the coordinate system. That is, the map 71 on the optical axis by the object designation device 63
Each specified point above (specific points 72, 73, 74, 75,
76) and points on the image (specific points 72, 73, 74, 7
5, 76) should be clarified.

To specify each point on the CCD plane and the map 71, for example, v = fy · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) −θ)} (3) or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h) )} (4) or v = {(f / sin θ) · y · sin β) / (h · cos (β−θ))} (5) u = x · (f / y) · (cos β / cos (Β-θ)) (6) However, it is possible to obtain by β = arcTan (h / y). As a result, the state in which the video image is displayed as an image is shown in FIG. 10B, and the deformable map 71 having the required line-of-sight angle is displayed as the image corresponding to the line-of-sight angle related to the camera projection. As a result, the designated points 72 on the image and the designated points 72 on the map 71 are first superposed (positioned).

Subsequently, in the object designation device 63 and the video map display device 64, as shown in FIG.
Is calculated by correcting the direction of the optical axis in consideration of the attitude and altitude of the aircraft (considering the line-of-sight angle from the front) and tilting the map 71 mainly so that the lower part is pulled toward you, that is, a video map. On the display device 64, the optical center is fixed to the line-of-sight direction from a predetermined direction at another position (including the line-of-sight line-of-sight angle in front view), and the vertical direction, that is, the contour of the map 71 is corrected by a necessary calculation, and the distance direction That is, the map 71 is tilted according to the setting of the line-of-sight angle from the front and the altitude, and the object designation device 63
And other one or more objects (or specific points) 72, 73, 74, 75, 7 on the vertical line, that is, with the line-of-sight angle from the front.
Find 6 and align. That is, the designated object 72
The elevation angle alignment of the map 71 including the alignments of ~ 76 is performed. Further, in the image map display device 64, the rotation of the coordinate system is obtained by the calculation means of the coordinate system by aligning each of the target objects 72 ... At one or more points other than the vertical line. The image and the map 71 can be superimposed. That is, as shown in FIG. 10 (C), the lateral rotation alignment of the camera is performed.

It should be noted that each object 72 is rotated along with the rotation of the map 71.
In order to perform the alignment of ..., W = {(f / sin (θ + (π / 2)) · Z · sinγ)} / {y · cos (γ− (θ + (π / 2)))} (7) where γ = arcTan (y / z) i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (β + (π / 2) ) − (Θ + (π / 2)))} (8) where β = arcTan (y / z) where W−i = {[(f / sin (θ + (π / 2)) · Z · sinγ) ] / [Y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h · sin (β + (π / 2))) / [ y · cos (β−θ)] ... (9) Furthermore, i′−W ′ = (i−W) / cos δ ... (10) where δ = arctan {(t−u) / (i−W)} t = S (f / Z) · {cosγ / (cosγ-θ- ( / 2))} (11) can correspond to that by using the application of.

As a result, as shown in FIG. 10D, in the object designating device 63 and the image map display device 64, the map 71 is tilted from the camera to the actual camera direction or the pseudo camera direction of the map 71. Map 71 schematically showing the relationship
It is possible to display an image in which the relationship between the camera and the camera position is set. In this case, the camera and each object 7 of the map 71 are displayed.
It is also possible to display the line of sight of multiple cameras connecting 2 ..., etc. Thus, the cameras move over time,
Also in this case, by applying the equations (3) to (11), the corresponding points of the respective objects 72, etc. on the map 71 and the points 72, etc. on the image are displayed. Basically, FIG. 10D clarifies the correspondence with FIG. 10C.

That is, as described above, the positions of the respective objects 72 ... Are aligned in the first one manual image frame and the second and subsequent image frames, and in the subsequent image frames, at least three corresponding points are set. Of the object 72 ... of the image map display device 64 or the like (not shown:
The position of the map 71 may be automatically adjusted according to the formulas (1) to (11) above. For example, the coordinate system of the map 71 shown in FIG. As shown in FIG. 11B, the optical center of the camera is aligned with the center of the map 71, and the map 71 is erected vertically. At this time, the object designation device 63 and the video map control device 64 are used to display the map 71 on the map 71. .. and the point of the optical center of the camera on the map 71 are clarified, and the lock-on display section 67 lock-on-displays the above-described image. At this time, the names of the target objects 72 ... And the attribute data can also be displayed on a part of the lock-on display section 67, for example, by starting the attribute recording / reproducing device 65.

Then, as shown in FIG. 11C, the image map display device 64 tilts the map 71 in such a manner that the lower part of the map 71 is drawn to the front in consideration of the altitude of the camera, and the vertical line of the map 71 is displayed. While maintaining the correspondence, the map is tilted around the horizontal axis of the screen and the positions of the objects 72, ... Are aligned on the vertical axis. Then, in the video map display device 64, as shown in FIG. 11D, the coordinate system of the map 71 is rotated about the vertical axis,
Further, the correspondences of the respective counterparts 72, ... At a point distant from the vertical line are taken, and the horizontal plane of the map 71 is matched. By going through these processing steps, it is possible to manually align the first map 71 with the video image.

Further, at this time, since the correspondence between the position on the map 71 and the position of each target object 72, etc. on the image is obtained, the camera position is known by the above calculation, and the camera Even when the map 71 moves, it is possible to obtain the amount of movement of the map 71 by the calculation function, and thus it is possible to recognize the amount of movement and the direction of movement of the map 71.
However, it is also possible to automatically perform the correspondence between the first image frame and the second image frame, calculate the camera position, and automatically obtain the image data for the second and subsequent images. At this time, it is necessary to clarify the correspondence between the video image captured by the camera and the objects 72, ... On the map 71.

In this case, on the basis of the relationship shown in FIG. 12, for example, the correspondence relationship between one or a plurality of or representative camera positions or video images taken by the cameras and objects on the map is clarified. Where y is the distance from directly below the camera to the object, v is the length of the object on the CCD surface in the camera, x is the length in the direction perpendicular to the line segment connecting the object 72 ... When θ is a dip angle of the camera in the lateral direction of the CCD surface in the camera, the above-mentioned formulas (1) to (6) or (7) to (12) are used.
As described above, it is also possible to calculate and obtain by appropriately using.

In FIG. 12, a point R is a point where the tip of the optical axis of the camera center corresponds to the map plane, and m is a point R from the optical center O.
, R is the line segment from the optical center O to the point R (Y
Axis), point P is a certain point (object) defined on the line segment r, point H
Is a point directly under the camera, h is a distance (line segment) from the optical center O to the point H, y is a line segment from the point H to the point P, b is a line segment from the points P and R, and B is a CCD surface. The point corresponding to point H above,
e is the distance (line segment) from the optical center O to the point B, v is CC
The image of the line segment corresponding to the line segment y on the D surface, d the image of the line segment corresponding to the line segment b, c the image of the line segment corresponding to the line segment r on the CCD surface, and F the image of the line segment corresponding to the line segment r. The center, f is the distance (line segment) from the CCD surface to the optical center O.

In summary, when the camera is aerial, the actual camera position can be known by recognizing known objects at three or more points on the ground. It is possible to recognize, identify, and track what kind of airplane the flying object is flying by enclosing it with a frame, and this is the same in the first and second embodiments.

Next, an image display device according to the fourth embodiment of the present invention will be schematically described. First, the first
Through adopting any one of the third image display method and the image display device and joining the Internet,
The URL is specified from the self side, that is, from the vehicle or aircraft side on which the camera and the image display device of this example are installed, or from the base station or the like side on which the image display device of this example is installed.
Lock-on image (name and attribute if necessary) of video image and map (including display of target object) generated by this image processing apparatus via L or ISDN, or line and provider such as satellite line or terrestrial digital broadcast W, for example, in real time on the output display unit (display) of the desired terminal station of image data (including data, camera position, camera direction, camera moving speed) (for example, a three-dimensional reproduction image).
eb distribution is possible. Conversely, by specifying the URL of the image display device (vehicle-mounted type, aircraft-mounted type, or base station-installed type) of this example from the client, the video image and the target generated by the image display device of this example are also specified. If image data of a map including objects (same as above) is delivered to the display of the terminal station on the client side, it is possible to display an image in real time.

In the recording medium according to the present invention, the algorithm of the concrete contents of the image display method described in the first to third and fourth embodiments, and the various calculation algorithms and expressions described above. It goes without saying that the formula (11) and the like from (1) may be recorded.

Next, with reference to FIGS. 13 to 19, the relationship between the first object to be measured in the first to fourth embodiments and the image of the object on the CCD surface will be described. In this example, description will be made assuming that a camera (not shown) is aimed at the road surface PR which is an object. First, as shown in FIG.
For example, the CCD surface cm of the camera is tilted downward at a predetermined height h by a predetermined angle, and the tip of the optical axis lp at the center of the camera is aligned with the point R on the road PR. In this case, on the road PR, the line segment HR from the point H on the road PR directly below the camera to the point R coincides with the Y axis. Further, on the road PR, a certain point P on the line segment HR is designated from the camera, and a point Q is designated at a position separated from the point P by a predetermined distance in the left-right direction to the line segment HR. In this case, the line segment PQ is parallel to the x axis. On the other hand, on the road PR, a point E is designated at a position separated by a predetermined distance from the point R in a direction to the left of the line segment HR in the sense of being used for calculation. Then, a line segment from the CCD surface cm to the optical center O to the point R is defined as a line segment l, and a line segment from the CCD surface cm to the optical center O to the point P is defined as a line segment n. In addition, line segment l and line segment HR
Is θ, the angle between the line segment m and the line segment HR is β, and the angle between the line segment 1 and the line segment n is β-θ. On the other hand, a line segment HP, that is, a line segment image u (indicated by reference numeral 81 in the figure) corresponding to the line segment Y appears on the CCD surface cm. Also, as shown in FIG. 14, the CCD surface c
On m, the image v of the line segment (corresponding to the line segment HR) is projected from the position corresponding to the point H on the line segment h toward the optical axis lp at a predetermined angle.

Further, as shown in FIG. 15, there is a case where it is tilted around the Y axis, a case where it is tilted about an axis parallel to the X axis passing through the point R, and a case where it is tilted synthetically by the required amount around both axes. But,
In that case, on the CCD surface cm, the image v of the line segment Y, and
The length of the image u of the line segment PQ (line segment x) changes and appears. However, even in that case, the intersection R of the optical axis and the ground surface
By rotating the image around the optical axis while keeping the position (the position of the image and the map) aligned, the coordinate conversion can be performed as described with reference to FIGS. In addition, FIG.
As shown in FIG. 16, when grid-like coordinates are virtually set on the road PR and the line segment y and the line segment x are photographed from directly above, the CCD plane cm is shown in FIG. As shown, the image u of the line segment y and the line segment x appear on the center side of one plane without any change in the line-of-sight direction from the front, and as shown in FIG. When the front of the road cm is photographed by giving the height h and the like, the virtual grid-like coordinates on the road cm change so as to converge toward the front of the road cm, and the line segment image u also changes. Due to the perspective, the image u is similar to the short line segment corresponding to the width of the road cm. In addition, in FIGS. 16A and 16B, the coordinates (0,
0) indicates the position of the CCD surface cm of the camera.

On the other hand, as shown in FIG. 17, a line segment 11 pointing upward at a predetermined angle along the Y axis from the optical center O is assumed, and the height of the optical center O coincides vertically upward from the point P. The line segment h2 and the line segment h3 that reaches the line segment ll vertically above the line segment h2 are set, and the line segment h3 is set.
The point S is designated at a position (directly above the point X) separated by a predetermined distance from the point Z reaching the apex of the line, ie, the point Z, in the horizontal right direction (parallel to the X axis). If CCD
On the surface cm, an image of the line segment corresponding to the line segment h2 and an image W of the line segment corresponding to the line segment h3 are projected on the extension of the image v of the line segment, and from the tip of the image W of the line segment. An image t of a line segment parallel to the image u of the line segment is captured. The angle γ formed by the line segments 11 and h3, the angle φ formed by the line segments 1 and h2, and the angle ε formed by the line segment n and the line segment extending vertically downward from the line segment h2 are
Expression (5), Expression (6), Expression (7), Expression (8) already described
Is used in.

As shown in FIG. 18, when a line segment connecting the points S and X is set on the road PR, the tip of the image u of the line segment and the image of the line segment are displayed on the CCD surface cm. A line segment image i ′ and a line segment image W ′ indicated by a dotted line are linearly imaged while connecting to the tip of t. As shown in FIG. 19 (A),
The lengths of the line segment image u and the line segment image t change, and
Needless to say, as shown in (B), the image u of the line segment, the image t of the line segment, the image i of the line segment, and the direction W of the image W of the line segment are changed.

As described above, the image reflected on the CCD surface cm is
By appropriately using the above equations (1) to (11), the position can be calculated and obtained, and by using this, a comparative search is performed and a reproduced image including a two-dimensional or three-dimensional shape corresponding to the object is obtained. It is also possible to specify or generate.

[0084]

Since the present invention is configured as described above, even if the video image changes from moment to moment, a peripheral map is displayed in the image as an image and the target 22 designated arbitrarily or specially. A mark or the like corresponding to ... etc. is always matched with the corresponding point on the map, and a moving image of a three-dimensional shape reproduced image or a continuous still image corresponding to the object 22 etc. that changes in time series with the movement of the camera is displayed. Since it was displayed sequentially as a group in contrast, for example, during normal times, emergencies, disasters, etc., the situation search of roads, buildings, rivers, etc. And the display area of the map on the image accompanied by the image display of the mark or the reproduced image corresponding to the target object 22 ...
It is also possible to separately display or overlap the display area for lock-on display of the three-dimensional shape reproduction image obtained by tracking 2 etc., so it is extremely easy to see the local situation In addition, the image can be displayed in an easy-to-understand manner, and even if the camera direction and camera position for in-vehicle use or aerial photography changes, the situation of the change is calculated by the required calculation, and the image is created using the required attribute data such as marks and numbers. If the information can be displayed, it is possible to easily understand the local situation including the inspection direction in more detail, and to promptly judge the local safety measures and response to damage situation, etc., and provide appropriate support. Close.

That is, in the present invention, this is a change in the relative angle and direction between the camera and the objects 22 ..., By designating one or more objects 22, 51, 72 ,.
And the characteristic of each image frame such as the object 22 ... Is detected while tracking the object 22 ... Etc. including the change of the distance, and an image of a characteristic having a correspondence relationship with the object 22 ... Search for data sequentially and search for target 22
This is because a reproduced image including a two-dimensional or three-dimensional shape corresponding to ... Is constructed, and the reproduced image is displayed by matching with corresponding points on the maps 23 to 71 in the image.
This makes it possible to quickly compare and check in detail the situation during normal times and the situation during a disaster, for example. It can be obtained and has a high social utility value.

Further, in the image data of various characteristics stored in the databases 13 to 33 and the like, in advance, each image frame of various reproduction objects, or the image data of characteristics corresponding to various constituent parts has the name and each attribute data. Since they are associated with each other, it is possible to construct a reproduced image including two-dimensional or three-dimensional shapes corresponding to the objects 22, 51, 72, ... In a very short time based on the recognition of the respective attribute data.
It has the advantage of promptly responding locally.

The image data of various characteristics stored in the databases 13 to 33 and the like include elements such as various reproduction objects, various straight lines, various curves, various angles, various edges, various planes, various curved surfaces, or the like in advance. In particular, since each attribute data including the combination relation between the image data of each feature is associated with each other, by referring to each attribute data,
It is possible to obtain reconstructed images of three-dimensional shapes such as an extremely large number of objects 22, 51, 72, ...

Further, even if the reproduced image including the two-dimensional to three-dimensional shapes corresponding to the objects 22, 51, 72, ...
Since the above-mentioned image display of a reproduced image of a two-dimensional or three-dimensional shape from various viewpoints such as 2 ...

Further, image data having a spatial distribution and a temporal distribution of the objects 22, 51, 72, etc. is obtained, and at least the three-dimensional shape corresponding to the objects 22, etc. is reproduced by the parallax information and the time difference information. Since image data of an image or the like is generated, and a three-dimensional model of the reproduced image is generated and the above-described image display can be performed, a very easy-to-understand image can be obtained.

In addition, 16 to 36 or 6 on the image display
7 or the like, or another base station or the like via a communication line, on the image display, objects 22, 51, 72 designated as necessary are displayed.
The surrounding maps 23 to 71 and the like are displayed as images, and the target object 22 and the like is reproduced at a corresponding position in the maps 23 to 71 and the like with a required mark or a two-dimensional to three-dimensional shape reproduction image. The image is displayed on the screen, and the map is 23 to 71.
Since it is possible to compare or overlap the display area such as the display area with the display area 67 of the reproduced image, or the like, even a first-time user can easily and properly use the display area. , It is very easy to use and does not confuse the user.

Further, among the objects 22, 51, 72, etc., a plurality of points are designated and the positional relationship between the known points and the camera position is calculated, and this arithmetic algorithm is used to measure other similar positional relationships. Therefore, it is possible to easily calculate the mutual points and distances of arbitrary objects 22 ... And the like, and easily calculate and measure the area or volume of the entire surface of the objects 22 ... This will enable the observer to recognize the local situation in more detail.

Further, with respect to the moving picture or a group of continuous still pictures which are sequentially displayed as a group, the camera position, the camera direction, and the camera moving speed are obtained by a required calculation, and the relevant matters are marked with a required mark. Since the images can be displayed with numeric characters and the like, it is possible to extremely efficiently assist the observer in moving to grasp the local situation.

Further, a region where the moving image or a group of continuous still images is sequentially displayed in contrast, or a region where the name and attribute data of the reproduced image corresponding to the objects 22, 51, 72 ... Alternatively, since the area for displaying the camera position, camera direction, and camera moving speed can be set arbitrarily according to the required operation or designation, it is possible to make settings that are extremely easy for the observer to understand and also to support the workability at the site. It has the advantage of being very easy to use.

Further, when the objects 22, 51, 72, etc. are designated, even if the area range wider or narrower than the objects 22, etc. is selected and designated, the objects 22 ,.
Since it is possible to extract and recognize only such as, etc., it has very good operability even when the vehicle or aircraft used is shaking, or even if the same selection and designation is made on the base station side, detailed operation is not required, Has the advantage of being easy to use.

Further, arbitrary objects 22, 51, 72 ...
Etc., or a shape formed by connecting a part of them, and an image display of attribute data including the position, distance, area, or volume of the shape, and a reproduction image including these two-dimensional to three-dimensional shapes as a whole. Since the image can be displayed, the situation can be grasped over a wide area, and there are advantages that it is convenient and easy to use.

Further, CG images of three-dimensional objects such as the objects 22, 51, 72, ... , Or because the image can be displayed in the required area, the object 22
… Easy to see.

On the other hand, according to the image display device of the present invention, in particular, an arbitrary object 22, 51, 72 ... Is designated from the image in the output display section 16 by the designation section (not shown), and the image is cut off. The unit 12 cuts out the existence other than the designated target objects 22, 51, 72, etc., and extracts only the target object 22 or the like, and the plurality of image frames or the features stored in the image storage unit 33 by the correlation calculation unit 14. Correlation with the image data of
The three-dimensional image construction unit 15 generates a two-dimensional to three-dimensional shape reproduction image corresponding to the target object 22 ... And the output display unit 1
In addition to displaying the video image and maps 23 to 71 and the like on the image 6, it is possible to sequentially output the moving image of the reproduced image or a group of continuous still images in a region set by a setting unit (not shown) in contrast to each other. With this configuration, it is possible to unify the reproduced image in the required display area in the output display unit 16 according to the required size standard and sequentially display the images as described above. Image name,
Also, the attribute data can be displayed at the required position in the same or other required display area once set, and the most usable setting can be found as the number of times of use increases. It is good and has the advantage that it is easily familiar to the user.

Further, according to the image display device of the present invention, in particular, the object 22, 51,
72, etc. are specified, the features of the objects 22, 51, etc. are generated as comparison images by the comparison image generation unit 32, and the comparison component search unit 34 searches the database 33 for comparison to determine the features of the comparison components. Image data of the
5, the comparative image data of the objects 22, 51, ...
2, 51, ..., Reproduction part images including a two-dimensional or three-dimensional shape are specified, and the names of the reproduction part images
In addition to searching and generating attribute data, obtaining a position to be fixed to the output unit 36 and displaying an image, and displaying the name of the remanufactured component image and the attribute data as necessary, the processing of each unit is independent. It becomes possible to construct a highly accurate reproduced image and display the image in addition to
Similar to the above, the image is easy to see and understand, and the name and attribute data of the reproduced image can be confirmed, so that it is possible to improve reliability when using the site.

Further, according to the image display device of the present invention, in particular, the image recording / reproducing output device 62 converts the image of the external world into an electric signal for recording and reproduction, and the image input device 61 also converts the external image into an electric signal. It is converted and input, and the image map designating device 64 superimposes it on a map on an output display unit (not shown), or simultaneously displays an image in an area relatively separated from the map. The object in the image at 63,2,5
1, 72, etc. are designated, and the target search designation device 66 performs comparative search for image frames of the corresponding features or image data of the features from a database (not shown) to search for the target object 2
A reproduced image including a two-dimensional or three-dimensional shape corresponding to 2, 51, etc. is generated and displayed on the lock-on display section 67, and the attribute recording / reproducing device 65 retrieves or generates a name of the reproduced image and attribute data. Since it is configured to record and reproduce, the lock-on display unit 67 is fixedly and exclusively handled, and no movement to search for a reproduced image is required during site observation. It is possible to direct the line of sight to the lock-on display position of the playback image based on the habit or without unknowingly getting rid of it, which is convenient in this respect, and a situation that hinders other tasks cannot occur. , It has the advantage of convenience.

Further, according to the method and apparatus of the present invention, the search adjusting unit (not shown) and the size and the line-of-sight angle of the objects 22, 51, 72, ...
By matching the size and line-of-sight angle with the image data of various reproduced objects or various features in the same, it is possible to very efficiently reproduce images including two-dimensional or three-dimensional shapes corresponding to the object 22 ... It can be comparatively searched and generated, and the processing speed can be improved.

Further, according to the method and apparatus of the present invention, several points 72, 73, 74, 75, 76, etc. in the video image are made to correspond to the map 71, and these are tracked along with changes in time series. The coordinates on the map 71 are read with and the camera position, camera direction, and camera moving speed are calculated from the coordinates of several points, and the video image and the map 71 are in the same coordinate system.
Alternatively, since it can be converted into a required coordinate system having a required line-of-sight angle, it corresponds to a mark or line-of-sight angle indicating the objects 22, 51, 72 ... At corresponding positions on the map 71 having any of the coordinate systems. A reproduced image including a two-dimensional or three-dimensional shape can be displayed as described above. Therefore, the reproduced image is always observed on the map 71 having a constant coordinate system even if the attitude of the aircraft or the flight direction changes. It is possible to
It is possible to easily recognize the situation at the site, that is, on the ground and the like, and to provide a very functionally excellent one.

Further, according to the method and apparatus of the present invention, one or a plurality of or representative camera positions and camera directions are calculated and obtained in relation to the outside world, and therefore, the map 23 on the image,
It is possible to display a mark indicating a camera position and a camera direction, a numeral character, or the like at a corresponding position on 71 or the like, which is also useful for selecting a travel route or a flight route.

Further, according to the method and apparatus of the present invention, the above-mentioned equation (1) or equation (2) is used to compute the above-mentioned one or a plurality of or representative camera positions. It has a short and accurate camera position and is highly reliable for local use.

Further, according to the method and apparatus of the present invention, the correspondence relationship between the one or more or a representative camera position or the video image photographed by the camera and the object on the map is calculated and clarified. However, since the calculation is performed using the above formulas (3) to (4), or the formulas (5) and (6), the processing time is short and the accurate camera position or the map is obtained as described above. It is possible to obtain an accurate correspondence relationship for each object, and it is highly reliable locally.

Further, according to the method and apparatus of the present invention, the relationship between the height apex Z of the object or the like and the height apex W of the image of the object on the CCD surface cm is calculated and clarified. In order to calculate and clarify the relationship between the height dimension of the target object and the height dimension i of the image of the target object on the CCD surface cm, the above equation (7) is used. Using equation (8), and further using equation (9) or equation (10) above to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD plane cm, , Since the above formula (11) is used to calculate and clarify the position of the image t of the portion extending laterally from the object on the CCD surface cm, each calculation can be performed in a short time. By appropriately using the above formulas (1) to (11) as necessary, the above object and the object In a short time for even a comparison search of the reproduced image containing the corresponding 2-dimensional or 3-dimensional shape, and can be easily performed, thereby improving the reliability of the field.

Furthermore, according to the method and apparatus of the present invention, any one or more of the expressions (1) to (11), or all the expressions (1), as required.
Since (11) to (11) are added to the attribute data, the comparison search can be efficiently and easily performed by referring to and decoding the attribute data.

Further, according to the method and apparatus of the present invention, the object is a road, a sign, a river, a canal, a building, various facilities, various facilities, a vehicle, a motorcycle, an aircraft, a hull, or a moving vehicle. , A two-wheeled vehicle, an aircraft, a hull, or any other structure having an operation function, it is possible to construct a two-dimensional to three-dimensional reproduced image corresponding to many objects and to display the above-mentioned image. Yes, it can be widely used, has high reliability and convenience, and has high social utility value.

On the other hand, according to the method and apparatus of the present invention, the object 22, 51, 72 on the map 23, 71 or the like is viewed from the self side.
The display status of the reproduced image corresponding to ... and the like, and if necessary, the name and attribute data, the camera position, the camera direction, the camera speed, and the like can be delivered on the Web, and vice versa.
Since the display status of the reproduced images on the maps 23 and 71 and the distribution of the name and attribute data can be distributed also by the request from the client, the output display units (not shown) of the plurality of terminal stations can display the local images. In addition to the advantage of being able to deliver the situation and provide information to many people, the fact that many people observe can also receive a lot of information and help to respond appropriately is there.

On the other hand, according to the storage medium of the present invention, at least the above-mentioned arithmetic algorithm according to the present invention, equations (1) to (11), and one or more images photographed by one or more cameras. Steps for calling comparison parts, which are constituent parts of the objects 22, 51, 72, etc., database 1 in which image data of abundant multiple types of characteristics are stored
3 to 33, etc., a step of retrieving the image data of the feature relating to the corresponding part of the comparison part, a step of determining whether or not the comparison part and the image data of the feature correspond, and the both do not correspond. In this case, a step for determining whether or not the calling of the predetermined comparison part has ended, a step for determining whether or not the processing for the predetermined part has ended if the calling has not ended, and a process for the predetermined part If not completed, the step of designating the next comparative part, the step of recognizing as the specific object images 22, 51, 72, etc. when both of the above correspond and specifying the next part, the object 22 , 51, etc., a step of determining whether or not the processing of a predetermined portion of the comparison image has been completed, and when the processing is completed, the objects 22, 51, etc., which are recognition results. Since it includes a step of sequentially outputting a moving image of a reproduced image including such a two-dimensional to three-dimensional shape or a group of continuous still images in a sequential manner, by distributing this storage medium, in many regions, Also, many observers can use the image display method according to the technical idea of the present invention, and the social benefit can be efficiently increased.

[Brief description of drawings]

FIG. 1 is a block diagram showing a main part of an image display device according to a first embodiment of the present invention.

FIG. 2 is an explanatory diagram illustrating a state of a former stage of an image display method of an object using the image display device according to the first embodiment.

FIG. 3 is an explanatory diagram illustrating a latter stage state of an image display method for an object using the image display apparatus according to the first embodiment.

FIG. 4 is an explanatory diagram illustrating a state of an image display method of another object using the image display device according to the first embodiment.

FIG. 5 is a block diagram showing a main part of an image display device according to a second embodiment of the present invention.

FIG. 6 is a block diagram showing a part of the image display device according to the second embodiment which executes a comparative search between a comparative image of an object and a plurality of comparative parts.

FIG. 7 is an explanatory diagram illustrating a specific example of an image display method using the image display device according to the second embodiment.

FIG. 8 is a flowchart showing a process of actually performing an image display method by the image display device of the second embodiment.

FIG. 9 is a block diagram showing a main part of an image display device according to a third embodiment of the present invention.

FIG. 10 is an explanatory diagram illustrating one specific example of an image display method using the image display device according to the third embodiment.

FIG. 11 is an explanatory diagram illustrating another specific example of the image display method using the image display device according to the third embodiment.

FIG. 12 is an explanatory diagram illustrating a photographing relationship between a camera CCD surface and a map plane used in the image display device according to the third embodiment.

FIG. 13 is an explanatory diagram for explaining the first stage of the relationship between the target object and the image of the target object on the CCD surface performed in each of the above-described embodiments.

FIG. 14 is an explanatory diagram for explaining a second stage of the relationship between the object and the image of the object reflected on the CCD surface, which is performed in each of the above embodiments.

FIG. 15 is an explanatory diagram for explaining a third stage of the relationship between the object and the image of the object reflected on the CCD surface, which is performed in each of the embodiments.

16 (A) is an explanatory view for explaining the relationship between the coordinates virtually set on the road and the object photographed from directly above, and FIG. 16 (B) shows the camera at a predetermined height position on the road. It is explanatory drawing explaining the relationship of the target and the virtually set coordinate which image | photographs the front and converges toward the distance on the road.

FIG. 17 is an explanatory diagram for explaining a fourth stage of the relationship between the target object and the image of the target object on the CCD surface performed in each of the above-described embodiments.

FIG. 18 is an explanatory diagram for explaining a fifth step of the relationship between the object and the image of the object reflected on the CCD surface, which is performed in each of the embodiments.

FIG. 19 is an explanatory diagram illustrating a case where an image reflected on the CCD surface changes.

[Explanation of symbols]

A ... Arrow 11 ... Input part 12 ... Image cutout part 13 ... Image storage part 14 ... Correlation calculation part 15 ... Three-dimensional image composition part 16 ... Output display part 20 ... Whole screen 21 ... Area 22A ... Object 22B ... Object 22a ... Enlarged image (reproduced image) 22b ... Enlarged image (reproduced image) 22c ... Enlarged image (reproduced image) 22d ... Enlarged image (reproduced image) 23 ... Area 24a ... Star mark (target object) 24b ... Star mark (target object) 25 ... Area 26 ... Enlarged image (reproduced image) 26a ... Enlarged image (reproduced image) 26b ... Enlarged image (reproduced image) 26c ... Enlarged image (reproduced image) 26d ... Enlarged image (reproduced image) 27 ... Position 28 ... Region 28a ... arrow 28b ... arrow 28c ... arrow 28d ... arrow 28e ... arrow 29A ... area 29B ... area 29C ... area 29D ... area 29E ... area 29a ... enlarged image (reproduction image) 29b ... enlargement Image (reproduced image) 29c ... Enlarged image (reproduced image) 29d ... Enlarged image (reproduced image) 29e ... Enlarged image (reproduced image) 31 ... Input unit 32 ... Comparison image generation unit 33 ... Database 34 ... Comparison component generation unit 35 ... Component identification part 35A ... Search 35B ... Recognition 35C ... Identification 35D ... Fixed 36 ... Output part Im1 ... Memory Im2 ... Image conversion part M1 ... Memory Ma ... Projection part M2 ... Storage area 41 ... Component identification device 51 ... Object ( Road sign) 52 ... Tree 53 ... General vehicle 54 ... Area 61 ... Video input device 62 ... Video recording / reproducing output device 63 ... Object designation device 64 ... Video map display device 65 ... Attribute recording / playing device 66 ... Object retrieval designation device 67 ... Lock-on display part 71 ... Map 72 ... Target (specific point) 73 ... Target (specific point) 74 ... Target (specific point) 75 ... Target (specific point) 76 Target (specific point) c ... Image of line segment d ... Image of line segment e ... Line segment f ... Line segment h2 ... Line segment h3 ... Line segment i ... Image of line segment i '... Line segment Image l ... Line segment m ... Line segment n ... Line segment r ... Line segment t ... Line segment image u ... Line segment image v ... Line segment image x ... Line segment y ... Line segment cm ... CCD surface es ... Connection Image plane lp ... Optical axis ll ... Line segment B ... Point E ... Point F ... Center H ... Point O ... Optical center P ... Point Q ... Point R ... Point S ... Point W ... Line segment image W ′ ... Line segment Image Z ... Point HR ... Line segment PQ ... Line segment PR ... Road

Claims (26)

[Claims] 1. One or a plurality of objects at arbitrary positions on an image display of a video image taken by one or a plurality of cameras in the outside world are designated by names or attributes, or are surrounded by a square with a mouse, or a single point position By clicking or by specifying with a light pen or touch panel, the target including the change in the relative angle and direction between the camera and the target and the change in distance while excluding the presence of the target surroundings. A database in which the features of each image frame of the target object or the features of each component of the target object are sequentially detected in time series while tracking an object, and abundantly stored image data relating to various features. From the image frame of the target object, or the image frame having the corresponding relation to the continuous change of the feature of each component, or the image data of the feature in order. A reproduced image including a two-dimensional or three-dimensional shape corresponding to the target object that is searched and pattern-matched corresponding to the search result is sequentially configured for each time series change, and the reproduced image is included. Match each image frame or image data of the feature to the required area on the above image display or other image display via communication line etc. to match the setting of the required size standard. Are displayed sequentially as a group of
An image display method characterized in that the name and the attribute data generated or attached to the reproduced image are also displayed in a required area as needed. 2. The image data of various characteristics stored in the database has respective attribute data associated with each image frame of various reproduction objects or image data of characteristics corresponding to various constituent parts in advance. When the above-described pattern matching is performed on the image data of at least one feature, a reproduction image including a two-dimensional or three-dimensional shape corresponding to the target object is immediately constructed based on the attribute data, and the reproduction is performed. The image display method according to claim 1, wherein the image is displayed as described above. 3. The image data of various characteristics stored in the database includes data including various reproduction objects, various straight lines, various curves, various angles, various edges, various planes, various curved surfaces, and the like in advance. In addition, each attribute data that clarifies the combination, assembly, and connection relationship between the image data of each feature is associated, and the pattern matching described above is performed for at least one set of the image data combinations of the feature and the like. If it is taken, the attribute data immediately corresponds to the one or more objects 2
A reproduced image including a three-dimensional shape or a three-dimensional shape is configured, and the above-described image display of the reproduced image is performed.
Alternatively, the image display method described in 2. 4. A reproduced image including a two-dimensional or three-dimensional shape corresponding to the object that has been subjected to the pattern matching, even if there is a change in the line-of-sight direction due to the movement of the camera,
4. The above-mentioned image display of a reproduced image including a two-dimensional to three-dimensional shape from various viewpoints of the object involved in the calculation and formation of the two-dimensional to three-dimensional shape. Image display method described. 5. An image data having a spatial distribution and a temporal distribution of the object is obtained by using a plurality of cameras, and a reproduced image including a three-dimensional shape corresponding to the object is obtained by at least parallax information and time difference information. 5. The image display method according to claim 1, wherein the image display is performed by generating a three-dimensional model of the reproduced image. 6. The reproduced image is sequentially and sequentially displayed as a group of the moving image or a series of continuous still images with a time series change, and is displayed on the image display or the other image display. Is an image display of a map around the object as needed, and an image display of the object at a corresponding position in the map with a required mark or a reproduced image based on the pattern matching. 6. The image display method according to claim 1, wherein the display area of the map and the display area of the reproduced image are compared with each other as necessary, or both are overlapped. 7. The positional relationship between the known point and the camera is calculated by designating a plurality of points on the object designated on the image display or on the other image display, and The calculation algorithm is also stored in the above database, and when other points are specified, the calculation algorithm is used to calculate each mutual point and distance among arbitrary objects, and the target object. 7. The image display method according to claim 1, wherein the area or volume of the entire surface or the surface of a part of the component parts is calculated and measured. 8. A camera position, a camera direction, and a camera moving speed are calculated by a necessary calculation for a moving image of the reproduced image or a group of continuous still images which are sequentially displayed as a group due to the camera movement. The image display method according to any one of claims 1 to 7, wherein the requested item is displayed as an image with a required mark or numeral character. 9. On the image display or on the other image display, an area to be sequentially displayed as a moving image of the reproduction image or a group of continuous still images, or a name or attribute of the reproduction image. 9. The image according to claim 1, wherein an area for displaying data, or an area for displaying the position of the camera, the direction of the camera, and the moving speed of the camera can be arbitrarily set according to a required operation or designation. Display method. 10. When designating the target object, even if the user selects and designates an area range wider or narrower than the target object with the mouse or the like, only the target object can be extracted and recognized. The image display method according to any one of claims 1 to 9, wherein: 11. When the objects or a part of the objects are designated, shapes formed by connecting the objects and attribute data including a position, a distance, an area, or a volume of the shapes, and their two-dimensional to three-dimensional shapes. The image display method according to any one of claims 1 to 10, wherein the reproduced image obtained by calculating a required correlation is displayed as described above. 12. A CG image of a three-dimensional object of the object is constructed on the image, and the CG image is sequentially updated in time series as the camera moves as a landscape simulation so as to be displayed on the image display or other images. The image display method according to any one of claims 1 to 11, wherein an image is displayed on a video image on a display, a map, or a required area. 13. One or a plurality of cameras for photographing the outside world, an input section for converting a time-sequentially changing video image photographed by the camera into an electric signal and inputting it at any time, and an arbitrary image from the output display section. A designation unit for designating one or more objects, an image cropping unit that crops out the existence other than the target object and extracts only the target object, each image frame of the target object, or an image of characteristics of each component. Data is stored in the image storage unit (ie, database) in abundance, and a pattern matching image frame or characteristic image data is calculated by calculating the correlation with each image frame or characteristic image data. Correspondingly, a correlation calculation unit that additionally stores image data of a new image frame or feature in the image storage unit, and an image frame or feature obtained by the correlation calculation unit. A three-dimensional image forming unit that generates a reproduced image including a two-dimensional to three-dimensional shape corresponding to the object based on the image data, a video image and a map of the reproduced image, and a moving image of the reproduced image or a continuous image. An output display unit that sequentially outputs as a group of still images in contrast, and a setting unit that sets a display area of the reproduction image in the output display unit, and reproduces the reproduction image generated by the three-dimensional image forming unit. Images are sequentially displayed in the required display area in the output display unit set by the setting unit on the basis of the required size as described above, and, if necessary, the name and attribute data associated with or generated in the reproduced image are also displayed. An image display device displaying at a required position of the same or another required display area set by the setting unit. 14. One or a plurality of cameras for photographing the outside world, an input section for converting a time-sequentially changing video image photographed by the camera into an electrical signal and inputting it at any time, and an arbitrary image from the output section. A designation unit that designates one or more target objects, a comparison image generation unit that excludes the existence of other than the target object and generates the characteristics of the entire target object or each component as a comparison image, and various reproduction targets. A database that abundantly stores image data of features of each object or various component parts, and a comparison component generation unit that generates image frames or image data of features of the comparison component by comparing and searching the database. The comparison image data of the object and the image data of the characteristics of the comparison part generated by the comparison part generation unit are compared and searched to include a two-dimensional to three-dimensional shape corresponding to the object. A part specifying unit that specifies the reproduced part image, searches or generates the name and attribute data attached to the reproduced part image as necessary, and obtains a position to be fixed with respect to the image display of the reproduced part image, In addition to displaying the video image and the map as an image, an output display unit that displays the reproduced component image at a position to be fixed on the map is provided, and the organization of the reproduced component image is fixed on the map in the output display unit. The required marks, numbers and characters, or moving images of two-dimensional or three-dimensional shape or a series of continuous still images are displayed in chronological order in sequence as images, and if necessary, the moving images or continuous images are displayed. An image display device characterized in that the name and attribute data associated with or generated by a typical still image are also displayed at required positions in the output display section. 15. One or a plurality of cameras for photographing the outside world, an image input device for inputting an electric signal of the time-sequentially changing video image, or a time-sequentially changing video image photographed by the camera. A video recording / reproducing output device for converting to an electric signal for recording and reproducing, and an object capable of designating one or more arbitrary objects from the time-sequentially changing video image by excluding unnecessary surroundings. A video map display device that displays a specified device and the video image and map on the required display screen in a superimposed or simultaneous manner, and retrieves or generates, records, and reproduces the name and attribute data of the object. Corresponding from the attribute recording / reproducing device and a database in which various image frames corresponding to the target object or the image data of the characteristics of various component parts are abundantly stored. 2D to 3D corresponding to the object by comparing and retrieving image frames or image data of features
A target object search and designation device for generating a reproduced image including a three-dimensional shape, and a sequential or comparatively locked as a group of moving images or continuous still images of a reproduced image including the two-dimensional to three-dimensional shape corresponding to the object. It is equipped with a lock-on display section for on-display, and is required at a position corresponding to the existing position of the object on the display screen on which the video image and the map are displayed in an overlapping manner or are displayed in parallel contrast. Is displayed, the reproduced images corresponding to the object are sequentially displayed on the lock-on display section as described above, and, if necessary, a desired position on the display screen or the lock-on display section. An image display device, which also displays the name and attribute data associated with or generated in the reproduced image at the required position of. 16. A search for matching the size and line-of-sight angle of the object with the size and line-of-sight angle of the various reproduction objects or image data of various characteristics that are abundantly stored in the database. An adjustment unit is provided, a reproduction image including a two-dimensional or three-dimensional shape corresponding to the object having the same size and line-of-sight angle is comparatively searched and generated, and the reproduction image is sequentially displayed as described above. 16. The image display method and the image display device according to claim 1. 17. Further, a plurality of points in the video image are made to correspond to a map, which is tracked along with a time-series change to read the coordinates on the map, and the camera position and the camera are determined from the read coordinates of the several points. A direction and camera speed are calculated and obtained, and a conversion means for converting the video image and the map into the same coordinate system or a required coordinate system having a required line-of-sight angle is provided. 2D to 3 corresponding to the mark or line-of-sight angle indicating the object at the corresponding position of
17. The reproduced images including the dimensional shape are sequentially displayed as described above, and if necessary, the names and attribute data associated or generated with the reproduced images are also displayed at required positions. An image display method and an image display device. 18. Further, there is provided computing means for computing and obtaining the one or more or a representative camera position, a camera direction, and a camera moving speed in relation to the outside world, and the camera position and the camera position are provided at corresponding positions on the map. The image display method and the image display apparatus according to claim 1, wherein a mark or a numeral character indicating a camera direction is displayed. 19. The one or more or each of the camera positions and the front of the camera (on the projection axis of the optical axis onto the ground plane).
In calculating the positional relationship of the observation target of, v = fy · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos θ (tan ⁻¹ (h / y) −θ)} ... (1) or v = f {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (2), and y is directly below the camera. 19. The image display method and image according to claim 1, wherein v is a length of the object on the CCD surface in the camera, and θ is a dip angle of the camera. Display device. 20. Correlation between a video image captured by the camera or the relationship between the position of one or more or each of the cameras, the ground surface and an object at an arbitrary position thereof, and an object on the map. In order to calculate and clarify the relationship, v = f · y · sin (tan ⁻¹ (h / y)) / sin θ / {h · cos (tan ⁻¹ (h / y) −θ)} (3 ) Or v = f · {tan ((π / 2) −θ) −tan ((π / 2) −θ−tan ⁻¹ (y / h))} (4) or v = {(f / sin θ ) .Y.sin.beta.) / (H.cos (.beta .-. Theta.))} (5) u = x. (F / y). (Cos β / cos (β-θ)) (6) where β = arcTan (h / y), where y is the distance from directly under the camera to the object, and v is the length of the object on the CCD surface in the camera. x is the object and the line segment orthogonally length connecting the camera, u is a CCD in the camera
20. The image display method and the image display device according to claim 1, wherein a horizontal length of the plane, θ is a dip angle of the camera. 21. To calculate and clarify the relationship between the height apex Z of an object or the like and the height apex W of the image of the object on the CCD surface, W = {(f / sin (θ + (Π / 2)) · Z · sin γ)} / {y · cos (γ− (θ + (π / 2)))} (7) where γ = arcTan (y / z), and f is the lens Focal length, θ is the dip angle of the camera, and φ = θ
+ (Π / 2), φ is the angle formed by the perpendicular of the line segment connecting the lens and the object on the camera intersection axis, and ε is β +
(Π / 2), y corresponds to Z, and the relationship between the height dimension of the target object and the height dimension i of the image of the target object on the CCD surface is calculated and clarified. However, i = {(f / sin θ + (π / 2)) · h · sin (β + (π / 2))} / {y · cos (β + (π / 2)-(θ + (π / 2))) )} (8) where β = arcTan (y / z), h is the height of the camera position, and f, θ, etc. are the same as in the case of the above formula (7). Further, in order to calculate and clarify the projection of the image corresponding to the height H + Z of the object on the CCD surface, the above Wi-{= ([f / sin (θ + (π / 2)) · Z · sin γ)] / [y · cos (γ− (θ + (π / 2)))]}-{[(f / sin (θ + (π / 2)) · h · sin (β + (π / 2))) / [Y ・ cos ( β−θ)]} (9) Further, i′−W ′ = (i−W) / cos δ (10) where δ = arcTan {(t−u) / (i−W)} According to the above equation (7), equation (8), equation (9), or equation (10), comparing the reconstructed image including the object and the two-dimensional or three-dimensional shape corresponding to the object. 21. The image display method and image display device according to claim 1, wherein the image display method is also capable of performing a search. 22. In order to calculate and clarify the position of the image t of the portion extending laterally from the object on the CCD surface, t = S (f / Z) · {cosγ / (cosγ−θ− (π / 2))} (11), S is a portion that extends laterally from the object, and
22. The use of 2) makes it possible to perform a comparative search between the target object and a reproduced image including a two-dimensional or three-dimensional shape corresponding to the target object.
An image display method and an image display device. 23. Any one or a plurality of mathematical formulas among the formulas (1) to (11), or all the mathematical formulas (1) to (11) are added to the attribute data. An image processing method and an image processing apparatus according to any one of claims 1 to 22. 24. The object is a road, a sign, a river, a canal, a building, various facilities, various facilities, a vehicle, a motorcycle, an aircraft, a hull, or a moving vehicle, a motorcycle, an aircraft, a hull, or other 24. The image display method and the image display device according to claim 1, wherein the image display method is a structure having an operation function. 25. A display status of the reproduced image corresponding to the one or more objects on the required map, and a name, attribute data, camera position, if necessary,
Web distribution is possible including the camera direction and camera movement speed, and the display status of the above-described reproduced image on the map as described above and any target instructed to the map are also requested by the client. The image display method according to any one of claims 1 to 24, characterized in that it is possible to distribute a display situation such as a mark or a numerical character indicating an object, the name or attribute data attached to or generated in the reproduced image, and the like. And an image display device. 26. At least a step of calling the above-mentioned arithmetic algorithm, expressions (1) to (11), and a comparative part which is a constituent part of one or more objects photographed by one or more cameras. , A step of retrieving the image data of the feature related to the corresponding part of the comparison part from a database storing abundant image data of a plurality of types of features, and whether or not the comparison part corresponds to the image data of the feature And a step for determining whether or not the calling of the predetermined comparison part has ended when both of them do not correspond, and whether or not the processing for the predetermined part has ended if the calling does not end. If both of the above correspond, the step of determining if In that case, the step of recognizing as a specific object image, the step of specifying the next part, the step of determining whether or not the processing of the predetermined portion of the comparison image in the object has ended, and the processing ends And a step of outputting the moving image of the reproduced image including the two-dimensional or three-dimensional shape of the target object as a recognition result or a series of continuous still images in comparison with each other in order. Storage medium recording method.