JP5780561B2 - Visibility video information generator - Google Patents

Description

  The present invention relates to a field-of-view video information generation device. In particular, in an apparatus for generating a view image of a landscape on the ground including an urban area, an apparatus using an aerial photograph taken from above in an aircraft or the like or a photograph taken on the ground on a road and an apparatus for generating view image information The present invention also relates to an apparatus for generating and displaying a three-dimensional city view image using photographs taken from various angles on the ground. In particular, in a field-of-view video device that uses photographs taken from various angles of the ground, it is corrected when switching between photographs that use the displacement of the view image display position due to parallax due to the distance in the height direction from the position reference plane of the object. And an apparatus for realizing smooth continuous display.

  Conventionally, as a method of displaying a view image of a landscape including a city, a 3D model of a city and terrain is created, and a texture is pasted on the surface of the generated 3D model by texture mapping to obtain a 3D model. There is a method for obtaining a field-of-view image from the obtained three-dimensional model by three-dimensional image processing. This method is widely used as a textbook method, but requires a lot of man-hours for creating a three-dimensional model and man-hours for attaching a texture to a polygon formed by the three-dimensional model. In particular, when an actual photograph is pasted by texture mapping, the amount of calculation increases, requiring a very large number of man-hours, resulting in a high cost, and also has a drawback of being responsive.

  Patent Document 1 discloses a technique for obtaining three-dimensional information of an object in real time on the same principle as stereo photogrammetry. That is, the three-dimensional information of the photographing target is obtained by the triangulation method using the information of the corresponding points of the panoramic images of the two common photographing targets. Each time the combination of panoramic images used for display changes, three-dimensional information of the object to be imaged is obtained by triangulation, and a polygon is formed by connecting adjacent corresponding points of the panoramic image. Texture mapping. Furthermore, it is a common practice to generate a 3D video field of view by matrix transformation from polygon information.

  Patent Document 2 discloses a method related to a panoramic video system that displays an undistorted image in real time. That is, a method of capturing a real image and generating a panoramic image as a moving image is disclosed. The system used in the method of Patent Document 2 is composed of a panoramic annular lens (PAL) system, a video camera, and a PC-based software system. The PAL system is 360 degrees with two mirrors, a hyperboloid mirror and an ellipsoidal mirror. A horizontal field of view and a vertical field of view of 90 degrees are taken with a video camera. At the time of reproduction, images taken by the software system are developed in real time into 360-degree video images that are seamless and have no distortion. The system for development uses a ray tracing program to convert a distorted spherical image into a planar panoramic image on Cartesian coordinates and obtain the image in real time. A graphic user interface (GUI) can be used to determine any breakpoint (panoramic image centerline), zoom in or zoom out functionality, and built-in calibration.

  Both of these methods require time and cost to generate a three-dimensional model of a city's three-dimensional structure, and in order to make the three-dimensional outer wall of a building or the like identical to the actual one, pattern information on the outer wall Alternatively, image information needs to be pasted (texture pasting). As information for texture pasting, information from all directions of the three-dimensional object is indispensable, and much labor and cost are required for acquisition and texture pasting. In this way, although it is strongly desired to display a landscape of an arbitrary city part from an arbitrary aerial route and line of sight as a three-dimensional image, the accuracy of texture pasting cannot be sufficiently realistic, or The update cycle of information has become longer, and the freshness and accuracy of the displayed video often have difficulties.

JP 2006-350852 JP Japanese Patent Application No. 2006-517853 Japanese Patent No. 4418857

  However, in the methods disclosed in Patent Documents 1, 2, and 3, if the height information of the ground structure is not used other than the DEM (digital elevation model) information on the ground surface, the viewpoint position, the line-of-sight direction, and the position of the photographed image In principle, it is impossible to eliminate the parallax due to the difference in viewpoint position caused by the height of the surface structure from the information alone. In other words, 3D information of urban structures is measured and a 3D model is generated, and a realistic view image based on an actual photograph can be obtained in a short time without the need for pasting a texture with an actual image on the surface. On the other hand, because it does not have 3D information such as buildings, it has the disadvantage that it will not be able to obtain a smooth view image due to the displacement of the image due to parallax at the time of switching the real image used for the view image generation. There is. The present invention has been made in view of such circumstances.

  As one embodiment of the present invention, the image capturing device captures images in a plurality of directions at each of a plurality of image capturing positions, captures a plurality of image areas divided into a plurality of sub areas, and acquires a plurality of pieces of image information. And when storing each of the plurality of pieces of image information in the memory, shooting position information and shooting direction information when the image information is acquired are stored in the memory in association with the image information. The sub-area captured when the image information is acquired with reference to the shooting position information and shooting direction information associated with the image information stored in the image information, and the specified sub-area, The image information, the shooting position information, and the shooting direction information are stored in the database in association with each other, and a plurality of positions arranged in order on the viewpoint path and the plurality of positions, respectively. The direction of the line of sight toward the shooting area is specified, the shooting direction information associated with the sub-area stored in the database is referred to, and the line of sight at each of the plurality of positions on the viewpoint path The image information including the sub-region located in the direction and the shooting position information associated with the image information is retrieved and read in the order, while switching the image information read in the order, In a view image information generating method including generating view image information with reference to shooting position information associated with the image information, generating the view image information in the direction of the line of sight from a position on the viewpoint path In doing so, from the image information to be used, the shooting position different from the shooting position of the image information and / or the shooting direction of the image information When switching to another image information in the shooting direction, the position image correction of the visual field image information is performed so that the display position of the area common to the image information and the other image information does not shift. A visual field image generation device characterized by the above is provided.

  As an embodiment of the present invention, the parallax for different viewpoints caused by the height of the surface structure that cannot be resolved only from the viewpoint position, the line-of-sight direction, and the position information of the photographed image is eliminated using image recognition technology. To do. In other words, even if the position-related attributes of the view image cannot be maintained, the point of interest of the specific feature exists in a close place in the view image, and the shape of the feature in the view image has high similarity. Use. This is because if the viewpoint is moved while capturing the same feature with the line of sight, the continuity of the view image cannot be maintained due to the parallax at the time of switching the actual image to be used, but the attention point of the specific feature in the view image This is because the viewpoints of viewing the feature before and after the switching of the real image are close and the line-of-sight direction is also close. Paying attention to this, image matching is performed around the point of interest of a specific feature in the view image, and the display coordinates of the view image at the time of switching the live image are shifted so that the positions of the matched images overlap, and if necessary Thus, the enlargement / reduction ratio is changed.

  In general, view images using real images taken from different viewpoints on the same feature do not completely match due to parallax. For this reason, it is necessary to perform image matching for registration centering on a specific part in the field of view. As one embodiment of the present invention, (1) the center part of the image and (2) the cursor are designated by selection. Part (3) The display coordinates of the view image at the time of actual image switching are shifted so that the matched images overlap with each other on the basis of the position on the view image screen detected by a line-of-sight detector such as an eye tracker.

  Further, as an embodiment of the present invention, the image matching process is performed in parallel with a separate processor while synchronizing with the view image generation process in order to perform the image matching process without affecting the view image generation time. As a form of this parallel processing, it is preferable to perform parallel processing using a CPU such as Intel (registered trademark) i7 that includes a plurality of cores in the same CPU chip and has a plurality of threads.

  In addition, as one embodiment of the present invention, since it is necessary to execute image matching at high speed, processing is performed in advance so that image matching can be easily performed on each photographed image, and in advance in the memory together with each photographed image. You can also remember it. Since such an image for image matching is exclusively used for calculation processing, an image processed so that the luminance distribution for each sub-region of the image is leveled is suitable.

  In the present invention, without generating a three-dimensional numerical model and texture pasting processing on the surface of a building, a visual image for an arbitrary viewpoint route and viewpoint direction is acquired for an arbitrary place using an actual image. Can be generated in a short time. In addition, it is possible to eliminate the discontinuity of the view image and obtain a smooth image.

It is a figure showing roughly the cityscape 3D picture generation and browsing system whole composition of the present invention. It is a figure which shows the concept of the aerial image acquisition in the cityscape three-dimensional video production | generation and browsing system shown in FIG. It is a conceptual diagram of the morphing in a three-dimensional image | video. It is a figure explaining the concept of the video generation by the morphing using the real image in the cityscape 3D video generation and browsing system of this invention. It is a figure explaining the operation example of the aerial image acquisition system by the aircraft of this invention. It is a figure explaining the processing flow of the aerial image acquisition system of this invention. It is a figure explaining the example of the airborne digital camera aggregate | assembly of this invention. It is a figure which shows the example of a setting of the course and imaging | photography point of the aerial image acquisition system by the aircraft of this invention. It is a figure which shows the structural example of the aerial image acquisition system of this invention. It is a figure which shows the processing flow of the flight navigation system part of the airborne image acquisition system of this invention. It is a figure which shows the structural example of the aerial image acquisition plan file of this invention. It is a figure which shows the example of the display screen of the flight navigation system part of the aerial image acquisition system of this invention. It is a figure which shows the information flow of the data acquisition recording system part of the aerial image acquisition system of this invention. It is a figure which shows the processing flow of the imaging | photography control system of the aerial image acquisition system of this invention. It is a figure which shows the structure of the aerial image primary file of this invention. It is a figure which shows the processing flow of the aerial image database production | generation registration system of this invention. It is a figure which shows the aerial image database structure of this invention. It is a figure which shows the relationship between the image and visual field in the haze effect removal process of this invention. It is a figure which shows the example of an image characteristic parameter graph in the haze influence removal process of this invention. It is a figure which shows the processing flow of the haze effect removal process of this invention. It is a figure which shows the example of a process result of the haze effect removal process of this invention. It is a figure which shows the aerial image index production | generation registration processing flow of the aerial image database production | generation registration system of this invention. It is a figure which shows the structure and term definition of the aerial image index mechanism of this invention. It is a figure which shows the structure of the aerial image index mechanism of this invention. It is a figure which shows the index structure of the solid angle cell of this invention. It is a figure which shows the correlation of the aerial image index of this invention, and an association table. It is a figure which shows the structural example of the visual field by the digital camera from which three types of focal distances differ of the digital camera aggregate | assembly of this invention. It is a figure which shows the structural example of the digital camera aggregate | assembly by the digital camera from which three types of focal distances differ of this invention. It is a figure which shows the imaging | photography range figure of the digital camera aggregate | assembly by three types of digital cameras from which focal distance differs of this invention. It is a figure which shows the structural example of the stable platform for aircraft mounting of this invention. FIG. 4 is a diagram illustrating a signal information flow of the stable platform of the present invention. It is a figure explaining the variable of the image generation by the morphing using the real image in the cityscape 3D image generation and browsing system of this invention. It is a figure which shows the aerial image generation process flow in the cityscape 3D image generation and browsing system of this invention. It is a figure which shows the processing flow of the original image search of the three-dimensional video generation process using the real image in the cityscape three-dimensional video generation and browsing system of this invention. It is the figure which showed the original image selection logic of the three-dimensional video generation process of this invention. It is a figure which shows the relationship between the visual field in the viewpoint in the three-dimensional video generation process of this invention, and an original image. It is a figure which shows the relationship between the gaze vector of this invention, and the intersection of Terrain. It is a figure which shows the relationship between the visual field in the viewpoint of this invention, and an original image. It is a figure which shows the visual field from the viewpoint P (t _i ) of the present invention. It is a figure explaining the mechanism in which the position shift in a visual field image | video occurs by parallax. 1 is a diagram schematically illustrating an overall configuration of a visual field image information generation device according to an embodiment of the present invention. FIG. 42 is a diagram illustrating a difference in visual field image due to parallax with respect to the same object in the image display system 100 of the visual field image information generation device illustrated in FIG. 41. It is a figure explaining the position shift correction of the visual field image | video by the parallax with respect to the same target object of the image switching smoothing system 110 of the visual field image | video information generation apparatus shown in FIG. It is a figure explaining the detailed structure of the image switching smooth system 110 of the visual field video information generation apparatus shown in FIG. It is a figure explaining the operation | movement time chart of the detailed structure of the image switching smooth system 110 of the visual field video information generation apparatus shown in FIG. It is a figure explaining the production | generation method of the image comparison information database 250 of the image switching smooth system 110 of the visual field video information production | generation apparatus shown in FIG. It is a figure explaining the histogram equalization of the image comparison information database shown in FIG. FIG. 47 is a diagram for describing an example of histogram leveling in the image comparison information database shown in FIG. 46. It is a figure explaining the position shift correction method by the parallax of the image switching smooth system 110 of the visual field video information generation device shown in FIG. It is a figure explaining the processing flow of the position shift correction process by the parallax of the image switching smooth system of the visual field video information generation device shown in FIG. FIG. 45 is a diagram illustrating a case where the time required for the image matching processing 255 is long in the operation time chart of the detailed structure of the image switching smoothing system 110 of the visual field image information generating device shown in FIG. 44. FIG. 45 is another diagram for explaining a case where the time required for the image matching processing 255 is long in the operation time chart of the detailed structure of the image switching smoothing system 110 of the visual field image information generating device shown in FIG. 44.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. The present invention is not limited to the following description, and various modifications can be made without departing from the scope of the invention.

(Embodiment 1)
Embodiment 1 of the present invention will be described with reference to FIGS. In the present embodiment, a system capable of capturing a wide range of multi-directional images with high-density mesh points in the air in an aircraft or the like is created, a large number of captured images are made into a database, and an arbitrary position in the city can be viewed from an arbitrary viewpoint. Build a system that searches for images with the least amount of parallax when viewed from any direction at high speed, and continuously smooth images from the image with the least amount of parallax corresponding to the specified viewpoint, line of sight, and location in the city. Morphing to generate

  The inventor of the present application is the biggest barrier in realizing the 3D video generation and viewing system for cityscapes, because there is no means to obtain inexpensive and frequent 3D information on buildings and structures such as urban buildings. It has been recognized that there is a lack of inexpensive and frequent means for acquiring and pasting images to be attached to the surface of a three-dimensional object. In the present invention, an image and a video mean content in which a moving image and a still image are displayed on an electronic display device including a liquid crystal screen. Moreover, in this application, although a content means landscape information exclusively, it is not limited to landscape information.

  Therefore, the inventor of the present application takes a landscape of any part of the city in advance from any direction as a method of obtaining a three-dimensional image of any part of the city from any route without following these means. In addition, the optimal captured image corresponding to the selected viewpoint path, line-of-sight direction, and target position is selected, and the position and optical axis direction of the camera that captured the image, and the difference between the selected viewpoint and line-of-sight are morphed. I came up with a method to compensate for this and obtain a smooth image.

  However, it is extremely difficult to take a picture of every part of the city in advance from all directions. In order to achieve this object, the present inventor must prepare photographs for all city points in directions ranging from a minimum of 2 degrees to a maximum of 25 degrees with a half-vertical angle when the solid angle is expressed as a cone. I know from experience that I must not. Since the solid angle is expressed in units of steradians, the steradians are calculated as 2π (1-cos θ) with respect to the half apex angle θ degrees, and therefore, the steradians range from a minimum of 0.004 to a maximum of 0.59. In order to solve this problem, many small digital cameras were used, and an arrangement method and a control method were conceived.

  That is, first of all, a large number of small digital cameras are formed as a collection of digital cameras, and they are stored in the holes of an aerial surveying camera on the floor of an aerial surveying aircraft. In order to avoid acquisition problems and, secondly, in order to efficiently take pictures from multiple directions, a camera for direct shooting is placed in the center of the digital camera assembly, and the surrounding area is radially inclined There are 8 to 25 digital cameras for photographing, or more if there is enough space.

  Third, even if the ground is photographed from various directions, it is not preferable that the resolution of the image changes depending on the depression angle at which the ground is viewed. The following description will be made using an off-nadir angle that generally refers to an angle formed from the vertical direction. A small off-nadir angle digital camera uses a short focal length telephoto lens, a large off-nadia angle digital camera uses a long focal length telephoto lens, and each of them is radially concentric. The inventor of the present application has come up with a digital camera assembly that acquires images over a wide range without degrading the resolution from directly below a flying object such as an aircraft to a region with a considerably large off-nadir angle.

  Fourth, the inventor of the present application uses an GPS device and an inertial navigation device on an aircraft in order to efficiently operate the digital camera assembly and obtain an image for every fixed solid angle for every point over a wide city space. After introducing a guidance system consisting of the above, a precise mesh-like shooting point was constructed in the air, and then a system for shooting and recording a shooting command was automatically issued to the digital camera assembly at the shooting point.

  Fifth, the present inventor conceived of shooting from a low altitude under the cloud to cloudy weather that is generally unsuitable for aerial surveying in order to reduce the shooting cost due to the necessity for frequent and frequent shooting. We have come up with a method to automatically compensate for image degradation due to airborne haze by digital image processing.

  Furthermore, the inventor of the present invention uses a solid angle in the range of a minimum of 0.004 to a maximum of 0.59 as a steradian (a range of 2 to 25 degrees in cone half apex angle) to generate an image for an arbitrary target point. In view of the need to search at high speed, we have come up with a high-speed image search mechanism suitable for this system. For example, the ground surface as an imaging region is divided into a square mesh shape by latitude / longitude or XY coordinates. One side of the square is sufficiently small, for example, set to about 50 m to 20 m. Each square on the mesh is managed by attaching a two-dimensional address. Each captured image includes the mesh square group included in the image range obtained by capturing the ground surface, and a vector connecting the camera position at the time of capturing and the center of the mesh square can be defined. This is normalized and defined as an original image vector. The original image vector can be used as an index for expressing an image for a designated ground mesh.

  That is, the zenith index is defined by quantizing the angle formed between the zenith vector indicating the position directly above and the original image vector, and the quantization criterion for the angle formed is 5 degrees or less. Since each zenith index includes 360 degrees of azimuth as horizontal components, it is subdivided into cells of solid angle units, and the angle between the original image vector and the zenith vector, the original Index by the horizontal component of the image vector. A necessary original image can be searched at high speed by this three-stage indexing. The original images required for 3D images are prepared for every urban location in solid angles from a minimum of 0.004 steradians to a maximum of 0.59 steradians. Since the viewpoint passes through this neighborhood, if the deviation of the solid angle between the actual viewpoint and the line of sight and the original image vector is a minimum of 0.004 steradian or less and a maximum of 0.59 steradian or less, the original image can be smoothly connected by morphing processing. . If the solid angle of the two images to be connected is smaller than a predetermined value, the original image may be switched without performing the morphing process.

  FIG. 1 is a diagram schematically showing an overall configuration of a browsing system in which the cityscape 3D video generation method according to the present embodiment is used. The browsing system in which the cityscape 3D image generation method is used has a portion 80 for generating a 3D image when the viewpoint is in the air. The part 80 is “preparing an image database (hereinafter referred to as“ original image ”) for each solid angle of 0.004 or more and 0.59 or less with respect to an arbitrary place in the city as an image database in advance. An original image whose viewpoint and line of sight are close to each other from the image database in accordance with the temporal change of the specified viewpoint path and the specified line-of-sight direction without generating a numerical model and performing texture pasting processing on the surface of the building and the ground surface. Are sequentially extracted and displayed by a method of smoothly connecting them while performing image morphing corresponding to the viewpoint path and the line of sight ". Furthermore, the browsing system according to the present embodiment includes a graphic user interface system 180, and a service is provided to the user 191 via the Internet 190.

  A portion 80 for generating a 3D image when the viewpoint is in the air includes an image acquisition system (hereinafter referred to as an aerial image acquisition system 100) for acquiring an aerial photograph in the air by an aircraft, and the acquired aerial photograph. The aerial image database generation / registration system 120 for processing and creating a database so that it can be easily used for 3D video generation, the aerial image database 140 created as a database, and the request of the user 191 required through the graphic user interface system 180 And an aerial 3D video generation system 160 that generates a 3D video based on it.

  FIG. 2 is a diagram showing the concept of aerial image acquisition in the cityscape 3D video generation and browsing system. One of the greatest and common technical features of the present invention is that, firstly, it is not necessary to generate a three-dimensional model of a city in the generation of a three-dimensional cityscape image, and secondly, the generation It is not necessary to apply the texture of the outer wall surface pattern or the outer wall surface photograph to the three-dimensional model. Instead of eliminating manual and costly model generation and texture pasting tasks, images are acquired from all directions in advance for any location in the cityscape, and can be selected and transformed as needed. can do. However, this method not only needs to capture a large number of images in advance, but also needs to search for an optimal captured image at high speed. Recent advances in digital cameras, reduction in the capacity and cost of various memories, When the conditions for realization were established by improving the capability and communication capability, the new scheme according to the present invention was first realized.

  In FIG. 2, the celestial sphere is divided into solid angles with a cone or regular hexagonal prism with a half apex angle of 2 to 40 degrees as shown in the sky sphere sphere 222 of a sufficiently small solid angle range i221 or solid angle range 1220 in FIG. This solid angle is calculated as 2π (1-cosθ) with respect to the half apex angle θ degrees, and the celestial sphere 222 except the vicinity of the ground surface is divided in the range of 0.04 to 0.59 steradians, and images of every place on the ground surface The concept of the aerial image acquisition system 100 for preparing for each solid angle is shown.

  FIG. 3 shows a case where the same technical method as that in FIG. 2 is applied to 90 when generating a three-dimensional image when the viewpoint is on the road, and is taken in the adjacent range i221 of the sufficiently small solid angle. The images for the same point look slightly different due to parallax, but the morphing process to obtain a smooth 3D image by linearly transforming the field of view between the slightly different images by linear transformation This shows the concept of. The field of view of the building shown in the relationship diagram 230 of the building and the line-of-sight direction viewed from the line-of-sight direction 1240 and the line-of-sight direction 2250 having a relatively small parallax is an image 241 viewed from the line-of-sight direction 1 and an image 251 viewed from the line-of-sight direction 2. Indicated. The angle formed by the line-of-sight direction 1240 and the line-of-sight direction 2250 is a solid angle having a half apex angle of 20 degrees. When the angle formed is smaller than a value of 1.47 steradians or less, or when the object is planar, the images can be approximated to each other by morphing processing by linear transformation. It is one of the features of the present invention that this morphing process is used for the interpolation process of an actual image of an adjacent viewpoint. Note that it is also possible to simply switch the image without performing the morphing process, which is one of the features of the present invention.

  FIG. 4 is a diagram for explaining the concept of generating a simulated field of view by simulated flight in the cityscape 3D video generation and browsing system according to the present invention. The viewpoint moves sequentially along the viewpoint path P (t) 270 as time t passes, and during that time, the ground surface is captured in the field of view along the target trajectory T (t) 280. From the position on the viewpoint path P (t) 270, the target trajectory T (t) 280 is a line of sight 271 at a time t 1, a line of sight 272 at a time t + δt, a line of sight 273 at a time t + 2δt, and a line of sight at a time t + 3δt 2. 274, a trajectory on the ground surface captured for each δt by a line of sight 275 at time t + 4δt and a line of sight 276 at time t + 5δt. In the present invention, as a method of generating a three-dimensional cityscape image for the target trajectory T (t) 280 from the viewpoint path P (t) 270, the parallax is close from the line of sight 271 at time t to the line of sight 276 at time t + 5δt. The few aerial images i260 and aerial image i + 1 261 are searched from the aerial image database 143, and the closest gaze is between the gaze 271 at time t1, the gaze 272 at time t + δt, and the gaze 273 at time t + 2δt. Using the image i260 as an original image, the parallax due to the difference in viewpoint between the line of sight 271 at time t1, the line of sight 272 at time t + δt, and the line of sight 273 at time t + 2δt and the line of sight of the aerial image i260 The aerial image i + 1 261 is determined to have less parallax due to a difference in viewpoint than the aerial image i260 from the line of sight 274 at time t + 3δt. , Line of sight 274 at time t + 3δt, time t The target trajectory that is smooth by correcting the parallax due to the difference in viewpoint between the line of sight 275 of + 4δt, the line of view 276 of time t + 5δt, and the line of sight of the aerial image i + 1 261 by morphing the aerial image i + 1 261 3D image generation of the cityscape along T (t) 280 is performed. Note that the aerial image i260 and the aerial image i + 1 261 may be switched without performing the morphing process.

  FIG. 5 shows an example of an aerial image acquisition method according to the present invention. A digital camera assembly containing a large number of digital cameras as shown in FIG. 7 is mounted on the aircraft 301, and the ground surface is photographed at photographing points 310 at regular intervals along the flight path 300. The flight path 300 and the shooting points 310 are densely set in a mesh pattern, and a plurality of digital cameras are mounted and multi-directional images are taken simultaneously as shown in the optical axis direction 320 of the digital camera assembly, thereby obtaining FIG. An image is acquired for each sufficiently small solid angle range i221 shown.

  FIG. 6 shows a processing flow of the aerial image acquisition system 100. Hereinafter, a detailed implementation method of the aerial image acquisition system 100 will be described with reference to FIG. First, in the shooting point setting process 330, a flight route and an imaging point are set so that aerial images can be taken in a mesh pattern at regular distance intervals. In this shooting plan, for example, flight routes and shooting points on the route are defined in the form of an aerial image acquisition plan file 101 whose configuration is shown in FIG. Next, in the shooting process 331, it is determined that the aircraft 301 has reached the shooting point 310 defined in the aerial image acquisition plan file 101, and shooting is performed by the aerial shooting control system 393 shown in FIG. The captured image in the digital camera is stored in the aerial image primary file 102 together with metadata including the shooting position and digital camera attitude data acquired in the imaging process at predetermined timings by the aerial shooting control system 393. .

  FIG. 7 is a diagram showing a configuration example of the digital camera assembly 360 constituting the aerial image acquisition system 100 according to the present invention. For the purpose of efficiently and densely photographing the ground surface from all viewpoints, a plurality of oblique digital cameras 350b to 350i are arranged in such a way that the horizontal circumferential direction is radially divided at equal intervals around the digital camera 350a in the direct downward direction. The digital camera assembly 360 is formed by arranging eight cameras so that the angle between the optical axis and the direction of gravity is the same. In particular, when shooting on board an aircraft, it is necessary to obtain an airworthiness certificate. For aerial photography, the hole on the aircraft floor is changed, or the camera is placed outside the aircraft from the floor hole. It is difficult to obtain permission and approval by laws and regulations etc. to use it in a pop-up form. Therefore, the digital camera assembly 360 is devised so that the digital camera assembly fits in the aircraft floor hole by using a small digital camera such as Canon PowerShot. The support structure other than the digital cameras 350a to 350i constituting the digital camera assembly 360 is required to have high shooting direction accuracy. Therefore, any support structure that is lightweight and rigid can be used. It can be composed of a plate or carbon resin.

  FIG. 8 shows an example of setting the route and the shooting point of the aerial image acquisition system using an aircraft. FIG. 8B shows a setting example of the shooting point 310. The flight path 300 flies in a reciprocating manner by making a U-turn as shown by a dotted line in FIG. 8 (b) through parallel flight routes 300 at equal intervals in order to form a network of imaging points 310 in the air. During this time, shooting is performed at the shooting point 310. The mutual interval between the shooting points 310 is defined by the flight interval shooting interval 372 and the flight direction shooting interval 371. When the flight altitude is set to about 800 m, the flight interval shooting interval 372 and the flight direction shooting interval 371 are set to about 80 m or less in order to obtain at least one image within the sufficiently small solid angle range i221 in FIG. It is preferable to do this. When shooting with the digital camera assembly 360, the range indicated by the shooting ranges 375a-i of the digital cameras 350a-i in FIG. The flight altitude is not limited to 800 m and can be set in the range of 200 m to 2500 m. When the flight altitude is lower than 200 m, the amount of captured images increases and the processing amount increases. Further, if the flight altitude is higher than 2500 m, it is difficult to acquire a detailed image on the ground.

  Overlap with the range captured by each digital camera in the imaging range 375a-i of the digital camera 350a-i by setting the flight altitude, the angle formed by the oblique direction digital camera 350b-i, and the focal length of the lens The degree varies. For the purpose of the present invention, it is desirable to set so that the ground surface is covered almost continuously while slightly overlapping each other. Note that the direction of the digital camera 350a that captures the direct direction with respect to the flight path 300 is the traveling direction of the flight path 300 in the horizontal direction of the image frame in FIG. However, the present invention is not limited to this, and the direction of the digital camera 350a that captures the direction directly below the flight path 300 may be the vertical direction of the image frame that is the traveling direction of the flight path 300.

  FIG. 9 is a diagram illustrating a configuration example of an aerial image acquisition system. The aerial image acquisition system includes a flight navigation system unit 385 and a data acquisition / recording system unit 390. The flight navigation system unit 385 is a device for guiding the aircraft 301 along the flight path 300 to the imaging point 310 defined in FIG. 8B. The aircraft position data is periodically obtained from the GPS 380, and the air meter Avionics information 387 such as the attitude, altitude and speed of the aircraft can be obtained from the equipment 388. These signal interfaces are not novel because they are standardized as aircraft bus signals. The attitude information of each digital camera can be calculated from the attitude of the aircraft and the position and orientation of each digital camera relative to the aircraft.

  The function of the flight navigation system unit 385 is in accordance with the contents of the aerial image acquisition plan file 101 of FIG. 11, and the flight navigation system unit of the aerial image acquisition system of FIG. 12 according to the processing flow of the flight navigation system unit of the aerial image acquisition system of FIG. The pilot is guided by the example of the display screen. The flight navigation system itself has already been publicly implemented and may not have any novelty. However, in order to realize the object of the present invention, the part that efficiently guides the aircraft to the shooting point 310 is a part related to the present invention. is there. First, it is determined which part of the city the sky image is to be acquired before the flight. Determine the shooting range on the map and formulate a flight plan. The flight plan is performed by setting the flight path 300 on the aerial map.

  The photographing point 310 should be set in a high-density mesh shape at a height in the range of 200 m to 2500 m, preferably 500 m to 2000 m, so as to achieve the object of the present invention. Based on the setting result, the flight path 300 is set to be constituted by parallel lines. The route number shown in the aerial image acquisition plan file 101 in FIG. 11 is assigned to each straight line portion, and shooting points 310 are assigned so as to form a mesh as a whole for each route, and the start coordinates and end coordinates of each route, and in between Set the number of shooting points and the coordinates of each shooting point by latitude and longitude and altitude. In this way, the aerial image acquisition plan file 101 of FIG. 11 is constructed. The graphic user interface related to the construction of the aerial image acquisition plan file 101 can be implemented as a map information system.

  The function of the flight navigation system unit 385 is shown by the processing flow of the flight navigation system unit of the aerial image acquisition system shown in FIG. In the processing block 420, the display shown in FIG. 12 is sequentially performed from the route numbers registered in the aerial acquisition plan file 101 to all the shootings for the route numbers scheduled for flight. Since the designated route number has a start coordinate, in order to start the route number in processing block 421, the position, altitude, traveling direction and speed of the route start point are designated and the contents of FIG. To display guidance. In order to perform photographing, it is preferable to satisfy the conditions defined in the processing block 421 with a certain error range, for example, a position error of 10 m to 30 m or less and a flight direction error of 5 degrees or less. If not satisfied, the flight route may be restarted. In the case of redoing, the guidance of the processing block 421 is performed again in the processing block 423. When the conditions of the processing block 422 are satisfied, the processing points 424 and 425 are sequentially processed one by one from the shooting point coordinates described in the aerial image acquisition plan file of the route number selected in the processing block 420 to the final shooting point. Guidance by FIG. 12 shows a display screen example of the flight navigation system unit of the aerial image acquisition system.

  In order to guide the aircraft 301 to the starting point of the route number designated in the processing block 420, the photographing guidance display 437 and the position deviation display 446 of FIG. The deviation of the aircraft position 373 from the flight route 300 can be known, and the deviation is eliminated by maneuvering the aircraft. Data on the position of the aircraft is obtained from GPS 380, and altitude, heading direction, speed, and pitch / yaw / roll attitude data is obtained as avionics information 387 from aircraft instrumentation 388, and is used for the display in FIG. In the shooting guidance display 437, the route number 439 indicates the route in flight, and the remaining shooting point number display 438 in the route indicates the number of shooting points that must be taken in the future. In the shooting guidance display 437, the shooting point 310 and the shooting allowable range 440 are displayed along the flight route 300 for the shooting point 310, so that the pilot passes through the shooting allowable range 440 for each shooting point. Maneuver like so. The aircraft guidance display 437 is displayed rolling from top to bottom according to the flight, so that the relationship between the flight path 300 and the aircraft position 373 that should always be taken as the latest photographing point is shown. The aircraft position display 373 is always at the bottom left of the screen. In addition, as the azimuth display 442, posture display 443, and position deviation display 446 of FIG. 12, those that are publicly implemented in the current aircraft instrumentation can be used.

  Next, the data acquisition / recording system unit 390 will be described in detail with reference to FIG. In FIG. 9, an aircraft 301 has an aircraft floor hole 397 for installing an aerial photographer below the fuselage. It is preferable to install the digital camera assembly 360 in this hole so as not to protrude outside the apparatus. In the example of FIG. 9, a structure that is suspended from the aircraft floor hole 397 by a stable platform device 395 described later is employed. Thus, in the stable platform control system 394 and the stable platform device 395, regardless of the attitude of the aircraft 301, the digital camera assembly 360 is always directed directly below the ground, and the direction is fixed in a specified direction. It should be noted that the stable platform device 395 may be omitted if the pitch and roll of the aircraft can always be controlled within 5 °. IMU is an acronym for English representation of inertial measurement device. By placing the IMU 396 on the stable platform device 395, the posture of the digital camera assembly 360 can be measured. The top surface of the digital camera assembly 360 may protrude above the aircraft floor 398.

  The data acquisition and recording system unit 390 stores an IMU 400 for observing the attitude of the aircraft 301, an aerial imaging control system 393 including a program for controlling the digital camera 350 and processing imaging data, and a large amount of data including image data. An aerial image primary file 102 composed of a memory device such as a capacity disk device and an aerial image acquisition plan file 101 storing shooting points for issuing shooting commands to the digital camera 350 are installed in the apparatus. A GPS380 antenna for measuring the position of the aircraft 301 is provided at a place where the field of view to the outside of the aircraft can be opened.

  FIG. 13 shows the flow of information between the components of the data acquisition / recording system unit 390 of FIG. FIG. 14 illustrates a processing flow of the imaging control system 393. The imaging control system 393 is configured using a CPU. The imaging control system 393 periodically acquires the attitude data 450 of the stable platform device 395 from the IMU 396. When the stable platform 395 is operating normally, it always maintains a constant attitude with respect to the inertial space regardless of the attitude of the aircraft 301. The IMU 400 fixed to the aircraft 301 periodically sends aircraft attitude data 451, and the GPS 380 periodically sends GPS antenna position data 452 to the imaging control system 393. The processing of the imaging control system 393 is described in detail in FIG. 14, but the contents of the aerial image acquisition plan file 101 and the obtained GPS antenna position data 452 and aircraft attitude data 451 are sequentially processed in processing block 460 and processing block 461. The route number is identified, the route number is identified, and from the end of shooting of the route number to the departure of the route, it is sequentially determined in processing block 462 whether it is passing the nearest point of the shooting point of the designated route and within the shooting allowable range 440 At the timing closest to the shooting point, a shooting command 457 is sent simultaneously to the digital cameras 350 constituting the digital camera aggregate in processing block 463, and at the same time, the aircraft data portion 472 in the aerial image primary data file shown in FIG. 15 is processed. Write at block 464.

  The digital camera 350 can have 32 GB of memory at the time of the present invention. Therefore, the captured aerial image data 455 can be held until at least one route number is completed. Aircraft 301 makes a U-turn flight until it finishes one route number and enters the next route number, but during this time, no image is taken, so aerial image data 455 in digital camera 350 is used as the aircraft. It is stored in the memory inside the digital camera 350 by transferring it to the aerial image primary file 102 composed of a memory device such as a large-capacity disk device installed in the processing block 465 or by replacing the memory held by the digital camera 350 itself. Delete the image information.

  FIG. 15 shows a configuration example of the aerial image primary file 102. An image header portion 470 and an image data portion 471 are prepared for each photographed image. The camera ID in the header part is a number for identifying each digital camera 350a-i constituting the digital camera assembly 360. The image data ID is an identification number assigned so that the images can be distinguished from each other. The shooting date / time is the shooting time corresponding to the aircraft data portion 472. In the aerial image database generation / registration processing 120, the digital data 350a is integrated. Used to calculate the optical axis direction of ~ i. The image data portion 471 is not particularly processed at this stage. Further, since the camera parameter of the image header section 470 is normally fixed during flight, the same set value is written.

  The aerial image database generation / registration system 120 will be described in detail with reference to FIGS. FIG. 16 describes the overall processing flow. In processing block 480, the aerial image primary file 102 accumulated during the day's flight in the aerial image acquisition system 100 is sequentially processed for all images after the flight is completed. To do. In a processing block 481, a header of the image header portion 490 of the aerial image database of FIG. 17 is created. At this time, as the header to be created, the header of the image header portion 470 of the aerial image primary file 102 in FIG. 15 can be directly copied. For the processing block 482, the camera parameters of the image header portion 470 of the aerial image primary file 102 are usually transferred as they are.

  In the processing block 483, the process of calculating the shooting parameters of the image header part 490 for each image will be described in detail below. The camera position can be defined by latitude, longitude, and altitude. In general, when the position measurement by the onboard aircraft GPS 380 is DGPS, an accurate position may not be obtained as it is because the flight altitude is low. If the position measurement by GPS380 is GPS, errors may be included, so coordinate correction data is obtained from a nearby DGPS station measured at the same time as the flight time after landing and corrected according to the shooting date and time. be able to.

The shooting parameters are calculated using linear algebra. Hereinafter, a description will be given with reference to FIG. The mathematical formulas written in brackets in the specification follow normal mathematical notation, but in the text, there are restrictions on the description of symbols, so the following principles are followed. A three-dimensional or two-dimensional point is written in uppercase Roman letters and has a three-dimensional or two-dimensional component. If there is a change in time, add () after it and write time information in it. In mathematical expressions described as images, vectors are written in italic capital roman letters, written in capital roman letters in the text, and "(vector)" is added before the letters when identification is necessary. A 3D or 2D point can also be expressed as a vector relative to the origin. In the mathematical expression described as an image, the matrix is written in bold uppercase Roman letters, written in uppercase Roman letters in the text, and "(matrix)" is added before the letters when identification is necessary. For the angle, Greek letters are used. For symbols indicating planes, lines, etc., the descriptions of planes, lines, etc. are added before the symbols. A unit vector in the Cartesian coordinate system pointing to the horizontal direction in the north is defined as (vector) N, which is the reference axis. The coordinate transformation by pitch, yaw, and roll of the airframe based on (vector) N is defined as (matrix) A. Further, the position of the aircraft data portion 472 in the aerial image primary file 102 is set to X in the orthogonal coordinate system, and the position deviation from the GPS antenna 380 to the digital camera assembly 360 in the body is set to (vector) D in vector expression. Further, the optical axis of each camera (represented by the suffix k = ai) and the four corner direction vectors with respect to the reference axis (vector) N are expressed as unit vectors as follows. The following are constants, which are determined by the focal length of the camera and the mounting direction of the digital cameras 350a to 350i of the digital camera assembly 360.
Image center optical axis direction vector (normalized) G _kc = ^t (G _kcx G _kcy G _kcz )
_Captured image lower right azimuth vector (normalized) G _k1 = ^t (G _k1x G _k1y G _k1z )
_Captured image upper right edge direction vector (normalized) G _k2 = ^t (G _k2x G _k2y G _k2z )
_Captured image upper left azimuth vector (normalized) G _k3 = ^t (G _k3x G _k3y G _k3z )
_Captured image lower left azimuth vector (normalized) G _k4 = ^t (G _k4x G _k4y G _k4z )
, The shooting position S is

Thus, the shooting position (latitude, longitude, altitude) shown in FIG. 17 is obtained. The normalized image vector (X, Y, Z) is obtained from AG _kC by (matrix) A and (vector) G _kC .

The normalized vector in the direction of the image 4 with the attitude transformation (matrix) A relative to the aircraft reference direction is used for each camera using (matrix) A and (vectors) G _k1 to G _k4 , respectively. _k1, AG _k2 (AG _k3, AG _k4 . Terrain data on the surface is given to the grid points on the surface by latitude, longitude, and altitude, so that the nearest three adjacent points form a triangle. The vertex coordinates are T ₁ , T ₂ , T _3, and T _c is the coordinate at which the plane of the triangle T ₁ T ₂ T ₃ and the (vector) G _kc intersect, where x represents the outer product,

Then, it can be said that T _c is inside the triangle T ₁ T ₂ T ₃ . That is, they intersect. If the image vector (normalized) G _kc from the shooting position S = (x, y, z) intersects the triangle T ₁ T ₂ T ₃ at T _c

Here, a and b are arbitrary scalar constants. Here, R is the image distance of the shooting parameter in FIG. The image distance is the distance from the shooting position until the shooting optical axis crosses Terrain 580. here,

Then,

Thus, the center point coordinates (latitude, longitude, altitude) on Terrain of the shooting parameters of FIG. 17 are obtained as _Tc . However, it is checked by calculating an outer product whether it is inside the triangle T ₁ T ₂ T ₃ .

Similarly, by calculating for (vector) G _k1 , G _k2 , G _k3 , G _k4 instead of (vector) G _kc, the nearest right coordinate (latitude, longitude, altitude) on Terrain, farthest right on Terrain The coordinates (latitude, longitude, altitude), the farthest left coordinate (latitude, longitude, elevation) on Terrain, and the latest left coordinate (latitude, longitude, elevation) on Terrain are obtained. Next, the frame rotation angle of the shooting parameter shown in FIG. 17 is obtained. If the attitude of the aircraft is the reference direction, the horizontal line passing through the optical axis of each digital camera frame is parallel to the horizontal line of the earth, but if the coordinate transformation by pitch, yaw and roll of the aircraft is (matrix) A, A rotation angle is generated from the horizontal line. The horizontal line of the screen frame when the attitude of the aircraft is in the reference direction is given below.

If the coordinate transformation by pitch, yaw, and roll of the airframe is (matrix) A, the horizontal line of the screen frame is given below.

Therefore, the frame rotation angle is

It becomes. Here, “·” represents the inner product, and the portion surrounded by two vertical lines represents the norm.

  Next, in the processing block 484, processing for removing the influence of airborne haze is performed. In the aerial three-dimensional video generation system 150 according to an embodiment of the present invention, images from various directions in a city are expressed using original images taken from different viewpoints. For this reason, if a shadow due to direct sunlight is present in the original image, the shooting time must be different for each original image, which makes it extremely difficult to see when the moving image is synthesized. In order to avoid this, it is preferable to capture an aerial image by shooting from below the cloud on a cloudy day without direct sunlight. For this reason, in the aerial image acquisition plan file 101, it is preferable to set the flight altitude to 700 to 800 m, which is lower than the altitude at which cumulus clouds or stratus clouds appear. Since shooting in cloudy weather is a condition where an aircraft cannot be used for aerial surveying, there is an advantage that surplus time of the aircraft can be used at low cost. On the other hand, since there is a lot of moisture in the air, there may be a drawback that the visibility is shorter than that in fine weather.

  Moisture is reflected and diffused in a straight line as minute water droplets, so that influence occurs depending on the passage distance in the air. In particular, in the case of oblique aerial photographs, the influence of moisture increases as the off-nadir angle increases. FIG. 18 is a diagram showing the relationship between the image and the field of view in the defocusing effect removal processing, and the lower end of the photographed image 495 is the closest portion of the photographed ground surface portion 499 in the foreground portion a496. The upper end portion of the photographed image 495 is a distant view portion 498 and is located at the farthest portion of the photographing ground surface portion 499. As the distance through which light passes through the air from the foreground portion a496 toward the far view portion 498 increases, the scattering of light by minute water droplets increases, the contrast decreases, and white turbidity occurs. As a result, as shown in FIG. 19 (a), when the histogram of the brightness of the pixels of the image in the absence of minute water droplets (moya) is a normalized histogram 500 distributed from the luminance lower limit value to the luminance upper limit value 507, In the portion a496, a foreground portion a histogram 501 is formed, and the width of the upper limit value and the lower limit value becomes narrower, and the median value shifts to the white side, that is, the higher luminance.

This tendency is most noticeable in the distant scene portion histogram 502, the value width 505 which is the difference between the histogram upper limit value 504 and the histogram lower limit value 503 is minimized, and the median value 506 is maximized. In the captured image 495 of FIG. 18, the range for which the histogram is taken is the entire range in the horizontal direction of the screen, and the vertical direction is a strip with a constant width such as the foreground portion a496 and the foreground portion b497, and is directed upward while shifting one line at a time. When a histogram of pixel brightness is calculated, characteristics as shown in FIG. 19B are obtained. Although the density of the haze may vary depending on altitude, it may be assumed that there is no difference in density in the horizontal direction, so the value width fitting curve 513 may be approximated as monotonically decreasing and the median fitting curve 512 may be approximated as monotonically increasing. Since these curves have unknown distribution due to the height of the haze, each pixel value is corrected with a correction coefficient of a value width and a median value obtained from the fitting curve for each row after being obtained by fitting from an actual measurement value. The brightness correction of each pixel is performed for each horizontal line (Equation 9)
Normalized histogram median + (pixel brightness-median fitting value)
* Obtained by normalized histogram value width / value width fitting value. The processing described above is the processing described from processing block 520 to processing block 524 in FIG. FIG. 21 shows an example of the processing result of the defocusing effect removal processing of the present invention. As a result of processing the pre-moisture removal image 530, a post-blur removal image 531 is obtained.

Next, the processing block 485 stores the results obtained in the processing blocks 481 to 484 in the corresponding image data portion of the aerial image database 143 shown in FIG. In the final processing block 486 of the processing flow of the aerial image database generation / registration system of FIG. 16, an index is searched for the purpose of searching the aerial image data at high speed for moving image generation using the aerial image data registered in the processing block 485. Generate. The contents of the processing block 486 are described in more detail in the aerial image index generation registration processing flow of the aerial image database registration system of FIG. Hereinafter, a description will be given with reference to FIG. The contents of the processing block 540 will be described with reference to the structure and term definition of the aerial image index mechanism in FIG. The image taken at the shooting position 550 includes shooting positions (latitude, longitude, altitude) 550, normalized image vector G = ^t (XYZ) 551, frame rotation angle 553, as shooting parameters of the aerial image database 133 in FIG. Image distance 552, Terrain center point coordinates (latitude, longitude, elevation) 554, Terrain latest right coordinate (latitude, longitude, elevation) 555, Terrain farthest right coordinate (latitude, longitude, elevation) 556, Terrain top The far left coordinate (latitude, longitude, elevation) 557 and the latest left coordinate (latitude, longitude, elevation) 558 on Terrain are already stored, and FIG. 23 illustrates this. The image photographed at the photographing position 550 captures the ground image range 549. Since the latitude and longitude are defined in the ground surface and Terrain 580, if meshing is performed every second, a cell of 30 m in the latitude direction and 25 m in the longitude direction can be formed around 35 ° latitude in the Tokyo region. If meshed every 0.4 seconds, a cell of 12 m in the latitude direction and 10 m in the longitude direction can be configured. Considering the 65km square city area, this number of cells is only 35M cells. The ground cell 559 included in the ground image range 549 can be easily obtained because the four end points of the ground image range 549 are known from the latitude and longitude.

  For each locality cell, it is necessary to be able to search for an image within a sufficiently small solid angle range i221 in all directions of the celestial sphere 222 shown in FIG. For this purpose, as shown in FIG. 24 (a), a zenith vector 570 in which the celestial sphere is directed vertically upward from the center of the ground cell 559 is defined. Hereinafter, the celestial spheres are cut into concentric circles based on the zenith vector. Define Zenith Index I 572. The zenith index i is defined sequentially as i = 0,1,2, ..., i = 0 is the zenith vector, and as i increases, the angle between the zenith vector increases with the same width, and the same concentric circle Defined from the zenith to the horizon. A solid angle cell center vector is defined from the surface cell 559 through the zenith index 572. The solid angle cell center vector 574 is always defined on the zenith index 572 at a constant azimuth angle interval.

  In FIG. 24A, the solid angle cell 573 corresponds to the sufficiently small solid angle range I 221 in FIG. 2, and is a basic unit for image retrieval from the ground cell 559. In FIG. 24 (b), the solid angle cell 573 is a cone having a half apex angle of 20 degrees with respect to the solid angle cell center vector 574, and is defined as a cone in which a vector whose solid angle is 0.4 steradian or less is contained. FIG. 24 is a view of the ground cell 559 viewed from the zenith direction. Solid angle cell 573 is defined adjacent to and overlapping each other with respect to zenith index I 572 and fills the entire circumference of zenith index i 572. The angle formed by the solid angle cell center vector 574 passing through the zenith vector 570 and the zenith index I 570 is called a zenith separation angle 571, and is a numerical value defining the zenith index I 572.

  FIG. 25 shows the index structure of the solid angle cell 573. FIG. 25B is a view of the ground cell center point 575 looking down from the zenith direction. A solid angle cell address (i, j) with a two-dimensional subscript is defined, i indicates a zenith index i572, and j indicates a solid angle cell in the jth direction on a concentric circle of the zenith index i. Solid angle cell address (0,0) Solid angle cell center vector of 584 is a zenith vector 570, surrounding this, a solid angle cell address (1,0) 585, solid angle cell address ( 1,1) 586 and solid angle cell address (1,2) 587 are defined below until the entire circumference of zenith index 1 is filled. Outside the zenith index 1581, along the zenith index 2, solid angle cell address (2,0) 588, solid angle cell address (2,1) 589, solid angle cell address (2,2) 590 are the zenith index 2 It is defined until the entire circumference is filled. The solid angle cells 573 overlap each other and fill the celestial sphere so that there is no gap. The solid angle of the solid angle cell is set so that three-dimensional moving image generation is performed smoothly. The number of solid angle cell addresses increases as the zenith index approaches the ground surface. In this way, for all the ground cells, an image obtained by photographing the ground cell can be searched by the direction on the celestial sphere.

  FIG. 26 shows the structure of the aerial image index relation table and its correlation. The structures of the zenith index and the zenith cell address are the same regardless of the ground cell, and this structure is defined by the zenith index parameter table ZNPRMT597 and the azimuth index parameter table DRPRMT598. In this structure, the zenith index parameter table ZNPRMT597 defines the zenith separation angle 571 with respect to the zenith index i572 sequentially from the zenith index 0, and the vertical direction component when the center vector 574 corresponding to the zenith index i572 is a normalized vector. Store as vertical component. Since the number of solid angle cells included in the zenith index i 572 increases as it approaches the horizon from the zenith, it is defined as the azimuth number NDRi corresponding to each zenith index. The direction index parameter table DRPRMT 598 defines the azimuth X component and the azimuth Y component of the solid angle cell center vector for the solid angle cell address corresponding to the azimuth number NDRi defined corresponding to the zenith index. The vector norm is normalized so as to be 1 together with the vertical component described in the zenith index parameter table ZNPRMT597.

Next, the structure of the image data index table corresponding to the ground cell will be described. The ground cell position index table TCINXT 600 in FIG. 26 is defined for all the ground cells 559, and the zenith index table address ADDR _ij 603 is designated corresponding to the ground cell. The subscript of ADDR _ij indicates that the longitude index LONINX 601 is i and the latitude index LATINX 602 is j corresponding to the position on the ground surface. The zenith index table address ADDR _ij (603) indicates the zenith index table ZNINXT 604 corresponding to the ground cell ij. In the zenith index table ZNINXT, an azimuth index table address ADDR _ijk (605) is defined corresponding to the selected zenith index k, and the azimuth index table DRINXT 606 is defined accordingly. Here, since there are the number of azimuths designated by the zenith index parameter table ZNPRMT597, if the azimuth index is m, ADDR _ijkm is designated as the address of the image address table 608. The image address table thus selected indicates the number of image data and the image data address included in the solid angle cell of the solid angle cell address (k, m) with respect to the ground cell ij. In this way, if the correlation between the aerial image index and the association table is defined according to FIG. 26, it is possible to retrieve an image taken from an arbitrary direction in the air from the position of the ground surface.

Based on the above description of the image data index mechanism, an aerial image index generation registration process flow of the aerial image database registration system shown in FIG. 22 will be described below. In processing block 540, if there is one image as shown in FIG. 23, all the ground cells included in the ground image range 549 are obtained. In processing block 541, processing is sequentially performed for all the ground cells obtained in processing block 540. In a processing block 542, a vector for predicting the photographing position 550 is obtained from the ground cell center coordinates. In the processing block 543 and the processing block 544, a solid angle cell address (k, m) including a vector for estimating the shooting position 550 from the ground cell center coordinates is calculated by the inner product calculation from the definitions of the zenith index parameter table ZNPRMT597 and the orientation index parameter table DRPRMT598. Ask. Based on this information, the number of image data in the image data table ADDR _ijkm is increased by 1 in a processing block 545 and an image data address is additionally written.

  FIG. 27 shows a configuration example of a visual field by three types of digital cameras having different focal lengths. In the generation of a three-dimensional moving image of a cityscape according to the present invention, the original image when the depression angle is small becomes an image with a large off-nadir angle, so that the image is inevitably far from the aircraft. In order to generate a moving image when the depression angle is small without reducing the resolution, it is necessary to consider that the resolution is not deteriorated when the image is captured when the off-nadir angle is large. For this reason, as shown in FIG. 27, as the off-nadir angle increases, the focal length of the telephoto lens is increased to maintain the resolution, and at the same time, the three types of off-nadir angle digital cameras are employed and the off-nadir angle increases. This is an example in which the focal length is increased to make the ground resolution uniform. In FIG. 28, a digital camera 350a is used for direct-point shooting, and eight digital cameras 350b to i are arranged with the same small off-nadir angle on the radiation, and the digital cameras 620j to y are arranged on the outside. Sixteen units are arranged on the radiation with the same large off-nadir angle. In recent years, digital cameras have been significantly reduced in size and performance, and can be sufficiently accommodated in the existing aircraft floor hole 398 for aerial photography cameras. FIG. 29 shows an example of the photographing range of a digital camera assembly using three types of digital cameras having different focal lengths. As a typical example, the digital camera 350a is oriented directly below with a 35mm film equivalent focal length of 50mm, the digital cameras 350b to i are oriented with a focal length of 70mm and an off-nadir angle of 35 °, and the digital cameras 620j to y are equivalent to a focal length of 105mm. There is a combination of off-nadir angle 60 ° orientation.

  FIG. 30 shows the simplest configuration example of the stable platform device 395 for mounting on an aircraft. The stable platform device 395 is installed in a shape that covers the aircraft floor hole 397 and covers the upper part, and includes a fixed part 635, a vertical movement part 636, and a rotating part 637 from the bottom. The rotating unit 637 has a rotating table structure, and an IMU 396 for posture detection is installed in close contact with the rotating table 637. A digital camera assembly 360 is suspended under the rotating unit 637 from the aircraft floor hole 397. All downward views can be accommodated in all digital cameras 350a-i without projecting outside. The stable platform device 395 can also shoot a digital camera aggregate 360 or 621 in the same manner using the digital cameras 350a-i and the digital cameras 620j-y. Note that the method of joining the rotating portion 637 and the digital camera assembly 360 or 621 needs to be fixed with rigidity. The stable platform device 395 is intended to maintain a constant attitude against the aircraft attitude disturbance, but the aircraft attitude disturbance that can be dealt with is a maximum of 7 to 10 degrees in pitch angle and roll angle, and the maximum in yaw angle. It is sufficient to assume 20 °. The fixing part 635 is fixed to the aircraft floor 398. The vertical movement unit 636 is held via the vertical drive mechanisms A to D641 to 644. The vertical drive mechanisms A to D 641 to 644 move up and down independently to correct the pitch angle and roll angle disturbances. The vertical drive may be a hydraulic mechanism or a conversion method of a rotary motion by a worm gear into a vertical motion. Further, a rotating part 637 is mounted on the vertical movement part 645 so as to be held by the bearing mechanism 645. The rotation drive mechanism 640 is fixed to the up-and-down moving part 636, and the rotating part 637 is installed in contact with the outer extension part. It is only necessary to transmit a rotational motion between the rotation unit 637 and the rotation drive mechanism 640, and the rotation unit 637 is rotated to cancel the disturbance to the yaw angle of the aircraft. FIG. 31 shows a signal information flow of the stable platform device. The stable platform control system 394 periodically inputs the pitch angle, yaw angle, and roll angle from the IMU 396, calculates the deviation from the reference value, and cancels the deviation of the pitch angle and roll angle from the vertical drive mechanism A to D641 to 644. To control the vertical movement 649. In order to cancel the deviation of the yaw angle, the rotation mechanism 640 is controlled to perform the rotation motion 650.

Next, aerial moving image generation processing in the cityscape three-dimensional moving image amusement system will be described with reference to FIGS. The concept of aerial moving image generation by morphing and description of variables will be described with reference to FIG. 32, and the overall processing flow will be described based on the aerial moving image generation processing flow in the cityscape three-dimensional moving image amusement system of FIG. In FIG. 32, the viewpoint moves with time t along the viewpoint path P (t) 270, and the ground surface is moved along the target trajectory T (t) 280 from the position on the viewpoint path P (t) 270 at time t. from ₁ to _{t 4, t i, V (} t 4) from line-of-sight vector V _{(t 1),} taken in a V _{(t i).} In the present invention, as a method of generating a three-dimensional moving image of the city landscape for the target trajectory T (t) 280 from the viewpoint route P (t) 270, the viewpoint P (t ₁ ) at times t ₁ , t ₂ , and t ₃ is used. The aerial image _m-1 666 that is close to the line of sight of 675, the viewpoint P (t ₂ ) 676, and the viewpoint P (t ₃ ) 677 and has little parallax is searched from the aerial image database 130, and the line of sight at times t ₁ , t ₂ , and t ₃ Between the vectors V (t ₁ ) 681, V (t ₂ ) 682, and V (t ₃ ) 683, the difference in parallax from the image vector G _m-1 672 is corrected by the morphing process for the aerial image _m-1 666. Connected to generate a smooth moving image. Switch the aerial image _m 667 to the original image is determined to be less parallax due to the difference towards the aerial image _m 667 is perspective than aerial image _m-1 666 from time t ₄ of the line-of-sight vector V (t ₄₎ 684, 3D smooth cityscape by correcting the difference in parallax between aerial images _m 667 to time viewing vector V (t ₄₎ of t ₄ 684 after the next original image is selected by the morphing process for the aerial image _m 667 Video generation is performed.

  FIG. 33 shows the above processing in a processing flow. In a processing block 690, the moving direction, moving speed, position, and line-of-sight vector to be calculated as the next moving image frame (next frame) from the graphic user interface system 180 are shown. take in. In a processing block 691, the viewpoint position of the next frame, the line-of-sight vector, and the target point coordinates on the ground surface are calculated based on this value. In processing block 692, it is determined whether the ground index position of the next frame is the same as that of the current frame based on the center point of the current frame and the next frame. If they are not identical, it is necessary to re-determine the optimal original image and go to processing block 694. If they are the same, it is determined in processing block 693 whether the line-of-sight vector of the next frame is within the same solid angle cell as the current frame. This determines whether the direction of the line of sight has changed to some extent from the current frame. If it is determined that the line of sight does not fall within the same solid angle cell, it is necessary to re-determine an optimal original image in processing block 694. If it is determined that the cells are in the same solid angle cell, the original image of the current frame is continuously used.

The processing contents of the processing block 694 are shown in more detail in the processing flow of the original image search in the aerial moving image generation of FIG. 34 and the original image selection logic diagram of the three-dimensional moving image generation processing of FIG. In the processing block 700, a ground cell address (i, j) existing at the center of the next frame is obtained, and a line-of-sight vector connecting the center point and the viewpoint of the next frame is created and normalized. In processing blocks 701 to 703, a solid angle cell address (k, m) is obtained from the normalized line-of-sight vector with reference to the zenith index parameter table ZNPRMT597 and the azimuth index parameter table DRPRMT598, and is combined with the ground cell address (i, j). Thus, the address ADDR _ijkm of the image address table is obtained. Since the original image candidate indicated by the first data of the image address table exists at the same solid-angle cell address for the same ground cell address, the visual field from the viewpoint P (t _i ) from among the processing block 704 and the processing block 705 It is determined whether the original image has taken all the four corner points. In FIG. 35A, the four end points T _i1 715, T _i2 716, T _i3 717, and T _i4 718 on the ground surface of the field of view of the viewpoint P (t _i ) are the original image shooting position S _A 710 and the original image shooting. Whether or not the image is taken from the position S _B 711 and the original image shooting position S _C 712 is determined by the processing block 705, and can be easily performed by vector cross product calculation. In the example of FIG. 35B, only the images from the original image shooting position S _A 710 and the original image shooting position S _B 711 are the four end points T _i1 715, T on the ground surface of the viewpoint P (t _i ). _{All of i2} 716, T _i3 717, and T _i4 718 are captured in the image. After obtaining all the original images satisfying the conditions of the processing block 705, the shortest image with the image distance closest to the ground cell address (i, j) is selected as the original image in the processing block 706. The image distance is calculated as one data of the shooting parameter of the aerial image database shown in FIG. 17, and selecting the original image with the shortest image distance is a procedure for eliminating an obstacle that blocks the view between the viewpoint and the target. is there.

Next, returning to FIG. 33, the contents of the morphing process of the processing block 697 will be described with reference to FIG. Note that by omitting the processing block 697, only the original image can be switched and displayed without performing morphing. When the original image shooting position S _m 671, the image vector G _m 673 (norm = 1), and Terrain (ground surface) are determined with respect to the original image _m , the intersection Z _{mc of the} image vector G _m 673 and the DEM and the four corners of the image Terrain intersections Z _m1 , Z _m2 , Z _m3 , and Z _m4 are determined. As shown in FIG. 37, because generally _{Z m1, Z m2, Z m3} , Z m4 are not coplanar, the midpoint Z _m23 street the Z _mc Z _m2, Z _m3, in the Z _m1, Z _m4 A vector parallel to the vector passing through the point Z _m41 is defined as Z _mV ,

The midpoint Z _m12 street the Z _mc Z _m1 and Z _{m @ 2,} a parallel vector and vector passing through the middle point Z _m34 of Z _m3 and Z _m4 is defined as Z _mH.

A surface that passes through Z _mc and is stretched by (vector) Z _mV and (vector) Z _mH is called a Terrain plane (plane) Z _m . The (plane) Z _m perpendicular is (vector) _nm (norm = 1). Through Z _mc, a plane perpendicular to the image vector _{G m} of (plane) N _m. (Plane) N _m is an image (field of view) photographed from the original image photographing position viewpoint S _m 671. There is a transformation to (plane) Z _m- > (plane) N _m , which is called Φ _A. This is a photographing action of the original image _m . For the viewpoint P _i , when the viewpoint coordinates P _i , the line-of-sight vector V _i (norm = 1), and the Terrain plane Z _m (common) are determined, similarly, T _iC and T _i1 , T _i2 , T _i3 , T _i4 are determined. . A plane passing through T _iC and perpendicular to the line-of-sight vector V _i is referred to as N _i . The plane T _mi constitutes a part of Z _{m in} the Terrain plane Z _m with a visual field based on the viewpoint P _i . Viewpoint _{P i} of the field of view (plan) N _i and (plane) T _mi is uniquely determined once the viewpoint _{P i} and (vector) _{V i} and the viewing angle theta _i. The mapping of (plane) T _mi to (plane) N _m is uniquely determined if the original image shooting position viewpoint S _m , (vector) G _m and the angle of view θ _m of the original image m are determined. (Plane) N _i- > (Plane) T _mi- > (Plane) N _m corresponding to θ _i of (Plane) N _i becomes a pixel corresponding to θ _m of (Plane) N _m If matched, the morphing algorithm is completed.

Generally X arbitrary point on the Terrain plane Z _m is

It is expressed. Here is the normal condition,

Substituting

Is obtained. Here, represents a vector dot product. Intersection T _iC centerline V _iC a Terrain plane Z _m of the field of view, T _iC is

Substituting this for satisfying

Than

And

Thus, the coordinates of the image center point of the viewpoint P _{i on} the Terrain plane Z _m are obtained. Similarly, normalized line-of-sight vectors at the four corners of the field of view P _i are V _i1 , V _i2 , V _i3 , and V _i4, and the intersections T _i1 , T _i2 , T _i3 , and T _i4 of these and the Terrain plane Z _m . The coordinates in the Terrain plane Z _m can be obtained. Further obtaining the conversion of the N _m plane of the coordinate. An arbitrary point X on the Terrain plane Z _m is

Expressed by the perpendicular condition,

From the four corner normalized eye vectors for V _i1 , V _i2 , V _i3 , V _i4 ,

However, k = 1,4 is satisfied

Than

And

Thus, the coordinates of the four corner points of the image of the viewpoint P _i on the Terrain plane Z _m are obtained. By associating the coordinates on the plane Z _m with the coordinates on the plane N _m , it is possible to know which point on the plane Z _m that is the original image should be displayed on the screen of the viewpoint P _i . (Plane) The coordinates T _iC , T _i1 , T _i2 , T _i3 , T _i4 (collectively T _ik , k = 1,2,3,4) of the center and four corner points of the viewpoint P _i on Z _m , c) corresponds to Q _iC , Q _i1 , Q _i2 , Q _i3 , Q _i4 (collectively Q _ik , k = 1, 2, 3, 4, c) on (plane) N _m . (Plane) When the horizontal line of the screen on the N _m plane is (vector) N _H , and the vertical line is (vector) N _V , an arbitrary point Y on the N _m plane is relative to the arbitrary constants a and b

It is represented by Since the image vector G _m is perpendicular to (vector) N _H , (vector) N _V

Q _ik and T _ik are on the same line as seen from the image shooting position S _m

So

And take the inner product with G _m

Because it becomes

It becomes.

Since T _ik has already been obtained, correspondence with the original screen m of the field of view of the viewpoint P _i is now complete.

Next, a conversion for moving (vector) G _m to the z-axis is obtained. here,

Then, it is rotated around the z-axis and placed on the yz plane, and then rotated around the y-axis to coincide with the z-axis. The rotation around the z axis is

Given in. Next, the rotation around the y axis is

Given in. Combining both rotations gives

It becomes. If the transformation that moves G _m to the z axis is applied to the intersection point Q _ik (k = 1,2,3,4, C) of the plane N _m , Q _ik (k = 1,2,3,4, C) Is converted to a point on the xy plane. This corresponds to the position on the screen of the original image m. That is morphing.

(Embodiment 2)
In the first embodiment, when the visual field image generation is performed without using the height information of the ground structure, the viewpoint position is determined based on the height of the ground structure from only the viewpoint position, the line-of-sight direction, and the position information of the photographed image. In principle, it is impossible to eliminate the parallax due to the difference. This will be described with reference to FIG. That is, a view image A 2220 and a view image B 2221 in which a point P 2212 having a height h 2215 from the ground surface is seen at the ground surface point O 2213 of the ground surface 214 from the viewpoint A 2210 and the viewpoint B 2211 are considered.

Image O _A 2218 in view video A 2220 of ground point O 2213 corresponds to the image O _B 2219 in view video B 2221. On the other hand, next to the image P _A 2216 is on view video A 2220 P 2212 points at the location of the height h 2215 from the surface 2214 as viewed from the viewpoint A 2210, it overlaps with the ground point R 2223. The place on the view image B 2221 of the point P 2212 viewed from the viewpoint B 2211 is an image Q _B 2225 and overlaps with the ground point Q 2222. Due to the difference in viewpoint, the position of the point P 2212 on the view image A 2220 and the view image B 2221 is different. This means that the feature on the surface of the ground surface jumps due to parallax if the viewpoint (shooting point) of the actual image to be used is different, and the view image shown in FIG. This means that each time the original image (aerial image i 260, aerial image i + 1 261) is switched, the image moves discontinuously. Such discontinuity on the visual field image is a phenomenon caused by the method of the first embodiment, and this embodiment solves the phenomenon while using the method of the first embodiment.

  FIG. 41 shows the configuration of a field-of-view video information generation device according to Embodiment 2 of the present invention. The image display system 2100 is a part according to the first embodiment, and the photographed image search engine 2104 performs the photographed image index mechanism 2102 according to the set input value of the viewpoint / line-of-sight input mechanism 2122 for inputting the viewpoint position and the line-of-sight direction including the keyboard or mouse. Referring to the real image database 2101, the optimal image (the viewpoint and line of sight at the time of shooting are closest to the viewpoint position and line of sight) is retrieved, and the visual field image generation system 2105 generates a visual field image to be output to the display device 2121. To do. One of the features of the present invention is an image switching smoothing system 2110, which solves the discontinuity on the visual field image that occurs when the real image used by the visual field image generation system 2105 is switched in the image display system 2100. . That is, discontinuity on the visual field image occurs at the point of interest position 2126 input by the attention point input mechanism 2123 of the operator 2130 including the mouse or eye tracker on the screen of the display device 2121 at the time of switching the real image screen. Do not. The image comparison information database 2111 and the visual field image position correction system 2112 are configured in detail below.

  42 and 43 are diagrams illustrating the function of the image switching smoothing system 2110. FIG. In FIG. 42, a relationship when the reference point 2233 is viewed from the line-of-sight direction 1 2231 and the line-of-sight direction 2 2232 so as to include the building 2239 in the field of view is shown as a building and line-of-sight direction relationship 2230. In this case, since the same building 2239 has a different viewpoint and line-of-sight direction, the image 2234 viewed from the line-of-sight direction 21 and a video 2235 viewed from the line-of-sight direction 22 appear different. FIG. 43 is a diagram for explaining the influence of the parallax on the position on the view image screen. Since the reference point 2233 is a point on the ground surface and 3D data is obtained from the DEM data 2103, as is well known in the 3D computer graphic technology, the position on the field-of-view image is accurately determined if the viewpoint and line of sight are determined. . The position of the reference point 2233 on the screen is determined in the image 234 viewed from the line-of-sight direction 21 in FIG. 43A and the image 2235 viewed from the line-of-sight direction 22 in FIG. When the photographed image to be used is switched while viewing the reference point 2233 from the viewpoint, as shown in FIG. 43 (c), the video 2234 and the line of sight viewed from the line-of-sight direction 21 due to parallax because there is no 3D information of the building 2239. The position of the image 2235 viewed from the direction 22 is generally shifted and displayed. In this case, for example, when the operator 2130 is gazing at the end point of the building 2239 as the attention point 2236, the view display shown in FIG.

  FIG. 43 (d) obtains the attention point of the operator 2130 by an input mechanism including a mouse cursor, a keyboard, or an eye tracker, and a visual field image is displayed at the attention point (in the case of FIG. 43 (d), the upper part of the building 2239). The position of the field-of-view video to be displayed on the display screen is shifted so as to match even when the original actual image is switched. It is an object of the present invention to provide smoothness at the time of screen switching of the view image by providing such a position adjustment function on the display screen, and the photographed image used in the technique of Embodiment 1 has three-dimensional information. This is a technology that eliminates the disadvantages of not doing so.

  FIG. 44 is a diagram for explaining a mechanism for realizing the position adjustment function on the display screen. The image switching smoothing system 2110 and the connection between the image display system 2100 and the image switching smoothing system 2110 are one of the features of the present invention, and the image switching smoothing system 2110 operates in synchronization with the image display system 2100. . The processing timing of each component is shown in the time chart of FIG. The column at the left end in FIG. 45 corresponds to each processing block in FIG. 44, and the row of each processing indicates processing timing on the time axis 2260. A curved arrow indicates that the process at the end of the arrow is started at the timing when the original process of the arrow ends.

  45 means that, in the image display system 2100, the photographed image search processing unit 2262, the photographed image development processing unit 2263, and the field-of-view image generation processing unit 2264 follow the viewpoint / line-of-sight input processing unit 2127. Are activated sequentially using. In the image switching smoothing system 2110, an image comparison information search processing unit 2251 and an image comparison information expansion processing unit 2257 are sequentially activated using the results of the previous stage based on the result of the real image search processing unit 2262 of the image display system 2100. . On the other hand, the comparison information region selection processing unit 2254 is activated together with the result of the attention point input processing unit 128, the image matching processing unit 2255 is activated, and the deviation calculation processing unit 2256 performs the deviation calculation processing based on the result. Is called. The result of the deviation calculation processing unit 2256 is combined with the result of the view image generation processing unit 2264 of the image display system 2100, and the processing of the screen output processing unit 2266 is performed. In one embodiment of the present invention, the image display system 2100 and the image switching smoothing system 2110 can be simultaneously processed in parallel as shown in FIG. 45, and are compatible with a recent multi-core CPU architecture.

  The display image 2124 displayed on the display device 2121 is updated every image update period 2261 in FIG. Information captured from the viewpoint / line-of-sight input mechanism 2122 is transmitted to the photographed image search engine 2104 as the viewpoint position / line-of-sight direction 2125 by the input processing unit 2127, and the photographed image database is referred to the photographed image index mechanism 2142 as image retrieval information. A live-action image is searched from 2143. The functions and processing of the photographed image search engine 2104 and the visual field image generation system 2105 have been described in the first embodiment. In the image switching smoothing system 2110, the image comparison information database 2250 corresponds to the photographed image database 2143 for each image and has the same structure. Therefore, the image comparison information database 2250 can be searched using the photographed image index mechanism 2142, and is stored in the image comparison information database 2250. The image comparison information search processing unit 2251 takes out the image comparison information data.

  The information in the image comparison information database 2250 (image comparison information) has the same structure as that of a live-action image, but since it is intended for use in image matching processing 2255 between images in which the viewpoint and the line-of-sight direction are close to each other, it can be efficiently shortened. The real image in the real image database 2143 is subjected to image processing so that image matching can be performed in time, and each real image is obtained by calculation in advance. FIG. 46 shows the structure of an example of such processing. A processing block 2300 indicates that all images in the live-action image database 2143 are sequentially processed. In a processing block 2301, a real-image image to be processed is extracted from the live-action image database 2143.

A processing block 2302 is a process for reducing the resolution of the photographed image to 1 / n for each of the vertical and horizontal (H, V). However, n is an integer of 1 or more. This process can reduce the calculation amount of the image matching processing unit 2255 to 1 / (n * n), but degrades the processing accuracy of the deviation calculation processing unit 2256 to 1 / n. It is necessary to determine n in consideration of processing capacity. The number of pixels in a live-action image is Hmax in the horizontal direction and Vmax in the vertical direction, and the brightness of each pixel in the live-action image is expressed as P (h, v, RGB). Here, h takes an integer value in the range of h = 0, Hmax-1 in the H (horizontal) direction coordinate, and v takes an integer value in the range of v = 0, Vmax-1 in the V (vertical) direction coordinate. . RGB is a distinction by color and takes an integer value of RGB = 0, 1, 2 and corresponds to three colors of R, G, and B. The brightness of each pixel in the 1 / n resolution image of the real image is expressed as Pn (hn, vn, RGB). Here, hn is an integer value in the range of hn = 0, Hmax / n-1 in the H (horizontal) direction coordinate, and v is in the range of vn = 0, Vmax / n-1 in the V (vertical) direction coordinate. Takes an integer value. RGB is a distinction by color. The operation to set the resolution to 1 / n is, for example,

Is required.

By making the processing block 2302 monochrome, the calculation amount of the image matching processing unit 2255 can be reduced to 1/3. However, the monochrome processing of the processing block 2302 is not necessarily performed. When the brightness of each pixel of a monochrome image is expressed by Pmn (hn, vn), the monochrome operation is, for example,

Is required.

  The histogram leveling of the processing block 2304 will be described with reference to FIG. In the image filter processing suitable for the image matching processing unit 2255, the resolution is changed to 1 / n, and the processing is performed on the monochrome photographed image 2310. The image filter 2311 performs the histogram leveling process described in the next paragraph while moving one pixel at a time from the upper left HV coordinate origin to the horizontal (H) axis 2312. When the right end is reached, the image filter 2311 is moved to the left end of the horizontal (H) axis 2312. Return and move one pixel to the vertical (V) axis 2313. In this state, while moving to the horizontal (H) axis 2312 one pixel at a time, the histogram leveling process described in the next paragraph is performed in the same way. Step through the entire screen until reaching the lower right corner of the live-action image 2310.

  Every time one pixel is moved to the horizontal (H) axis 2312 or the vertical (V) axis 2313, the following histogram leveling process is performed. In general, the luminance distribution of an image varies depending on the location of the image. Such bias is a luminance distribution of low spatial frequency components, and it is necessary for image processing to improve matching detection accuracy in advance to remove spatial frequency components lower than the size of the matching filter used in the image matching processing unit 2255. is there. For this purpose, the luminance distribution is leveled according to the size of the image filter 2311. The image filter 2311 has a size of m * m pixels. The image filter 2311 moves pixel by pixel while scanning the image from the upper left corner to the lower right corner of a real image 2310 (hereinafter referred to as a real image 2310) that has been changed to 1 / n and converted into monochrome. Each pixel of the live-action image 2310 in the image filter 2311 has a luminance value. When the luminance expression is 8 bits, it has a value from 0 to 255. FIG. 47B is a histogram that graphs the number of pixels by luminance with respect to m * m pixels in the image filter 2311. The luminance histogram axis 2315 is the number of pixels with respect to the luminance on the luminance axis 2314. FIG. 47B shows an example of the luminance histogram (distribution) of the pixels in the region where the image filter 2311 has moved on the photographed image 2310.

  An example of the luminance distribution in an area of the image filter 2311 range with respect to an arbitrary place of the photographed image 2310 is shown by a luminance histogram 2318. The luminance histogram 2318 is distributed between the luminance lower limit value 2316 and the luminance upper limit value 2317, but generally exists in a biased manner between the luminance lower limit value 2316 and the luminance upper limit value 2317 as shown in FIG. This is because there is a bias in brightness and contrast depending on the location of the screen, which hinders image collation. To solve this, histogram leveling processing is performed in processing block 2304. That is, the luminance histogram 2318 is extended along the luminance axis 2314 so that the luminance distribution in the region of the image filter 2311 range exists over the entire luminance lower limit value 2316 and luminance upper limit value 2317 as shown in FIG. . Note that the decompression is an example of the histogram leveling process of the processing block 2304, and any process other than the above may be used as long as it is suitable for image collation.

  The calculation result of the image filter 2311 stores the luminance value at the center of the area of m * m pixels as the result of the filtering process. Therefore, it is preferable that the number of m pixels on one side of the image filter is an odd number. Further, by adopting such a result storing method, the image filter 2311 is matched with scanning by moving the actual image 2310 pixel by pixel. FIG. 48 shows an example of luminance histogram leveling, FIG. 48 (a) shows a monochromatic actual image, and FIG. 48 (b) shows an example of the histogram leveling. The content of the processing block 2304 has been described above, but the result is stored in the image comparison information database 2250 in the processing block 2305.

  Returning to the description of FIG. 44, the description of the image switching smoothing system 2110 is continued. The viewpoint position / line-of-sight direction 2125 is taken in the row of the viewpoint / line-of-sight input processing unit 2127 in FIG. 45 and is fetched prior to the image update period 2261 and sent to the photographed image search engine 2104 of the photographed image search engine 2104. In this way, the most appropriate photographed image for the viewpoint position / line-of-sight direction 2125 is extracted from the photographed image database 2143. Since the image comparison information database 2250 is set to have the same structure as the photographed image database 2143, the image comparison information retrieval processing 2251 obtains the image retrieval information as image comparison information data retrieval information 2258 from the photographed image retrieval processing 2104. Thus, image comparison information data corresponding to the photographed image can be obtained from the image comparison information database 2250.

  In the comparison information area selection processing 2254, it is necessary to input the current display image comparison information 2253 and the next display image comparison information 2252. For this reason, the image comparison information retrieval processing 251 needs to read out the image comparison information data corresponding to the actual image to be displayed next in addition to the actual image of the current display. The viewpoint position and the line-of-sight direction can be predicted at this point in time, and using this predicted value, search information for a live-action image to be switched and displayed in the same process as the real-action image currently being displayed can also be obtained. The image comparison information search processing 2251 also searches and reads out the actual image to be switched and displayed from the image comparison information database 2250.

  As shown in FIG. 45, the image comparison information expansion process 2257 is executed subsequent to the image comparison information search process 2251. The shooting point position, shooting direction, viewpoint position / line-of-sight direction 2125 of the searched real image and the display screen are displayed. The image comparison information expansion processing unit 2257 obtains the currently displayed field image displayed on the display device 2121 from the field angle and the field image corresponding to the actual image displayed after switching. This result is transmitted to the comparison information area selection processing unit s254 as current display image comparison information 2253 and next display image comparison information s252 (FIGS. 44 and 45). The contents of the image comparison information development process 2257 are the image comparison information data searched by the image comparison information search processing unit 2251 from the image comparison information database 2250, corresponding to the shooting point and shooting direction of the corresponding real image. The viewpoint position / gaze direction 2125 is close but different. This difference is publicly known as a three-dimensional graphic, or is converted into a view image by linear conversion according to the first embodiment.

  Although the current display image and the next display image are in the vicinity where the shooting point and the shooting direction are adjacent to each other, they do not match, so there is a parallax, and it is not a plane to match both images for the entire display field screen of the display device 2121 It cannot be done in principle. However, since the operator 2130 is interested in a part of the display screen, the attention point position 2126 is input by the attention point input mechanism 2123 and the attention point input processing 2128, and is input to the comparison information region selection processing 2254. The comparison information region selection processing unit 2254 selects a part to be processed by the image matching processing unit 2255 from the display image comparison information 2253 and the next display image comparison information 2252.

FIG. 49 illustrates the function of the comparison information area selection processing unit 2254. FIG. 49A shows a field-of-view screen 2340 that is currently being displayed, and an attention point window 21 2330 shows an area that the operator is paying attention to based on the attention point position 2126. The HV coordinate of the point of interest position 2126 is (Ht, Vt). The region surrounded by the attention point window 21 2330 in the image comparison information 21 2333 in FIG. 49B corresponding to the currently displayed visual field screen 340 becomes the reference pattern 2335 in the image matching processing unit 2255. If the point of interest input mechanism 2123 is not provided, the center of the field of view display screen may be automatically set (Note that FIGS. 49B and 49D are examples of image comparison information, and FIGS. 49A and 49B). It does not correspond to the building of.) If the reference pattern 335 is an area of m pixels × m pixels and is described in a grammar such as C language, the following is obtained.
for (int i = 0; i <m; i ++) {
for (int j = 0; j <m; j ++) {
Pattern [i] [j] = M [Ht-m / 2 + i] [Vt-m / 2 + j];
}
}
Here, Pattern [i] [j] is the luminance array of the reference pattern, and M [Ht-m / 2 + i] [Vt-m / 2 + j] is the luminance array of the image comparison information 21 2333.

  FIG. 49C shows a visual field screen 341 for switching display. The point-of-interest window 21 2330 in FIG. 49C is at the same position on the display screen as FIG. 49A, but a portion of the building 2239 different from FIG. 49A is displayed due to the parallax. In the example of FIG. 49, the attention point window 21 2330 of FIG. 49 (a) is displayed as the attention point window 22 2331 in FIG. 49 (c). The positional deviation between the attention point window 21 2330 and the attention point window 22 2331 in FIG. 49C is a value calculated by the image matching processing unit 2255 and the deviation calculation processing unit 2256. The attention point window 21 2330 in FIG. 49A is determined corresponding to the attention point position 2126. The attention point window 21 2330 of FIG. 49 (b) is set at the same position on the visual field screen 2340 that is currently displayed. FIG. 49D shows the image comparison information 22 2334 corresponding to the switching display field-of-view screen 2341 in FIG. 49C, and the same position in FIG. 49B as the attention point window 21 2330 and FIG. 49D. An enlarged attention point window 2332 is set by enlarging the periphery of the attention point window 21 2330 in FIG. The center point of the enlarged attention point window 2332 is the same as that of the attention point window 21 2330, and the vertical and horizontal magnification is a standard until the maximum outer edge touches the image comparison information 22 2334, but can be further enlarged as necessary.

The point with the highest correlation is obtained in the enlarged attention point window 2332 of FIG. 49 (d) where the reference pattern 2335 has the highest correlation. If the enlarged attention window 2332 is N times as large as the attention window 21 2330, and the algorithm is described in C language grammar
long Convolution [N * m] [N * m];
for (int I = 0; I <N * m; I ++) {
for (int J = 0; J <N * m; j ++) {
Convolution [I] [J] = 0;
for (int i = 0; i <m; i ++) {
for (int j = 0; j <m; j ++) {
Convolution [I] [J] + = Pattern [im / 2] [jm / 2]
* M [Ht-N * m / 2 + Im / 2 + i] [Vt-N * m / 2 + Jm / 2 + j];
}
}
}
}
The product-sum calculation (convolution [I] [J]) is performed for each point in the expanded attention window 2332, and the maximum value (Htnext, Vtnext) is the most highly correlated with the reference pattern 2335. This is a point corresponding to the point of interest position 2126 of the visual field screen 2340 currently displayed on the visual field screen 2341 for switching display.

The deviation of the coordinates of the attention point window 21 2330 and the enlarged attention point window 2332 is a value obtained by the deviation calculation processing unit 2256,
Horizontal displacement = Htnext−Ht (pixels)
Vertical misalignment = Vtnext-Vt (pixels)
Is required. This value is passed to the screen output processing unit 2266 as display image position correction information 2259, and is used to correct the display position of the view image at the time of switching the actual image to be used. FIG. 50 shows the above processing in a processing flow. FIG. 45 shows the anteroposterior relationship and dependency relationship in the timing of each process in FIG.

  51 and 52 show a parallel processing method different from that in FIG. 45 for the processing of the image display system 2100 and the image switching smoothing system 2110. The image update period 2261 of the display screen is generally about 30 ms, and the processing of the image matching processing unit 2255 may take the longest in the processing of the image switching smoothing system 2110. When the image matching processing unit 2255 does not end within the image update cycle 2261, as shown in FIG. 51, the deviation calculation processing 256 may be transmitted to the screen output processing 2266 after the double of the image update cycle 2261, In the case of FIG. 52, the image is transmitted to the screen output process 2266 after three times the image update cycle 2261. As described above, when the time required for the processing of the image matching processing unit 2255 is longer than the image update cycle 2261, it can be dealt with by delaying the synchronization timing. However, when the delay becomes large, the response to the change of the target point position 2126 deteriorates. To do.

  As a method of processing the processing of the image matching processing unit 2255 at high speed, there is a method of processing the processing of the image matching processing unit 2255 in parallel with a plurality of cores or threads using a multi-core and multi-thread CPU. The method of parallel processing is to divide the enlarged attention window 2332 into multiple regions in the vertical or horizontal direction, and perform the correlation operation (multiply-accumulate operation) with the reference pattern 2335 for each division, and then select the maximum What takes a value should just be adopted.

  In the method described above, the actual image to be displayed next is predicted based on the changing direction and changing speed of the viewpoint position / gaze direction 2125 and the viewpoint position / gaze direction 2125, but the prediction may be other than linear extrapolation. Since there is no method, it may be impossible to cope with a sudden change in the direction of change of the viewpoint position / gaze direction 2125. In order to solve this problem, from all the live-action images that may be displayed next to a plurality of CPU cores or threads other than the CPU core or thread that performs the image display system 2100, or from the live-action images that may be displayed. The processing of the image switching smoothing system 2110 can be performed in parallel on a plurality of images in descending order of possibility of display. There is a method of determining the possibility of display next in order from the smallest angle formed by the viewpoint and line-of-sight direction of the photographed image and the viewpoint position and line-of-sight direction 2125 input by the operator 2130.

  If the viewpoint is moved while capturing the same feature with the line of sight, the method according to the first embodiment cannot maintain the continuity of the visual field image due to the parallax at the time of switching the actual image to be used. If the attention points of the feature are limited, the viewpoints of viewing the feature before and after the switching of the actual image are close and the line-of-sight direction is also close. That is, even if the position-related attribute of the visual field image cannot be maintained, the attention point of the specific feature exists at a close location in the visual image, and the shape of the feature in the visual image is highly similar. Paying attention to this, image matching is performed around the point of interest of a specific feature in the view image, and the display coordinates of the view image at the time of switching the live image are shifted so that the positions of the matched images overlap, and if necessary Thus, the enlargement / reduction ratio is changed.

  The field-of-view video information generation apparatus according to the present invention eliminates image disturbance or discontinuity that occurs when a live-action image is switched when a three-dimensional view image is generated using a real-shot image, and generates a smooth video. Therefore, it can be used industrially as a field-of-view video information generation device using a photographed image. Furthermore, the visual image information generating apparatus of the present invention can be used industrially in a wide range of fields as urban space landscape simulation, video guidance, situational grasp and monitoring, advertising, commercial activities, and amusement infrastructure, and in advance a specific target It can be used for the purpose of disaster prevention and security maintenance by taking advantage of the features that can be observed from various viewpoints and directions with respect to a specific object at the time of image display, by photographing a certain area with high density without limiting objects.

2100 Image display system
2101 Live-action image database
2102 Image index mechanism
2103 DEM data
2104 Live image search engine
2105 Visual image generation system
2110 Smooth image switching system
2111 Image Comparison Information Database
2112 Vision image position correction system
2120 User interface system
2121 Display device
2122 Viewpoint / Gaze input mechanism
2123 Attention point input mechanism
2127 Viewpoint / Gaze input processor
2128 Viewpoint / Gaze input processor

Claims (11)

Shooting in a plurality of directions at each of a plurality of shooting positions by a shooting device, shooting a shooting region divided into a plurality of sub-regions, acquiring a plurality of image information and storing them in a memory,
When storing each of the plurality of pieces of image information in the memory , the shooting position information and the shooting direction information when the image information is acquired are stored in the memory in association with the image information,
Identifying a sub-area captured when the image information is acquired with reference to imaging position information and imaging direction information associated with the image information stored in the memory;
Storing the image information, the shooting position information, and the shooting direction information in a database in association with the identified sub-region;
Specify a plurality of positions arranged in order on the viewpoint path and a direction of the line of sight toward each imaging region at each of the plurality of positions,
Image information including the sub-region positioned in the direction of the line of sight at each of the plurality of positions on the viewpoint path with reference to shooting direction information associated with the sub-region stored in the database; Retrieve the shooting position information associated with the image information and read in the order,
While switching the image information read by said order, in view video information generating apparatus comprising generating a reference to view the video information and the photographing positional information associated with the picture image information,
When generating the field-of-view video information in the line-of-sight direction from a position on the viewpoint path, the image information to be used is different from the shooting position of the image information and / or the shooting direction of the image information. When switching to another image information in the shooting direction, the positional deviation of the visual field video information is corrected so that a display positional deviation of an area common to the image information and the other image information does not occur. View image generator.

  2. The field-of-view video generation apparatus according to claim 1, wherein the region in the field-of-view video information for performing positional shift correction so as not to cause a display position shift is a central portion of the field-of-view video information being displayed on the field-of-view video generation device. The field-of-view image generation device according to claim 1.   2. The field-of-view image generating apparatus according to claim 1, wherein an area in the field-of-view video information for performing position shift correction so as not to cause a display position shift is a field-of-view image of a field-of-view image currently displayed on the field-of-view image generating apparatus. 2. The field-of-view image generation apparatus according to claim 1, wherein the field-of-view image generation apparatus includes an area specified by a position pointing device.   The visual field image generation device according to claim 3, wherein the position pointing device designates a portion in the visual field image information by a line of sight detected by a line-of-sight detection device.   5. The visual field image generation apparatus according to claim 3 or 4, wherein when the positional deviation correction is performed so that the positional deviation of the visual field video information does not occur, the display position of the region in the visual field video information including the line of sight is corrected. A field-of-view image generation apparatus characterized by performing a smooth switching of an image by adjusting a display magnification based on a point where position deviation correction is performed so as not to cause a deviation and / or a distance from the region.   6. The field-of-view video generation device according to claim 1, wherein image information during display of position shift correction and image information after switching display are displayed so that a display position shift does not occur in the field-of-view video. A field-of-view video generation apparatus characterized by performing image recognition between the two.   7. The field-of-view image generation apparatus according to claim 6, wherein the image information after the switching display is predicted from a movement of a cursor or a movement of a line of sight.   8. The field-of-view image generation apparatus according to claim 6, wherein the image recognition is performed by calculating in advance information for use in image recognition for positional deviation correction and storing the information in a memory. A visual field image generation device characterized by the above. 9. The visual field image generation device according to claim 1, wherein the visual field image information generation processing and the image recognition processing for positional deviation correction are processed in parallel while being synchronized. Visibility image generator.   10. The visual field image generation apparatus according to claim 9, wherein the parallel processing is performed by being shared by a plurality of cores or threads in an LSI package of the same CPU. 9. The field-of-view image generation apparatus according to claim 8, wherein the information used for image recognition that is calculated in advance and stored in a memory for each image information is such that a luminance histogram for each sub-region of the image is leveled. A field-of-view image generation apparatus characterized by being processed information.