JP2009217524A - System for generating and browsing three-dimensional moving image of city view - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system and service for supplying three-dimensional city view moving images using real images. <P>SOLUTION: A three-dimensional moving image amusement system includes: a system for imaging a city with a plurality of digital cameras mounted on an air craft, a vehicle, and the like from the sky and on the ground from many directions with high density; a system for storing the images in a database and searching for an image corresponding to an arbitrary viewpoint path and line of sight direction with respect to a position of an arbitrary city; and a system for generating a smooth three-dimensional continuous moving image by performing morphing the searched image in accordance with a parallax between an actual viewpoint and the photographic image. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to a system that generates a three-dimensional video of a landscape on the ground including an urban area, and more particularly, to a system that uses an aerial photograph taken from above in an aircraft or the like, or a photograph taken on the ground on a road. The present invention relates to a system for generating and displaying a moving image of a three-dimensional cityscape that uses photographs taken from various angles on the ground.

  Conventionally, 3D animation display of cityscapes is done by measuring the 3D information of urban buildings with a laser profiler and creating a 3D model, and then pasting the wall images of the buildings by rendering processing. An arbitrary viewpoint path and an image in an arbitrary gaze direction along the line-of-sight direction are generated by the 3D image processing for the three-dimensional city model obtained in this way, and are continuously displayed in time to generate a three-dimensional moving image. Is what you get. As another method for obtaining three-dimensional information of a city building, a method of generating a three-dimensional city model by extracting information in the height direction by stereo image processing using images obtained by photographing the same point on the ground from different directions. There is also. In addition to this, there is also a method of creating a moving image by three-dimensional image processing by constructing a three-dimensional city space model that does not necessarily correspond to an actual city on a one-to-one basis.

Both of these methods require time and cost to generate a three-dimensional model of a city, that is, a three-dimensional model. Further, in order to make a solid outer wall of a building or the like identical to the actual one, pattern information on the outer wall or The image information needs to be pasted (rendered), and the information from all directions of the three-dimensional object is indispensable, and the acquisition and the rendering require a great amount of labor and cost. . In this way, although it is strongly desired to display a landscape of an arbitrary urban area as a three-dimensional video from an arbitrary aerial route and line of sight, rendering accuracy cannot be sufficiently realistic, In many cases, the update cycle becomes long, and the freshness and accuracy of the displayed video are difficult.

Some amusement systems that use a three-dimensional simulated field of view, such as Ace Combat in Non-Patent Document 1, process and use surface images taken from satellites. However, the purpose is to give the game a realistic experience to the last. This does not necessarily match the 3D landscape. In addition, the urban space three-dimensional simulated view seen in the second life of Non-Patent Document 2 represents a virtual city three-dimensionally but does not correspond to an actual map. Non-Patent Document 3 COCOR meet-me is a 3D amusement system of virtual city space that corresponds to a real map as a more realistic representation of city space for Second Life, The detailed structure does not match the reality. In addition, the technology according to the present invention is to shoot and accumulate a large amount of aerial photographs in a multi-viewpoint and multi-line-of-sight direction in advance. Aerial cameras that use digital one-dimensional line sensors or two-dimensional image sensors have been put into practical use. Furthermore, due to the high performance of general-purpose digital cameras, it can be applied to the aerial photography field. Patent Document 1 proposes a system for creating an ortho image from an oblique photograph taken with a digital camera using dedicated software. In aerial photography for surveying purposes, a reference point was set on the ground in advance, and the camera position and orientation were calculated with reference to the reference point to obtain the coordinates of the feature. However, GPS (Global Positioning System) and IMU Equipped with an (Inertial Measurement Unit), it can measure the position and orientation of the camera at the time of imaging. (Patent Document 2) Patent Document 3 proposes a technique for capturing an aerial photograph for surveying with three CCD line sensors having different imaging directions.

http://www.acecombat.jp/ http://www.secondlife.com/ http://www.mmet-me.jp/ JP 2002-357419 A. JP 2004-245741 A. Japanese Patent No. 2876222.

  However, the method of Non-Patent Document 1 is a three-dimensional video amusement system that pursues realism using high-resolution satellite images as landscape material, but the target is an overseas battlefield, and the image itself is also produced. Since it is not updated from, it is different from the realization of a cityscape 3D animation. In the system proposed in Non-Patent Document 2, the cityscape three-dimensional video display of the human eye is performed, but all the displays are fictitious worlds generated on a computer, which differs from the real world and conforms to the reality It does not match the purpose of displaying the cityscape in three dimensions. The method of Non-Patent Document 3 is an improvement of the method of Non-Patent Document 2 and constructs buildings etc. corresponding to the map of the actual city, but the landscape itself pursues the identity with the reality. It is not a reality.

Patent documents relating to the acquisition of aerial photographs include the following, but the system proposed in Patent Document 2 acquires two oblique images, but only one oblique image requires a large amount necessary for the present invention. It is difficult to acquire multi-directional images with high density. The system proposed in Patent Document 3 captures images using CCD line sensors in three different directions, but these three CCD line sensors are arranged in parallel and linearly, and thus acquire a stereo image for surveying. Although it is suitable for the purpose of the present invention, the amount of information is overwhelmingly insufficient for acquiring an original image for generating a three-dimensional moving image of a cityscape as in the object of the present invention.

In Patent Documents 1 and 2, it is conceivable to increase the number of cameras in order to capture the ground in multiple directions, in large quantities and at high density. However, the shooting openings on the floor of the aircraft for aerial photography have dimensions and positions. However, it is strictly prescribed as a target for the airspace certification, and it is not possible to enlarge the opening or add a camera that protrudes out of the machine from the opening. Therefore, the realization means suitable for the cityscape three-dimensional animation amusement system by the real image which needs to image the ground from multiple directions with high density has not been considered so far.

  The present invention has been made in view of such circumstances, constructing a system capable of capturing a wide range of multi-directional images with high-density mesh points in the air in an aircraft or the like, creating a database of these large amounts of images, Establish a system that quickly searches for the image with the least parallax when viewing any position in the city from any direction, and the most parallax corresponding to the specified viewpoint, line of sight, and position in the city A morphing system that generates continuous and smooth moving images from less images is constructed. Furthermore, in the present invention, in order to create a realistic moving image of the human eye even on the ground, a dedicated vehicle-mounted system that acquires images in all directions at high density on every road or passage in the city was constructed and acquired. Create a database of a large number of images, build a system that quickly searches for images with the least parallax when viewing any position on the streets of the city from any direction with any human eye, and further, the specified viewpoint, line of sight, A morphing system that generates continuous and smooth moving images from images with the least parallax corresponding to the position in the city was constructed.

  As a result of diligent research in view of the above problems, the present inventor is the biggest barrier in the realization of a three-dimensional amusement system for cityscapes. The three-dimensional information on buildings and structures such as urban buildings is inexpensive and frequent. We have come to recognize that there is no means to acquire, and that there is a lack of means to acquire and paste images pasted on the surface of these three-dimensional objects at low cost and frequently. For this reason, the present inventor has taken a landscape of any part of the city in advance from any direction as a method for obtaining a three-dimensional video of any part of the city from any route without following these means. In addition, the optimal captured image corresponding to the selected viewpoint path, line-of-sight direction, and target position is selected, and the position and optical axis direction of the camera that captured the image, and the difference between the selected viewpoint and line-of-sight are morphed. And came up with an idea of how to obtain a smooth moving image.

However, it is extremely difficult to take a picture of every part of the city in advance from all directions. Experience has shown that in order to achieve this goal, the inventor must prepare photographs from every direction for every city point every few degrees in solid angles. In order to solve this problem, a lot of small digital cameras were used, and the arrangement method and control method were made novel. That is, first of all, a large number of small digital cameras are formed as a collection of digital cameras, and they are stored in the holes of an aerial surveying camera on the floor of an aerial surveying aircraft without being overhanging. In order to avoid acquisition problems and, secondly, in order to efficiently take pictures from multiple directions, a camera for direct shooting is placed in the center of the digital camera assembly, and the surrounding area is radially inclined There are 8 to 25 digital cameras for photographing, or more if there is enough space.

Third, even if the ground is photographed from various directions, it is not preferable that the resolution of the image changes depending on the depression angle at which the ground is viewed. Therefore, an angle formed from the vertical direction is generally called an off-nadir angle, but a small off-nadia angle is used. A long focal length telephoto lens is used for a large off-nadir angle digital camera, and these are arranged radially and concentrically. In this way, we have devised a digital camera assembly that can acquire a wide range of images from just below the aircraft to a considerably large off-nadir angle region without degrading the resolution.

Fourthly, the present inventor uses the GPS camera and the inertial navigation device in the aircraft in order to efficiently operate the digital camera assembly and obtain an image of every solid angle for every point over a wide city space. In order to construct a precise mesh-like shooting point at the same altitude in the air, a system was established to automatically issue a shooting command to the digital camera assembly at the shooting point.

Fifth, in order to reduce the cost of photography due to the need to shoot a large amount of images, it is generally conceived that images are taken from low altitudes below the clouds in cloudy skies where aerial surveys are inappropriate, and image degradation due to cloudy aerial fog We have invented a method for automatically compensating for image by digital image processing.

  Further, the present inventor needs to search a captured image having a solid angle of about several degrees at a high speed for generating a moving image with respect to an arbitrary target point. Devised. In other words, the surface of the earth is divided into square meshes by latitude and longitude or XY coordinates. One side of the square is sufficiently small, for example, set to about 50 m to 20 m. Each square on the mesh is managed by attaching a two-dimensional address. Each captured image includes the mesh square group included in the image range obtained by capturing the ground surface, and a vector connecting the camera position at the time of capturing and the center of the mesh square can be defined. This is normalized and defined as an original image vector. The original image vector can be used as an index for expressing an image for a designated ground mesh.

That is, the zenith index is defined by quantizing the angle formed between the zenith vector indicating the position directly above and the original image vector, and the quantization criterion for the angle formed is 5 degrees or less. Since each zenith index includes 360 degrees of azimuth as horizontal components, it is subdivided into cells of solid angle units, and the angle between the original image vector and the zenith vector, the original Index by the horizontal component of the image vector. A necessary original image can be searched at high speed by this three-stage indexing. Although the original image necessary for the three-dimensional moving image is prepared every several degrees with a solid angle with respect to an arbitrary city position, since the viewpoint passes through this neighborhood, the actual viewpoint, the line of sight and the original image vector Deviations of several degrees or less are smoothly connected by morphing processing for the original image.

  Since the present invention is to display the cityscape as a moving image in real, the original image used for the three-dimensional moving image must be acquired even on the ground. The present inventor considers that 3D animation of the cityscape on the ground is to reproduce the landscape from the human perspective, and the cityscape of every place is taken in advance by the human perspective. We have devised a method to generate a 3D movie in a manner similar to that of the scenery. The human gaze on the ground is restricted by the fact that humans move on roads or passages, and the position of the eyes is 1.5-1.8 m above the ground, and the degree of freedom of viewpoint is considerably higher than the view from the air. small.

The inventor of the present invention has devised a digital camera assembly equipped with a large number of digital cameras similar to those mounted on an aircraft for in-vehicle use. That is, a camera that shoots in the horizontal direction in all directions 8 to 12 and the diagonally upward direction with an elevation angle from the horizontal in all directions 8 to 12 is configured as a digital camera assembly, installed on the roof of the vehicle, Is moved on a predetermined road by a car navigation system, and based on the detailed position measurement results by GPS and IMS, a shooting command is issued to all the cameras constituting the digital camera assembly at intervals of several meters, and the result is It is recorded with the position data. In this way, after building a road image database, a method for generating a three-dimensional moving image from the sky with respect to an arbitrary place in the city, an arbitrary place on the road or a passage, and a human eye from an arbitrary line of sight. It was devised based on the generation of 3D video. The method includes a process of selecting an optimal original image and a process of generating a smooth moving image by a morphing process according to the movement state of the viewpoint, the line-of-sight direction, and the position in the city.

However, unlike in the air where the viewpoint can be freely specified by placing the viewpoint in a completely free space, the place where the viewpoint can exist is limited on the road or road. Furthermore, since the method of the present invention is a method for generating a three-dimensional video without generating a three-dimensional model for a structure in an urban space, an image taken on a road or a passage is pasted on Terrain in the case of an aerial photograph. As noted, the target to be pasted must be selected and modeled. Considering the structure of the city as a representation of the city space from the human perspective, a model that meets this objective is a zero-thickness mainly composed of a plane vertically along the boundary line of the road or passage and other urban areas. A wall was set up and used as a projection surface of the original image.

Mathematically expressing the projection plane along this road or passage is equivalent to using DEM as a model of the ground surface in aerial photographs, but the road or passage has a graph structure. Model by expression. Further, what is important in image projection is the wall or projection plane existing on the boundary line of the road or passage. Further, the road or passage is refracted, branched, intersected, merged, started, and ended. The inventor has devised a mathematical expression method of a road or a passage suitable for the purpose of projecting an original image by a graph expression. As a result, we succeeded in expressing an arbitrary cityscape from a human eye at an arbitrary position on a road or a passage as a moving image from a captured original image by morphing.

As described above, in the cityscape three-dimensional video amusement system according to the present invention, it is possible not only to jump freely in the air and see an arbitrary place with an arbitrary line of sight but also to an arbitrary human eye from an arbitrary road or passage. You can get a video. For this reason, a simulated cockpit for simulated flight in the air is provided as a graphical user interface via the Internet, and the azimuth, elevation, and speed can be freely controlled and specified, and the azimuth and tilt angles of the camera can be switched to a simulated camera. The zoom amount can be changed freely to display a three-dimensional video. On the road or on the aisle, you can move forward and backward with a human eye via the graphic user interface, change the position freely by turning left or right, and change the direction of the line of sight to express the field of view in a free direction as a three-dimensional video. Can do. Furthermore, the feature of the cityscape three-dimensional video amusement system of the present invention is that it can freely fly down from the air, walk on the road or on the passage, and soar from the ground to the sky and fly freely in the air. . For this reason, the amusement was improved by introducing a graphic user interface for landing by designating a landing point on the ground from a simulated aerial view, and conversely, a graphic user interface for jumping from the ground to the air.

  As described above, according to the cityscape three-dimensional video amusement system of the present invention, the image is taken in the air without generating a costly three-dimensional numerical model of the urban space and rendering the building surface. A real image can be used to generate a three-dimensional moving image for an arbitrary viewpoint route and viewpoint direction in the air for an arbitrary place in the city. In addition, it is possible to generate a three-dimensional moving image with respect to an arbitrary viewpoint route and a viewpoint direction with a human eye at a point on an arbitrary road or passage in a city using an actual image taken on the road or the passage.

  Hereinafter, the best mode for carrying out the three-dimensional moving image amusement system of a cityscape of the present invention will be described in detail with reference to the accompanying drawings. FIG. 1 is a diagram schematically showing an overall configuration of a cityscape three-dimensional video amusement system in the present embodiment. The urban landscape 3D video generation amusement system is common to both the part for generating 3D video when the viewpoint is in the air and the part for generating 3D video when the viewpoint is on the road A graphic user interface system 180 is provided, and services are provided to the user 191 via the Internet 190.

In order to generate a 3D video when the viewpoint is in the air, the aerial image acquisition system 100 for acquiring an aerial photo in the air by an aircraft and the acquired aerial photo to be easy to use for generating a 3D video. An aerial image database generation / registration system 120 for processing and creating a database, an aerial image database 140 created as a database, and an aerial 3 that generates a three-dimensional moving image based on a request from a user 191 requested through a graphic user interface system 180 A three-dimensional video generation system 160 is configured.

The part for generating 3D video when the viewpoint is on the road is processed to make it easy to use 3D video generation on the road image acquisition system 110 for acquiring a photo on the road with a vehicle etc. A road image database generation / registration system 130 for creating a database, a road image database 150 created as a database, and a three-dimensional road image for generating a three-dimensional moving image based on a request from a user 191 requested through a graphic user interface system 180 The video generation system 170 is configured. The graphic user interface system 180 common to both has a function of switching between the field-of-view generation when the viewpoint is in the air and the field-of-view generation when the viewpoint is on the road based on a request from the user 191. Hereinafter, after explaining the basic principle of 3D animation generation according to the present invention, each system constituting the cityscape 3D animation amusement system according to the present invention will be described in detail.

FIG. 2 is a diagram showing the concept of aerial image acquisition in a city landscape 3D video amusement system. The greatest feature of the present invention is that a city 3D model is not generated first in the city landscape 3D video generation. Secondly, it is not to render the outer wall pattern or outer wall photo on the generated 3D model, and instead of eliminating these manual and costly work, every position of the cityscape is An image is acquired from a direction, and this is selected and transformed as necessary. However, this method not only needs to capture a large number of images in advance, but also needs to search for an optimal captured image at high speed. Recent advances in digital cameras, reduction in the capacity and cost of various memories, When the conditions for realization were established by improving the capability and communication capability, the new scheme according to the present invention was first realized.

In FIG. 2, the sky sphere sphere 222 in the city 200 is three-dimensionally divided at an angle of about 2 ° to about 5 ° like a sufficiently small solid angle range i 221 or a solid angle range 1220, and the surface near the surface of the sphere 222 is covered. The concept of an aerial image acquisition system 100 for preparing images everywhere on the surface of the earth for all the sufficiently small solid angles is shown.

  FIG. 3 shows a slightly different view of the image of the same spot taken in the adjacent sufficiently small solid angle range i221 of FIG. 2 due to parallax, but the images are continuously displayed by linear conversion of the field of view between the slightly different images. The concept of the morphing process for obtaining a smooth three-dimensional moving image is shown. The field of view of the building shown in the relationship diagram 230 of the building and the line-of-sight direction viewed from the line-of-sight direction 1240 and the line-of-sight direction 2250 having a relatively small parallax is an image 241 viewed from the line-of-sight direction 1 and an image 251 viewed from the line-of-sight direction 2. As shown, when the angle formed by the line-of-sight direction 1240 and the line-of-sight direction 2250 is small, or when the object is planar, the images can be approximated to each other by morphing processing using linear transformation. It is a feature of the present invention that this morphing process is used for an interpolation process of an actual image of an adjacent viewpoint.

FIG. 4 is a diagram for explaining the concept of generation of a simulated field of view by simulated flight in the cityscape three-dimensional video generation amusement system according to the present invention. The viewpoint moves with time t along the viewpoint path P (t) 270, and during that time, the ground surface is captured in the field of view along the target trajectory T (t) 280. From the position on the viewpoint path P (t) 270, the target trajectory T (t) 280 has a line of sight 271 at a time t 1, a line of sight 272 at a time t + δt, and a time t + 2δt.
The trajectory on the ground surface is captured for each δt by the line of sight 273, the line of sight 274 at time t + 3δt, the line of sight 275 at time t + 4δt, and the line of view 276 at time t + 5δt. In the present invention, as a method of generating a three-dimensional cityscape movie for the target trajectory T (t) 280 from the viewpoint path P (t) 270, the parallax is close from the line of sight 271 at time t to the line of sight 276 at time t + 5δt. A few aerial images i260 and aerial images i + 1 261 are searched from the aerial image database 143, and the line of sight 271 at time t1, the line of sight 272 at time t + δt, and the time t + 2δt
Using the aerial image i260 having the closest line of sight as the original image during
Parallax due to the difference in viewpoint between the line of sight 271 of time t + δt 2, and the line of sight 273 of time t + 2δt and the line of sight of the aerial image i 260 is corrected by the morphing process for the aerial image i 260, From the line of sight 274, it is determined that the aerial image i + 1 261 has less parallax due to a difference in viewpoint than the aerial image i260, and the aerial image i + 1 261 is switched to the original image, and the line of sight 274 at time t + 3δt t + 4δt
The target trajectory T ((s) smoothed by correcting the parallax due to the difference in viewpoint between the line of sight 275 at time t + 5δt and the line of sight of the aerial image i + 1 261 by morphing processing on the aerial image i + 1 261. t)
The 3D animation of the cityscape along 280 is generated.

  The method according to the present invention requires an enormous amount of aerial images. FIG. 5 shows an example of the acquisition method. A digital camera assembly containing a large number of digital cameras as shown in FIG. 7 is mounted on the aircraft 301, and the ground surface is photographed at photographing points 310 at regular intervals along the flight path 300. The flight path 300 and the shooting points 310 are densely set in a mesh pattern, and a plurality of digital cameras are mounted and multi-directional images are taken simultaneously as shown in the optical axis direction 320 of the digital camera assembly, thereby obtaining FIG. An image is acquired for each sufficiently small solid angle range i221 shown.

  FIG. 6 shows a processing flow of the aerial image acquisition system 100. Hereinafter, a detailed implementation method of the aerial image acquisition system 100 will be described with reference to FIG. First, the flight route and imaging points are set so that the imaging point setting process 330 can capture aerial images in a mesh pattern at regular intervals. In this shooting plan, shooting points on the flight route and the anti-road are defined in the form of an aerial image acquisition plan file 101 whose configuration is shown in FIG. Next, in the shooting process 331, it is determined that the aircraft 301 has reached the shooting point 310 defined in the aerial image acquisition plan file 101, and shooting is performed by the aerial shooting control system 393 shown in FIG. The captured images in the digital camera are stored in the aerial image primary file 102 together with metadata including position and orientation data acquired in the imaging process at predetermined timings by the aerial shooting control system 393.

  FIG. 7 is a diagram showing a configuration example of the digital camera assembly 360 constituting the aerial image acquisition system 100 according to the present invention. For the purpose of efficiently and densely photographing the ground surface from all viewpoints, a plurality of oblique digital cameras 350b to 350i are arranged in such a way that the horizontal circumferential direction is radially divided at equal intervals around the digital camera 350a in the direct downward direction. The digital camera assembly 360 is formed by arranging eight cameras so that the angle between the optical axis and the direction of gravity is the same. In particular, when shooting on an aircraft, it is necessary to obtain a flight proof, change the hole on the aircraft floor for aerial photography, or install a camera outside the aircraft from the floor hole. Since it is practically impossible to use it in a pop-up form, the digital camera assembly 360 is designed to fit within the aircraft floor hole using a small digital camera such as Canon PowerShot. It is a thing. The support structure other than the digital cameras 350a to 350i constituting the digital camera assembly 360 is required to have high shooting direction accuracy. Therefore, any support structure that is lightweight and rigid can be used. It can be composed of a plate or carbon resin.

  FIG. 8 shows an example of setting the route and the shooting point of the aerial image acquisition system using an aircraft. FIG. 8B shows a setting example of the shooting point 310. The flight path 300 flies in a reciprocating manner while making U-turns as shown in FIG. 8B by connecting the flight routes 300 parallel to each other by dotted lines in order to form a network of the photographing points 310 in the air. During this time, shooting is performed at the shooting point 310. The mutual interval between the shooting points 310 is defined by the flight interval shooting interval 372 and the flight direction shooting interval 371. When the flight altitude is set to about 800 m, the flight interval shooting interval 372 and the flight direction shooting interval 371 are set to about 80 m or less in order to obtain at least one image within the sufficiently small solid angle range i221 in FIG. It is preferable to do this. When shooting with the digital camera assembly 360, the range indicated by the shooting ranges 375a-i of the digital cameras 350a-i in FIG. Overlap with the range captured by each digital camera in the imaging range 375a-i of the digital camera 350a-i by setting the flight altitude, the angle formed by the oblique direction digital camera 350b-i, and the focal length of the lens The degree varies. For the purpose of the present invention, it is desirable to set so that the ground surface is covered almost continuously while slightly overlapping each other. Note that the orientation of the digital camera 350a that captures the direct direction with respect to the flight path 300 is set such that the horizontal direction of the image frame is the traveling direction of the flight path 300 in FIG. 8A, but the vertical direction of the image frame is the flight path 300. May be in the direction of travel.

  FIG. 9 is a diagram showing a configuration example of the aerial image acquisition system, which includes a flight navigation system unit 385 and a data acquisition / recording system unit 390. The flight navigation system unit 385 is a device for guiding the aircraft 301 along the flight path 300 to the imaging point 310 defined in FIG. 8B. The aircraft position data is periodically obtained from the GPS 380, and the air meter Avionics information 387 such as aircraft attitude, altitude and speed can be obtained from the equipment 388. These signal interfaces are not novel because they are standardized as aircraft bus signals.

The function of the flight navigation system unit 385 is in accordance with the contents of the aerial image acquisition plan file 101 of FIG. 11, and the flight navigation system unit of the aerial image acquisition system of FIG. 12 according to the processing flow of the flight navigation system unit of the aerial image acquisition system of FIG. The pilot is guided by the example of the display screen. The flight navigation system itself has already been publicly implemented and has no novelty. However, in order to realize the object of the present invention, a part that efficiently guides the aircraft to the photographing point 310 is related to the present invention. First, determine which part of the city you want to get the sky image of before the flight. Determine the shooting range on the map and formulate a flight plan. The flight plan is performed by setting the flight path 300 on the aerial map.

The shooting point 310 must be set at a uniform height from the ground surface at the same height so as to achieve the object of the present invention, and based on the result, the flight path 300 is composed of parallel lines. Set as follows. The route number shown in the aerial image acquisition plan file 101 in FIG. 11 is assigned to each straight line portion, and the shooting points 310 are assigned so as to form a mesh as a whole for each route, and the start coordinates and end coordinates of each route, The number of shooting points and the coordinates of each shooting point are set in latitude and longitude and altitude. In this way, the aerial image acquisition plan file 101 of FIG. 11 is constructed. The graphic user interface related to the construction of the aerial image acquisition plan file 101 is publicly implemented as a map information system.

The function of the flight navigation system unit 385 is shown by the processing flow of the flight navigation system unit of the aerial image acquisition system shown in FIG. 12 are sequentially displayed from the route numbers registered in the aerial acquisition plan file 101 in the processing block 420 until all the shootings for the route numbers scheduled for flight are completed. Since the designated route number has a start coordinate, in order to start the route number in processing block 421, the position, altitude, traveling direction and speed of the route start point are designated and the contents of FIG. To display guidance. In order to take a picture, the conditions specified in the processing block 421 must be satisfied within a certain error range, for example, a position error of 10 m to 30 m or less and a flight direction error of 5 ° or less. If not, the flight route is restarted, and the guidance of the processing block 421 is performed again in the processing block 423. When the conditions of the processing block 422 are satisfied, the processing points 424 and 425 are sequentially processed one by one from the shooting point coordinates described in the aerial image acquisition plan file of the route number selected in the processing block 420 to the final shooting point. Guidance by FIG. 12 shows a display screen example of the flight navigation system unit of the aerial image acquisition system.

In order to guide the aircraft 301 to the starting point of the route number designated in the processing block 420, the photographing guidance display 437 and the position deviation display 446 of FIG. The deviation of the aircraft position 373 from the flight route 300 can be known, and the deviation is eliminated by maneuvering the aircraft. Data on the position of the aircraft is obtained from GPS 380, and altitude, heading direction, speed, and pitch / yaw / roll attitude data is obtained as avionics information 387 from aircraft instrumentation 388, and is used for the display in FIG. In the shooting guidance display 437, the route number 439 indicates the route in flight, and the remaining shooting point number display 438 in the route indicates the number of shooting points that must be taken in the future. In the shooting guidance display 437, the shooting point 310 and the shooting allowable range 440 are displayed along the flight route 300 for the shooting point 310, so that the pilot passes through the shooting allowable range 440 for each shooting point. Maneuver like so. The aircraft guidance display 437 is displayed rolling from top to bottom according to the flight, so that the relationship between the flight path 300 and the aircraft position 373 that should always be taken as the latest photographing point is shown. The aircraft position display 373 is always at the bottom left of the screen. Note that the azimuth display 442, posture display 443, and position deviation display 446 in FIG. 12 are all publicly implemented in the current aircraft instrumentation.

Next, the data acquisition / recording system unit 390 in FIG. 9 will be described in detail. In FIG. 9, an aircraft 301 has an aircraft floor hole 397 for installing an aerial photographer below the fuselage, and the digital camera assembly 360 is installed in this hole so as not to protrude outside the aircraft. In the example of FIG. 9, a structure that is suspended from the aircraft floor hole 397 by a stable platform device 395 to be described later is adopted. However, the stable platform control system 394 and the stable platform device 395 are digital camera assemblies regardless of the attitude of the aircraft 301. Since the 360 has a function of always directing 360 directly below the ground and fixing the azimuth in a specified direction, the stable platform device 395 may be omitted if the pitch roll of the aircraft can always be kept within 5 ° by maneuvering. The IMU 396 is an acronym for the English expression of the inertial measurement device, and the posture of the digital camera assembly 360 can be measured by placing it on the stable platform device 395. The top surface of the digital camera assembly 360 may protrude above the aircraft floor 398.

The data acquisition and recording system unit 390 stores an IMU 400 for observing the attitude of the aircraft 301, an aerial imaging control system 393 including a program for controlling the digital camera 350 and processing imaging data, and a large amount of data including image data. An aerial image primary file 102 composed of a capacity disk device and an aerial image acquisition plan file 101 storing shooting points for issuing shooting commands to the digital camera 350 are installed in the apparatus. A GPS380 antenna for measuring the position of the aircraft 301 is provided outside the aircraft.

FIG. 13 shows a flow of information between components of the data acquisition / recording system unit 390 of FIG. FIG. 14 illustrates a processing flow of the imaging control system 393. The photographing control system 393 is composed of a CPU. The imaging control system 393 periodically acquires the attitude data 450 of the stable platform device 395 from the IMU 396. When the stable platform 395 is operating normally, it always maintains a constant attitude with respect to the inertial space regardless of the attitude of the aircraft 301. The IMU 400 fixed to the aircraft 301 periodically sends aircraft attitude data 451, and the GPS 380 periodically sends GPS antenna position data 452 to the imaging control system 393. The processing of the imaging control system 393 is described in detail in FIG. 14, but the contents of the aerial image acquisition plan file 101 and the obtained GPS antenna position data 452 and aircraft attitude data 451 are sequentially processed in processing block 460 and processing block 461. The route number is identified, the route number is identified, and from the end of shooting of the route number to the departure of the route, it is sequentially determined in processing block 462 whether it is passing the nearest point of the shooting point of the designated route and within the shooting allowable range 440 At the timing closest to the shooting point, a shooting command 457 is sent simultaneously to the digital cameras 350 constituting the digital camera aggregate in processing block 463, and at the same time, the aircraft data portion 472 in the aerial image primary data file shown in FIG. 15 is processed. Write at block 464.

Since the digital camera 350 can hold a 32 GB memory at the time of the present invention, it can hold the captured aerial image data 455 until at least one route number is completed. Aircraft 301 makes a U-turn flight until it finishes one route number and enters the next route number, but during this time, no image is taken, so aerial image data 455 in digital camera 350 is used as the aircraft. The processing is transferred to the aerial image primary file 102 composed of the mounted large-capacity disk device at processing block 465, and the memory inside the digital camera 350 is emptied.

FIG. 15 shows a configuration example of the aerial image primary file 102, but an image header portion 470 and an image data portion 471 are prepared for each captured image. The camera ID in the header part is a number that identifies each digital camera 350a-i constituting the digital camera assembly 360. The image data ID is an identification number assigned so that the images can be distinguished from each other. The shooting date / time is the shooting time corresponding to the aircraft data portion 472. In the aerial image database generation / registration processing 120, the digital data 350a is integrated. Used to calculate the optical axis direction of ~ i. The image data portion 471 is not particularly processed at this stage. Further, since the camera parameter of the image header section 470 is normally fixed during flight, the same set value is written.

  The aerial image database generation / registration system 120 will be described in detail with reference to FIGS. FIG. 16 describes the overall processing flow. In processing block 480, the aerial image primary file 102 accumulated during the day's flight in the aerial image acquisition system 100 is sequentially processed for all images after the flight is completed. To do. In the processing block 481, the header of the image header portion 490 of the aerial image database of FIG. 17 is created, but the header of the image header portion 470 of the aerial image primary file 102 of FIG. For the processing block 482, the camera parameters of the image header portion 470 of the aerial image primary file 102 are usually transferred as they are.

In the processing block 483, the shooting parameters of the image header 490 are calculated for each image, and the processing will be described in detail below. The camera position is defined by latitude, longitude, and altitude. However, when the position measurement by the aircraft-mounted GPS 380 is usually DGPS, an accurate position may be obtained as it is because the flight altitude is low. Since the error is included when the position measurement by the GPS 380 is GPS, coordinate correction data is obtained from a nearby DGPS station measured at the same time as the flight time after landing, and the correction is performed according to the shooting date and time.

The calculation of shooting parameters is performed using linear algebra, and will be described below with reference to FIG. Unit vector in Cartesian coordinate system pointing to the horizontal direction in the north
And the reference axis.
Coordinate conversion by pitch, yaw and roll of the aircraft based on
And In addition, the position of the aircraft data portion 472 in the aerial image primary file 102 is represented in an orthogonal coordinate system.
The positional deviation from the GPS antenna 380 in the aircraft to the digital camera assembly 360 is expressed in vector.
And Furthermore, the reference axis
The optical axis of each camera (indicated by the subscript k = a to i) and the direction vectors at the four corners are expressed as unit vectors as follows. The following are constants, which are determined by the focal length of the camera and the mounting direction of the digital cameras 350a to 350i of the digital camera assembly 360.
Image center (normalized) G _kc = ^t (G _kcx G _kcy G _kcz )
_Captured image lower right azimuth vector (normalized) G _k1 = ^t (G _k1x G _k1y G _k1z )
_Captured image lower right azimuth vector (normalized) G _k2 = ^t (G _k2x G _k2y G _k2z )
_Captured image lower right azimuth vector (normalized) G _k3 = ^t (G _k3x G _k3y G _k3z )
_Captured image lower right azimuth vector (normalized) G _k4 = ^t (G _k4x G _k4y G _k4z )
, The shooting position S is S =
+
Thus, the shooting position (latitude, longitude, altitude) shown in FIG. 17 is obtained. Normalized image vector (X, Y, Z) is
It is obtained by G _kc .

Aircraft attitude
The normalized vector in the direction of the edge of the image 4 taking into account for each camera,
G _k1,
G _k2,
G _k3,
Gk4 . Since the Terrain data of the surface is given to the lattice points of the surface by latitude, longitude, and altitude, a triangle is composed of the nearest three adjacent points, and the vertex coordinates are defined as T ₁ , T ₂ , T ₃ and T _c is the coordinate where the plane of the triangle T ₁ T ₂ T ₃ and G _kc intersect, x represents the outer product,
x
0
x
0
x
0
Then, it can be said that T _c is inside the triangle T ₁ T ₂ T ₃ . That is, they intersect.
If the image vector (normalized) G _kc from the shooting position S = (x, y, z) intersects the triangle T ₁ T ₂ T ₃ at T _c (Equation 2)
T _c -S = RG _kc
T _c −T ₁ = a (T ₂ −T ₁ ) + b (T ₃ −T ₁ ) where a and b are arbitrary scalar constants. Here, R is the image distance of the shooting parameter in FIG. The image distance is the distance from the shooting position until the shooting optical axis crosses Terrain 580.
here,
= ^t (T _1x T _1y T _1z )
= ^t (T _2x T _2y T _2z )
= ^t (T _3x T _3y T _3z )
= ^t (T _4x T _4y T _4z )
Then,
=
=
+ a (
-
) + b (
-
)
The center point coordinates (latitude, longitude, elevation) on Terrain of the shooting parameters in FIG.
It is obtained as However, it is checked by calculating an outer product whether it is inside the triangle T ₁ T ₂ T ₃ .

Similarly, by calculating with respect to G _k1 , G _k2 , G _k3 , G _k4 instead of G _kc, the nearest right coordinate (latitude, longitude, altitude) on Terrain, the farthest right coordinate (latitude, longitude, on Terrain) (Elevation), the farthest left coordinate (latitude, longitude, elevation) on Terrain, and the latest left coordinate (latitude, longitude, elevation) on Terrain.
Next, the frame rotation angle shown in FIG. 17 is determined. If the attitude of the aircraft is the reference direction, the horizontal line passing through the optical axis of each digital camera frame is parallel to the horizontal line of the earth, but coordinate transformation by pitch, yaw and roll of the aircraft is not possible.
If so, a rotation angle is generated from the horizon of the earth.
The horizontal line of the screen frame when the attitude of the aircraft is in the reference direction is given below.
Coordinate transformation by pitch, yaw and roll of the aircraft
Then the horizontal line of the screen frame is given by
Therefore, the frame rotation angle is
It becomes. Where ・ represents inner product,
Represents the norm.

Next, in the processing block 484, processing for removing the influence of airborne haze is performed. The aerial three-dimensional video generation system 150 of the present invention uses the original images taken from different viewpoints to represent videos from various directions in the city. For this reason, if a shadow due to direct sunlight is present in the original image, the shooting time must be different for each original image, which makes it extremely difficult to see when the moving image is synthesized. In order to avoid this, an aerial image is acquired by shooting from below the cloud on a cloudy sky without direct sunlight. For this reason, in the aerial image acquisition plan file 101, shooting is performed by setting the flight altitude to 700 to 800 m, which is lower than the altitude at which cumulus clouds or stratus clouds appear. Since shooting in cloudy weather is a condition where the aircraft cannot be used for aerial surveying, there is an advantage that the surplus time of the aircraft can be used at low cost. On the other hand, since there is a lot of moisture in the air, there is a drawback that the visibility is shorter than that in fine weather.

Moisture is reflected and diffused in a straight line as minute water droplets, so that influence occurs depending on the passage distance in the air. In particular, in the case of oblique aerial photographs, the influence of moisture increases as the off-nadir angle increases. FIG. 18 is a diagram showing the relationship between the image and the field of view in the defocusing effect removal processing, and the lower end of the photographed image 495 is the foreground part a496 and the nearest part of the photographed surface part 499. The upper end portion of the photographed image 495 is a distant view portion 498 and is located at the farthest portion of the photographing ground surface portion 499. As the distance through which light passes through the air from the foreground portion a496 toward the far view portion 498 increases, the scattering of light by minute water droplets increases, the contrast decreases, and white turbidity occurs. As a result, as shown in FIG. 19A, when the histogram of the image without a haze is the normalized histogram 500 distributed from the lower limit luminance value to the upper luminance limit value 507, the foreground portion a histogram 501 in the foreground portion a496. Thus, the range between the upper limit value and the lower limit value becomes narrower, and the median value shifts to the white side, that is, the higher brightness level.

This tendency is most noticeable in the distant scene portion histogram 502, the value width 505 which is the difference between the histogram upper limit value 504 and the histogram lower limit value 503 is minimized and the median value 506 is maximized. In the photographed image 495 of FIG. 18, the range for which the histogram is taken is the entire range in the horizontal direction of the screen, and the vertical direction is a strip with a constant width such as the foreground portion a496 and the foreground portion b497. When the histogram is calculated, characteristics as shown in FIG. 19B are obtained. Although the density of the haze may vary depending on altitude, it may be assumed that there is no difference in density in the horizontal direction, so the value width fitting curve 513 may be approximated as monotonically decreasing and the median fitting curve 512 may be approximated as monotonically increasing. Since these curves have unknown distribution due to the height of the haze, each pixel value is corrected with a correction coefficient of a value width and a median value obtained from the fitting curve for each row after being obtained by fitting from an actual measurement value.
The luminance correction of each pixel is performed for each horizontal line (Equation 8).

Normalized histogram median + (pixel brightness-median fitting value)
* Obtained by normalized histogram value width / value width fitting value. The processing described above is the processing described from processing block 520 to processing block 524 in FIG. FIG. 21 shows an example of the processing result of the defocusing effect removal processing of the present invention. As a result of processing the pre-moisture removal image 530, a post-blur removal image 531 is obtained.

Next, the processing block 485 stores the results obtained in the processing blocks 481 to 484 in the corresponding image data portion of the aerial image database 143 shown in FIG. In the final processing block 486 of the processing flow of the aerial image database generation / registration system of FIG. Generate. The contents of the processing block 486 are described in more detail in the aerial image index generation registration processing flow of the aerial image database registration system of FIG. Hereinafter, a description will be given with reference to FIG. The contents of the processing block 540 will be described with reference to the structure and term definition of the aerial image index mechanism in FIG. The image taken at the shooting position 550 includes shooting positions (latitude, longitude, altitude) 550, normalized image vector G = ^t (XYZ) 551, frame rotation angle 553, as shooting parameters of the aerial image database 133 in FIG. Image distance 552, Terrain center coordinates (latitude, longitude, elevation) 554, Terrain latest right coordinates (latitude, longitude, elevation) 555, Terrain farthest right coordinate (latitude, longitude, elevation) 556, Terrain top The far left coordinate (latitude, longitude, altitude) 557 and the latest left coordinate (latitude, longitude, altitude) 558 on Terrain have already been stored, and FIG. 23 illustrates this. The image photographed at the photographing position 550 captures the ground image range 549. Since the latitude and longitude are defined in the ground surface and Terrain 580, if meshing is performed every second, a cell of 30 m in the latitude direction and 25 m in the longitude direction can be formed near 35 ° latitude in the Tokyo region. If meshed every 0.4 seconds, a cell of 12 m in the latitude direction and 10 m in the longitude direction can be configured. Considering the 65km square city area, this number of cells is only 35M cells. The ground cell 559 included in the ground image range 549 can be easily obtained because the four end points of the ground image range 549 are known from the latitude and longitude.

For each locality cell, it is necessary to be able to search for an image within a sufficiently small solid angle range i221 in all directions of the celestial sphere 222 shown in FIG. For this purpose, as shown in FIG. 24 (a), a zenith vector 570 in which the celestial sphere is directed vertically upward from the center of the ground cell 559 is defined. Hereinafter, the celestial spheres are cut into concentric circles based on the zenith vector. Define Zenith Index I 572. The zenith index i is defined sequentially as i = 0,1,2, ..., i = 0 is the zenith vector, and as i increases sequentially, the angle formed with the zenith vector increases with the same width, and becomes the same concentric circle Defined from zenith to horizon. A solid angle cell center vector is defined from the surface cell 559 through the zenith index 572. The solid angle cell center vector 574 is always defined on the zenith index 572 at a constant azimuth angle interval.

In FIG. 24A, the solid angle cell 573 corresponds to the sufficiently small solid angle range I 221 in FIG. 2, and is a basic unit for image retrieval from the ground cell 559. The solid angle cell 573 is defined as a cone that accommodates a vector whose angle formed with respect to the solid angle cell center vector 574 is, for example, 3.5 ° or less. FIG. 24 is a view of the surface cell 559 as viewed from the zenith direction. Solid angle cell 573 is defined adjacent to and overlapping each other with respect to zenith index I 572 and fills the entire circumference of zenith index i 572.
The angle formed by the solid angle cell center vector 574 passing through the zenith vector 570 and the zenith index I 570 is called a zenith separation angle 571, and is a numerical value defining the zenith index I 572.

FIG. 25 shows the index structure of the solid angle cell 573. FIG. 25B is a view of the ground cell center point 575 looking down from the zenith direction. A solid angle cell address (i, j) with a two-dimensional subscript is defined, i indicates a zenith index i572, and j indicates a solid angle cell in the jth direction on a concentric circle of the zenith index i. Solid angle cell address (0,0) Solid angle cell center vector of 584 is a zenith vector 570, surrounding this, a solid angle cell address (1,0) 585, solid angle cell address ( 1,1) 586 and solid angle cell address (1,2) 587 are defined below until the entire circumference of zenith index 1 is filled. Outside the zenith index 1581, along the zenith index 2, solid angle cell address (2,0) 588, solid angle cell address (2,1) 589, solid angle cell address (2,2) 590 are the zenith index 2 It is defined until the entire circumference is filled. The solid angle cells 573 overlap each other and fill the celestial sphere so that there is no gap. The solid angle of the solid angle cell is set so that three-dimensional moving image generation is performed smoothly. The number of solid angle cell addresses increases as the zenith index approaches the ground surface. In this way, for all the ground cells, an image obtained by photographing the ground cell can be searched by the direction on the celestial sphere.

FIG. 26 shows the structure of the aerial image index relation table and its correlation. The structures of the zenith index and the zenith cell address are the same regardless of the ground cell, and this structure is defined by the zenith index parameter table ZNPRMT597 and the azimuth index parameter table DRPRMT598. In this structure, the zenith index parameter table ZNPRMT597 defines the zenith separation angle 571 with respect to the zenith index i572 sequentially from the zenith index 0, and the vertical direction component when the center vector 574 corresponding to the zenith index i572 is a normalized vector. Store as vertical component. Since the number of solid angle cells included in the zenith index i 572 increases as it approaches the horizon from the zenith, it is defined as the azimuth number NDRi corresponding to each zenith index. The direction index parameter table DRPRMT 598 defines the azimuth X component and the azimuth Y component of the solid angle cell center vector for the solid angle cell address corresponding to the azimuth number NDRi defined corresponding to the zenith index. Normalization is performed so that the vector norm becomes 1 together with the vertical component described in the zenith index parameter table ZNPRMT597.

Next, the structure of the image data index table corresponding to the ground cell will be described. The ground cell position index table TCINXT 600 of FIG. 26 is defined for all the ground cells 559, and the zenith index table address ADDR _ij 603 is designated corresponding to the ground cell. The subscript of ADDR _ij indicates that the longitude index LONINX 601 is i and the latitude index LATINX 602 is j corresponding to the position on the ground surface. The zenith index table address ADDR _ij 603 indicates the zenith index table ZNINXT 604 corresponding to the ground surface cell ij. In the zenith index table ZNINXT, an orientation index table address ADDR _ijk 605 is defined corresponding to the selected zenith index k, and the orientation index table DRINXT 606 is defined accordingly. Since there are the number of azimuths specified in the zenith index parameter table ZNPRMT 597, if the azimuth index is m, ADDR _ijkm is specified as the address of the image address table 608. The image address table thus selected indicates the number of image data and the image data address included in the solid angle cell of the solid angle cell address (k, m) with respect to the ground cell ij. In this way, if the correlation between the aerial image index and the association table is defined according to FIG. 26, it is possible to retrieve an image taken from an arbitrary direction in the air from the position of the ground surface.

Based on the above description of the image data index mechanism, an aerial image index generation registration process flow of the aerial image database registration system shown in FIG. 22 will be described below. In processing block 540, if there is one image as shown in FIG. 23, all the ground cells included in the ground image range 549 are obtained. In processing block 541, processing is sequentially performed for all the ground cells obtained in processing block 540. In a processing block 542, a vector for estimating the shooting position 550 is obtained from the center coordinates of the ground cell. In the processing block 543 and the processing block 544, a solid angle cell address (k, m) including a vector for estimating the photographing position 550 from the center coordinate of the ground cell by the inner product operation from the definitions of the zenith index parameter table ZNPRMT 597 and the orientation index parameter table DRPRMT 598. ) Based on this information, the processing block 545 _increments the number of image data in the image data table ADDR _ijkm by 1, and additionally _writes the image data address.

FIG. 27 shows a configuration example of a visual field by three types of digital cameras having different focal lengths. In the generation of a three-dimensional moving image of a cityscape according to the present invention, the original image when the depression angle is small becomes an image with a large off-nadir angle, so that the image is inevitably far from the aircraft. For this reason, in order to generate a moving image when the depression angle is small without reducing the resolution, it is necessary to consider that the resolution is not deteriorated when the image is captured when the off-nadir angle is large. Therefore, as shown in FIG. 27, as the off-nadir angle increases, the focal length of the telephoto lens is lengthened and the resolution is maintained. At the same time, a digital camera having three types of off-nadir angles is adopted, and the off-nadir angle increases. This is an example in which the focal length is increased to make the ground resolution uniform. In FIG. 28, a digital camera 350a is used for direct-point shooting, and eight digital cameras 350b to i are arranged with the same small off-nadir angle on the radiation, and the digital cameras 620j to y are arranged on the outside. Sixteen units are arranged on the radiation with the same large off-nadir angle. Digital cameras have been significantly improved in size and performance, and can be sufficiently accommodated in the existing aircraft floor hole 398 for aerial camera. FIG. 29 shows an example of the photographing range of a digital camera assembly using three types of digital cameras having different focal lengths. As a typical embodiment, the digital camera 350a is oriented directly below at a focal length of 35 mm equivalent to 50 mm, the digital cameras 350b to i are oriented at an off-nadir angle of 35 ° equivalent to a focal length of 70 mm, and the digital cameras 620j to y are off-nadia equivalent to a focal length of 105mm. There is a combination of 60 ° angle orientation.

FIG. 30 shows the simplest configuration example of the stable platform device 395 for mounting on an aircraft.
The stable platform device 395 is installed in a shape that covers the aircraft floor hole 397 and covers the upper part, and includes a fixed part 635, a vertical movement part 636, and a rotating part 637 from the bottom. The rotating unit 637 has a rotating table structure, and an IMU 396 for posture detection is installed in close contact with the rotating table 637. A digital camera assembly 360 is suspended below the rotating unit 637 from the aircraft floor hole 397. All downward views can be accommodated in all digital cameras 350a-i without projecting outside. The stable platform device 395 can also shoot a digital camera aggregate 360 or 621 in the same manner using the digital cameras 350a-i and the digital cameras 620j-y. Note that the method of joining the rotating portion 637 and the digital camera assembly 360 or 621 needs to be fixed with rigidity. The stable platform device 395 is intended to maintain the attitude of the shadow against the disturbance of the aircraft attitude, but the aircraft attitude disturbance that can be dealt with is a maximum of 7 to 10 degrees in the pitch angle and the roll angle, and the maximum in the yaw angle. It is sufficient to assume 20 °. The fixing part 635 is fixed to the aircraft floor 398. The vertical movement unit 636 is held via the vertical drive mechanisms A to D641 to 644. The vertical drive mechanisms A to D 641 to 644 move up and down independently to correct the pitch angle and roll angle disturbances. The vertical drive may be a hydraulic mechanism or a conversion method of a rotary motion by a worm gear into a vertical motion. Further, a rotating part 637 is mounted on the vertical movement part 645 so as to be held by the bearing mechanism 645. The rotation drive mechanism 640 is fixed to the vertical movement unit 636 and installed in contact with the outer extension of the rotation unit 637. It is only necessary to transmit a rotational motion between the rotation unit 637 and the rotation drive mechanism 640, and the rotation unit 637 is rotated to cancel the disturbance to the yaw angle of the aircraft. FIG. 31 shows a signal information flow of the stable platform device. The stable platform control system 394 periodically inputs the pitch angle, yaw angle, and roll angle from the IMU 396 to obtain the deviation from the reference value and cancel the deviation of the pitch angle and roll angle from the vertical drive mechanism A to D641 to 644. To control the vertical movement 649. In order to cancel the deviation of the yaw angle, the rotation mechanism 640 is controlled to perform the rotation motion 650.

Next, aerial moving image generation processing in the cityscape three-dimensional moving image amusement system will be described with reference to FIGS. The concept of aerial moving image generation by morphing and description of variables will be described with reference to FIG. 32, and the overall processing flow will be described based on the aerial moving image generation processing flow in the cityscape three-dimensional moving image amusement system of FIG. In FIG. 32, the viewpoint moves along the viewpoint path P (t) 270 along with the time t, and during that time, the ground surface is moved to the target trajectory T (t).
A line of sight vector V (t ₁ ) to V (t ₄ ), V (t _i ) is taken from a position on the viewpoint path P (t) 270 along time 280 from time t ₁ to t ₄ , t _i . In the present invention, as a method of generating a three-dimensional moving image of the city landscape for the target trajectory T (t) 280 from the viewpoint route P (t) 270, the viewpoint P (t ₁ ) at times t ₁ , t ₂ , and t _3. The aerial image m-1666 that is close to the line of sight of 675, the viewpoint P (t ₂ ) 676, and the viewpoint P (t ₃ ) 677 and has little parallax is searched from the aerial image database 130, and the line-of-sight vectors at times t ₁ , t ₂ , and t ₃ are obtained. Between V (t ₁ ) 681, V (t ₂ ) 682, and V (t ₃ ) 683, the parallax difference from the image vector G _m−1 672 is corrected by the morphing process for the aerial image m−1666, and smooth A moving image is generated. From the line-of-sight vector V (t ₄ ) 684 at time t ₄ , it is determined that the aerial image m 667 has less parallax due to a difference in viewpoint than the aerial image m-1666, and the aerial image m 667 is switched to the original image. t ₄
3D animation generation of a smooth cityscape by correcting the difference in parallax from the aerial image m 667 with the morphing process for the aerial image m 667 until the next original image is selected after the line-of-sight vector V (t ₄ ) 684 Do it.

FIG. 33 shows the above processing in a processing flow. In a processing block 690, the moving direction, moving speed, position, and line-of-sight vector to be calculated as the next moving image frame (next frame) from the graphic user interface system 180 are shown. take in. In a processing block 691, the viewpoint position of the next frame, the line-of-sight vector, and the target point coordinates on the ground surface are calculated based on this value. In processing block 692, it is determined whether the ground index position of the next frame is the same as that of the current frame based on the center point of the current frame and the next frame. If they are not identical, it is necessary to re-determine the optimal original image and go to processing block 694. If they are the same, it is determined in processing block 693 whether the line-of-sight vector of the next frame is within the same solid angle cell as the current frame. This determines whether the direction of the line of sight has changed to some extent from the current frame. If it is determined that the line of sight does not fall within the same solid angle cell, it is necessary to re-determine an optimal original image in processing block 694. If it is determined that the cells are in the same solid angle cell, the original image of the current frame is continuously used.

The processing contents of the processing block 694 are shown in more detail in the processing flow of the original image search in the aerial moving image generation of FIG. 34 and the original image selection logic diagram of the three-dimensional moving image generation processing of FIG. In the processing block 700, a ground cell address (i, j) existing at the center of the next frame is obtained, and a line-of-sight vector connecting the center point and the viewpoint of the next frame is created and normalized. In processing blocks 701 to 703, a solid angle cell address (k, m) is obtained from the normalized line-of-sight vector with reference to the zenith index parameter table ZNPRMT597 and the azimuth index parameter table DRPRMT598, and is combined with the ground cell address (i, j). Thus, the address ADDR _ijkm of the image address table is obtained. Since the original image candidate indicated by the first data of the image address table exists at the same solid-angle cell address for the same ground cell address, the visual field from the viewpoint P (t _i ) from among the processing block 704 and the processing block 705 It is determined whether the original image has taken all the four corner points. In FIG. 35A, the four end points T _i1 715, T _i2 716, T _i3 717, and T _i4 718 on the ground surface of the field of view of the viewpoint P (t _i ) are the original image shooting position S _A 710 and the original image shooting. position S _B 711, a determination of process block 705 to determine whether a picture is taken image from the original image capturing position S _C 712, can be easily by a vector product computation. In the example of FIG. 35B, only the images from the original image shooting position S _A 710 and the original image shooting position S _B 711 are the four end points T _i1 715, T on the ground surface of the viewpoint P (t _i ). _{All of i2} 716, T _i3 717, and T _i4 718 are captured in the image. After all the original images satisfying the conditions of the processing block 705 are obtained, the shortest image with the image distance closest to the ground cell address (i, j) is selected as the original image in the processing block 706. The image distance is calculated as one data of the shooting parameter in the aerial image database in FIG. 17, and selecting the original image with the shortest image distance is a procedure for eliminating an obstacle that blocks the view between the viewpoint and the target.

Next, returning to FIG. 33, the contents of the morphing process of the processing block 697 will be described with reference to FIG.
Original image shooting position with respect to original image m
671 and image vector
Once 673 (norm = 1) and Terrain (ground surface) are determined, the image vector
The intersection of 673 and DEM
And the intersection of the four corners of the image and Terrain
,
,
,
Is decided. As shown in FIG.
,
,
,
Are not on the same plane,
Through
,
Midpoint
When,
,
Midpoint
A vector parallel to the vector passing through
(
=
),
Through
When
Midpoint
When,
When
Midpoint
A vector parallel to the vector passing through
(
=
).
Through
When
The surface that is stretched by is called the Terrain plane
It expresses.
The perpendicular of
(Norm = 1).
The street, image vector
A plane perpendicular to
That's it.
Is the original image shooting position perspective
It is an image (field of view) taken from 671.
->
There is a conversion to
That's it. This is a photographing action of the original image m. point of view
, Viewpoint coordinates
And gaze vector
(Norm = 1) and Terrain plane
(Common) is determined as well
When
,
,
,
Is decided.
Street gaze vector
A plane perpendicular to
That's it. Plane
What is Terrain surface
In perspective
In the field of view
Part of point of view
Vision
When
Is
When
And viewing angle
If it is decided, it is decided uniquely. Plane
Plane
The mapping to
When
And angle of view of original image m
If it is decided, it is decided uniquely.
― ＞
― ＞
By the correspondence of
of
Each pixel corresponding to
of
The morphing algorithm is completed if it is associated with the pixel corresponding to.

Terrain plane in general
Any point on
Is
= a
+ b
+
It is expressed.
_{Here is the normal condition,}
= 0 and
Substituting = 0
=
Is obtained. Here, represents a vector dot product.
Center line of sight
And Terrain plane
Intersection of
Is
_But
=
+ c
Substituting this for satisfying
(C
+
) =
C =
And
=
+
=
And perspective
Terrain plane at the center of the image
The coordinates at were obtained.

Similarly, the perspective
Normalized gaze vectors at the four corners of the field of view
,
,
,
And these and Terrain plane
Intersection of
,
,
,
Terrain plane
The coordinates at can be obtained. In addition to this coordinate
Find the transformation on the surface.
Terrain plane
Any point on
Is
= a
+ b
+
Represented by
_{It is a perpendicular condition,}
= 0 and
= 0, the normalized gaze vectors at the four corners
,
,
,
Against
=
+ c
However, k = 1,4 is satisfied
(C
+
) =
C =
And
=
+
=
And perspective
Terrain plane at the four corner points of the image
The coordinates at were obtained. This plane
Coordinates on the plane
The plane that is the original image by associating it with the coordinates above
View any point above
You can see if it should be reflected on the screen.
Perspective on
Screen center and four corner point coordinates
_,
,
,
,
(collect
_{, K = 1,2,3,4, c)}
Above
_,
,
,
,
(collect
_{, K = 1, 2, 3, 4, c).}
The horizontal line of the screen on the screen
The vertical line
If
Any point on the face
For arbitrary constants a and b
=
= a
+ b
+
It is represented by Image vector
Is
,
Because it ’s perpendicular to
= 0,
= 0, and
When
Is the image shooting position
Because it ’s on the same line.
_{= d (
-}
) +
So _{d (
-}
) +
= a
+ b
+
And
D =
Because it becomes
=
It becomes.
=
+
Already in
So now we have a perspective
Correspondence with the screen of the original screen m of the field of view was completed.

next,
Find the transformation that moves to the z-axis. here,
= 1
=
Then, it is rotated around the z-axis and placed on the yz plane, and then rotated around the y-axis to coincide with the z-axis.
The rotation around the z axis is
Given in.
Next, the rotation around the y axis is
Given in.
Combining both rotations gives

=
It becomes.
Plane
Intersection of
_{for (k = 1,2,3,4, C)}
To the z axis,
_{(k = 1,2,3,4, C)} is converted to a point on the xy plane. This corresponds to the position on the screen of the original image m. That is morphing.

The description about the aerial 3D video generation of the city landscape 3D video generation amusement system according to the present invention is finished, and the description shifts to the description on the road 3D video generation of the city landscape 3D video generation amusement system. A three-dimensional moving image is generated with a human eye from an arbitrary place on the road or passage and an arbitrary line-of-sight direction for an arbitrary place in the city. In generating a 3D moving image on the road, a 3D moving image is generated using an actual image without generating a 3D model related to a structure in an urban space, as in the case of generating an aerial 3D moving image. On the road, the degree of freedom of the viewpoint is smaller than in the air where you can specify the line of sight freely in free space, and the location is limited to about 1.7 m above the ground on the passage, but the passage has a structure different from the Terrain model by DEM Therefore, the passage is modeled by a graph representation. In the method of the present invention, the projection plane of the actual image taken in advance must be set, such as Terrain as the projection plane of the image in the case of an aerial photograph.

As a result of examining the viewpoint as a human eye and the range in which the line of sight can move, a vertical plane along the boundary line between the passage and the other urban areas is the main projection plane of the original image taken on the road, and the forward view or on the passage For the rear view, a vertical plane perpendicular to the passage and separated by a certain distance is used as an auxiliary projection plane. Since the method of the present invention captures a cityscape in advance with an arbitrary line of sight from an arbitrary point where a human eye can be present on the passage, it is diagonally above the horizontal direction with 8 to 16 directions in all directions and with an elevation angle from the horizontal. A digital camera assembly that shoots the entire direction from 8 to 16 directions is installed on the roof of the vehicle, and a road image database is constructed. Note that the diagonally upper part can be omitted by using a wide-angle lens. The moving image generation includes a process of selecting an optimal original image and a process of generating a smooth moving image by a morphing process according to the moving state of the viewpoint, the line-of-sight direction, and the position in the city.

Hereinafter, the road image acquisition system 110, the road image database generation / registration system 130, the road image database 150, and the road three-dimensional video generation system 170 will be described in detail according to the block diagram of the city landscape 3D moving image amusement system. FIG. 40 shows a processing flow of the road image acquisition system. Hereinafter, a detailed method of realizing the road image acquisition system 110 will be described with reference to FIG. First, a moving route and imaging points or intervals are set so that the imaging point setting process 741 can capture images at regular intervals along roads and passages in the city, and the road image acquisition plan file 111 shown in FIG. 53 is configured. And In this shooting plan, a road ID number and a shooting point on the road are defined. The description of the road structure will be described later because it must be strictly defined including the road ID number. Next, in the shooting process 742, it is determined that the vehicle 803 has reached the shooting point 766 shown in FIG. 42 defined in the road image acquisition plan file 111, and shooting is performed by the road shooting control system 806 shown in FIG. The captured images in the digital camera are stored in the road image primary file 112 together with the position and orientation data acquired in the imaging process at predetermined timings.

  FIG. 41 is a diagram showing a configuration example of the digital camera assembly 755 constituting the road image acquisition system 110 according to the present invention. For the purpose of photographing high density efficiently from various viewpoints on the road, for the purpose of photographing diagonally above the digital cameras 750a-h where the horizontal digital cameras 750a-h are arranged radially at equal intervals around the circumference. Digital cameras 750i-p are arranged to form a digital camera assembly. The support structure other than the digital cameras 750a to 750p constituting the digital camera assembly 755 is required to have high photographing direction accuracy, and any structure that is lightweight and highly rigid may be used. It can be composed of a plate or carbon resin.

  FIG. 42 shows a setting example of the route and the shooting point of the road image acquisition system by the vehicle. FIG. 42 (b) shows an example of setting the shooting point 766. The road image acquisition system 765 performs shooting at the shooting point 766 while moving along the road according to the shooting points 766 configured in a row on the road in the road image acquisition plan file 111. The mutual interval of the shooting points 766 is selected so that the shot images become smooth moving images by morphing in the road three-dimensional moving image generating system. When the digital camera aggregate 755 is photographed, the range indicated by the photographing ranges 760a-p of the digital cameras 750a-p in FIG. Depending on the setting of the focal length of the lens, the overlap between the digital camera 750a-p and the range captured by each digital camera in the imaging range 760a-p changes. For the purpose of the present invention, it is desirable to set so that the ground surface is covered almost continuously while slightly overlapping each other.

FIG. 43 shows the relationship between the vertical shooting pattern and the image projection plane of the road image acquisition system. In the aerial three-dimensional video generation, the projection plane of the captured image is Terrain, but in the road three-dimensional video generation, the human eye is a realization target, and Terrain cannot be the projection plane. Further, since the image to be captured is also information with a horizontal direction and an elevation angle, the projection plane of the image captured from the road is a plane that stands perpendicular to the boundary line between the road and the building as shown in FIG. The boundary surface between the road and the building may be approximately the boundary surface of the road. Since the definition of the image projection plane must be strictly performed together with the definition of the road and made into a database, it will be described later. In addition, this method may be applied anywhere as long as it is a human eye passage, regardless of the possibility of passage of the vehicle, and can be applied if there is a human passage even in an underground passage or a building. FIG. 43 illustrates the shooting range and direction of the digital cameras 750i to 750p and the shooting range and direction of the digital cameras 750a to 750h.

  In the road 3D video generation system of the urban landscape 3D video amusement system of the present invention, the original image of the moving image must be searchable from any location on the road because the boundary plane of the road or passage is the projection plane of the image. And must be able to register. The search requires high speed, and it must be possible to register and search images from all directions from the human viewpoint. A road database system for this purpose will be described with reference to FIGS. This description is the technical basis of the road three-dimensional video generation system.

  FIG. 44 describes a road description method using a graph. Every road has a width, and the center line forms a graph to form the basis of road expression. The centerline of this road is called a centerline graph 780. Consists of end points and sides. There are always end points at road intersections, junctions, branch points, and turning points. A road is always defined as a set of continuous edges and endpoints whose endpoints are both ends. 777 is uniquely assigned. FIG. 45 shows a method for searching for a road from latitude and longitude. For example, when the mesh is made in units of 0.1 seconds in latitude and longitude, in the Tokyo region, the mesh becomes 2.5 m in the longitude direction and 3 m in the latitude direction. When meshes are created in this way, each mesh always has a road or not. This mesh is referred to as a surface cell 781. If there is a road, the road ID No. can be assigned to the surface cell 781 to access the road if there is a road from any point where the latitude and longitude are known. If defined in this way, one mesh may be related to a plurality of road ID numbers. A perpendicular line is dropped from the ground cell center point 782 to the center line graph, and the road ID No. having the shortest distance from the foot is defined as the road ID No. of the ground cell. In the example of the intersection enlarged in the lower right of FIG. 45, two perpendicular lines can be dropped on the center line graph 780. In this case, the perpendicular line which is the shorter of the perpendicular line A783a and the perpendicular line B783b. B783b road ID No. Is selected.

FIG. 46 defines the structure of a road database for describing the structure of the road. The ground cell road index table CLLRDINXT indicates a corresponding graph portion of a road graph data table in which road ID No. and road graph data corresponding to the road ID No. are described from meshed latitude and longitude indexes (i, j). You can search for a relative record address. The road graph address table RDGRADRT designates the start address in the road graph data table RDGDT corresponding to the road ID No. This is because the road graph data table RDGDT has different end points and side sizes depending on the road ID No. The road graph data table RDGDT describes the road structure as a graph. The first record is road ID No., and the second record is the total number of endpoints. The road is sequentially repeated from the start point until the end of the road range specified by the road ID No. in the form of describing the end point coordinates and the attributes of the side to the next end point. There are the following three types of attributes.
Type 1 attribute Attribute a on the side to the next end point a Projection plane code 0 No projection plane on either side of the side.
There is a projection plane only on the left side in the direction of travel of one side.
There is a projection plane only on the right side in the direction of travel of the two sides.
3 There are projection planes on both sides in the direction of travel.
b Width Indicates the distance from the center line graph at both ends of the road.
Type 2 attribute Information on road connection at the end point a Connected road ID No.
b Relative record address of connection point in road graph data table RDGDT corresponding to connection destination road ID No. This information is used to determine whether the road connection is crossing, branching or merging.
Type 3 attribute Indicates the end of the road at the end point.

In the following, a method for constructing a road graph data table will be described corresponding to a case that actually exists.
FIG. 50a shows a method for describing an orthogonal intersection. Road ID No. = k 777k and Road ID No. = l (El) 777l is the end point k, i + 2 792k, i + 2 and end point l, j + 2
Crossing at 793l, j + 2. The reason why two subscripts are used in the end point notation is that the first subscript represents the road ID No. and the second subscript represents the order of the end points within the same road ID No. This end point is the same end point named separately on both roads. The structure description of road ID No. = k 777k will be described with reference to FIG. 50a-1 road graph data table RDGDTk 795k and FIG. 50a. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). As a result of entering the intersection at the end point k, i + 1 792k, i + 1, the projection planes on both sides disappear (attribute code = 0), and the width is undefined and blank (end point k, i + 1). Endpoint k, i + 2
792k, i + 2 crosses road ID No. = l (el) 777l (attribute code = l (el)) attribute 2 is the end point of record ID 2 of road ID No. = l (el) 777l Indicates crossover (end points k, i + 2). As a result of exiting the crossing point at the end points k, i + 3 792k, i + 3, it has projection planes on both sides (attribute code = 3), and the width becomes w _k (end points k, i + 3). Furthermore, it continues after the end point k, i + 4 792k, i + 4. The structure description of road ID No. = l (el) 777l will be described with reference to FIG. 50a-2 road graph data table RDGDTl (el) 795l and FIG. 50a. Road ID No. = l 777l starts from the end point l, j 793l, j at the lower end and has projection planes on both sides up to the next end point l, j + 1 793l, j + 1 (attribute code = 3). Is w _l (end point l, j). As a result of entering the intersection at the end point l, j + 1 793l, j + 1, the projection planes on both sides disappear (attribute code = 0), and the width is undefined and blank (end point l, j + 1). Intersection with road ID No. = k 777k at end point l, j + 2 793l, j + 2 (attribute code = k), attribute 2 intersects at the end point where the record address of road ID No. = k 777k is 2 (End point l, j + 2). As a result of exiting the intersection at the end point l, j + 3 793l, j + 3, it has projection planes on both sides (attribute code = 3), and the width becomes w _l (end point l, j + 3). Furthermore, it continues after the end point l, j + 4 793l, j + 4.

FIG. 50b shows a method for describing a three-way. Road ID No. = k 777k to road ID No. = l (El) 777l branches at the end point k, i + 2 792k, i + 2 and the end point l, j 793l, j at the center of the 3 fork . The structure description of road ID No. = k 777k will be described with reference to FIG. 50b-1 road graph data table RDGDTk 795k and FIG. 50b. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). As a result of entering the 3-way at the end point k, i + 1 792k, i + 1, the projection plane is only on the right side (attribute code = 2), and the width is w _k (end point k, i + 1). At the end point k, i + 2 792k, i + 2, branch with road ID No. = l (el) 777l (attribute code = l (el)), attribute 0 is road ID No. = l (el)
Indicates that the record address of 777l branches at the end point of 0, that is, the start point (end point k, i + 2). Endpoint k, i + 3
792k, i + 3 in 3 results exits the fork path, will have a projection plane on both sides (attribute code = 3) road width is w _k (end point k, i + 3). Furthermore, it continues after the end point k, i + 4 792k, i + 4. The structure description of road ID No. = l (el) 777l will be described with reference to FIG. 50b-2 road graph data table RDGDTl (el) 795l and FIG. 50b. Road ID No. = l 777l starts from the end point l, j 793l, j of the 3 fork road ID
Branch from No. = k 777k (attribute code = k), attribute 2 indicates that the record address of road ID No. = k 777k is branched at the end point 2 (end point l, j), There are no projection planes on both sides in the 3 fork to the end point l, j + 1 793l, j + 1 (attribute code = 0), the width is undefined and blank (end point l, j). As a result of exiting the 3 fork at the end point l, j + 1 793l, j + 1, there are projection planes on both sides (attribute code = 3), and the width is w ₁ (end point l, j + 1). Furthermore, it continues after the end point l, j + 2 793l, j + 2.

FIG. 50c shows a branch path description method. Road ID No. = k 777k to road ID No. = l (El) 777l branches at the end points k, i + 2 792k, i + 2 and the end points l, j 793l, j at the center of the branch road. The structure description of road ID No. = k 777k will be described with reference to FIG. 50c-1 road graph data table RDGDTk 795k and FIG. 50c. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). As a result of entering the 3-way at the end point k, i + 1 792k, i + 1, the projection plane is only on the right side (attribute code = 2), and the width is w _k (end point k, i + 1). At the end point k, i + 2 792k, i + 2, branch with road ID No. = l (el) 777l (attribute code = l (el)), attribute 0 is road ID No. = l (el)
Indicates that the record address of 777l branches at the end point of 0, that is, the start point (end point k, i + 2). Endpoint k, i + 3
792k, i + 3 result leaves the branch passage, the now has a projection plane on both sides (attribute code = 3) road width is w _k (end point k, i + 3). Furthermore, it continues after the end point k, i + 4 792k, i + 4. The structure description of road ID No. = l (el) 777l will be described with reference to FIG. 50c-2 road graph data table RDGDTl (el) 795l and FIG. 50c. Road ID No. = l 777l starts from the end point l, j 793l, j of the branch road, and the road ID
Branch from No. = k 777k (attribute code = k), attribute 2 indicates that the record address of road ID No. = k 777k is branched at the end point 2 (end point l, j), There are no projection planes on both sides of the branch path up to the end point l, j + 1 793l, j + 1 (attribute code = 0), the width is undefined, and it is blank (end point l, j). A projection plane exists on the left side at the end point l, j + 1 793l, j + 1 (attribute code = 1), and the width is w ₁ (end point l, j + 1). The end point l, j + 2 exits the branch path at 793l, j + 2, and projection surfaces exist on both sides (attribute code = 3), and the width becomes w ₁ (end point l, j + 2). Furthermore, it continues after the end point l, j + 3 793l, j + 3.

FIG. 50d shows a method of describing road change. The structure description of road ID No. = k 777k will be described with reference to FIG. 50d-1 road graph data table RDGDTk 795k and FIG. 50d. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). The direction is changed at the end points k, i + 1 792k, i + 1, but the projection planes are left and right (attribute code = 3), and the width is w _k (end points k, i + 1). Furthermore, it continues after the end points k, i + 2 792k, i + 2.

FIG. 50e shows a method for describing a non-orthogonal intersection. Road ID No. = k 777k and Road ID No. = l (El) 777l crosses at the end point k, i + 3 792k, i + 3 and the end point l, j + 3 793l, j + 3 is doing. The structure description of road ID No. = k 777k will be described with reference to FIG. 50e-1 road graph data table RDGDTk 795k and FIG. 50e. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). As a result of entering the intersection at the end points k, i + 1 792k, i + 1, the projection plane is only on the left side (attribute code = 1), and the width is w _k (end points k, i + 1). At the end points k, i + 2 792k, i + 2, the projection surfaces on both sides disappear (attribute code = 0), and the width is blank (end points k, i + 2). At the end point k, i + 3 792k, i + 3, intersect with road ID No. = l (el) 777l (attribute code = l (el)) attribute 3 is road ID No. = l (el)
Indicates that the record address of 777l intersects at the end point of 3 (end point k, i + 3). Endpoint k, i + 3
792k, the projection plane on the right side in the i + 3 appeared (attribute code = 3), the road width is w _k (end point k, i + 4). Endpoint k, i + 5
792k, i + 5 result passes through the intersection, now has a projection plane on both sides (attribute code = 3) road width is w _k (end point k, i + 5). Furthermore, it continues after the end point k, i + 6 792k, i + 6. The structure description of road ID No. = l (el) 777l will be described with reference to FIG. 50e-2 road graph data table RDGDTl (el) 795l and FIG. 50e. Road ID No. = l 777l starts from the lower left end point l, j 793l, j, and has projection planes on both sides until the next end point l, j + 1 793l, j + 1 (attribute code = 3) The width is w _l (end points l, j). Endpoint l, j + 1
793L, result entered the intersection at j + 1 left projection surface disappears (attribute code = 2), the road width is w _l. The projection planes on both sides disappear at the end point l, j + 3 793l, j + 3 (attribute code = 0), and the width is undefined and blank (end point l, j + 3). At the end point l, j + 3 793l, j + 3 intersects with road ID No. = k 777k (attribute code = k), attribute 3 intersects at the end point where the record address of road ID No. = k 777k is 3 (End point l, j + 3). A projection plane appears on the left side at the end point l, j + 4 793l, j + 4 (attribute code = 1), and the width becomes w ₁ (end point l, j + 4). As a result of exiting the intersection at the end point l, j + 5 793l, j + 5, it has projection planes on both sides (attribute code = 3), and the width becomes w _l (end point l, j + 5). Furthermore, it continues after the end point l, j + 6 793l, j + 6.

FIG. 50f shows a description method of a 5-fork. Road ID No. = k 777k and Road ID No. = l (El) 777l intersects at the end point k, i + 3 792k, i + 3 and the end point l, j + 2 793l, j + 2 at the center of the intersection ing. Road ID No. = m 777m merges with Road ID No. = l (El) 777l at the end points l, j + 3 792l, j + 3. The structure description of road ID No. = k 777k will be described with reference to FIG. 50f-1 road graph data table RDGDTk 795k and FIG. 50f. Road ID No. = k 777k starts from the left end point k, i 792k, i and has projection planes on both sides up to the next end point k, i + 1 792k, i + 1 (attribute code = 3). There is a w _k (end point k, i). As a result of entering the intersection at the end points k, i + 1 792k, i + 1, the projection plane is only on the left side (attribute code = 1), and the width is w _k (end points k, i + 1). Endpoint k, i + 2
At 792k, i + 2, the projection planes on both sides disappear (attribute code = 0), and the width is blank (end points k, i + 2). Endpoint k, i + 3
792k, i + 3 intersects road ID No. = l (el) 777l (attribute code = l (el)) The attribute 2 is the end point where the record address of road ID No. = l (el) 777l is 2 Indicates that they intersect (end points k, i + 3). A projection plane appears on the left side at the end point k, i + 4 792k, i + 4 (attribute code = 1), and the width is w _k (end point k, i + 4). As a result of exiting the intersection at the end points k, i + 5 792k, i + 5, it has projection planes on both sides (attribute code = 3), and the width becomes w _k (end points k, i + 5). Furthermore, it continues after the end point k, i + 6 792k, i + 6. The structure description of road ID No. = l (el) 777l will be described with reference to FIG. 50f-2 road graph data table RDGDTl (el) 795l and FIG. 50f. Road ID No. = l 777l starts from the end point l, j 793l, j at the lower end and has projection planes on both sides up to the next end point l, j + 1 793l, j + 1 (attribute code = 3). Is w _l (end point l, j). As a result of entering the intersection at the end point l, j + 1 793l, j + 1, the projection planes on both sides disappear (attribute code = 0), and the width is blank. Intersection with road ID No. = k 777k at end point l, j + 2 793l, j + 2 (attribute code = k), attribute 3 intersects at the end point with record address 3 of road ID No. = k 777k (End point l, j + 2). At end point l, j + 3 793l, j + 3, merge with road ID No. = m 777m (attribute code = m), and attribute n joins at the end point where the record address of road ID No. = m 777m is n (End point l, j + 3). A projection plane appears on the right side at the end point l, j + 4 793l, j + 4 (attribute code = 2), and the width becomes w ₁ (end point l, j + 4). As a result of exiting the intersection at the end point l, j + 5 793l, j + 5, it has projection planes on both sides (attribute code = 3), and the width becomes w _l (end point l, j + 5). Furthermore, it continues after the end point l, j + 6 793l, j + 6. The structure description of road ID No. = m 777m will be described with reference to FIG. 50f-3 road graph data table RDGDTm 795m and FIG. 50f. Road ID No. = m 777m ends at the end point m, n 793m, n of the merge path, merges to road ID No. = l 777l (attribute code = l), attribute 3 is road ID No. = l 777l Indicates that the record addresses of the two are merged at the end point 3 (end point m, n). From the previous end point m, n-1 793m, n-1, there is no projection plane on both sides in the branch path (attribute code = 0), width is undefined and blank (end points m, n-1). A projection plane exists only on the right side from the previous end point m, n-2 793m, n-2 (attribute code = 2), and the width is w _m (end point m, n-2). Projection planes exist on both sides from the previous end point m, n-3 793m, n-3 (attribute code = 3), and the width is w _m (end point m, n-3).

  FIG. 51 is a diagram illustrating a configuration example of a road image acquisition system, which includes a car navigation system unit 801 and a data acquisition / recording system unit 800. The car navigation system unit 803 is a device for guiding the vehicle 803 to the photographing point 766 defined in FIG. 42B along the movement path 765, and vehicle position data is periodically obtained from the GPS 380. The function of the car navigation system unit 803 is based on the contents of the road image acquisition plan file 111 of FIG. 53, and the car navigation system unit of the road image acquisition system of FIG. 54 according to the processing flow of the car navigation system unit of the road image acquisition system of FIG. The vehicle is guided by the example of the display screen. The car navigation system itself has already been publicly implemented and has no novelty. However, in order to realize the object of the present invention, the part that efficiently guides the vehicle to the photographing point 766 is related to the present invention. First, determine which part of the city road you want to get before driving. Determine the shooting range on the map and formulate a driving plan. The shooting points 766 must be set uniformly and densely along the road or passage so as to achieve the object of the present invention, and the designated shooting route 825 is set based on the result. The road ID No. shown in the road image acquisition plan file in FIG. 53 is assigned to each road, and a shooting point 766 is assigned. The start coordinates and end coordinates of each road ID No., and the number of shooting points and the shooting point coordinates between them are assigned. Set by latitude and longitude. In this way, the road image acquisition plan file of FIG. 53 is constructed. The graphic user interface related to the construction of the road image acquisition plan file is publicly implemented as a map information system.

The function of the car navigation system unit 801 is shown by the processing flow of the car navigation system unit of the road image acquisition system shown in FIG.
54 is displayed in sequence until all shooting is completed for the road ID No. scheduled to be shot from among the road ID Nos. Registered in the road acquisition plan file in processing block 810. Since the designated road ID No. has a start coordinate, in order to start the road ID No. in the processing block 811, the position, traveling direction and speed of the travel start point are designated, and the contents of FIG. 54 are displayed on the monitor. To display guidance. In order to take a picture, the condition specified in the processing block 811 must be satisfied with a certain error range, for example, a position error with an accuracy of 1 m to 5 m or less. The processing points 812 and 813 sequentially guide the shooting points from the shooting point coordinates described in the road image acquisition plan file of the road ID No. selected in the processing block 810 to the final shooting point one by one.

The data acquisition and recording system unit 800 stores an IMU 400 for observing the attitude of the vehicle 803, a road imaging control system 806 including a program for controlling the digital camera 750 and processing imaging data, and a large amount of data including image data. A road image primary file 112 composed of a capacity disk device and a road image acquisition plan file 121 storing shooting points for issuing shooting commands to the digital camera 750 are mounted. A GPS 380 antenna for measuring the position of the vehicle 803 is provided outside the apparatus.

FIG. 55 shows a flow of information between components of the data acquisition / recording system unit 800 of FIG. FIG. 56 illustrates a processing flow of the road imaging control system 806. The on-street imaging control system 806 is composed of a CPU. The on-road imaging control system 806 periodically captures attitude data 450 of the stable platform device 395 from the IMU 396. When the stable platform 395 is operating normally, a constant posture is always maintained with respect to the inertial space regardless of the posture of the vehicle 803. The IMU 400 fixed to the vehicle 803 periodically sends vehicle attitude data 828, and the GPS 380 periodically sends GPS antenna position data 830 to the road imaging control system 806. The processing of the road imaging control system 808 is described in detail in FIG. 56, but the contents of the road image acquisition plan file 111 and the obtained GPS antenna position data 380 and vehicle attitude data 828 are processed in processing blocks 834 and 835. Sequentially collate, identify the road ID No., determine whether the road ID No. is passing through the nearest point of the designated road ID No. in the sequential processing block 836 until the end of the shooting of the road ID or the route departure At the timing closest to the shooting point within the shooting allowable range, the shooting command 457 is simultaneously sent to each digital camera 750 constituting the digital camera assembly at processing block 837, and at the same time, the vehicle data in the road image primary file shown in FIG. Part 840 is written at process block 838. Since the digital camera 750 can hold a 32 GB memory at the time of the present invention, it can hold the captured road image data 827 until at least one road ID number is completed. The vehicle 803 finishes one road ID No. and until it enters the next road ID No., the road image data 827 in the digital camera 750 is a road image primary file composed of an on-vehicle large-capacity disk device. Processing block 839 transfers the data to 112, and the internal memory of the digital camera 750 is emptied.

FIG. 57 shows a configuration example of the road image primary file 112, but an image header portion 470 and an image data portion 471 are prepared for each photographed image. The camera ID in the header part is a number for identifying each digital camera 750a-p constituting the digital camera assembly 755. The image data ID is an identification number assigned so that the images can be identified with each other, and the shooting date / time is the shooting time corresponding to the vehicle data portion 840. In the road image database generation / registration processing 130, each digital camera 750a is integrated. Used to calculate the optical axis direction of ~ i. The image data portion 471 is not processed. Further, since the camera parameter of the image header section 470 is normally fixed during running, the same set value is written.

FIG. 58 shows the simplest configuration example of the in-vehicle stable platform device 395.
The stable platform device 395 has the same structure as the aircraft-mounted stable platform shown in FIG. 30, but is installed on the vehicle ceiling plate, and an in-vehicle digital camera assembly 755 is installed on the stable platform device 395.

The road image database generation registration system 130 and the road image database 150 will be described in detail with reference to FIGS.
Since the road image database is searched from the image projection plane, the line of sight, and the viewpoint, first, the structure of the image projection plane and the structural relationship of the road graph database 151 must be defined. FIG. 63 illustrates an image projection plane based on the direction of the road graph and the direction of the line-of-sight vector. On the road, the road image of the entire circumference along the passage is projected onto the projection plane at the road boundary, and morphing is performed according to the positional relationship between the line of sight and the viewpoint. Since it is elongate, an extremely long road boundary surface is required to project an image, which is difficult to realize.

In order to solve this problem, the forward virtual projection plane 878 and the rear virtual projection plane 879 are not shown for the video in the traveling direction, the direction opposite to the traveling direction, or the direction close thereto, as shown by dotted lines in FIG. An actual image is projected on the image. The positions of the front virtual projection plane 878 and the rear virtual projection plane 879 move according to the road viewpoint. In the example of FIG. 63, if the traveling direction of the road, that is, the angle 882 formed by the road graph and the line of sight is in the range of 20 ° left and right, the front virtual projection plane 878 is used, and the front image taken by the road image acquisition system 110 is used. If the angle 882 formed by the road graph and the line of sight is in the range of 160 ° to 180 ° on the left and right, the rear virtual projection plane 879 is used, and the rear image captured by the road image acquisition system 110 is used. If the angle 882 formed by the road graph and the line of sight is in the range of 20 ° to 160 ° on the left side, the left projection plane 880 is used, and the left diagonally forward, left and left rear images with respect to the traveling direction photographed by the road image acquisition system 110. If the angle 882 formed by the road graph and the line of sight is in the range of 20 ° to 160 ° on the right side, the right projection plane 881 is used and the right front side and the right side with respect to the traveling direction photographed by the road image acquisition system 110 are used. Use the right rear image. Here, an example has been described in which the projection plane is switched at an angle 882 between the road graph and the line of sight of 20 ° forward and left and 160 ° backward right and left, but this power can be made variable according to the actual reality. However, in the following, 20 ° is selected as a specific example. Hereinafter, the road image database generation / registration system 130, the road image database 150, and the road three-dimensional moving image generation system 170 are constructed in accordance with the projection plane method described above.

Next, the structure of the image projection plane will be described with reference to FIG. When the viewpoint and line of sight are determined, the road three-dimensional moving image generating system of the present invention determines a projection plane, searches for an optimal actual image, and performs morphing according to parallax. For this reason, there must be a real image with little parallax with respect to a given viewpoint and line of sight, and the real image must be searched at high speed. In order to specify an arbitrary point of road ID No.m, an end point sequence is defined, and the “end point m, i coordinate for the position of the m-th end point from the beginning of the road and the subsequent side section in the graph. (X _mi , y _mi ) ”, which is determined by the form of the road and is too long for the definition of the projection plane. For this reason, as shown in FIG. 64, the left and right projection plane sections are defined as projection plane sections, and the left and right projection planes corresponding to the sides are defined as “projection plane sections” and are composed of end points and sides. Subdivide as a graph. The projection plane segment on the left side in the direction 883 of the road graph corresponding to the end point m, i coordinate (x _mi , y _mi ) 890 is
0th projection plane section mil0 coordinate (x _mil0 , y _mil0 ) 891
The first projection plane section mil1 coordinate (x _mil1 , y _mil1 ) 892
The second projection plane section mil2 coordinates (x _mil2 , y _mil2 ) 893
The third projection plane section mil3 coordinates (x _mil3 , y _mil4 ) 894
The same is defined below. The subscript l indicates the left side.
Similarly, the right projection plane section in the direction 883 of the road graph is
The 0th projection plane section mir0 coordinate (x _mir0 , y _mir0 ) 895
The first projection plane section mir1 coordinate (x _mir1 , y _mir1 ) 896
The second is the projection plane section mir2 coordinate (x _mir2 , y _mir2 ) 897
The third projection plane section mir3 coordinates (x _mir3 , y _mir4 ) 898
The same is defined below. The subscript r indicates the right side. Each projection plane section is represented by an edge in the direction 883 of the road graph following the end point coordinates shown above.

Next, an index system is constructed so that an image captured by the road image acquisition system 110 is searched as an original image by the following search procedure when the latitude and longitude of the start point are designated.
Latitude / Longitude-> Road ID
No .-> End point coordinates-> Projection plane section coordinates Projection plane section coordinates and line-of-sight direction-> Original image In FIG. 64, a digital camera aggregate 755 exists at the original image shooting position R903, and one of the digital cameras 750 therein. The image vector R904 is taken as a normalized vector representing the camera optical axis, the image vector 904 and the image right end R906 intersect with the projection plane section mir2 coordinates (x _mir2 , y _mir2 ) 897, and the image left end R905 is projected. Crosses the plane section mir2 coordinate (x _mir2 , y _mir2 ) 897. Since the usage of the projection plane in FIG. 63 differs depending on the angle formed by the image vector and the direction 883 of the road graph, the original image is registered by linking to the projection plane division according to the following principle.
(1) 0 °
Angle formed by image vector for direction 883 of road graph
20 ° left
Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the left side of the direction 883 of the road graph Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the right side of the direction 883 of the road graph (2 ) Left 20 °
Angle formed by image vector for direction 883 of road graph
160 ° left
Projection plane section where image vectors intersect on the left side of road graph direction 883 (3) 160 ° left
Angle formed by image vector for direction 883 of road graph
180 ° left
Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the left side of the direction 883 of the road graph Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the right side of the direction 883 of the road graph (4 ) 0 °
Angle formed by image vector for direction 883 of road graph
20 ° right
Projection plane segment with perpendicular foot to the projection plane segment of the viewpoint on the right side of the road graph direction 883 Projection plane segment with perpendicular leg to the projection plane segment of the viewpoint on the left side of the road graph direction 883 (5 ) 20 ° right
Angle formed by image vector for direction 883 of road graph
160 ° right
Projection plane section where image vectors intersect on the left side of road graph direction 883 (6) Right 160 °
Angle formed by image vector for direction 883 of road graph
180 ° right
Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the left side of the direction 883 of the road graph Projection plane segment where the foot of the perpendicular to the projection plane segment of the viewpoint exists on the right side of the direction 883 of the road graph Since 1), (3), (4), and (6) are images with respect to the front-rear direction of the direction 883 of the road graph, they are registered in the left and right projection plane sections.

In accordance with the above principle, the image vector R904 from the original image shooting position R903 in FIG. 64 corresponds to the case of (5) above, and the original image is registered by linking to the projection plane section mir2 coordinates (x _mir2 , y _mir2 ) 897. The The image vector L900 from the original image shooting position L899 in FIG. 64 corresponds to the case (2) described above, and the original image is registered by being linked to the projection plane section mil2 coordinates (x _mil2 , y _mil2 ) 893. Also, the image vector F908 from the original image shooting position F907 in FIG. 65 corresponds to the case of (4) above, and the original image has the projection plane section mir1 coordinates (x _mir1 , y _mir1 ) 896 and the projection plane section mil1 coordinates (x _mil1 , y _mil1 ) 892 and registered. The image vector B912 from the original image shooting position B911 in FIG. 65 corresponds to the case of (3) above, and the original image has the projection plane section mil2 coordinates (x _mil2 , y _mir2 ) 893 and the projection plane section mir2 coordinates (x _mir2 , y _mir2 ) 907 is registered by linking.

Next, with reference to FIG. 66, a method of dividing an original image according to the angle formed by the image vector and the direction 883 of the road graph when the original image is registered in the projection plane section will be described. The original image registered by linking to the projection plane section 917 is further classified according to what line-of-sight vector is used to view the projection plane section 917. That is, for an image in which the projection plane section 917 is put in the field of view, the sector from 931 to 937 into which the line connecting the original image position and the projection plane section center point 918 enters is classified. Original image shooting position a 916a is angle 140-160 ° sector
In 937, the original image shooting position c 916c is divided into sectors 80, 120 ° at an angle 80-120 °. The image vector d 919d exists in a range 920 in which the angle between the image vector and the direction 883 of the road graph is 20 ° to the left and right. The angle 20 Divide the original image into -40 ° sector 930. The image vector e 919e exists in the range 921 where the angle between the image vector and the direction 883 of the road graph is 20 ° backward left and right. -180 ° Divide the original image into 938 sectors.

The projection plane and image index method for the road has been described above. FIG. 62 illustrates a specific structure of the road image index mechanism. The road graph data table RDGDT 790 described in FIG. 46 is expanded and a projection plane original image index table address 873 is added next to the attribute column to project the road original image for each end point and side constituting the road graph data table RDGDTm 790m. It can be further subdivided into the projection plane sections for use. That is, the address PJINXTADRmi of the projection plane original image index table PJINXT 871 is linked to the end point of the road record ID No.m relative record address i, and the projection plane sections belonging to the end points m and i are sequentially encoded from the start point. Corresponding to the address 874, the left side (left wall) and the right side (right wall) of the direction 883 of the road graph are described separately. The number of cells is the number of projection plane sections belonging to the end points m and i. The representation of the projection plane segment is indicated by its starting coordinates for each projection plane segment. The projection plane original image index table PJINXT 871 has a projection plane original image address table address 875 PJADRTADR for each projection plane section of the left and right walls, and is linked to the projection plane original image address table PJADRT 872 corresponding to FIG. The original road image corresponding to the projection plane segmentation is segmented according to the line-of-sight angle for which the projection plane segment is expected, the number of original images included in each segment is 869 on the road image data, and the original image is the road image data address 870. be written. In this way, the following high-speed search mechanism is completed.
Latitude / Longitude-> Road ID
No .-> End point coordinates-> Projection plane segment coordinates Projection plane segment coordinates and line-of-sight direction-> Original image

Next, the structure of the road image database 153 will be described with reference to FIG. The road image database 153 must be configured to be consistent with the road image index mechanism 152 and the road graph database 151 based on the information acquired in the road image primary file 112. The road image database 153 of FIG. 60 is generated and registered according to the processing flow of the road image database generation / registration system of FIG. In a processing block 843, after the shooting of the road image primary file 112 accumulated during the day running in the road image acquisition system 110 is finished, all images are sequentially processed. In the processing block 844, the header of the image header portion 850 of the road image database of FIG. 60 is created, but the header of the image header portion 470 of the road image primary file 112 of FIG. For the processing block 845 as well, the camera parameters in the image header part 470 of the road image primary file 112 are usually transferred as they are. In the processing block 846, the shooting parameters of the image header unit 850 are calculated for each image, and the processing will be described in detail below.

The camera position is defined by latitude, longitude, and altitude. However, when the position measurement by the in-vehicle GPS 380 is usually DGPS, the accurate position may be obtained as it is because it is the ground surface. Since the error is included when the position measurement by the GPS 380 is GPS, coordinate correction data is obtained from a nearby DGPS station measured at the same time as the travel time after traveling, and correction is performed according to the shooting date and time. In the case of the road image acquisition system 110, the calculation of the shooting parameters is different from the case of the aerial image because it only moves two-dimensionally on the road and the distance between the shooting point and the shooting object is closer than the angle of view. You may calculate by approximating by plane geometry. If the road ID No is not known as the camera position (road ID No. end point relative record address, end point coordinates (x, y), end point distance, centerline left-right distance) of the second record, the ground cell road index in FIG. Search from the table CLRRDINXT 785, road ID No, road graph address table RDGRADRT from relative record address
Further, the road graph data table RDGDT 790 is searched, and the road ID No. end point relative record address and end point coordinates (x, y) are immediately determined.

Road graph direction
883 is obtained by normalizing (x _{m, i + 1} −x _{m, i} , y _{m, i + 1} −y _{m, i} ), and the vehicle traveling direction
853 is a normalized vector obtained from the vehicle data portion 840 of the road image primary file. Set the direction of any digital camera α with respect to the vehicle traveling direction.
Rotation matrix for the digital camera setting orientation
Assuming 859, this value is determined if the structure of the digital camera assembly 755 is determined.
Endpoint distance Ds
852 is Ds =
Is required. However,
=
Centerline left / right distance Dv
853 is Dv =
It is calculated at the right or left of the road graph
Judgment is based on the sign of.
Original image shooting position
Image vector from 916
The point where 904 intersects on the projection plane is that for any scalar a, b,
However,
,
,
,
Solving this,
The lengths of the projection plane segments are usually the same, but since the coordinates of the projection plane segment are defined, projection plane segment mir0 coordinates (xmir0, ymir0) 895 up to any projection plane segment miri coordinates (xmiri, ymiri) The total projection surface length from is known. From the above n, the projection plane section relative address and the position dc 856 in the image vector projection plane section can be obtained by simple subtraction. Similarly, the position dl 857 in the image left end projection plane section and the position dr 858 in the image right end projection plane section can be obtained.

Next, the processing block 847 stores the results obtained in the processing blocks 844 to 846 in the corresponding image data portion 851 of the road image database 153 shown in FIG. In the last processing block 848 of the processing flow of the road image database generation / registration system of FIG. 59, the road image index mechanism shown in FIG. 62 for generating a moving image using the road image data registered in the processing block 847 is generated. The contents of the processing block 848 are described in the road image index generation processing flow of the road image database generation registration system 130 in FIG. The content of the processing is based on the road image index mechanism described in detail in the description of FIGS. 62 to 66. In processing block 861, the direction of the road graph is obtained from the end point number in the road graph data table, and the image vector is obtained. Branches according to the angle formed with (photographing optical axis).
When the image vector direction is 0 ° to 20 ° on the left side and 0 ° to 20 ° on the right side with respect to the direction of the road graph, the angle of the projection surface original image address table of the nearest left projection surface is processed as the front projection surface processing in the processing block 862 The original image data address is registered in the 160-180 ° section and the angle 0-20 ° section of the projection surface original image address table of the right projection plane.
When the image vector direction is 160 ° to 180 ° on the left side and 160 ° to 180 ° on the left side with respect to the direction of the road graph, the angle 0 of the projection surface original image address table of the nearest left projection surface is used as the rear projection surface processing in the processing block 863. The original image data address is registered in the angle 160-180 ° section of the projection surface original image address table of the right 20 ° section and the nearest right projection plane.
When the image vector direction is 20 ° to 160 ° on the left side with respect to the direction of the road graph, the left projection plane in which the photographed image is in the field of view is selected as the left projection plane processing in processing block 864 and selected in processing block 865. For the projection plane, an original image data address is registered in a corresponding angle portion in the projection plane original image address table corresponding to the angle formed by the camera position and the projection plane midpoint.
When the image vector direction is 20 ° to 160 ° on the right side with respect to the direction of the road graph, the left projection plane in which the captured image is in the field of view is selected as the right projection plane processing in processing block 866, and is selected in processing block 867. For the projection plane, the original image data address is registered in the corresponding angle portion in the projection plane original image address table corresponding to the angle formed by the camera position and the projection plane midpoint.

Next, a road animation generation processing system in the cityscape three-dimensional animation amusement system will be described. The road moving image is generated according to the model shown in FIG. 63 by selecting the original image closest to the viewpoint and the line of sight, calculating the parallax with respect to the image projection plane, and performing the morphing process. Similar to the algorithm at 160, but because it is a human eye, the line of sight may move on a horizontal plane and the vertical axis may not tilt. For this reason, morphing is processed two-dimensionally. The entire process is shown in FIG.

In a processing block 940, the moving direction, moving speed, position, and line-of-sight direction vector to be calculated as the next moving image frame (next frame) are fetched from the graphic user interface system 180. In the processing block 941, the viewpoint position of the next frame, the line-of-sight vector, and the target point coordinates on the ground surface are calculated based on this value. In processing block 942, it is determined whether the field of view of the next frame is within the range included in the original image of the current frame or compared with the shooting parameters of the original image in the road image database. If not, it is necessary to re-determine the optimal original image and go to processing block 944. If included, processing block 943 is set to continue using the original image of the current frame. In the example of FIG. 68, the viewpoint R903 indicates that the right-front-view image captured at the original image capturing position a 916a can be the original image on the right projection plane.

The processing content of the processing block 944 is shown in more detail in FIG. 63 and the processing flow of the original image search in the processing of the road three-dimensional moving image generation system of FIG. Hereinafter, a description will be given with reference to FIG. In a processing block 950, it is determined whether an image is generated on the front and rear virtual projection planes according to the section of FIG. 63 or an image is generated on the left and right projection planes along the road based on the angle 882 formed by the road graph and the line of sight. When left and right projection plane view is selected, the road ID No. end point address and coordinates are obtained from the previous position and current position in processing block 953, and further, the relationship between the viewpoint position, line-of-sight vector direction, and road graph vector In accordance with 63, whether the left projection plane or the right projection plane is determined. Further, in processing block 954, the projection plane segment where the current gaze vector, the left field vector of the visual field, and the right field vector of the current field intersect is predicted from the variation amount of the previous position and the current visual field vector, and the variation amount of the previous visual line vector and the current visual line vector. To do. Usually, the projection segment of the current frame is continuously used or is an adjacent projection segment. In a processing block 955, a projection plane segment where the line-of-sight vector intersects is searched for around the predicted value. In the processing block 956, in accordance with the section of FIG. 66, the original image address of the angle section closest to the angle formed by the projection plane section intersecting with the current line-of-sight vector is retrieved and obtained by the projection mechanism original image address table PJADRT by the search mechanism of FIG. . If it is determined in processing block 950 that images are to be generated on the front and rear virtual projection planes in FIG. 63, the road ID No. end point address and coordinates are obtained from the previous position and current position in processing block 951. Further, the left projection plane or the right projection plane is determined from the relationship between the viewpoint position, the line-of-sight vector direction, and the road graph vector. In the case of the front-rear view, the left and right projection planes are not used as the projection plane, but this determination is made because they are registered linked to the left and right projection planes depending on the original image shooting position. In the processing block 952, according to the classification in FIG. 63, it is determined from the relationship between the line-of-sight vector direction and the road graph vector whether it is a forward view or a backward view, the latest projection plane section is obtained, and the angle 0 of the projection plane original image address table is obtained. An image in a section of −20 ° (forward view) or an angle of 160 to 180 ° (forward view) is searched as an original image. Returning to the processing block 945 in FIG. 72, the retrieved new original image is used as the original image.

Processing block 946 branches on the basis of FIG. 63, and the left and right projection plane morphing processing of processing block 948 or the front-rear view morphing processing of processing block 947 is performed. The contents of the left and right projection plane morphing processing in the processing block 948 will be described with reference to FIGS. 68 and 69 with the following text and expressions.
Original image shooting position with respect to original image m
672 and image vector
Once the projection plane is determined as 673 (norm = 1), the image vector
673 intersection of projection plane
And left edge of image, intersection of right edge and projection plane
When
Is decided. A vector representing the projection plane
Is the direction of the road graph
The same normalized vector as 883.
The perpendicular of
(Normalization).
Is an image vector
673
Is where the projection plane intersects,
The street, image vector
A plane perpendicular to
That's it.
Is the original image shooting position perspective
It is an image (field of view) taken from 671.
->
There is a conversion to
That's it. This is a photographing action of the original image m. point of view
, Viewpoint coordinates
And gaze vector
(Norm = 1) and projection plane
(Common) is determined as well
When
,
Is decided.
Street gaze vector
A plane perpendicular to
That's it. Plane
Is the projection plane
In perspective
In the field of view
Part of
point of view
Vision
When
Is
When
And viewing angle
If it is decided, it is decided uniquely. Plane
Plane
The mapping to
When
And angle of view of original image m
If it is decided, it is decided uniquely.
― ＞
― ＞
By the correspondence of
of
Each pixel corresponding to
of
The morphing algorithm is completed if it is associated with the pixel corresponding to.
Generally flat
Any point on
Is
= a
+
It is expressed.
_{Here is the normal condition,}
Substituting = 0
=
Is obtained. Here, represents a vector dot product.
Center line of sight
And projection plane
Intersection of
Is
_But
=
+ c
Substituting this for satisfying
(C
+
) =
C =
And
=
+
=
And perspective
Projection plane of the image center point
The coordinates at were obtained.

Similarly, the perspective
Normalized gaze vector at the left and right ends of the field of view
,
And these and the projection plane
Intersection of
,
Projection plane
The coordinates at can be obtained. Projection plane
Any point on
Is
= a
+
Expressed by the _{perpendicular condition,}
= 0, the normalized gaze vector at the left and right ends
,
Against
=
+ c
However, k = L is satisfied
(C
+
) =
C =
And
=
+
=
And perspective
The left and right projection planes of the image
The coordinates at were obtained. This plane
Coordinates on the plane
The plane that is the original image by associating it with the coordinates above
View any point above
You can see if it should be reflected on the screen.
Perspective on
Screen center and left and right end point coordinates
_,
,
(collect
_{K = L, R, C)}
Above
_,
,
(collect
_, K = L, R, C).
The horizontal line of the screen on the screen
If
Any point on the face
Is for an arbitrary constant a
=
= a
+
It is represented by Image vector
Is
Because it ’s perpendicular to
= 0, and
When
Is the image shooting position
Because it ’s on the same line.
_{= d (
-}
) +
So _{d (
-}
) +
= a
+ b
+
And
D =
Because it becomes
=
It becomes.
=
+
Already in
So now we have a perspective
Correspondence with the screen of the original screen m of the field of view was completed.

next,
Find the transformation that moves to the y-axis. here,
= 1
=
When rotating around the z axis,
Is on the x-axis.
The rotation around the z axis is

Given in.
Plane
Intersection of
_{for (k = L, R, C)}
To the z axis,
(k = L, R, C) is converted to a point on the x-axis. This corresponds to the position on the screen of the original image m. That is morphing. Since the above is two-dimensional morphing, the correspondence on the horizontal line in FIG. 71 was obtained in units of pixels. The correspondence in the original image 730 of the visual field 731 at the viewpoint is shown on the horizontal line. The vertical direction is vertically symmetric, and the enlargement / reduction ratio is the same as that on the horizontal line. If the movement of the line of sight is not limited to a horizontal plane, a 3D morphing process in generating an aerial 3D moving image may be used privately. In the front-rear view morphing process of the processing block 947 in FIG. 72, the view of the viewpoint F 907 is generated from the original image at the shooting point a 916a by the same algorithm as described in the processing block 948 by two-dimensional morphing.
This is the end of the description of the on-road 3D animation generation of the city landscape 3D animation generation amusement system according to the present invention.

FIG. 71 shows another configuration example of the on-vehicle digital camera assembly 973. In the generation of a three-dimensional moving image of a cityscape of the present invention, the distance between the imaging target and the digital camera assembly is shortened due to the road structure of the city, and the parallax tends to increase between adjacent original images. As a result, the smoothness of the generated moving image may deteriorate. In order to solve this problem, by halving the angle pitch of the side digital camera with respect to the traveling direction 976 and employing a wide-angle lens as a whole, it is possible to collect original images more suitable for generating a three-dimensional moving image. .

Next, explanation will be given on the graphic user interface system 180 of the three-dimensional moving image generating amusement system of the cityscape with reference to FIG. FIG. 75 shows the overall structure of the graphic user interface system 180. The graphic user interface system 180 is constituted by the processing flow of FIGS. 75 and 76 and the monitor display and input functions of FIGS. 77, 78 and 79. The amusement system for generating a three-dimensional moving image of a cityscape according to the present invention is characterized in that it can freely switch between aerial simulation flight and walking on the road. In processing block 974, it is determined whether switching from the air to the ground or from the ground to the air is performed. If landing from the air to the ground is selected, the processing block 975 reads the landing point and the traveling direction after landing. If the landing button 996 is pressed for a short time in the aerial 3D video graphic user interface of FIG. 77, it will land on the ground at a specified diagonal angle, such as 45 degrees below the direction of travel, and if it is pressed for a long time, There is also a method of approaching the point directly below. It is common sense that the gaze direction on the S ground maintains the orientation during flight. Go to processing block 974 together with the case where there is no change in the road simulation from the previous time, read the direction of travel and speed instruction from the screen in FIG. 78 which is a graphic user interface for the ground, update the position and speed vector and update the 3D video on the road Tell generation system 170. If it is determined in the process block 974 that the road 3D moving image generation is switched to the aerial 3D moving image generation, the process goes to the processing block 977. Switching from the ground to the air occurs when the jumping button 984 is pressed in FIG. 78, and depending on the length of time pressed, when the time is short, the traveling direction is obliquely backward at a constant angle, for example, a constant altitude of 45 °, for example 100 m. There is a method such as making the sky above, and making it fly forward as the time pressed becomes longer.

Next, the process goes to the aerial graphic user interface system of the processing block 978 together with the case where the aerial three-dimensional moving image was generated from the previous time. This content is further described in FIG. In processing block 979, the advancing direction, slot, and elevation instruction are read from the screen from the aerial graphic user interface of FIG. 77, and the position and velocity vectors are updated. In the aerial 3D video generation, not only the cockpit display is performed on the 3D video display 990, but also the camera display can be switched by selecting with the function button 1001. This corresponds to the aerial three-dimensional moving image generation system 160 being able to freely take the trajectory of the start position and the trajectory of the line-of-sight vector. If cockpit display is selected in processing block 980, processing proceeds to processing block 983 where the line-of-sight vector is set to the direction of travel, the viewing angle is set to a specified value, and information is transmitted to the aerial three-dimensional video generation system 160. If it is determined in processing block 980 that the camera has been selected, processing block 981 captures the camera direction, tilt angle, and zoom setting values from the aerial graphic user interface of FIG. 77, and processing block 982 determines the viewpoint, line of sight, and zoom. The viewing angle is calculated and information is transmitted to the aerial three-dimensional moving image generation system 160.

Hereafter, the part which has not been demonstrated so far about FIG. 77, 78, 79 is demonstrated. 77 indicates the flight direction, and the altitude indicator indicates the flight altitude. A map display 988 displays the current flight position on a map. This function is publicly implemented as GIS. FIG. 78 is a video display for generating a three-dimensional moving image on the road, and displays the city with a human eye. The function button 1001 allows functions to be added in preparation for a later date, and the map display 983 is arranged on the central road so that the guide function is improved. The comment display window 1000 can display advertisements depending on the region. FIG. 79 shows another method for specifying the direction of travel and the line of sight for generating a three-dimensional moving image on the road. The screen is divided into regions, and the meaning of the operation input is changed depending on the position of the cursor on the screen using a mouse or the like. If the location of advance 998 is specified, it moves forward or backward in the direction of the arrow, and if it is specified the location of left turn right turn 997, it proceeds to the left or right turn in the direction of arrow and specified the location of the left and right gaze change 999 In this case, change the line of sight to the left or right in the direction of the arrow.

  The aerial photography imaging system of the present invention can be industrially used as a control device and an imaging data processing device for a camera mounted on an aircraft.

It is a figure showing roughly the cityscape three-dimensional animation amusement system whole composition of the present invention. It is a figure which shows the concept of the aerial image acquisition in the cityscape three-dimensional moving image amusement system shown in FIG. It is a conceptual diagram of the morphing in a three-dimensional moving image. It is a figure explaining the concept of the moving image production | generation by the morphing using the real image in the cityscape three-dimensional moving image amusement system of this invention. It is a figure explaining the operation example of the aerial image acquisition system by the aircraft of this invention. It is a figure explaining the processing flow of the aerial image acquisition system of this invention. It is a figure explaining the example of the airborne digital camera aggregate | assembly of this invention. It is a figure which shows the example of a setting of the course and imaging | photography point of the aerial image acquisition system by the aircraft of this invention. It is a figure which shows the structural example of the aerial image acquisition system of this invention. It is a figure which shows the processing flow of the flight navigation system part of the airborne image acquisition system of this invention. It is a figure which shows the structural example of the aerial image acquisition plan file of this invention. It is a figure which shows the example of the display screen of the flight navigation system part of the aerial image acquisition system of this invention. It is a figure which shows the information flow of the data acquisition recording system part of the aerial image acquisition system of this invention. It is a figure which shows the processing flow of the imaging | photography control system of the aerial image acquisition system of this invention. It is a figure which shows the structure of the aerial image primary file of this invention. It is a figure which shows the processing flow of the aerial image database production | generation registration system of this invention. It is a figure which shows the aerial image database structure of this invention. It is a figure which shows the relationship between the image and visual field in the haze effect removal process of this invention. It is a figure which shows the example of an image characteristic parameter graph in the haze influence removal process of this invention. It is a figure which shows the processing flow of the haze effect removal process of this invention. It is a figure which shows the example of a process result of the haze effect removal process of this invention. It is a figure which shows the aerial image index production | generation registration processing flow of the aerial image database production | generation registration system of this invention. It is a figure which shows the structure and term definition of the aerial image index mechanism of this invention. It is a figure which shows the structure of the aerial image index mechanism of this invention. It is a figure which shows the index structure of the solid angle cell of this invention. It is a figure which shows the correlation of the aerial image index of this invention, and an association table. It is a figure which shows the structural example of the visual field by the digital camera from which three types of focal distances differ of the digital camera aggregate | assembly of this invention. It is a figure which shows the structural example of the digital camera aggregate | assembly by the digital camera from which three types of focal distances differ of this invention. It is a figure which shows the imaging | photography range figure of the digital camera aggregate | assembly by three types of digital cameras from which focal distance differs of this invention. It is a figure which shows the structural example of the stable platform for aircraft mounting of this invention. FIG. 4 is a diagram illustrating a signal information flow of the stable platform of the present invention. It is a figure explaining the variable of the moving image production | generation by the morphing using the real image in the cityscape three-dimensional moving image amusement system of this invention. It is a figure which shows the aerial moving image production | generation processing flow in the cityscape three-dimensional moving image amusement system of this invention. It is a figure which shows the processing flow of the original image search of the three-dimensional moving image production | generation process using the real image in the cityscape three-dimensional moving image amusement system of this invention. It is the figure which showed the original image selection logic of the three-dimensional moving image production | generation process of this invention. FIG. It is a figure which shows the relationship between the visual field in the viewpoint in the three-dimensional moving image production | generation process of this invention, and an original image. It is a figure which shows the relationship between the gaze vector of this invention, and the intersection of Terrain. It is a figure which shows the relationship between the visual field in the viewpoint of this invention, and an original image. It is a figure which shows the visual field from viewpoint P (t _i ) of this invention. It is a figure which shows the processing flow of the road image acquisition system of this invention. It is a figure which shows the vehicle-mounted digital camera aggregate structure example of this invention. It is a figure which shows the example of a setting of the path | route and imaging | photography point of a road image acquisition system of this invention, and an imaging | photography range. It is a figure which shows the relationship between the road image acquisition system vertical direction imaging | photography pattern of this invention, and an image projection surface. It is a figure explaining the description of the road by the graph of this invention. It is a figure explaining the road search system by the latitude and longitude of this invention. It is a figure explaining the structure of the road graph database of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road graph data table RDGDT of this invention. It is a figure which shows the structural example of the road image acquisition system of this invention. It is a figure which shows the processing flow of the car navigation system part of the road image acquisition system of this invention. It is a figure which shows the structural example of the road image acquisition plan file of this invention. It is a figure which shows the example of a display screen of the car navigation system of the road image acquisition system of this invention. It is a figure which shows the information flow of the data acquisition recording system part of the road image acquisition system of this invention. It is a figure which shows the processing flow of the data acquisition recording system part of the road image acquisition system of this invention. It is a figure which shows the structure of the road image primary file of this invention. It is a figure which shows the structural example of the vehicle-mounted stable platform of this invention. It is a figure which shows the processing flow of the road image database production | generation registration system of this invention. It is a figure which shows the structure of the road image database of this invention. It is a figure which shows the road image index production | generation processing flow of the image database production | generation registration system of this invention. It is a figure which shows the structure of the road image index mechanism of this invention. It is a figure explaining the animation production | generation process by the direction of the road graph of this invention, and the direction of a gaze vector. It is a figure explaining the process in case the line-of-sight direction has faced the projection surface in the road image database production | generation registration system and road three-dimensional moving image production system of this invention. It is a figure explaining the process in case a gaze direction is substantially parallel to a projection surface in the road image database production | generation registration system and road three-dimensional moving image production system of this invention. It is a figure which shows the relationship between a projection surface and an original image imaging | photography optical axis in the road image database production | generation registration system and the road three-dimensional moving image production system of this invention. It is a figure which shows the calculation method of the imaging | photography parameter of the road image database of this invention. It is a figure explaining selection of the original picture in the road three-dimensional animation generation system of the present invention. It is a figure which shows the system of the morphing with respect to the projection surface in the road three-dimensional moving image production | generation system of this invention. It is a figure which shows the morphing system with respect to the visual field substantially parallel to a projection surface in the three-dimensional moving image production | generation system on the road of this invention. It is a figure which shows the processing flow of the road three-dimensional moving image production | generation system of this invention. It is a figure which shows the processing flow of the road three-dimensional moving image production | generation system of this invention. It is a figure which shows the processing flow of the original image search in the process of the road three-dimensional moving image production | generation system of this invention. It is a figure which shows another structural example of the vehicle-mounted digital camera aggregate | assembly of this invention. It is a figure which shows the processing flow of the graphic user interface system of this invention. It is a figure which shows the processing flow of the aerial graphic user interface system of the process of the graphic user interface system of this invention. It is a figure which shows the example of a display of the aerial graphic user interface system of the process of the graphic user interface system of this invention. It is a figure which shows the example of a display of the road graphic user interface system of the process of the graphic user interface system of this invention. It is a figure which shows the direction work cursor function of the road graphic user interface system of the process of the graphic user interface system of this invention.

Explanation of symbols

100 Aerial image acquisition system
101 Aerial image acquisition plan file
102 Aerial image primary file
110 Road image acquisition system
111 Road image acquisition plan file
112 Road image primary file
120 Aerial image database generation registration system
121 DEM file
122 DGPS file
130 Road image database generation and registration system
140 Aerial image database
142 Aerial image index mechanism
143 Aerial image database
150 Road image database
151 road graph database
152 Road image index mechanism
153 Road image database
160 Aerial 3D video generation system
165 Aerial image search engine
170 Road 3D video generation system
175 Road image search engine
180 graphic user interface system
190 Internet
191 users
200 cities
210 eyes
220 sufficiently small solid angle range 1
221 sufficiently small solid angle range i
222 celestial sphere
230 Relationship between building and line of sight
240 Gaze direction 1
241 Image seen from line of sight 1
250 Gaze direction 2
251 Image viewed from line of sight 2
260 Aerial image i
261 Aerial image i + 1
270 View path P (t)
271 Line of sight at time t
272 Line of sight at time t + δt
273 Line of sight at time t + 2δt
274 Line of sight at time t + 3δt
275 Line of sight at time t + 4δt
276 Line of sight at time t + 5δt
280 Target trajectory T (t)
300 flight paths
301 aircraft
310 Shooting points
320 Digital camera assembly optical axis direction
330 Shooting point setting process
331 Shooting process
332 Aerial image primary file generation process

350a ~ i digital camera
360 digital camera assembly
370 Shooting direction
371 Flight direction shooting interval
372 Shooting interval between flight routes
373 Aircraft position
375a to i Digital camera 350a to i shooting range
380 GPS antenna
385 Flight Navigation System
386 Flight Navigation System
387 Avionics Information
388 Air Instrumentation
390 Data acquisition and recording system
391 metadata files
393 Aerial shooting control system
394 Stable platform control system
395 Stable platform equipment
396 IMU
397 Aircraft floor hole
398 aircraft floor
400 IMU
420 ~ 425 processing block
430 Route No.1 shooting coordinate file
431 Route No.2 shooting coordinate file
432 Route No.m shooting coordinate file
437 Shooting guidance display
438 Number of remaining shooting points in route
439 Route No. display
440 tolerance
441 Flight direction
442 Orientation display
443 Posture display
444 Horizontal tilt display
445 Elevation posture axis display
446 Position deviation display
447 Horizontal deviation axis
448 Altitude deviation axis
450 Stable platform attitude data
451 Aircraft attitude data
452 GPS antenna position data
453 Digital camera shooting parameters
454 Camera parameters
455 Aerial image data
456 Imaging point data
457 shooting command
460 ~ 465 processing block
470 Image header
471 Image data part
472 Aircraft data part
480 ~ 486 processing block
490 Image header
491 Image data part
495 Images taken
496 Foreground part a
497 Foreground part b

498 Distant view
499 Shooting surface
500 Normalized histogram
501 Histogram for a foreground
502 Histogram partial histogram
503 Lower limit of histogram
504 Histogram upper limit
505 Price range
506 Median
507 Upper brightness limit
508 Luminance axis
509 pixel count axis
510 Value axis
511 Distance axis from digital camera
512 median fitting curve
513 Value Fitting Curve
514 Foreground part a value range
515 Foreground part a median
516 Distant view median
517 Far view partial value width
520 to 524 processing block
530 Image before removing haze
531 Image after removing haze effect
540 to 545 processing block
549 Ground image range
550 Shooting position
551 Image vector (normalized)
552 image distance
553 Frame rotation angle
554 Terrain upper center point coordinates
555 Terrain recent right coordinates
556 farthest right coordinate on Terrain
557 farthest left coordinate on Terrain
558 Terrain latest left coordinate
559 surface cell
560 Terrain
570 Zenith vector
571 Zenith angle
572 Zenith Index i
573 solid angle cell
574 solid angle cell center vector
575 Ground cell center point
580 Zenith Index 0
581 Zenith Index 1
582 Zenith Index 2
583 Zenith Index 3
584 Solid angle cell address (0.0)
585 Solid angle cell address (1.0)
586 Solid angle cell address (1.1)
587 Solid angle cell address (1.2)
588 solid angle cell address (2.0)
589 Solid angle cell address (2.1)
590 solid angle cell address (2.2)
597 Zenith Index Parameter Table ZNPRMT
598 Direction index parameter table DRPRMT
599 bearings
600 Ground cell position index table TCINXT
601 Longitude index LONINX
602 Latitude Index LATINX
603 Zenith Index Table Address
604 Zenith Index Table ZNINXT
605 Direction index table address
606 Azimuth index table DRINXT
607 Image address table address
608 Image address table GADDRT
610 digital camera with long telephoto lens
611 Digital camera with short telephoto lens
612 standard lens digital camera
613 Standard lens digital camera shooting range
614 Range of digital camera with short telephoto lens
615 Range of digital camera with long telephoto lens
620j ~ y Digital camera
621 Digital camera assembly
630j to y Digital camera 550 j to y shooting range
635 fixed part
636 Vertical movement
637 Rotating part
640 Rotation drive mechanism
641 Vertical drive mechanism A
642 Vertical drive mechanism B
643 Vertical drive mechanism C
644 Vertical drive mechanism D
645 Bearing mechanism
646 Stable platform attitude data
647 Low, pitch direction control
648 Control of yaw direction
649 Vertical movement
650 rotational motion
666 Aerial image m-1
667 Aerial image m
670 Original image shooting position S _m-1
671 Original image shooting position S _m
672 Image Vector G _m-1
673 Image Vector G _m
675 Viewpoint P (t ₁ )
676 Viewpoint P (t ₂ )
677 Viewpoint P (t ₃ )
678 Viewpoint P (t ₄ )
679 viewpoints P (t _i )
680 Eye vector trajectory V (t)
681 Eye vector V (t ₁ )
682 Eye vector V (t ₂ )
683 Eye vector V (t ₃ )
684 Eye vector V (t ₄ )
685 Gaze vector V (t _i )
690-697 processing block
700 to 705 processing block
710 Original image position S _A
711 Original image shooting position S _B
712 Original image position S _C
713 Shield
714 Visible range on Terrain
715 Terrain recent right coordinate Z _m1
716 Terrain farthest right coordinate Z _m2
717 Terrain farthest left coordinate Z _m3
718 Terrain recent left coordinate Z _m4
719 Terrain upper center coordinate Z _mC
720 Terrain Upper left middle point Z _m34
721 Terrain upper right middle point Z _m12
722 Terrain upper middle point Z _m23
723 Terrain upper and lower middle point Z _m41
724 Terrain upper field of view vertical vector Z _mV
725 Terrain upper field of view horizontal vector Z _mH
726 Terrain top view vertical vector _nm
727 Terrain plane
730 Original image
731 Field of view
732 Frame angle (horizontal)
733 Frame angle of view (vertical)
734 Field of view lower right corner point V _i1
735 Field of view upper right corner V _i2
736 Field of view upper left corner point V _i3
737 Field of view lower left point V _i4
741-743 processing blocks
750a-w digital camera
755 Digital camera assembly
760a to p Digital camera 750a to p shooting range
765 Route
766 Shooting points
767 Shooting direction
768 road border
769 road or walkway
770 image projection plane
771 Building
775 endpoint
776 sides
777k ~ n Road ID No. k to n
779 endpoint coordinates
780 Centerline graph
781 latitude and longitude cells
782 Latitude and longitude cell center point
783a-b Vertical foot AB
785 Ground Cell Road Index Table CLLRDINXT
786 Longitude index
787 Latitude Index
788 Road graph address table RDGRADRT
789 Road ID No
790 Road graph data table RDGDT
791 Attributes
792k, i ~ 792k, i + 6 Endpoint k, i ~ k, i + 6
793l, j to 793l, j + 6 End point l, j to l, j + 6
794m, n-3 to 794m, n End point m, n-3 to m, n
795k ~ m Road graph data table RDGDT k ~ m
796 Relative record address
800 Data acquisition and recording system
801 Car navigation system
802 Car navigation system terminal
803 vehicles
804 Metadata file
805 Road image data file
806 Road imaging control system
807 Vehicle ceiling board
810-813 processing block
820 Road ID No.1 shooting coordinate file
821 Road ID No.2 shooting coordinate file
822 Road ID No.m shooting coordinate file
825 Designated shooting route
826 Shooting point data
827 Road image data
828 Vehicle attitude data
829 Stable platform attitude data
830 GPS antenna position data
831 Digital camera shooting parameters

832 Road image data
834-839 processing block
840 Vehicle data part
843-848 processing blocks
850 Image header part
851 Image data part
852 Endpoint distance
853 Centerline left / right distance
854 Vehicle traveling direction
855 Position within the projection plane section of the original image shooting position
856 Position in image vector projection plane section
857 Position within left projection plane section
858 Image right edge projection plane position
859 Digital camera setting direction
860 ~ 867 processing block
869 Number of road image data
870 Road image data address
871 Projection surface original image index table PJINXT
872 Projection plane original image address table PJADRT
873 Projection plane original image index table address
874 Relative record address
875 Projection surface original image address table address
876
877 street perspective
878 Virtual front projection plane
879 Virtual rear projection plane
880 Left projection plane
881 Right projection plane
882 Angle between road graph and line of sight
883 Road graph direction
884 Image taking optical axis
885 Front projection plane processing range
886 Left projection plane processing range
887 Rear projection plane processing range
888 Right projection plane processing range
890 End point m, i coordinate (x _{m, i} , y _{m, i} )
891 Projection plane section mil0 coordinates ( _xmil0 , _ymil0 )
892 Projection plane section mil1 coordinates ( _xmil1 , _ymil1 )
893 projection plane division mil2 coordinates ( _xmil2 , _ymil2 )
894 projection plane section mil3 coordinates ( _xmil3 , _ymil3 )
895 Projection plane section mir0 coordinates (x _mir0 , y _mir0 )
896 Projection plane section mir1 coordinates (x _mir1 , y _mir1 )
897 Projection plane section mir2 coordinates (x _mir2 , y _mir2 )
898 projection plane section mir3 coordinates (x _mir3 , y _mir3 )
899 Original image shooting position L
900 image vector L
901 Left image L
902 Right edge L
903 Original image shooting position R
904 Image vector R
905 Left edge of image R
906 Image right end R
907 Original image shooting position F
908 Image Vector F
909 Left edge F
910 Image right edge F
911 Original image shooting position B
912 Image vector B
913 Left edge of image B
914 Image right edge B
915 Reverse direction of road graph
916a to d Original image shooting positions a to d
917 Projection plane division
918 Projection plane section center point
919a-e image vector ae
920 Front left and right 20 ° range
921 Range of rear left and right 20 °
930 Angle 0-20 ° Sector
931 Angle 20-40 ° Sector
932 Angle 40-60 ° sector
933 angle 60-80 ° sector
934 Angle 80-100 ° Sector
935 angle 100-120 ° sector
936 Angle 120-140 ° Sector
937 angle 140-160 ° sector
938 Angle 160-180 ° sector
940-948 processing block
950-956 processing block
960 Original image m
961 Original image m + 1
962 Image vector G _m
963 Viewpoint P _i
964 Eye vector V _ic
965 Left line of sight vector V _iL
966 Gaze right edge vector V _iR
967 Z _m perpendicular
968 Projection plane Z _m
969 Original image shooting position S _m
970 _Left end Q _iL on the original image
971 _Right edge Q _iR on the original image
972 Original image right end Q _iR
973 Digital camera assembly
974 to 983 Processing block
984 Jumping button
985 landing button
986 Direction change button
987 altitude indicator
988 Show map
989 Current location name display
990 3D image display
991 Orientation indicator
992 Control button
993 throttle
994 Camera front / rear direction specification
995 Camera left / right direction designation
996 Zoom designation
997 Turn left Turn right
998 back and forth
999 Left / right gaze change
1000 comment display window
1001 Function button

Claims (30)

In an urban landscape 3D video amusement system that displays a landscape viewed from an arbitrary viewpoint route in the air and an arbitrary line of sight in a three-dimensional manner for an arbitrary location in the city, for an arbitrary location in the city Images that can be viewed at a certain sufficiently small solid angle are prepared in advance as an image database, and images with close viewpoints and lines of sight are sequentially extracted from the image database in accordance with temporal changes in the specified viewpoint path and specified line-of-sight direction. A system for displaying a three-dimensional moving image by smoothly connecting the viewpoint path and the line of sight while performing image morphing, including an aerial image acquisition system, an aerial image database generation and registration system, an aerial database, and an aerial 3D It consists of a video generation system and a graphic user interface system. Cityscape 3-dimensional video amusement system that. In a cityscape three-dimensional video amusement system that displays a three-dimensional moving image of an arbitrary viewpoint route of a human eye and an arbitrary line-of-sight direction with respect to an arbitrary place facing a city passage, Prepare in advance an image database as an image database for every small azimuth that is sufficiently small with respect to an arbitrary location facing the passage, and form a vertical plane facing the passage as an image projection plane, and a designated viewpoint path Images with close viewpoints and line of sight are sequentially extracted from the image database in accordance with temporal changes in the specified line-of-sight direction, and are smoothly connected while performing image morphing corresponding to the viewpoint path and the line of sight on the image projection plane. 3D moving image display system, road image acquisition system, road image database generation registration system, road image database, road 3D Cityscape 3D video amusement system comprising image generation system, and that it is composed of a graphic user interface system. The city landscape three-dimensional video amusement system according to claim 1, wherein the aerial image acquisition system comprises an aircraft-mounted digital camera of a flying aircraft. The aerial image acquisition system of the cityscape three-dimensional video amusement system according to any one of claims 1 and 3, wherein the flight route interval of the aircraft is set at an equal interval and the altitude from the ground surface is constant, It is possible to compose a photographing point and compose a digital camera assembly with a plurality of aircraft-mounted digital cameras so as to be able to photograph and store not only the direction of gravity but also a plurality of oblique directions at the photographing point. A featured aerial image acquisition system. 5. The aerial image acquisition system of the cityscape three-dimensional moving image amusement system according to claim 4, wherein the flight control device comprising a GPS and an inertial navigation device is located at or near the mesh-shaped imaging point. An aerial image acquisition system for recording an aircraft position and an attitude of the aerial camera assembly on a recording device at the same time as issuing a shooting command signal to each digital camera constituting the digital camera assembly . In the aerial image acquisition system of the cityscape three-dimensional animation amusement system according to any one of claims 4 and 5, a telephoto lens having a long focal length is used for a direction having a larger angle than the direction of gravity, and the direction of gravity is determined. An aerial image acquisition system characterized in that a reduction in resolution of an image with a small depression angle is prevented by using a telephoto lens having a short focal length in a direction with a smaller angle. In the aerial image acquisition system of the cityscape three-dimensional moving image amusement system according to any one of claims 4 or 5 or 6, the angle of the optical axis of the lens and the direction of gravity of the multiple-direction oblique digital camera is the same, An aerial image acquisition system, wherein the horizontal circumferential direction is radially divided into equal intervals. The aerial image acquisition system for a cityscape three-dimensional moving image amusement system according to any one of claims 4 to 7, wherein a digital camera for oblique shooting is pointed around a digital camera whose lens optical axis is in the direction of gravity. A digital camera assembly is configured by symmetrically arranging radially, and in particular, it is arranged so that projections do not protrude outside the imaging cylindrical gap on the floor surface of the aerial imaging aircraft. Aerial image acquisition system. 9. An aerial image acquisition system for a cityscape three-dimensional video amusement system according to claim 8, wherein at least two types of telephoto lenses having focal lengths are prepared, each of which is oblique with a gravitational direction camera as a center. A camera for direction shooting is arranged radially around the point in a symmetrical manner to form a digital camera assembly, and in particular, it is arranged so that protrusions do not protrude outside the cylindrical gap on the floor surface of an aircraft for aerial shooting. An aerial image acquisition system characterized by: 10. The aerial image acquisition system for a cityscape three-dimensional moving image amusement system according to claim 8, wherein the digital camera assembly is suspended from or mounted on a stable platform mechanism, and a detection signal of an inertial navigation device is mounted. An aerial image acquisition system that eliminates the effects of aircraft body attitude disturbance. The aerial image acquisition system of the cityscape three-dimensional video amusement system according to any one of claims 3 to 10, wherein a flying altitude under a cloud in a cloudy sky to obtain a smooth three-dimensional video while avoiding shadows caused by direct sunlight. An aerial image acquisition system and method, characterized in that an image is taken from below the cloud. In the aerial image database generation / registration system of the urban landscape three-dimensional video amusement system according to any one of claims 1 and 3, in order to remove the influence of water vapor in the air, the line-of-sight direction is made more than the direction of gravity in the image. The aerial image acquisition system acquires the image processing so that the histogram characteristics are the same regardless of the angle formed from the gravitational direction by measuring a histogram that continuously integrates each color element or each color element corresponding to the corner. An aerial image database generation / registration system which is performed on an image and registered in the aerial image database. The aerial image database generation / registration system of the cityscape three-dimensional video amusement system according to any one of claims 1 and 3, wherein an original image of each frame image of the cityscape three-dimensional video is quickly transmitted from the aerial image database. For the purpose of searching, as metadata accompanying the image data, the position coordinates such as latitude and longitude and elevation on the ground surface of the four corners as the metadata, the position coordinates such as latitude and longitude and elevation of the point where the shooting optical axis intersects the ground surface, the shooting position and For the image acquired by the aerial image acquisition system that calculates the distance until the imaging optical axis intersects the ground surface, and all or part of the angle at which the image frame rotates around the imaging optical axis with respect to the horizontal plane An aerial image database generation / registration system that performs registration in the aerial image database together with a shooting position and a shooting optical axis direction. The aerial image database generation / registration system of the cityscape three-dimensional video amusement system according to any one of claims 1 and 3, wherein an original image of each frame image of the cityscape three-dimensional video is quickly transmitted from the aerial image database. For the purpose of searching, for the image acquired by the aerial image acquisition system or the image registered in the aerial image database, the ground surface is divided into mesh cells by latitude and longitude or other horizontal coordinates, and the image is The mesh cell being photographed is selected and indexed so that the image can be retrieved for each mesh, and the azimuth angle from the center point of the mesh of the photographing point of the image capturing the mesh, Indexing the image so that the image can be searched from the mesh cell at an elevation angle and a distance, and registering the image in the aerial image database Image database generation registration system. In the aerial three-dimensional video generation system of the cityscape three-dimensional video amusement system according to any one of claims 1 and 3, a viewpoint path P (t) defined by a function of a time axis t moving in the air, The line-of-sight vector trajectory V (t) indicating the line-of-sight direction from the viewpoint path P (t), and the trajectory of the point at which the line-of-sight vector trajectory V (t) intersects the target point of the image are referred to as the target trajectory T (t). The aerial image database is searched for an image having the closest attribute consisting of a shooting optical axis vector, a shooting position, and a viewing angle in the line-of-sight vector trajectory V (t _i ) at t = t _i . And each frame image of the three-dimensional cityscape based on the attribute of the original image is assigned to the attribute of the line-of-sight vector, viewpoint, and viewing angle of each frame image. Aerial 3D video generation system characterized by a smooth moving picture by joining on the generated time axis respective frame image by morphing conversion from the difference in the attribute of the original image. The aerial three-dimensional video generation system of the cityscape three-dimensional video amusement system according to claim 15, wherein the line-of-sight vector trajectory V (t _i ) at t = t _i includes a shooting optical axis vector, a shooting position, and a viewing angle. In the process of retrieving an image having the closest attribute from the aerial image database to obtain an original image, the angle formed by the photographic optical axis vector of the original image and the line-of-sight vector trajectory V (t _i ) is greater than a predetermined angle. The original image has a fixed viewing angle depending on the focal length of the camera with the photographing optical axis vector as the center, and a polygon in which the quadrangular pyramid forming the visual field intersects the ground surface is the line-of-sight vector trajectory V ( wherein the quadrangular pyramid constituting a field of view to simulate about a t _i) is selected to include a polygon intersecting the surface 3-dimensional moving image generating system in. The aerial three-dimensional video generation system of the cityscape three-dimensional video amusement system according to claim 16, wherein when the original image is selected from the aerial image database, the line-of-sight vector trajectory V (t _i ) at the time t _i and the target In order to avoid obstacles existing between the points, the image having the smallest vector length from the plurality of original images satisfying the selection condition to the point where the photographing optical axis vector intersects the ground surface is defined as the original image. An aerial three-dimensional video generation system characterized by selecting. 3. The urban landscape three-dimensional video amusement system according to claim 2, wherein the road image preparation system is composed of a vehicle-mounted digital camera of a moving vehicle or a digital camera that is moved, pulled, or propelled by a moving person. A 3D movie amusement system featuring urban landscape. The road image acquisition system of the cityscape three-dimensional moving image amusement system according to any one of claims 2 and 18, wherein the road image acquisition system is an in-vehicle digital camera, or a digital camera that is stretched, pulled, or propelled by a person. The digital camera is moved along the direction of travel of the path, and the path along the direction of travel of the path so that there is an image that can be seen at a predetermined angle that is sufficiently small relative to an arbitrary point facing the path. Set up shooting points for each fixed distance above, and compose and shoot a digital camera assembly with multiple digital cameras so that not only the front, back, left and right but also the middle direction and diagonally upward direction can be shot and stored on the horizontal plane A road image acquisition system characterized by that. The road image acquisition system of the cityscape three-dimensional video amusement system according to claim 19, wherein the digital camera assembly is suspended or mounted on a stable platform mechanism, and the vehicle body posture is disturbed by a detection signal of an inertial navigation device. A road image acquisition system characterized by eliminating the influence. 21. The imaging point defined in claim 19, wherein in the road image acquisition system of the cityscape three-dimensional video amusement system according to any one of claims 19 and 20, a car navigation device comprising a GPS and an inertial navigation device is provided. And detecting the position of the digital camera assembly and issuing a shooting command signal to each camera constituting the digital camera assembly, and simultaneously recording the position and orientation of the digital camera assembly on a recording device. A road image acquisition system characterized by that. The road image database generation registration system and road image database in the city landscape three-dimensional video amusement system according to any one of claims 2 and 18, wherein an original image of each frame image of the city landscape three-dimensional video is the road image For the purpose of high-speed retrieval from the database, the structure of the passage is expressed as a graph with the center line of the passage as a graph, and this is referred to as a center line graph. Describes the position and structure of the passage by describing the presence or absence of the image projection plane for each of the left and right traveling directions, road intersection, branching, and merge information, and further dividing the ground surface into mesh cells by latitude and longitude or other horizontal coordinates And an index mechanism for linking the mesh cell to the passage recognition number described in the center line graph. The road image database that. In the road image database production | generation registration system of the cityscape three-dimensional animation amusement system of any one of Claim 2 and 18 and 22,
The image projection plane that accompanies the centerline graph describing the passage structure for the purpose of quickly retrieving the original image of each frame image of the cityscape three-dimensional video from the road image database along the edges of the centerline graph The image obtained by the road image acquisition system or the image captured by the image registered in the road image database is retrieved, and the image capturing the image from the segment is searched for the road image. An index is provided so that the image can be retrieved from a database, and the image is recorded in the road image database after being indexed by the azimuth and distance from the central point of the image of the image capturing the image. Aerial image database generation and registration system. The viewpoint path P (t) defined by the function of the time t moving on the passage in the on-road 3D animation generation system of the cityscape 3D animation amusement system according to claim 2, and the viewpoint path P (t) The line-of-sight vector trajectory V (t) indicating the line-of-sight direction from the image line, and the line-of-sight vector trajectory V (t) intersects the virtual image projection plane that stands perpendicular to the boundary line of the building or object or passage that is the object of the image. The trajectory of the point can be defined as the target trajectory T (t), and the line-of-sight vector trajectory V (t _i ) at t = t _i is closest to the attribute composed of the photographing optical axis vector, the photographing position, and the viewing angle. An image having a thing is retrieved from the image database as an original image, and each frame image of the cityscape three-dimensional moving image is converted into each frame image based on the attribute of the original image. Line-of-sight vector, perspective, road 3D video generation system and generating a respective frame image by morphing conversion from the difference in the attribute of the attribute and the original image consisting of the viewing angle. The road three-dimensional video generation system of the cityscape three-dimensional video amusement system according to claim 24, wherein the line-of-sight vector trajectory V (t _i ) at t = t _i includes a shooting optical axis vector, a shooting position, and a viewing angle. In the process of retrieving an image having the closest attribute from the road image database to obtain an original image, the original image has a fixed viewing angle depending on the focal length of the camera around the photographing optical axis vector. The range on the horizontal plane of the human line of sight that intersects the image projection plane that is the subject of the image is on the horizontal plane of the human line of sight that simulates the visual field trajectory V (t _i ) around the line-of-sight vector trajectory V (t _i ). encompasses a range, yet, the angle of the line-of-sight vector of the original image and the visual line vector trajectory V (t _i) is less than the predetermined angle Road 3D video generation system characterized by selecting as. 26. The road three-dimensional video generation system of the cityscape three-dimensional video amusement system according to claim 24, wherein the line-of-sight vector trajectory at time t _i is selected when the original image is selected from the road image database. In order to avoid an obstacle existing between V (t _i ) and the target point, a plurality of the original images satisfying the selection condition up to a point where the imaging optical axis vector and the image projection plane intersect A road three-dimensional moving image generation system, wherein an image having the smallest vector length is selected as an original image. The graphic user interface system of the cityscape three-dimensional video amusement system according to any one of claims 1 and 3, and 15 to 17, is displayed on a monitor or via a dedicated simulation device. Referring to the map for specifying the flight route, while watching the simulated field of view generated by the 3D video generation system, operate the simulated control stick to raise and lower the flight direction and flight altitude, and operate the slot lever to adjust the flight speed By controlling the flight path and speed, the graphic user interface system displays the front view of the simulated cockpit on a monitor or a dedicated simulation device. 26. The graphic user interface system of a cityscape three-dimensional moving image amusement system according to claim 25, further comprising a mechanism for switching the simulated view display of the simulated cockpit to an on-board simulated camera display, wherein the simulated camera has an orientation, zooming, and tilt. An aerial graphic user interface system that can control corners. 27. A graphic user interface system of a cityscape three-dimensional video amusement system according to any one of claims 2 and 23 to 26, wherein a map displayed on a monitor is referred to and generated by the road three-dimensional video generation system. A graphic user interface system characterized in that the direction and speed can be controlled by operating a displayed cursor while viewing the simulated field of view, and the direction of the eyes can be controlled. In the graphic user interface system of the cityscape three-dimensional video amusement system according to any one of claims 27 and 28,
32. When the flight altitude of the real-time flight simulation falls below a certain value, the landing point is designated obliquely downward in front of the field of view, and the viewpoint position is shifted to the road point of the landing point, and the city on the road according to claim 31 hereinafter. Switch to the landscape 3D video generation system. In addition, when a take-off icon is clicked while generating a 3D video of a cityscape on the road, the position is shifted to a position above the point on the take-off road or above a certain altitude where the point on the take-off road is seen diagonally forward in accordance with the selection of the icon. 29. A graphic user interface system capable of switching to a flight simulation in the air of the cityscape three-dimensional moving image amusement system according to any one of claims 27 and 28.