JP5196472B2 - Image display processing program and image display processing system - Google Patents

Description

  The present invention relates to an image display processing program that can calculate corresponding points required for morphing without performing image matching.

Each time it travels a predetermined distance in one direction along the shooting route such as roads, rivers, railway lines, etc., it processes the shooting route and a plurality of images taken around the shooting route so that it can run along the shooting route. An image processing system that displays an image on a display screen of a display device is known (see, for example, Patent Document 1).
In the above-described image processing system, when an image is displayed on the display device so as to travel at a predetermined speed Vm / s along the shooting path, the number of images (frame rate) NV drawn per second is expressed by the following equation: It is represented by
NV = V / D
Here, D is an image acquisition interval. The above equation means that the frame rate NV decreases when the speed specified for the image processing system is low. Therefore, when the maximum frame rate that can be output by the display device is Nmax and Nmax> NV, the image cannot be displayed smoothly. That is, when the designated speed V is low, the frame rate NV decreases even if there is a margin in the output frame rate of the display device, so that smooth continuous image display cannot be realized.
In order to realize smooth continuous image display when the designated speed V is low, it is only necessary to decrease D and increase the number of images to be captured. However, in this case, since the number of captured images increases, the imaging cost increases. Further, when a system for registering captured images in the database of the server computer and transmitting the images from the server computer to the client computer is constructed, the capacity of the database for registering the image increases, so the data processing cost and data Storage and management costs will increase.
Therefore, instead of reducing D, a method called morphing is used between the previous image and the subsequent image, which are the two images immediately before and after the image captured before and after, and between the previous image and the subsequent image. An interpolated image that interpolates between the previous image and the subsequent image may be combined and displayed.
In morphing, it is necessary to obtain a corresponding point that is a pixel position in the subsequent image corresponding to an arbitrary pixel position in the previous image. That is, it is necessary to obtain image coordinates corresponding to the previous image and the subsequent image at every point in the image. A method for obtaining the corresponding points by performing image matching is known (see, for example, Patent Document 2).
JP 2003-219403 A Japanese Patent Laid-Open No. 2002-232908

However, according to the method for obtaining corresponding points by image matching, it is necessary to search for similar points in the entire image in the previous image and the subsequent image, which increases the data construction cost, and for each image pair. Matching information must be managed.
In addition, performing image matching throughout the entire image during display without performing image matching during data construction is expensive and is not a suitable method for generating an interpolated image in real time.
The present invention has been made in view of the above problems, the corresponding points between the required image morphing, image display processing program can be obtained without performing image matching, and provides an image display processing system .

The image display processing program of the present invention captures and displays on a display device a front image and a rear image obtained by photographing the photographing route and the periphery of the photographing route every time a predetermined distance is advanced in one direction along the photographing route. By creating an interpolated image between the previous image and the rear image, the front image, the interpolated image, and the rear image are combined and continuously displayed on the display device so as to move at the specified speed along the shooting path. In the image display processing program for causing the computer to function as a means for display, the image display processing program includes a corresponding point calculation processing program for causing the computer to function as a means for calculating the corresponding points necessary for creating the interpolation image, Corresponding point calculation processing program creates a simple peripheral 3D model that simplifies the surrounding situation of the shooting path, and converts it to the image coordinates of any pixel position in the previous image. Then, calculate the azimuth angle from the shooting position of the previous image on the simple peripheral 3D model to see any pixel position of the previous image, and the straight line extending from the shooting position of the previous image to the azimuth indicated by the above azimuth angle is simple Calculate the intersection that intersects with the surrounding stereo model, calculate the azimuth angle when viewing the above intersection from the shooting position of the rear image on the simple surrounding stereo model, and based on this azimuth angle, Image coordinates are calculated, and the computer is made to function as means for setting an arbitrary pixel position of the subsequent image to a corresponding point corresponding to an arbitrary pixel position of the previous image.
The corresponding point calculation processing program creates a simple peripheral solid model having a left wall and a right wall parallel to each other at the left and right of the shooting position, and causes the computer to function as a means for calculating an intersection that intersects the left wall and the right wall. Also features.
The corresponding point calculation processing program causes the computer to function as means for setting the distance between the shooting position and the left wall or the right wall.
An interpolation image number determination processing program for causing a computer to function as means for determining the number of interpolation images to be created according to a designated speed is also provided.
According to the image display processing system of the present invention, each time a predetermined distance is advanced in one direction along the shooting path, a server computer that registers the previous image and the rear image obtained by shooting the shooting path and the periphery of the shooting path; Interpolate the front image and the rear image from the server computer, create an interpolated image to be displayed between the front image and the rear image, and move the front image and the interpolated image as if moving at a specified speed along the shooting path An image processing system comprising: an image display processing unit that synthesizes the image and a subsequent image and continuously displays the combined image on a display device, wherein the image display processing unit includes the image display processing program. .

According to the present invention, corresponding points between images necessary for morphing can be calculated at high speed using a simple peripheral three-dimensional model without using image matching. Therefore, more interpolated images between the previous and subsequent images can be created in real time, and a smooth continuous image display can be realized. In addition, without increasing the amount of image data to be registered in the database of the server computer, the client can perform smooth continuous image display without decreasing the display frame rate according to the speed designated by the client computer.
By using a simple peripheral solid model including a left wall and a right wall, for example, an image display process including a road and buildings arranged on both sides of the road can be appropriately performed.
By appropriately setting the distance between the shooting position and the left and right walls, the image correlation is increased, and the connection between images can be made smooth.
Since the interpolation image number determination processing program is provided, the number of interpolation images to be created can be determined according to the designated speed, and smooth continuous image display can be performed without reducing the display frame rate. In particular, when a slow speed is designated, a smooth continuous image display can be realized by creating more interpolated images between the preceding and succeeding images in real time .

  1 to 7 show the best mode of the present invention, FIG. 1 shows a block configuration of an image display processing system, FIG. 2 shows an image acquisition method, and FIG. 3 shows a front image, a rear image, and an interpolated image. 4 shows the relationship between pixel positions and orientations, FIG. 5 shows an image coordinate system, FIG. 6 shows a simple surrounding solid model representing road surrounding conditions, and FIG. 7 shows a method for obtaining corresponding points.

  As shown in FIG. 1, the image display processing system 1 includes a server computer 2 and an image display processing device 3. The server computer 2 includes a database 4. The image display processing device 3 includes a client computer 5, an image display processing program 6 that causes the client computer 5 to function as image display processing means, an input device 7, and a display device 8. The image display processing program 6 includes a corresponding point calculation processing program 11, a morphing processing program 12, and an interpolation image number determination processing program 13.

  An outline of an image display processing method by the image display processing system 1 will be described. First, as shown in FIG. 2, each time a Dm travels in one direction A along a road 19 as a shooting route, a plurality of panoramic still images (hereinafter referred to as “panoramic still images”) are taken. 20 is registered in the database 4 of the server computer 2. The server computer 2 transmits the image 20 to the image display processing device 3 in response to a request from the image display processing device 3. The image display processing device 3 continuously displays images on the display screen of the display device 8 at a speed instructed via the input device 7 using the plurality of images 20 sent. That is, an image is displayed as if a person watching the display screen is traveling on the road. In this case, the image display processing device 3 performs a morphing process between the front image I (i) and the rear image I (i + 1) which are the two images 20; 2; 3, as shown in FIGS. 2 and 3, an interpolated image Iins (I (i), I (i +) which is an interpolated image 21 for interpolating between the previous image I (i) and the subsequent image I (i + 1). 1), t) is synthesized and displayed.

For example, the plurality of images 20 are individually attached to the positions of the black spots of the car 25 shown in FIG. 2 with the cameras 26 facing upward, forward, left, right, left rear, and right rear of the car 25. Every time you travel Dm while traveling on the road in one direction (traveling direction) A, the above six cameras 26 will simultaneously drive the car 25 upward, the car 25 forward, the car 25 left, the car 25 right The direction, the left rear direction of the car 25, and the right rear direction of the car 25 were obtained. In other words, a plurality of cameras 26 were used to shoot at different angles, and a plurality of images obtained thereby were combined to form a panoramic still image.
A panoramic still image may be acquired by using a highly efficient fish-eye lens or a similar lens and mounting the lens camera to shoot.

  A sequence number i = 1,... Nimg is assigned to the plurality of individual images 20 obtained by photographing each time Dm travels along the road 19 in the order of photographing, and a series of images I (i) is obtained in the database 4. be registered. Further, each image 20 is given specific information including information on the shooting position (Xi, Yi, H) where the image 20 was shot and information on the shooting direction (azimuth angle in the traveling direction) θi.

  As shown in FIG. 3, all the images 20 are omnidirectional panoramic images created by equirectangular projection images (images linearly resampled to the horizontal and vertical angles from the shooting position). As shown in FIG. 5, the omnidirectional panoramic image is an image 20 in which the column (col) direction corresponds to 360 degrees in the horizontal direction and the line (row) direction corresponds to 180 degrees in the vertical direction. The direction of travel. The image size of the omnidirectional panoramic image is col × row.

  As shown in FIG. 4, the horizontal azimuth is an angle measured clockwise from north, and the shooting azimuth θd is the center of the image 20, that is, the horizontal azimuth in the traveling direction of the vehicle. The photographing position is measured in a three-dimensional coordinate system in which X (E) is eastward, Y (N) is north, and Z is the vertically upward direction of the camera 26. θh and θv are a horizontal azimuth angle θh and a vertical angle θv viewed from the photographing position. Here, the inclination of the road 19 is ignored, and the upper direction of the camera 26 is assumed to coincide with the vertical axis.

As shown in FIG. 5, in the image coordinate system of the upper left origin, when the image coordinate of an arbitrary position P on the image 20 is (x, y), the horizontal azimuth angle θh and the vertical position when the position P is viewed from the shooting position. The angle θv is expressed by the following formula.
θvi = 90-y × (180 / row)
θhi = x × (360 / col) + θd-180
... (1)
The inverse transformation is expressed by the following equation.
y = (90-θvi) × (row / 180)
x = (θhi-θd + 180) × (col / 360) //
... (2)

  In the best mode 1, the rear image I (i) corresponding to the position Pi on the image of the front image I (i) using the information on the photographing position, the information on the photographing direction, and the simple road model representing the road surrounding situation. The position P (i + 1) on the image of (+1) is obtained as a corresponding point. Hereinafter, a method for obtaining corresponding points will be described.

  The shooting position of the front image I (i) is (Xi, Yi), the shooting direction of the front image I (i) (the azimuth angle in the traveling direction of the vehicle) is θi, and the shooting position of the rear image I (i + 1) is ( X (i + 1), Y (i + 1)), the shooting direction of the rear image I (i + 1) is θ (i + 1), and the camera height from the ground to the camera 26 at the time of shooting is H.

The first step for determining the corresponding points is a simple peripheral solid that represents the surrounding situation of the imaging position (Xi, Yi, H) and the imaging position (X (i + 1), Y (i + 1), H). Assume model 30. The simple surrounding solid model 30 may be any one that simplifies the surrounding environment.
For example, in the case of a straight road or a road where buildings are lined up along a gentle curve, as a simple peripheral solid model 30, a rectangular parallelepiped shape having left and right walls 31; 32 parallel to the traveling direction as shown in FIG. A simple peripheral three-dimensional model 30 is created. In other words, the shooting position of the previous image I (i) is (Xi, Yi, H), and the bottom surface is a rectangular parallelepiped with z = 0 and the long side direction parallel to the moving direction.

  As other parameters for determining the simple peripheral three-dimensional model 30, the left distance DL to the left wall 31 on the left side from the shooting position toward the shooting direction, the right distance DR to the right wall 32 on the right side, the front to the wall in the front direction There is a distance DF, a rear distance DB to the rear wall, and a height HS to the upper surface. DB, DF, and HS are based on the distance D between the shooting position of the previous image I (i) and the shooting position of the rear image I (i + 1) (X (i + 1), Y (i + 1)). Is set to a sufficiently large value (for example, 100 times may be infinite).

When the simple rectangular solid model 30 of the rectangular parallelepiped is adopted, the left distance DL and the right distance DR are determined with reference to a separately provided road width or determined by the following procedure.
Now, it is assumed that the partial area AL in the previous image I (i) is on the left wall 31 side of the simple peripheral three-dimensional model 30. The corresponding point at I (i + 1) of the jth (j = 1,..., N) pixel position P (i, j) in the area AL of the previous image I (i) is P (i + 1, j, DL). A resampling image in which the pixel value at pixel position Pij is replaced by P (i + 1, j, DL) of I (i + 1)
I (i + 1, DL)
Then, in the correct DL, it is expected that I (i) and I (i + 1, DL) are almost the same in the region A. That is, I (i) and I (i + 1, DL) have high image correlation, and the connection between images can be made smooth.
Therefore, DL is determined by the following procedure.
(1) Prepare a list Dk (k = 1, .. n) of n distances.
(2) For all Dk, a resampled image I (i + 1, Dk) is created for the region AL.
(3) For each k, a correlation coefficient Rk between I (i) and I (i + 1, Dk) is calculated.
(4) The distance Dk that maximizes Rk gives the optimal left wall distance DL.
(5) The distance DR of the right wall can also be determined by the same method as described above.

  In the second step for determining the corresponding point, first, for an arbitrary pixel position (xi, yi) of the previous image I (i), an azimuth angle (θhi, θvi) corresponding to the pixel position (xi, yi) Is obtained using equation (1). Then, as shown in FIG. 7, an intersection where a straight line extending from the shooting position (Xi, Yi, H) of the previous image I (i) to the direction (θhi, θvi) intersects the left wall 31 of the simple peripheral solid model 30. PS (XSi, YSi, ZSi) is obtained.

  In the third step for determining the corresponding points, first, as shown in FIG. 7, from the shooting position (X (i + 1), Y (i + 1), H) of the rear image I (i + 1). The azimuth angles (θh (i + 1), θv (i + 1)) of the viewed intersection PS (XSi, YSi, ZSi) are obtained. Then, the pixel position (x (i + 1), y (i + 1)) of the post-image I (i + 1) corresponding to the azimuth angle (θh (i + 1), θv (i + 1)) It calculates | requires using Formula (2).

  Thus, the pixel position (x (i + 1), y (i + 1)) of the subsequent image I (i + 1) corresponding to the arbitrary pixel position (xi, yi) of the previous image I (i) is obtained. .

  Thus, the pixel positions (x (i + 1), y (i + 1)) of the subsequent image I (i + 1) corresponding to all the pixel positions (xi, yi) of the previous image I (i) are By obtaining the corresponding points for all the corresponding points in the image I (i) and the image I (i + 1), the interpolation image 21 at an arbitrary interpolation position t is obtained using the following equation (3). Can be requested.

  When the corresponding points are obtained, the image Iins (I (i), I (i + 1), t) at an arbitrary interpolation position between the previous image I (i) and the image I (i + 1) is obtained by morphing. Generate.

The image Iins (I (i), I (i + 1), t) obtained by morphing is synthesized as follows.
When the position Pi of the image I (i) on the image corresponds to the position P (i + 1) of the image I (i + 1) on the image, the interpolated image 21 is interpolated by the parameter t. The display color Iins (Pt) of the image at the position Pt = (1-t) × Pi + t × P (i + 1) in the image 21 is the display colors I (i) (Pi) and P (i +) of Pi. The display color I (i + 1) (P (i + 1)) of 1) is used to be obtained by the following equation (3).
Iins (Pt) = (t-1) × I (i) (Pi) + t × I (i + 1) (P (i + 1)) (3)

That is, the client computer 2 of the image display processing device 3 loads an image display processing program 6 including a corresponding point calculation processing program 11, a morphing processing program 12, and an interpolated image number determination processing program 13 from a storage device (not shown) such as a ROM. The following processing is performed in accordance with the procedure of the capture and image display processing program 6. First, the client computer 5 performs the following processing according to the procedure of the corresponding point calculation processing program 11. A rectangular parallelepiped simple peripheral solid model 30 including a left wall 31 and a right wall 32 is created. Substituting the image coordinates (xi, yi) of an arbitrary pixel position (xi, yi) of the previous image I (i) into the equation (1), from the shooting position (Xi, Yi, H) of the previous image I (i) An azimuth angle (θhi, θvi) when an arbitrary pixel position (xi, yi) of the previous image I (i) is viewed is calculated. The distances DL and DR between the shooting position on the simple peripheral three-dimensional model 30 and the left wall 31 and the right wall 32 are set to values instructed via the input device 7. A straight line extending from the shooting position (Xi, Yi, H) of the previous image I (i) to the azimuth (θhi, θvi) indicated by the azimuth angle (θhi, θvi) is the left wall 31 of the simple peripheral solid model 30 or An intersection PS (XSi, YSi, ZSi) that intersects the right wall 32 is calculated. Orientation of the intersection PS (XSi, YSi, ZSi) viewed from the shooting position (X (i + 1), Y (i + 1), H) of the rear image I (i + 1) on the simple peripheral solid model 30 The angles (θh (i + 1), θv (i + 1)) are calculated. Substituting this azimuth (θh (i + 1), θv (i + 1)) into equation (2) to obtain image coordinates (x (i + 1), y (i + 1)) Corresponding point in the subsequent image I (i + 1) corresponding to the arbitrary pixel position (xi, yi) of the previous image I (i) with the image coordinates (x (i + 1), y (i + 1)) Determine as. Similarly, corresponding points in the subsequent image I (i + 1) corresponding to all the pixel positions of the previous image I (i) are determined.
Next, the client computer 5 follows the procedure of the morphing processing program 12 and images Iins (I (i), I (i +) at an arbitrary interpolation position between the previous image I (i) and the image I (i + 1). 1), t) is generated.
At this time, when the speed is designated via the input device 7, the client computer 5 determines the number of interpolation images 21 to be created by morphing according to the procedure of the interpolation image number determination processing program 13. For example, when a fast speed is designated, the number of interpolation images 21 created by morphing is reduced, and when a slow speed is designated, the number of interpolation images 21 created by morphing is increased. The interpolated image number determination processing program 13 includes a correspondence table between a designated speed and the number of interpolated images (not shown), and the client computer 2 determines the number of interpolated images with reference to the correspondence table when the speed is designated. .

According to the best mode, corresponding points between images necessary for morphing can be calculated at high speed using the simple peripheral three-dimensional model 30 without using image matching. More can be created in real time, and smooth continuous image display can be realized. Therefore, smooth continuous image display can be performed without decreasing the display frame rate according to the speed designated by the client computer 5 without increasing the amount of image data registered in the database 4 of the server computer 2.
By using the simple peripheral three-dimensional model 30 including the left wall 31 and the right wall 32, continuous image display of an omnidirectional panoramic image including the road 19 and buildings lined on both sides of the road 19 can be appropriately performed. In this case, when the distances DL and DR between the shooting position and the left wall 31 and the right wall 32 are appropriately set, the image correlation becomes high, and the connection between the images can be made smooth.
Since the interpolated image number determination processing program 13 is provided, smooth continuous image display can be realized in accordance with the designated speed.

  When there are almost no buildings in the surroundings, an upper hemisphere having a bottom surface with z = 0 as shown in FIG. The diameter R of the hemisphere is set to a sufficiently large value (for example, 100 times) for the moving distance between I (i) and I (i + 1). Even when there are almost no buildings around the intersection, the upper hemisphere with the bottom z = 0 is adopted as the simple peripheral solid model 30. The diameter of the hemisphere approximately matches the road width.

  INDUSTRIAL APPLICABILITY The present invention can be applied to an application that uses omnidirectional panoramic image data obtained by photographing a street or the like, such as road management / car navigation data creation, power / communication facility management, and outdoor advertisement management.

The block block diagram of an image display processing system. Explanatory drawing which shows the image acquisition method. The figure which shows a front image, a back image, and an interpolation image. The figure which shows the relationship between a pixel position and an azimuth | direction. The figure which shows an image coordinate system. The figure which shows the simple periphery solid model showing the road periphery condition. Explanatory drawing which shows the method of calculating | requiring a corresponding point. The figure which shows the other example of a simple periphery solid model.

Explanation of symbols

1 image display processing system, 2 server computer, 3 image display processing device,
4 databases, 5 client computers,
6 image display processing program, 11 corresponding point calculation processing program,
12 morphing processing program, 13 interpolation image number determination processing program.

Claims (5)

  Each time the camera travels a predetermined distance along the shooting path, the front image and the rear image obtained by shooting the shooting path and the periphery of the shooting path are captured and displayed on the display device, and between the front image and the rear image. By creating an interpolated image, the computer functions as a means for combining the previous image, the interpolated image, and the subsequent image so that they are moving at a specified speed along the shooting path and continuously displaying them on the display device. In the image display processing program to be executed, the image display processing program includes a corresponding point calculation processing program for causing a computer to function as a means for calculating corresponding points necessary for creating an interpolated image. Create a simple peripheral 3D model that simplifies the surrounding situation of the image, and based on the image coordinates at any pixel position in the previous image, Calculate the azimuth angle when viewing any pixel position in the previous image from the shooting position, and calculate the intersection where the straight line extending from the shooting position of the previous image to the azimuth indicated by the above azimuth intersects the simple peripheral solid model The azimuth angle when the intersection is viewed from the shooting position of the rear image on the simple peripheral stereo model is calculated, and the image coordinates of an arbitrary pixel position of the rear image are calculated based on the azimuth angle. An image display processing program for causing a computer to function as means for setting an arbitrary pixel position as a corresponding point corresponding to an arbitrary pixel position of a previous image.   Corresponding point calculation processing program creates a simple peripheral solid model with left and right walls and bottom parallel to each other at the left and right of the shooting position, and functions as a computer to calculate intersections that intersect the left and right walls The image display processing program according to claim 1, wherein:   The image display processing program according to claim 2, wherein the corresponding point calculation processing program causes the computer to function as means for setting the distance between the photographing position and the left wall or the right wall.   The image according to any one of claims 1 to 3, further comprising an interpolation image number determination processing program that causes a computer to function as means for determining the number of interpolation images to be created according to a designated speed. Display processing program. A server computer that registers a front image and a rear image obtained by photographing the photographing route and the periphery of the photographing route every time a predetermined distance is traveled in one direction along the photographing route, and fetches the front image and the rear image from the server computer Creates an interpolated image to be displayed between the previous image and the rear image, and synthesizes the previous image, the interpolated image, and the rear image so that it moves at the specified speed along the shooting path. An image processing system comprising image display processing means for continuously displaying, wherein the image display processing means comprises the image display processing program according to any one of claims 1 to 4. Image display processing system .

Images