JP4530214B2 - Simulated field of view generator - Google Patents

Description

  The present invention relates to a simulated visual field generator for displaying a visual field outside a window in real time according to a driving operation by a steering wheel, for example, for an automobile simulator.

In the prior art that generates a simulated field of view, a method of rendering an object such as a building on a roadside or a vehicle on the road with polygons and rendering it with a three-dimensional computer graphics technique is generally used. Graphics LSIs have greatly improved performance and are now widely used. In order to increase the reality of the appearance, a technique such as a photo texture in which an image obtained by photographing an actual object is pasted on a polygon is often used. However, in order to input a real cityscape as a three-dimensional model, accurate position coordinates of objects such as buildings are required to a detailed level, and data creation work using modeling software is also reliant on human labor. There is a problem that takes.
On the other hand, an apparatus for simulating the outside view of a train simulator window by storing an image captured by a video camera on a laser disk or the like and controlling the frame display speed from the laser disk according to the traveling speed is used. This method has an advantage that an image of an actual route can be used for simulation relatively easily. Disadvantages include, for example, in order to simulate a vehicle entering a railroad crossing or how a person falls from the home, image processing that combines a two-dimensional image with an accurate position and size for each frame is required. Takes time and effort.

  The problem to be solved is to generate images of cityscapes taken with a video camera and roads, vehicles, signals, etc. created as a three-dimensional model according to the virtual viewpoint position and line-of-sight direction, and combine them on one screen. To be able to display in real time.

The present invention captures panoramic image data in which the position coordinates and the main shooting direction of a shooting point previously shot on a road are shot, recorded together with the shooting direction, and a predetermined area can be cut out by the shooting direction vector . A photographed image data storage means for holding a set and an image in which a wall-shaped rectangle is defined on both sides of a road in a model built in a computer and a specific area of the wall-shaped rectangle is viewed from a virtual viewpoint In consideration of the position coordinates of the shooting point and the line-of-sight direction vector from the virtual viewpoint toward the center of the specific area of the wall-shaped rectangle, the distance from the virtual viewpoint is short. A shooting direction vector from the shooting point to the center of the specific area, and a gaze direction vector from the virtual viewpoint to the center of the wall-like rectangular area. From the panoramic image captured and recorded at the shooting point, the specific area that is a wall-shaped rectangle with respect to the direction of the shooting direction vector is desired. An image area corresponding to the viewing angle is cut out and pasted on the wall-shaped rectangular area to generate images on both sides of the road, and from the panoramic image acquired at the shortest shooting point from the virtual viewpoint, An image area corresponding to a desired viewing angle of a rectangular area is cut out and pasted to the rectangular area to generate a front image, and a 3D model modeled with polygons is rendered using 3D computer graphics techniques. Which outputs color data and depth data from the viewpoint to the wall rectangle and front rectangle A three-dimensional model image is generated by rendering a three-dimensional model modeled with a polygon using a three-dimensional computer graphics technique and outputting color data for each pixel and depth data from a virtual viewpoint to the rendered object. A set of color data for each pixel by the panorama image generation unit and a set of depth data from a virtual viewpoint to a wall-like rectangle and a front rectangle, or a predetermined plurality of sets, and a pixel by the three-dimensional model image generation unit Each color data and one set of depth data from the virtual viewpoint to the object or a predetermined plurality of sets are input, and all depth data are compared in size for each pixel on one frame, and the virtual viewpoint is the most virtual for each pixel. Image synthesizing means for selecting color data of pixels close to and outputting as video signals.

  According to the simulated visual field generation device of the present invention, a model constructed in a computer is defined by connecting wall-shaped rectangles on both sides of a road, and an image in which a specific region of the wall-shaped rectangle is viewed from a virtual viewpoint is obtained. Considering the position coordinates of the shooting point and the vector from the virtual viewpoint toward the specific area of the wall-shaped rectangle from the shot image data such as the cityscape from the road shot with a video camera, etc. The image area of the panoramic image viewed from the shooting point closest to the vector direction is cut out and pasted on the wall-shaped rectangle corresponding to the specific area to generate images on both sides of the road and set the front image to the front Generated by cutting out and pasting the front panorama image area to the front rectangular area and color data for each pixel and the depth data from the viewpoint to the wall-shaped rectangle. By using images by 3D computer graphics in the foreground parts such as roads, vehicles, signals, etc., the simulated view fields corresponding to the virtual viewpoints are synthesized on one screen for both the foreground and foreground parts from both sides of the road in real time. Can be displayed.

  FIG. 1 is a functional block diagram of one embodiment illustrating a simulated visual field generation apparatus according to the present invention. 101 is a driving operation input device, 102 is a vehicle motion simulation unit, 103 is a left side distant view image generating unit, 104 is a right side distant view image generating unit, 105 is a front distant view image generating unit, 106 is a foreground image generating unit, and 107 is an image. A synthesizing device 108 is a display device. The left side distant view image generation unit 103, the right side distant view image generation unit 104, and the front distant view image generation unit 105 constitute a panoramic image generation unit. The foreground image generation means 106 constitutes a three-dimensional model image generation means.

The exercise operation input device 101 is an operation device for a simulated vehicle, such as a steering, a brake pedal, and an accelerator pedal. The vehicle motion simulation unit 102 connects the output of the motion operation input device 101, that is, the operation amount of the operating device, simulates the motion of the simulated vehicle, and determines the position of the driver's viewpoint, that is, the virtual viewpoint, the direction of the vehicle, the vehicle The speed of the virtual viewpoint, etc. is output as the speed of.
The left side distant view image generation unit 103, the right side distant view image generation unit 104, and the front distant view image generation unit 105 are respectively provided with photographed image data storage units 1031, 1041, and 1051, and the positions of the photographing points previously photographed on the road. A set of panoramic image data whose coordinates and main shooting direction are known is held.

(Acquisition of omnidirectional video images)
The acquisition of an omnidirectional video image will be described. Aside from the operation of the simulated visual field generation device, a video camera capable of photographing all directions is mounted on the vehicle in advance, and is shot while traveling on an actual road to obtain an omnidirectional image (360-degree panorama). At the same time, the position coordinates of the shooting point and the azimuth angle (main shooting direction) of the traveling direction of the vehicle are measured by GPS or the like and recorded together with the image. An example of this photographing is shown in FIG. The measurement vehicle 201 is equipped with an omnidirectional video camera 202, travels in an area to be simulated, and passes through points X1, X2, and X3 according to the passage of time t-1, t, and t + 1. An omnidirectional image that is visible from each of those points at the time is obtained.

(Create panorama image)
Creation of panoramic image data to be stored in the photographed image data storage means 1031, 1041, 1051 will be described. As described above, a video image captured by a video camera is taken into a computer and converted into digital data. For example, a predetermined area can be cut out by a shooting direction vector as a direction with respect to the main shooting direction on a horizontal plane including the main shooting direction. recorded in, to create a panoramic image data of time T. This is repeated for each time, and panoramic image data for each time is created from all the video images, and stored in the disks constituting the respective photographed image data storage means 1031, 1041, and 1051.

(Simulation of scenery using panoramic images)
It is assumed that the virtual viewpoint of the driver in the simulated vehicle on the road is located at P as shown in FIG. The generation of an image of a distant view from the road viewed from the virtual viewpoint P at this time will be described.
Assume a screen that continues like a wall on both sides of a virtual road. Therefore, it is assumed that the left side far view image generation unit 103 has a screen Sl (left). The right side distant view image generation means 104 assumes a screen Sr (right). These screens Sl and Sr are assumed to be located on the wall of the building facing the road, and the objects located farther away than the building facing the road are all assumed to be located on the wall or screen of the building. To generate an image.
A landscape viewed from a virtual viewpoint P traveling on a road is generated from a panoramic image. That is, the scenery seen from a certain virtual viewpoint P on the road is an image p (t-1) taken from the point X1 by the measuring vehicle at time t-1, as shown in FIG. A landscape viewed in the direction is generated using an image p (t) taken from a point X2 at time t, and a landscape seen in the C direction is generated using an image p (t + 1) taken from a point X3 at time t + 1.
For the building X in the real world, first, for example, a shooting point group that is relatively close to the virtual viewpoint P is extracted from the shooting image data storage unit 1031 of the left side distant view image generation unit 103, and heads toward the center of the building X from the virtual viewpoint P. A shooting point where the angle formed by the line-of-sight vector and the shooting direction vector from the shooting point toward the center of the building X is searched is searched, and the building X of the panoramic image recorded in the shooting direction vector direction at the shooting point is obtained. An existing area, that is, an area having a certain horizontal and vertical viewing angle range in which the building X is desired from the photographing point is cut out in a strip shape that is long in the vertical direction and pasted on the screen. At this time, the search is not performed for each individual object, but the line-of-sight vector 401 is changed so that the screen in the field of view is filled as shown in FIG. , 405... Are cut out in a strip shape and pasted on the screen Sl406. The distance from the virtual viewpoint P to the wall where the building is located is calculated, and the distance is defined as d. For example, the center point of the location where each shooting direction vector is in contact with the wall at a distance d is the boundary point Q1, Q2, Q3, Q4, Q5... Between the regions of the panoramic image of each shooting direction vector (in the vertical direction on the screen surface). It is possible to make the distant view of a building or the like look closer to reality by setting the boundary line).
FIG. 4 illustrates the processing in the left side far view image generation unit 103, but the same applies to the right side far view image generation unit 104.
The screens Sl and Sr are defined as a set of polygons, and the panoramic image is registered as a texture. For example, by using glGLex Coord2f () of the Open GL graphics library, the correspondence between these polygons and textures can be specified so that panoramic images can be continuously pasted on the screen as described above. Cut out the shape. Screen Sl and screen Sr are defined in the same coordinate system.
The front image in the traveling direction is generated from the panoramic image taken at the point closest to the virtual viewpoint by the front distant view image generation means 105. As shown in FIG. 5, it is assumed that the virtual viewpoint 501 by the simulation vehicle is traveling on the virtual road with the right direction in the figure in front. The left screen S <b> 502 and the right screen Sr <b> 503 are located on both sides (vertical direction in FIG. 5) of the virtual viewpoint 501. A predetermined viewing angle 504 in front of the virtual viewpoint 501 is set, and an appropriate height equal to, for example, the height of the cityscape and the intersection of the vertical plane extending to the left and right of the viewing angle 504, the left screen Sl502 and the right screen Sr503 surface. A polygon of a rectangle Sf505 formed by the horizontal line that has been passed is set. The position of the virtual viewpoint 501 every moment is input by the vehicle motion simulation unit 102, and the front distant view image generation means 105 uses the panoramic image obtained from the image taken at the point X2 closest to the position of the virtual viewpoint 501 as the texture. The front image is generated by cutting out, cutting out an area corresponding to the rectangle Sf in the panoramic image, and pasting the region on the rectangle Sf505. The rectangle Sf is defined in the same coordinate system as the screen Sl and the screen Sr.
The left side distant view image generation unit 103, the right side distant view image generation unit 104, and the front distant view image generation unit 105 generate the distant view image for each virtual viewpoint that enters from the vehicle motion simulation unit 102 and changes from moment to moment.

(Simulation of road surface etc. and image generation by 3D model)
An object that cannot be regarded as a roadside, such as a road surface, guardrail, pedestrian bridge, vehicle, traffic light, etc., is modeled in the foreground image generation means 106 as a three-dimensional model composed of polygons by a three-dimensional computer graphics technique. Create using. The sky is simulated by, for example, a set of hemispherical polygons with a cloud texture.
Image generation of the object in the foreground image generation means 106 is performed by drawing processing (rendering) of an image viewed from a virtual viewpoint every moment entering from the vehicle motion simulation unit 102 by a depth buffer method using a three-dimensional graphics chip. Is called. The three-dimensional model is defined in the same coordinate system as the screen S1, the screen Sr, and the rectangle Sf.
In addition, a three-dimensional model modeled with polygons is rendered, and color data for each pixel and depth data for each pixel from the virtual viewpoint to the rendered object are output.

(Generation of simulated images using panoramic images)
As described above, the three-dimensional models such as the screen Sl, the screen Sr, the rectangle Sf, and the road surface are defined in the same coordinate system. The screen S1, the screen Sr, and the rectangle Sf are a set of polygons with textures attached. The left side distant view image generation unit 103, the right side distant view image generation unit 104, and the front distant view image generation unit 105 each use a 3D graphics chip, and display processing (rendering) of an image viewed from a virtual viewpoint by a depth buffer method. ) And depth data for each pixel from the virtual viewpoint to the object (surface of each screen Sl, screen Sr, and rectangle Sf) is output in addition to the color data for each pixel of the displayed image.
It should be noted that the panoramic image data size is large in the left side distant view image generation unit 103, the right side distant view image generation unit 104, and the front distant view image generation unit 105, and the right side distant view image generation unit 104 and the front distant view image generation unit 105 respectively. When all the data cannot be stored in the texture memory of the image data, the panorama image is sequentially read from the high-speed disk and the contents of the texture memory are updated.

(Synthesis of simulated image using 3D model and simulated image using panoramic image)
As described above, the simulated image based on the three-dimensional model is generated by the foreground image generating unit 106, and the simulated image using the panoramic image is generated by the left side far view image generating unit 103, the right side far view image generating unit 104, and the front background image generating unit 105. In each case, rendering is performed by the depth buffer method, and the color and depth for each pixel are output. Based on these data, the image compositing device 107 performs concealment determination based on the depth of each pixel using, for example, a comparator circuit and a selector circuit, selects a color closest to the viewpoint, and performs D / A conversion. The video signal is output and the combined result is displayed on the display device 108 in real time. For example, the depth data is set so that the three-dimensional model of the road surface generated by the foreground image generation means 106 is in front of the road surface of the panoramic image to be compared with this, so the road surface of the three-dimensional model is always synthesized. Displayed as an image.

It is a functional block diagram of one Example of the simulation visual field generator which concerns on this invention. It is explanatory drawing which showed acquisition of the omnidirectional video image. It is a figure explaining one object in a certain direction from a virtual viewpoint, and the point which image | photographs it. It is a figure explaining the relationship between a side screen and the image to paste. It is a figure explaining the relationship between the front and the image to paste.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 ... Driving operation input device, 102 ... Vehicle motion simulation part, 103 ... Left side distant view image generation means, 104 ... Right side distant view image generation means, 105 ... Front distant view image generation means, 106 ... Near view image generation means, 107 ... Image Synthesizer 108, display device.

Claims (2)

A set of panoramic image data that can shoot all directions in which the position coordinates and the main shooting direction of the shooting point previously shot on the road are known, recorded together with the shooting direction, and cut out a predetermined area by the shooting direction vector is held. Photographic image data storage means;
In the model built in the computer, a wall-shaped rectangle is defined by connecting both sides of the road, and an image of a specific area of the wall-shaped rectangle viewed from a virtual viewpoint is captured from the captured image data. Considering the position coordinates of the point and the line-of-sight direction vector from the virtual viewpoint toward the center of the specific area of the wall-like rectangle , the shooting point is selected from among the shooting points that are close to the virtual viewpoint. The shooting point that minimizes the angle formed by the shooting direction vector from the virtual viewpoint toward the center of the specific area and the gaze direction vector from the virtual viewpoint toward the center of the specific area of the wall-like rectangle is selected, and the shooting point The image region corresponding to the desired viewing angle is cut out from the panoramic image captured and recorded in step S1, and the specific region that is a wall-shaped rectangle with respect to the direction of the photographing direction vector at the photographing point is extracted. The generates the road both sides of the image by adhering to its wall-like rectangular areas, an image area corresponding to the view angle overlooking the front rectangular area from the panoramic image obtained by the shortest photographing point from the virtual viewpoint A front image is generated by cutting out and pasting to the rectangular area , a 3D model modeled with polygons is rendered by 3D computer graphics techniques, and the wall-shaped rectangle is viewed from the color data and viewpoint of each pixel. Panoramic image generation means for outputting depth data up to the front rectangle;
3D model image generation means for rendering a 3D model modeled with polygons by a 3D computer graphics technique and outputting color data for each pixel and depth data from the virtual viewpoint to the rendered object;
One set of color data for each pixel by the panoramic image generation means and depth data from a virtual viewpoint to a wall-like rectangle and a front rectangle or a predetermined plurality of sets, and a color for each pixel by the three-dimensional model image generation means Pixel that is closest to the virtual viewpoint for each pixel by inputting data and one set of depth data sets from the virtual viewpoint to the object, or by comparing a plurality of depth data for each pixel on one frame And an image composition means for selecting the color data and outputting it as a video signal.
2. The simulated visual field generation device according to claim 1, wherein the panoramic image generation means generates one or more of an image on the left side of the road, an image on the right side of the road, or an image on the front.

Images