JPWO2004008744A1 - Planar development image processing method, reverse development image conversion processing method, plane development image processing device, and reverse development image conversion processing device for plane object images such as road surfaces - Google Patents

Abstract

It is a method and apparatus for developing and displaying a plan view as represented by a map by converting oblique images obtained by perspective of various monitor cameras. Information necessary for plane development is read from the perspective image obtained from a normal camera, developed into a plan view like a map by mathematical formulas, and combined and connected to form a large road development view. For example, a normal video camera is mounted around the bus, and the entire road is composed of multiple video cameras. The road surface outside the bus is photographed at a certain angle, that is, a depression, and this is calculated. The road surface image is developed like a map image, the image developed like the map is processed, connected, and generated as a road surface image around the bus. This is obtained by calculation using mathematical formulas.

Description

The present invention relates to a planar developed image processing method for a planar object image such as a road surface in which the screen is photographed in perspective when photographed with a normal camera but is developed in a plane like a map screen, and vice versa. The present invention relates to a developed image conversion processing method, a flat developed image processing apparatus, and a reverse developed image processing apparatus.
Specifically, a video shot with a normal camera is developed, for example, images around the bus are shot with a plurality of cameras, each is developed as a flat road surface on the road surface, and the road surface around the bus is As seen on the map from the top of the road, the developed images are combined and displayed as a single flat road surface, and by this flat image development, it becomes possible to combine images by becoming a linear scale, In addition, image recognition and measurement processing can be performed on the target object in the developed flat image, and the road surface is developed into a flat surface using this image conversion principle in reverse. By returning to the image, it is possible to obtain a video with a different viewpoint from the first perspective image, and this image development principle is applied not only to the road surface but also to any plane in the image .
In addition, an optical flow (or parallax, matching, etc.) processing is performed on the flatly developed image to extract an arbitrary plane from other planes. Further, the unevenness of the plane is detected, and the deviation of the unevenness from the plane is detected. In addition, by using the optical flow, the moving distance speed direction is extracted, an object composed of a CG (computer graphics) image is placed on the flatly developed image, and a texture corresponding to the object is pasted. The camera's dip angle θ can be obtained by displaying as a perspective image, or by making the line components that should be parallel lines in the screen parallel when the perspective image is flattened, or in the perspective image By calculating the point where the line components that should be parallel lines in the screen intersect, that is, the vanishing point, the camera's dip angle θ And how to deploy in a plane that allows or correcting the camera shake by moving the image to a constant apparatus further relates its application.
Furthermore, it is possible to reconstruct a desired moving image by transmitting and receiving the plane-developed image information and the camera position information so that moving image data can be transmitted and received at high speed while minimizing the data transmission amount. To do.

Conventionally, it has been natural that a CCTV (Closed Circuit Television) camera or various monitor cameras are photographed in perspective due to the nature of the lens. For example, with a camera that captures the situation behind the bus that arrives at the rear of the bus, unless the camera is mounted vertically downward, a perspective image is naturally captured, and an image of the road surface seen from directly above is obtained. I couldn't. Moreover, it was a reality that it was partly visible, had a blind spot, or looked as a distorted image when viewed by a rearview mirror etc., including the sides, the front, and the rear of both sides of the bus. In other words, if the image was taken with a normal camera, the image was taken in perspective, and it was not possible to draw an image developed on a plane like a map.
In addition, as a technology for combining images, there is a technology that combines oblique images, but in principle, it is a combination of multiple images taken from the same shooting position, changing the viewpoint, and the camera position can be changed arbitrarily. There is no technology that unfolds, combines, and displays similar images to real objects. In addition, there has not been a method or apparatus for obtaining a video with a different viewpoint from a video image in the past, and a method for detecting unevenness of a plane from a plurality of perspective images by image processing technology is also available from the uneven surface. There was no way to determine the deviation.
There is neither a method nor an apparatus for extracting the moving distance speed direction using the image developed from the perspective image of the video, but also deleting the image other than the target from the image developed on the plane and converting it into a CG image. There is no method or device for pasting textures and displaying them as a plane or perspective image, and there is no exact method and equation for extracting and calculating the variables of the equations necessary for plane development from within the image Met.
Moreover, even if a mathematical solution of plane development is to be obtained, 6 to 12 or more actually measured corresponding points are required to associate with the actual coordinates, and that condition is satisfied particularly in moving images. Is virtually impossible.
Furthermore, when moving images such as video images are transmitted between a plurality of devices or the like, transmission data is compressed. However, the conventional compression method uses a motion on a still background in a moving image. A method of separating and compressing only a certain part is adopted (for example, MPEG2 method). With such a compression method, a sufficient compression effect is obtained for a moving image in which the camera itself moves and the entire image has a motion component. As a result, data transmission could not be performed.
Therefore, the present invention was created in view of the existing circumstances as described above, and an image obtained by a camera perspective method is used without using a plurality of actual measurement points of an actual object for each image. In addition, it is intended to develop a plane based only on the photographing conditions of the camera and information in the photographed image, and further intends to develop and display it on a plan view that is expressed by a map.
For example, when viewing the external situation from a bus, etc., using the image seen from the front, the image seen from the side, and the image seen from the rear, developed it, joined it, and viewed from above the bus The surrounding road surface is displayed like a map, and the position of the host vehicle is displayed in a plan view to make driving safer.
That is, by converting oblique images obtained by the perspective of various monitor cameras, it is intended to develop and display a plan view as represented by a map. For example, when viewing external conditions from a bus or the like, Using the image seen from the front, the image seen from the side, and the image seen from the back, expand it, connect it, and try to display the road surface around it as seen from above the bus like a map A plane development image processing method for a plane object image such as a road surface, which enables a combination of images and further measurement within the image by forming a linear scale by such plane image development. It is an object of the present invention to provide a reverse development image conversion processing method, a flat development image processing apparatus thereof, and a reverse development image processing apparatus.
In addition, as an application of the above principle, after expanding the road surface to a plane, the viewpoint is moved and then restored to the perspective image, thereby obtaining a video with a different viewpoint from the first perspective image. When extracting an arbitrary plane other than a road, an optical flow (or parallax, matching, etc.) is used to extract the target image by using the optical flow (or parallax, matching, etc.), or to extract a plane from multiple perspective images. A method of detecting unevenness, making it possible to obtain the deviation of the unevenness from the plane, using optical flow from a plurality of perspective images, extracting the moving distance speed direction, and CG to the image developed on the plane Place an object consisting of an image, paste the texture corresponding to it, display it as a flat or perspective image, or actually measure the perspective image when flat By making the line components that should be parallel lines in the screen parallel so that they can be used as little as possible, the camera's dip angle θ should be obtained, or it should be parallel lines in the screen from perspective images By finding the point where the line components intersect, that is, the vanishing point, it is possible to obtain the camera's dip angle θ, or to correct the camera shake by moving the image so that the dip angle θ is constant. It is an object of the present invention to provide a plane development image processing method, a reverse development image conversion processing method, a plane development image processing apparatus, and a reverse development image processing apparatus for a planar object image such as a road surface.
Since a desired three-dimensional image can be reconstructed from a plurality of images obtained by plane development of perspective images and camera position information, the original moving image can be obtained on the receiving side by transmitting only the plane development image and camera position information. The plane of a plane object image such as a road surface that can transmit data of a desired object even on a narrow-band line or the like can be reproduced. It is an object of the present invention to provide a developed image processing method, a reverse developed image conversion processing method, a flat developed image processing apparatus, and a reverse developed image processing apparatus.

In order to achieve the above-described object, in the present invention, information necessary for plane development is read from a perspective image obtained from a normal camera, and is developed into a plan view like a map by mathematical formulas. Combination, stitching, and making a single large road development. For example, a normal video camera is mounted around the bus, and the entire road is composed of multiple video cameras. The road surface outside the bus is photographed at a certain angle, that is, a depression, and this is calculated. The image of the road surface is developed like a map image, and the image that can be developed like the map and the image of the wall surface of the building are processed in a similar manner, joined together, and generated as the image of the road surface around the bus and the building It is to try. Alternatively, an image of a floor surface and a wall surface inside a building is photographed, and an image photographed obliquely is developed on a flat surface, for example, a room wall is opened, and a drawing in which the room is just unfolded is obtained. This is obtained by calculation using mathematical formulas.
Then, a desired three-dimensional moving image can be reconstructed from a plurality of image data expanded in plane and position information data of the camera. Thus, for example, even when the shooting camera and the monitor display unit are separated from each other, the original moving image can be reproduced on the monitor side by transmitting the plane development image and the camera position information from the camera side. Is possible. A flat image is a still image, and its data volume is much smaller than a moving image of an oblique image. Therefore, data can be transmitted freely even between devices connected by a narrow bandwidth line, etc. By reconstructing with, the moving image of a desired object can be data-communicated.
Specifically, the oblique image plane development device as the real-time processing includes, for example, a CCTV camera or a digital still camera as a video input device, a video playback unit, an image correction unit, a spherical aberration correction unit, a video development plane processing unit, a developed image It consists of a coupling part, a display part, and a recording part.
In addition, the oblique image plane development apparatus as offline processing includes a video reproduction apparatus that has recorded oblique images, an image correction unit, a video development plane processing unit, a development image combining unit, and a display unit.
As a plane development, with respect to an image obtained by obliquely shooting an object of a real scene including a plane, the plane originally composed of a plane is made proportional to the plane of the real scene (similar) by mathematical calculation. As a plane image, it is displayed in a flat plane.
As a combination of the plane development, a plurality of plane development images obtained by the above method are combined and expressed as one large plane development image.
In addition, for omnidirectional display and direct display using a plurality of CCTV images, a plurality of CCTV images are developed in a plane using the above-described apparatus, and the images are combined into one image to display the entire target area. In response to this, an oblique video of CCTV corresponding to the displayed location is also displayed at the same time.
As continuous connection in the moving direction, images of the moving direction obtained by loading a camera on a moving vehicle, aircraft, ship, etc. are imaged in a plane and continuously combined into a single image. is there.
In addition, when a moving object is captured when combining multiple images that partially include overlapping objects, a combined image of only a stationary object is generated by combining the images while avoiding the moving object image. To do.
In other words, when developing an oblique image obtained by CCTV or the like as a θ method as a planar image, the optical axis position θ is read from the oblique image, and the camera height h, the f value of the photographing lens, or the virtual f value on the monitor is read. And the coordinates of the target location
y = v · 2 ^1/2 · H · cos (π / 4-θ) · cos (β-θ) / (f · sinβ) (1)
x = u · h · cos (β−θ) / (f · sin β) (2)
By using an expression expressed as follows and an expression having the same meaning, information that can be read from the captured image and θ without giving the position coordinates and size of the object in the real world as known information By providing information on imaging conditions such as h, r, etc., coordinate conversion is performed by associating the real world coordinate system with the coordinate system on the image monitor.
Where θ is the angle formed by the optical axis of the camera and the road surface, f is the focal length of the camera, h is the height of the camera, β is a distance h + y from directly below the camera, and the line segment connecting the camera and the road An angle formed by the plane, v is a coordinate in the vertical direction from the origin on the CCD (acquired video) plane which is a projection plane in the camera, and u is a coordinate in the horizontal direction from the origin on the CCD (acquired video) plane. In addition, y is a distance or coordinate that travels further in the direction of the optical axis from the point that is advanced from right below the camera on the road surface, and x is a lateral distance or coordinate on the road surface. In addition, when the vertical wall surface is developed on a plane, the coordinates may be inclined by 90 degrees. It should be noted that the mathematical formula is not limited to this formula, and may be a mathematical formula that associates another similar perspective with a plane.
Here, a numerical value necessary for calculation, that is, an angle θ formed by the target plane and the optical axis, a value β such as an angle β formed by a camera or the like corresponding to an arbitrary point in the image, and (β−θ), etc. Is a physical quantity in the real world (live-action video), so it can be obtained by actual measurement. However, as an actual problem, it is practically impossible to measure each moving image that the camera moves at multiple locations while actually measuring. From the nature of this conversion formula, It can be obtained from within the image.
In many cases, the position of the optical axis in the image is the optical axis position, but this can be achieved accurately by finding the optical center of the camera or other photographic equipment with a collimator. Once measured, the optical axis position can be obtained as a point in the image as a value unique to the camera system including the lens.
Next, an example of the most important θ measurement in the image in the conversion formula is described. The parallel line part in the real world is empirically searched from the image, and the extension of the parallel line is the intersection (disappearance) in the image. A point, that is, a point that intersects at a distant point when drawing a drawing in perspective, and is a Vanishing Point in a perspective view etc. For example, when a straight road is written in perspective, the road becomes one point in the distance And the point is a vanishing point.), The plane a is a plane parallel to the target plane including the optical axis, and the plane b is a plane parallel to the target plane including the intersection. And d can be determined as a ratio (arcTand d / f) between d and the virtual focal length f.
Here, as an example of obtaining the virtual focal length, the distance on the optical axis is such that the angle at which an arbitrary object in the real space is viewed in advance and the angle at which the same object in the displayed image is viewed are the same. It may be obtained on the display image, and if the unit at this time is expressed in pixels, it is a value specific to the camera system including the lens, and it is only necessary to obtain it once. Furthermore, since a parallel line portion of an object in the real world is searched and it is represented as an intersection line having an intersection on the extension line in the image, the intersection line becomes parallel when it is expanded in a plane. Can be obtained or fine adjustment of θ can be performed.
Note that θ can also be obtained by actual measurement. For example, by using a protractor, which is a simple method, it is possible to accurately determine how much the camera attached to a moving vehicle such as a car is facing downward. If it is intended, it can be obtained by measuring with a dedicated angle measuring device.
In addition, in the process of obtaining the vanishing point in the acquired planar image, it is possible to stabilize the image that shakes due to camera shake or the like by moving and displaying the image so as to fix the position of the vanishing point of each image. . That is, this extends the part of the parallel line component in the target plane in the image, finds the intersection point, and when it becomes the vanishing point, the position of the vanishing point fluctuates as the camera moves or shakes. Although the image is blurred, the entire image is moved and displayed so that the position of the vanishing point is fixed, thereby stabilizing an image that shakes due to camera shake or the like.
A unit time movement amount of a minute area in a moving image composed of a plurality of different planes obtained by plane development is referred to as an optical flow (in this specification and drawings, “Opt.F” may be omitted). A certain planar image can be separated by obtaining the same component from the component distribution map and obtaining the same component by a required method. When using this optical flow method, the optical flow of perspective images is generally not the same value in the moving direction, even in the same plane, and takes different values depending on the distance. The optical flow uses the property of taking the same value. In other words, even if an image is obtained by overlapping a plurality of different planes obtained by plane development, each plane image can be separated by extracting the same component from the optical flow component distribution overall view. In the developed image, the wall surface of a building such as a building that forms a parallel plane, the guardrail surface of the road, and the streetlight surface can be separated by an optical flow value.
The optical flow is a flow showing how each corresponding point moves in a plurality of images. If there is a movement in a plurality of images, there is a flow of the optical flow. If there is no movement, the flow of the optical flow is not represented by a line, and it is important to know whether or not the corresponding point has moved in a plurality of images.
Further, an optical flow distribution map is generated in the plane moving image obtained from the plane converted moving image, and the unevenness of the plane is detected as a deviation from the plane from the minute difference, or a different image in the plane moving image is detected. The parallax is detected by comparing and calculating the plane image obtained from the corner, and the uneven component in the plane is detected from the component distribution, and the deviation from the plane of each point of the original plan view is detected by the detected uneven value. A modified plan view is generated, and a plurality of plane images developed in a plurality of plan views are compared and calculated by a method such as a correlation method or a matching method, so that each of the plurality of plane images such as a road surface is displayed. For each small area, the amount of movement of the corresponding small area is obtained by the parallax method or the optical flow method, and three-dimensional data such as unevenness on the road surface is detected from the distribution of the components. And, or with the detected three-dimensional irregularities value it can generate a modified plan view, including deviation from the plane of each point in the original plan. In other words, a distribution map of optical flow is generated, the unevenness of the plane is detected as a deviation from the plane from the minute difference using the above principle, and the parallax is calculated by comparing the plane images obtained from different angles of view. And detecting a concavo-convex component in the plane from the component distribution, or generating a corrected plan view including a deviation from the plane of each point of the original plan view with the detected concavo-convex value. In addition, if there are undulations or irregularities on the plane, and if a small difference in the optical flow of the converted plane image can be detected, it means a deviation from the plane, so an uneven distribution map can be generated. Can do it. In addition, since there are multiple images developed in the plan view of the road surface observed from different directions by moving images or images acquired by multiple cameras, they are compared and calculated using a method such as correlation or matching, and roads By calculating the parallax for each minute area of a plurality of planar images such as a surface, etc., by calculating the amount of movement from the difference in the components of that minute area, and by calculating the corresponding points in combination, the unevenness of the road surface Is detected.
This means that by acquiring a flat developed image from moving images with overlapping fields of view and obtaining parallax or optical flow from them, not only the unevenness of the road surface, but also all objects that overlap in the flat developed image This means that three-dimensional data can be detected.
It should be noted that all the operations by optical flow can be performed by parallax, and can be performed by matching. Therefore, the term “optical flow” in the present invention means that any processing such as optical flow, parallax, or matching may be performed.
In addition, the average optical flow value of the continuous image developed on the plane or the movement distance of the matching corresponding position is obtained, and the movement distance / movement speed / movement direction of the target plane or the movement distance / movement of the photographed camera is obtained from the value. Speed and moving direction can be obtained. That is, since the optical flow of the same plane developed in a plane takes a constant value, the moving speed of the camera can be obtained from the optical flow of the target plane, which is the relative position between the camera and the target plane, Since it is a relative speed, it is the same even if the stationary system and the moving system are reversed. Further, here, the parallax has the same meaning as the optical flow geometrically, and the optical flow or the parallax of a wide area of the continuous image obtained by plane development is obtained and used to move the movement distance of the target plane, The moving speed, moving direction or moving distance, moving speed, and moving direction of the photographed camera can be obtained.
By pasting the texture of the object plane in the single plane that has been separated and developed into planes onto the object plane in the CG (computer graphics) image or map image corresponding to the place, a live-action image is displayed on the CG image or map image. And can be displayed as a perspective image by plane conversion or inverse transformation. That is, because the same plane developed in the plane has the same optical flow, it is possible to cut out only the texture of the target plane from a plurality of mixed planes. By pasting the texture of the object plane in the separated single plane that has been separated into planes onto the corresponding object plane in the corresponding CG image or map image, the real image is taken into the CG image or map image, Alternatively, it is inversely transformed and displayed as a perspective image.
Further, in the above formulas (1) and (2), when f and h are first given, and when the parallel line of the object is an intersection line forming an intersection point in the image, the intersection line is expanded in plane. By selecting θ so as to be parallel to the angle θ, θ can be obtained, and θ to be selected can be finely adjusted. This is because a specific object such as a road uses the property that the object itself often has a part that becomes a parallel line. In the perspective image, the extended line segment forms an intersection, and θ is obtained by selecting θ so that the line segment intersecting at the intersection point becomes a parallel line in the plane image obtained by plane conversion. It can be done.
A parallel line portion in a live-action image is extracted from the image, and a distance between a plane a that is a plane parallel to the target plane formed by the intersection and a plane b that is a plane parallel to the target plane including the optical axis point is calculated. d can be obtained, and the virtual focal length can be represented by f. From the ratio of d and f, θ can be obtained as θ = arcTan (d / f). That is, the angle θ between the target plane and the optical axis is a physical quantity in the real world (actual video), and is an amount to be actually measured, but it is calculated from the image and each frame image in the moving image. In order to obtain this, the parallel line part in the real world (live-action image) is empirically searched from the image, and the plane parallel to the target plane formed by the intersection is defined as plane a, and parallel to the target plane including the optical axis point. The plane is the plane b, the distance between the plane a and the plane b is d, the virtual focal length is f, and θ is determined as θ = arcTan (d / f) from the ratio of d and f. This is to find the angle θ between the target plane, such as the road surface, and the optical axis of the camera, and find the part of the parallel line in the real world empirically from the image, and the road created by the intersection and the parallel line part A plane parallel to the target plane such as a road that includes the optical axis point, with the target plane such as a plane and the position of the optical axis in the image measured in advance, or the geometric center of the image as the approximate optical axis Is obtained by obtaining the ratio of the distance to the virtual focal length and obtaining the arc tangent thereof.
In addition, multiple simultaneous images of the same point are acquired by a plurality of ordinary cameras installed at different installation locations, and parallax is detected by comparing and calculating the plane development images of the multiple images of the same point simultaneously. A three-dimensional shape of an object can be generated. In the above example, the image from one camera is developed in a plane, or a plane development image is obtained by combining a plurality of cameras with different viewpoints. By using a camera or the like, parallax is detected in the flat developed image of the overlapping portion after acquiring the video of the same spot taken from different points as a flat developed image. In other words, until now, detection of 3D data by detection of parallax has always been obtained from the original image, that is, the perspective image itself, but here a new method of detecting parallax after performing planar development image processing, It becomes possible to obtain three-dimensional data with high positional accuracy, and thereby, it is possible to easily acquire direct three-dimensional shape data with higher accuracy than before. Here, the parallax obtained from multiple images with overlapping fields of view often has the same meaning as the optical flow in the stationary coordinate form in the video, but the accuracy of the three-dimensional coordinates is improved when the object changes over time. In the case of raising the image, it is effective to obtain a plurality of simultaneous images of the same point by such a plurality of cameras.
On the other hand, a map or a plan view is prepared from the beginning, and based on any plan view, plan photograph, CG (computer graphics) image, etc., the inverse transform is performed opposite to the previous transform. It is possible to generate a perspective video from an arbitrary viewpoint different from the video of the previous viewpoint. In addition, by continuously inversely transforming each frame image of the video image, it is possible to generate a video moving image based on a virtual moving camera viewpoint that is not actually photographed by repeating viewpoint movement.
Specifically, a plane development view that is generated by converting a perspective image including a plane video into a plan view, or a video that includes a plurality of plane images taken from a plurality of directions is developed into a plan view. Then, based on a single large-screen plane development view or a plan view-like CG (computer graphics) image or map generated by overlapping corresponding points, the above formulas (1) and (2 ) Can be used to generate a virtual perspective image viewed from an arbitrary viewpoint, or by continuously processing to generate a moving image with a moving camera viewpoint, and after changing the viewpoint The specific inverse transformation formula based on the inverse transformation method is based on the following formulas (3) and (4).
v = y · f · sin β / (2 ^1/2 · H · cos (π / 4-θ) · cos (β-θ))
………………… (3)
u = x · f · sin β / (h · cos (β−θ)) (4)
Where h is the height of the camera from the road surface, θ is the angle formed by the optical axis of the camera and the road surface, f is the focal length of the camera, and β is advanced y from the point h ahead of the camera. The angle formed by the line connecting the point and the camera lens and the road surface, x is the vertical coordinate from the line obtained by orthogonally projecting the optical axis of the camera onto the road surface, that is, the horizontal coordinate when viewed from the camera, y Is the coordinate in the direction of the optical axis when the point advanced from right below the camera is the origin, v is the vertical coordinate on the CCD plane which is the projection plane of the camera, u is on the CCD plane which is the projection plane of the camera The horizontal coordinate. It should be noted that the mathematical formula is not limited to this formula, and may be a mathematical formula that associates another similar perspective with a plane.
In addition, it is possible to perform various recognition processing such as measurement processing on the image, image recognition processing, etc. by the image developed on the plane, the scale of the image developed on the plane becomes a linear scale, and measurement on the image, Image processing, image recognition, etc. are performed very easily. Since the optical flow is also obtained in a form proportional to the relative speed with the camera, the relative speed of the object is also expressed in a linear scale without depending on the distance. Therefore, not only in measurement but also in image processing recognition. It is greatly simplified.
Examples of applied planes include road surfaces, sea surfaces, lake surfaces, river surfaces, ground surfaces, vertical walls, vertical virtual planes created by objects arranged in the same plane, building wall floors, ships Airport facilities such as the deck surface and runway taxiway.
An example of a vehicle as an applied device is a peripheral road surface, a building surface, an electric pole array surface, a roadside tree array surface, a guardrail array surface, etc. in a land vehicle such as a bus, and the sea surface of a marine vehicle such as a ship. Omnidirectional display such as deck, wall surface, runway of aircraft, ground surface, etc., or target area surface display.
Furthermore, in a building structure as another application example, a plane portion such as a floor surface or a wall surface of a building is displayed in a plane development display and in a plane combination display.
In addition, 3D map creation as an application example is not only continuous shooting of the road surface, ground surface and water surface with a plurality of cameras, moving vehicles, aircraft, ships, etc., but also vertical surfaces such as building wall surfaces, etc. By continuously photographing a target having a virtual vertical plane in which a plurality of utility poles, guard rails, etc. are regularly arranged in a plane, the plane expanded image is coupled and extended in the moving direction, and simultaneously includes a vertical plane. A three-dimensional map is created by creating a broader plane and vertical development.
On the other hand, the flat developed image processing apparatus and the reverse developed image processing apparatus directly used in the above-described flat developed image processing method and reverse developed image conversion processing method of a planar object image such as a road surface acquire perspective images. A video input unit, a video playback unit that reproduces an oblique video captured by the video input unit, an image correction unit that corrects a shooting rotation angle by the video input device, and a spherical aberration correction in the video input device. A spherical aberration correction unit, a video development plane processing unit that converts a perspective image into a plane development view, a developed image combining unit that combines the images that have undergone the video development process, and a display unit that displays the combined image Is.
In addition, an optical flow map generation unit that generates and illustrates an optical flow of the developed video and an optical flow extraction unit that extracts only the target optical flow from the optical flow map are provided, and the same point from different positions is provided. A parallax extraction unit that detects parallax from the image of the image, a development image comparison unit that compares a plurality of development images at the same point, an image comparison unit that extracts road surface unevenness by calculation, and the unevenness And a modified plane generation unit.
Furthermore, a video input unit that generates video by a camera such as a CCTV camera or a digital still camera, an input image display unit that stabilizes and displays an input image, a video recording unit that records the input video, and a playback of the recorded image A video reproduction unit, and an image correction unit that performs coordinate transformation for correcting distortion of the image due to a lens such as spherical aberration and aligns a target plane image with a plane in the image in order to correct a camera rotation angle; A video development plane processing unit that generates a plan view from a perspective image by mathematical operation, an optical flow map generation unit that generates and illustrates an optical flow of the developed video, and a target optical from the optical flow map An optical flow extractor that extracts only the flow, and a parallax that detects parallax from video at the same point from different positions Combining the output part, the target image processing part that leaves unnecessary objects, deletes unnecessary images, and inserts new video, and the processed images that are developed in a plane A developed image combining unit for generating a developed image, a developed image display unit for displaying them, a recording unit for recording them, an arbitrary viewpoint image generating unit for inversely converting to an arbitrary viewpoint, and displaying the image Arbitrary viewpoint image display unit, unfolded image comparison unit that compares unfolded images at the same point, an image comparison unit that extracts road surface unevenness by calculation, and a modified plane generation unit that considers the unevenness are appropriately combined. It is a thing.
Similarly, a reverse development image processing apparatus used directly in a plane development image processing method and a reverse development image conversion processing method of a planar object image such as a road surface is provided with an arbitrary viewpoint that is converted back to an arbitrary viewpoint and displayed. An image generation unit and an arbitrary viewpoint image display unit that displays the image can be provided.
Furthermore, the plane development image processing apparatus and the reverse development image processing apparatus include a video input unit that acquires a perspective image, and one or more planes that form a three-dimensional space from the perspective image captured by the video input unit. A plane decomposition unit that decomposes into an image, a position detection unit that detects a three-dimensional position of the video input unit, a plane image decomposed by the plane decomposition unit, and a three-dimensional position of the video input unit that is detected by the position detection unit And a display unit for reconstructing and displaying a three-dimensional image. A configuration may be provided that includes a position notation unit that indicates the three-dimensional position of the video input unit detected by the position detection unit in the plane image decomposed by the plane decomposition unit. Further, when the video input unit moves, the position notation unit can be configured to continuously describe the three-dimensional position of the moving video input unit in the plane image decomposed by the plane decomposition unit. .
When the display unit for reconstructing the three-dimensional image is disposed apart from the plane decomposition unit and the position detection unit, one or more plane images are provided from the plane decomposition unit and the position detection unit to the display unit. A transmission / reception means for transmitting a signal and a three-dimensional position signal of the video input unit may be provided.
In the plane development image processing method, the reverse development image conversion processing method and the plane development image processing apparatus, and the reverse development image processing apparatus of the planar object image such as road surface according to the present invention configured as described above, video input A perspective image, which is an oblique image acquired by the apparatus, is converted into a plan development view by equations (1) and (2) and displayed as a practical map-like image.
Combining acquired and generated flat development views is displayed as a large expanded image, for example, including the situation around the acquisition location / place, like a map, full display of the entire target area, direct display of a specific area, etc. And can easily compare with the surrounding situation by direct display of the input video simultaneously.
In addition, when there are multiple images developed on a plan view of the road surface observed from different directions by moving images or images acquired by multiple cameras, they are compared and calculated using a method such as correlation or matching, and roads For each small area of a plurality of planar images such as a surface, the amount of movement is calculated from the difference of the components of the small area by a parallax method or an optical flow method, and the corresponding points are combined and calculated. Surface irregularities are detected, and when there are undulations or irregularities on the plane, a corrected plan view including a deviation from the plane of the detected irregularity value is generated.
This is to obtain a flat developed image from moving images with overlapping fields of view, and obtain parallax or optical flow from them, not only for unevenness on the road surface, but for all objects that overlap in the flat developed image, Detect 3D data.
A plane development image generated by a plurality of videos obtained by shooting a video of the same point from different points by a plurality of cameras with overlapping viewpoints causes parallax to be detected in the plane development image of the overlapping portion.
The generated plane development is developed on the land, sea, airport facilities, etc., and various displays such as plane displays on building structures, etc. A three-dimensional map is also created by extending the joints of vertical development.
Also, a video input unit, an input image display unit, a video recording unit, a video playback unit, an image correction unit, a video development plane processing unit, an optical flow map generation unit, an optical flow extraction unit, a parallax extraction unit, a target image processing unit, The development image combining unit, the development image display unit, the recording unit, the arbitrary viewpoint image generation unit, the arbitrary viewpoint image display unit, the development image comparison unit, the image comparison unit, the correction plane generation unit, and the like are appropriately combined. Depending on the purpose, for example, the perspective image road surface is developed into a flat image, the building wall image displayed in perspective is developed into a flat image, the building wall surface and the guardrail image are separated, or the flat image is displayed. After conversion, change the viewpoint and convert it back to the perspective image again, display the road or building surface with the texture attached, or convert it to the perspective image by reverse conversion, Or to obtain a parallax image by combining images from different viewpoints, then allows for a wide range of applications, such as or calculate the unevenness of the road surface.
The desired three-dimensional image, 3D map, etc. can be reconstructed from the plurality of image data and the position information data of the camera developed, so that the camera for photographing and the display unit for monitoring are placed apart from each other. Even in such a case, it is possible to reproduce the original moving image on the reception side (monitor side) by transmitting the plane development image and the camera position information from the video acquisition side (camera side). The image data developed on a plane is a still image, and the amount of data is much smaller than a moving image of an oblique image. Accordingly, by transmitting and receiving the flat developed image and the camera position information data for reconstructing it, it is possible to perform data communication of a moving image with a data transmission amount as small as possible.

FIG. 1 is a block diagram of a plane development apparatus for a plane object image such as a road surface showing an embodiment of the present invention.
FIG. 2 is a block diagram showing another embodiment of the apparatus of the present invention.
FIG. 3 is a block diagram showing another embodiment of the apparatus of the present invention.
FIG. 4 is a block diagram showing a plane development apparatus for video.
FIG. 5 is a block diagram showing an unevenness detecting device on the road.
FIG. 6 is a block diagram showing a viewpoint movement / texture pasting apparatus that similarly performs viewpoint movement and texture pasting by the road surface development method.
FIG. 7 is a block diagram in an embodiment in which an image is planarly developed by the optical flow method.
FIG. 8 is a schematic explanatory diagram in the case where a flat developed image is generated from the acquired image and an oblique image is formed by moving the viewpoint.
FIG. 9 is a schematic explanatory view showing a case where unevenness on the road surface is similarly detected.
FIG. 10 is a schematic diagram for obtaining θ from the vanishing point.
FIG. 11 is an example of a flat development when the θ values are different when the flat development is performed. FIG. 11A shows a plane development before the perspective development as it is, and FIG. (C) shows the case where the value of θ is the same as the actual value, that is, θ is obtained correctly.
FIG. 12 is a block diagram of the same three-dimensional map generation apparatus.
FIG. 13 is an image view of a plane similarly extracted by the optical flow.
FIG. 14 shows a plane extracted by the optical flow and an image of a three-dimensional map generated by three-dimensional transformation.
FIG. 15 is an image diagram of a state where the locus of the camera position is described on the three-dimensional map.
FIG. 16 is a block diagram of the moving object detection solid generation apparatus.
FIG. 17 is a block diagram of a texture pasting apparatus in the same three-dimensional map generating apparatus.
FIG. 18 is a block diagram of the object recognition apparatus.
FIG. 19 is a specific example of an image obtained by a traffic monitoring video camera as an example of the object recognition apparatus of FIG. 18. (a) is a monitoring video camera image (far and near image), and (b) is a book. The flat developed image converted by the invention, (c) is a region analysis display showing the object recognized by the present invention.
FIG. 20 is a flowchart showing processing steps for object recognition in a traffic monitoring video camera as an example of the object recognition apparatus of FIG.
FIG. 21 is a list summarizing the results obtained by the object recognition process shown in FIG.
FIG. 22 is a situation diagram in which a plurality of cameras for photographing a road around a bus are arranged, for example.
FIG. 23 is an image diagram developed on the road surface like a map.
FIG. 24 shows, for example, nine road image diagrams (1) to (9) obtained by taking an oblique image of the road.
FIG. 25 is a plan image diagram like a map obtained by processing based on the same road image diagram.
FIG. 26 shows another embodiment. For example, (1)... (16), 16 indoor image views, each of which is taken as an oblique image of a plane portion such as a floor surface or a wall surface of a building structure. is there.
FIG. 27 is a composite image diagram obtained by planarly combining the image diagrams shown in FIG.

Embodiments of the present invention will be described below with reference to the drawings.
As shown in FIG. 1, a camera 1 such as a CCTV camera or a digital still camera as an input device, a video playback unit 2 that plays back an oblique video shot by these cameras 1, an image correction unit 3, and spherical aberration A correction unit 4, a video development plane processing unit 5 that develops an oblique video into a plan view according to formulas (1) and (2), which will be described later, a developed image combining unit 6 that joins the developed videos in an appropriate manner, The developed image display unit 7 displays the joined video and the recording unit 8 records the joined video on a recording medium. The oblique image is developed in a plane as real time processing. That is, the image correction unit 3 corrects the rotation angle of the camera 1 while the image is input from the camera 1 and reproduces the image in real time so as to be configured as an oblique image plane development device as real-time processing. The correction unit 4 corrects the spherical aberration and the purpose of the camera 1, and the image development plane processing unit 5 converts the perspective image into a plane development view like a map and displays it on the development image display unit 7. To do. If necessary, the developed images are combined with the developed images by the developed image combining unit 6, and the images are displayed and recorded in the recording unit 8. is there.
Further, as shown in FIG. 2, in the oblique image plane developing apparatus for offline processing, an oblique image reproducing unit 11, an image correcting unit 12, an image developing plane processing unit 13, a developed image combining unit 14, The developed image display unit 15 is used to reproduce the image from the oblique image reproduction unit 11 in which the image captured by the normal camera 1 is recorded, and the image correction unit 12 is intended for spherical aberration, camera rotation angle, and the like. The image development plane processing unit 13 develops a flat image such as a map and displays it on the development image display unit 15. If necessary, a plurality of images developed by the development image combination unit 14 are then displayed. The images are joined together by an appropriate method, and the joined images are displayed on the developed image display unit 15.
Further, as shown in FIG. 3, the oblique image plane development apparatus includes a camera side for image acquisition (transmission side) and a monitor side for displaying, recording, and reconstructing a planar image decomposed and developed from the oblique image ( It can be installed separately from the receiving side. In this case, as shown in FIG. 3, on the camera side (transmission side), there are a camera 1 as an input device, a video reproduction unit 2, an image correction unit 3, a spherical aberration correction unit 4, and a video development plane processing unit 5. On the monitor side (reception side), a developed image combining unit 6, a developed image display unit 7, and a recording unit 8 are provided. Further, the camera side is provided with a transmission unit 5a that transmits the developed and decomposed planar image signal to the monitor side, and the monitor side is provided with a reception unit 6a that receives the planar image signal transmitted from the camera side. These transmission / reception units 5a and 6a are connected via a communication line so that data communication is possible. As a result, the plane development image of the moving image obtained on the camera side and the predetermined information such as the camera position information are transmitted to the monitor side via the communication line, and the monitor side transmits the moving image based on the received plane development image. An image can be reconstructed, and a desired moving image can be transmitted and reproduced while reducing the transmission data amount as much as possible.
Note that FIG. 3 shows the real-time processing oblique image plane development apparatus shown in FIG. 1 separated and installed. However, the real-time processing and the offline processing can be separated and installed. Of course.
As a plane development form, an image obtained by obliquely photographing an object of a real scene including a plane is proportionally related to the plane of the real scene (similar shape) by a mathematical operation. A flat image is developed and displayed on a flat surface. This is an image obtained by photographing a scene including a plane with a normal camera, that is, an image photographed from an oblique direction, and transforms a plane originally composed of a plane into a plane image having a similar relationship with the real plane by mathematical operation. For example, if it is a road surface, it is developed on a plane and developed like a map screen.
Furthermore, when combining these, a plurality of plane development images obtained by the above method are combined and expressed as one large plane development image. This is an image taken with a normal camera on a plane. This means that the plurality of flat developed images are combined by an appropriate method and expressed as one large flat developed image. That is, when a plurality of road surfaces are developed into a plane development view such as a map, they are connected to form one large plane development image. Of course, since it is an image developed in a plane, any number of images can be combined freely, and images in the perspective view cannot be combined unless the camera positions are the same.
Furthermore, it is possible to perform omnidirectional display and direct display based on images acquired by a plurality of CCTV cameras. The images of a plurality of CCTV cameras are developed in a plane by the above-described apparatus, and the images are combined to form a single sheet. The entire area of the target area is displayed as an image, and if necessary, an oblique video of the CCTV camera corresponding to the displayed position can be displayed at the same time.
That is, the images of a plurality of CCTV cameras are developed into a planar image by the above-described apparatus, and the respective images are combined by matching corresponding points to form one image, and the entire target area is displayed. Furthermore, if necessary, an image as it is, that is, an oblique image of the CCTV camera corresponding to the displayed location can be displayed, so that the purpose effect such as monitoring can be improved.
FIG. 4 shows a processing block diagram when an image is developed on a plane. First, when a video image 21 or a still image image 22 is input to the apparatus as an input image, spherical aberration correction, rotation correction, and the like are performed by the correction unit 23 at the time of input. Next, the video plane development processing unit 24 performs video plane development processing using the following formulas (1) and (2).
That is, when the oblique video obtained by the video image 21 or the still image video 22 is converted into a planar image, in the present invention, it is obtained by a method called the θ method. For example, an oblique video obtained by a CCTV camera or the like is obtained. In developing the flat image, the optical axis position θ is read from the oblique image, the camera height h, the f value of the photographing lens, or the virtual f value on the monitor is read, and the coordinates of the target location are expressed by the following equation (1). , (2).
y = v · 2 ^1/2 · H · cos (π / 4-θ) · cos (β-θ) / (f · sin β) ... (1)
x = u · h · cos (β−θ) / (f · sin β) (2)
By using the mathematical expressions expressed by the expressions (1) and (2) and the mathematical expressions having the same meaning, it is possible to shoot without giving the position coordinates and the size of the object in the real world as known information. By providing information that can be read from the video and information on photographing conditions such as θ, h, and r, coordinate conversion is performed by associating the real world coordinate system with the coordinate system on the image monitor. It is.
However, the plane development coordinates are x, y, the in-image coordinates are u, v, θ is the angle between the optical axis of the camera and the road surface, f is the focal length of the camera, h is the height of the camera, and β is the camera height. The angle between the point h + y from the bottom and the line connecting the camera and the road surface, v is the vertical coordinate from the origin on the CCD plane that is the projection surface of the camera, and u is the horizontal axis from the origin on the CCD plane It is a coordinate of direction. In addition, y is a distance or coordinate that travels further in the optical axis direction from the point that has advanced h from directly below the camera on the road surface, and x is a lateral distance or coordinate on the road surface.
Therefore, the perspective image obtained by photographing the road surface is converted into a planar image such as a map and developed using the equations (1) and (2).
The plane image thus converted and expanded is displayed and recorded by the plane expanded image display / recording unit 25. Next, a perspective image is developed on a plane, and a number of the plane development images are connected to a developed image combining unit 26. The developed image combining unit 26 obtains a large-scale flat image. The combined developed image display / recording unit 27 displays and records this.
Next, an arbitrary viewpoint is designated from the developed video to generate an arbitrary viewpoint, and this is performed by the arbitrary viewpoint generation unit 28. Then, the inverse expansion conversion processing unit 29 uses the following expressions (3) and (4), which are the inverse conversion expressions of the above expressions (1) and (2), to change the perspective from the original viewpoint. The reversely developed image display / recording unit 30 displays and records a legal image.
That is, the equation for inverse transformation is as follows.
v = y · f · sin β / (2 ^1/2 · H · cos (π / 4-θ) · cos (β-θ))
………………… (3)
u = x · f · sin β / (h · cos (β−θ)) (4)
Where h is the height of the camera from the road surface, θ is the angle formed by the optical axis of the camera and the road surface, f is the focal length of the camera, and β is advanced y from the point h ahead of the camera. The angle formed by the line connecting the point and the camera lens and the road surface, x is the vertical coordinate from the line obtained by orthogonally projecting the optical axis of the camera onto the road surface, that is, the horizontal coordinate when viewed from the camera, y Is the coordinate in the optical axis direction when the origin is a point advanced from directly under the camera, v is the vertical coordinate on the CCD surface, which is the projection surface of the camera, and u is the horizontal coordinate on the CCD surface. .
Note that the inverse transformation formula is not limited to these formulas (3) and (4), but may be a mathematical formula that relates other similar perspectives and planes.
In FIG. 5, a new block diagram is added to a part of the block diagram of FIG.
That is, the video image 21 and the still image image 22 are input, and the perspective image is developed into a plane image by the equations (1) and (2), and this is performed by the image plane development processing unit 24. The developed image comparison unit 31 compares the developed images. The developed image comparison unit 31 obtains the unevenness of the road surface by comparing the images from different viewpoints. It is supposed to be processed by.
In FIG. 6, not only the road surface is developed, but also buildings on the roadside, roadside trees, guardrails, etc. are each developed on a plane, converted into an arbitrary viewpoint image, or a texture is attached to the building. A configuration for performing viewpoint movement by a road surface expansion method and texture pasting so as to create 3DCG using a live-action texture is shown.
That is, first, the road surface plane development unit 42 obtains a road surface plan development from the image input from the video image input unit 41, while the side vertical plane development unit 43 planarizes a roadside portion such as a building wall. Expand in the figure. Thereafter, an optical flow map is generated by the optical flow map generation unit 44, and a target portion is selectively extracted by the next optical flow map selection extraction unit 45. Accordingly, the curb surface extraction unit 46 extracts road curbs, and the building surface extraction unit 47 extracts building wall surfaces and the like. Further, the sidewalk surface development unit 48 performs sidewalk surface development, matches the data from the road surface plane development unit 42 described above, and the road horizontal plane integration unit 49 combines the sidewalk portion and the roadway portion.
On the other hand, the road tree is extracted from the optical flow map selection / extraction unit 45 by the plane tree extraction unit 50, and the road surface is extracted by combining the data from the building surface extraction and sidewalk surface development unit with the road vertical plane integration unit 51. The vertical component surface on the side can be configured. In addition, when forming images in the road horizontal plane integration unit 49 and the road vertical plane integration unit 51, deletion or insertion of partial images according to the purpose is performed by passing through the image processing unit 58, and a predetermined necessity The various images are created and displayed in an integrated manner.
Next, alignment is performed by the 3DCG alignment unit 52, the vertical texture is pasted by the vertical surface texture pasting unit 53 such as a building, and the 3DCG to which the real photograph texture is pasted is configured by the real photograph texture 3DCG generation unit 54. Further, the data from the road horizontal plane integration unit 49 and the data from the road vertical plane integration unit 51 are combined in the road vertical horizontal plane integration unit 55 and integrated to form a three-dimensional map (three-dimensional map) having a horizontal plane and a vertical plane. ) Is obtained. The arbitrary viewpoint perspective display unit 56 converts the image into an image from an arbitrary viewpoint, and displays the perspective image from the arbitrary viewpoint with the viewpoint changed.
Note that the moving direction and moving distance of the object (target plane), the moving direction and moving speed of the photographing camera, and the moving direction and moving speed of the photographing camera are detected by the moving speed direction detecting unit 57 by the optical flow map generating unit 44 and the optical flow map selecting and extracting unit 45. -The movement distance is detected.
FIG. 7 shows an example constituting the apparatus of the present invention, and the video input unit 61 is a part for inputting a real video image obtained by a camera such as a CCTV camera or a digital still camera.
The video recording unit 62 is a part for recording the input video, the video reproduction unit 63 is a part for reproducing the recorded image, and the image correction unit 64 performs coordinate conversion for correcting distortion of the image due to a lens such as spherical aberration. In order to correct the camera rotation angle, a portion where the target plane image is aligned with the plane in the image, the image development plane processing unit 65 is based on the above formulas (1) and (2) by a mathematical operation. A portion that is generated by developing a plan view from an image, an optical flow map generation unit 66 generates an optical flow of the expanded image, and a portion that illustrates the optical flow, and an optical flow selection / extraction unit 67 is a target optical from the optical flow map. The part that extracts only the flow, the image processing unit 68, deletes unnecessary object images from the image, leaving only the necessary objects, Is a portion to insert the image. The developed image combining unit 69 combines the developed and processed individual images to generate a single continuous image. The generated developed image is displayed by the display unit 70 and recorded by the recording unit 71. Is done. Furthermore, the arbitrary viewpoint image generation unit 72 is a part for inversely converting to an arbitrary viewpoint and displaying it as a perspective image, the developed image comparing unit 73 is a part for comparing developed images at a plurality of identical points, and the image comparing unit 74 is a road surface unevenness by calculation Is a part to extract.
Thus, for example, a perspective image road surface corresponding to each purpose is developed into a planar image, a building wall image displayed in perspective is developed into a planar image, and further, a building wall image and a guardrail image are displayed. Separated, converted to a flat image, then changed the viewpoint and converted back to a perspective image again, or displayed a road or building surface with a texture attached, or converted to a perspective image by reverse conversion In addition, a parallax image obtained by combining images from different viewpoints can be obtained, and then processing such as calculating the unevenness of the road surface can be performed.
FIG. 8 shows a procedure for performing viewpoint movement, moving object deletion, and the like. In the figure, a perspective image of a road surface viewed from a certain viewpoint is shown at the top. As represented by three types of arrows for plane development, left plane development, road plane plane development, and right plane development are performed in order from the left. Below that, a number of squares are overlaid, which, as described above, develops perspective images on three types of planes for many screens.
Then, after developing the image on three planes, a moving object on the road surface, for example, a front vehicle that cannot be seen in the perspective is removed, and a vehicle image is obtained from the road surface image. It is possible to obtain a plan view from which is removed. Below the combined / combined arrows, a left plane developed image, a road plane developed image, and a right plane developed image developed in three planes are obtained. In the left plane development image and the right plane development image, street trees, guardrails, building wall surfaces, and the like are separated by optical flow calculation.
As a result, the road surface is displayed as a map, and the surfaces on both sides thereof are displayed as shown in FIG.
Finally, as shown by the arrows in the θ method inverse transformation, the perspective is changed by changing the viewpoint and then performing the inverse transformation after changing the viewpoint, as shown in the figure written at the bottom of FIG. Will be obtained. The inverse transformation formula at this time is the one shown in the above formulas (3) and (4).
FIG. 9 shows a procedure for detecting an uneven surface on a road surface.
That is, in image A, there is a hole on the road surface in front of the right lane, and in image B, there is also a hole in front of the right lane, but because the camera is moving on the image, The hole in image B is closer to the camera. When the road surface is developed, the positions of the holes appear differently as in the previous image indicated by the arrows of viewpoint change 1 and viewpoint change 2. From these two images, the three-dimensional appearance of the hole is synthesized using parallax. The same thing can be said even if two cameras are attached to the front and rear and the parallax or optical flow of the corresponding point of each video is taken out.
In this way, the uneven portion of the road surface can be detected, and the depth of the hole or the like can be measured as depicted in the bottom diagram in FIG.
For example, FIG. 9 shows an example of the depth of the hole. In addition to the depth of the hole, it is possible to measure a raised portion of asphalt after road construction or a rudder caused by traveling of a car.
An example of a device that measures the unevenness of road surfaces in the world at present is one that measures using a laser, but its price is quite expensive. In the method of the present invention, the unevenness of the road surface can be measured by an inexpensive method by processing the video with software as described above.
FIG. 10 shows a specific method for obtaining θ. A part of the parallel line in the real world (live-action video) is empirically searched from the image, and the extension of the parallel line is displayed as an intersection in the image. Therefore, the plane is a plane parallel to the target plane created by the intersection. The distance between a and a plane b that is a plane parallel to the target plane including the optical axis point is d, the virtual focal length is f, and from the ratio of these d and f, θ = arcTan (d / f) Is to obtain θ.
Here, as an example of obtaining the virtual focal length, the distance on the optical axis is such that the angle at which an arbitrary object in the real space is viewed in advance and the angle at which the same object in the displayed image is viewed are the same. It may be obtained on the display image, and if the unit at this time is expressed in pixels, it is a value specific to the camera system including the lens, and it is only necessary to obtain it once. Furthermore, since a parallel line portion of an object in the real world is searched and it is represented as an intersection line having an intersection on the extension line in the image, the intersection line becomes parallel when it is expanded in a plane. Θ can be obtained by selecting and further fine adjustment of θ can be performed.
FIG. 11 shows a method of measuring the actual value of θ by developing a perspective image on a plane.
FIG. 11A shows the perspective image as it is, and FIG. 11B shows the state when plane development is performed with θ set to a certain value. At this time, the parallel line in the real world, that is, the road surface is not a parallel line but a line having an intersection in the perspective as shown in FIG. Further, when the plane is developed, if the value of θ is different from the actual value, the line segments on both sides of the road that should be parallel lines do not become parallel lines as shown in FIG. On the other hand, when θ is correctly obtained, the line segments on both sides of the road become parallel lines on the side development as shown in FIG. Thereby, θ can be obtained.
Next, a case where a three-dimensional map is created will be described with reference to FIGS. Here, a case where a three-dimensional map is generated from an image photographed in a substantially traveling direction from a video camera mounted on a vehicle traveling on a road will be described as an example.
In FIG. 12, a moving image input unit 81 inputs an image captured in a substantially traveling direction from a video camera mounted on a vehicle traveling on a road, and the acquired moving image is converted into a plurality of planes. Decomposed by the decomposition unit 82. The reference plane designating unit 83 interprets that the video is composed of a plurality of planes, and sets a plurality of planes in the video. When the vehicle is traveling on a road, the road plane is set as the reference plane. And set.
On the other hand, the arbitrary-purpose plane designating unit 84 sets a street lamp plane on the assumption that it is within one plane because, for example, a plurality of street lamps are regularly installed. Similarly, the curb plane and the street tree plane A plurality of planes such as a building whole plane can be set.
FIG. 13 shows an image of a plane extracted by the optical flow.
As shown in the figure, if the vertical distance of the plane to which each object belongs from the standard position of the camera is D, all planes can be separated and extracted as a plurality of parallel plane groups. At this time, as with the roadside tree shown in the figure, for a curved object, even for a single object, even if it is a curved object having points or faces that do not ride on one plane, Is treated as a collection of a plurality of planes, and given a distance from the reference plane (street tree (1) in the figure), it can be grasped as information on one object belonging to that plane.
The θ and h detection unit 85 reads the angle θ between the road surface and the optical axis and the distance h between the camera system optical center and the road surface from the image. By setting f and h in the expressions (1) and (2), the intersection line is set to be parallel when the plane is expanded, or the ratio of d and f related to the planes a and b. This θ is obtained from the equation (1), and this θ may be read by actual measurement if it can be actually measured. The coordinate conversion unit 86 calculates θ and h by substituting them into the plane expansion conversion equation according to the above formulas (1) and (2), and acquires a plane expansion image by the plane expansion unit 87 of the image.
Opt. The F (optical flow) value calculation unit 88 divides the image into small regions because the video is a moving image, and calculates the optical flow of each part of the image by calculation using matching and correlation methods. The F (optical flow) map generation unit 89 displays the calculation result as an image map.
The reference plane image extraction unit 90 obtains the reference plane image by extracting only the unique optical flow value indicating the road surface. That is, on the road surface that is developed in a plane, the relative flow in the same plane is always constant, so the optical flow is the same, and the road surface can be extracted easily. Note that in general perspective images, the relative speed changes even with the same plane in the image depending on the distance, so that not only the reference plane cannot be extracted with a unique optical flow value, but also depending on the distance. Because the size changes, the comparison is not simple.
The parallel plane extraction unit 91 is configured to obtain an optical flow value different from that of the reference plane in the same manner as the reference plane is extracted. Since the plane parallel to the reference plane is different from the reference plane in its eigenvalue, the parallel plane can be obtained separately from the reference plane.
The plane image construction unit 92 treats each obtained plane as an image plane as it is, and can acquire an image in the set plane. The three-dimensional map generation unit 93 generates a three-dimensional map having a reference plane and a plane parallel to the reference plane by assembling each parallel plane with three-dimensional coordinates. However, since all the objects cannot be expressed only by the reference plane and a plane parallel to the reference plane, it is necessary to form a plane image in the same manner for another plane different from the reference plane.
Further, one of a plurality of initially set planes can be handled in the same manner as the reference plane, and can be generated by creating a three-dimensional map in the same process. Here, since θ and h must be converted into respective planes, it is necessary to convert θ and h into respective arbitrary planes from the obtained three-dimensional coordinates of the reference plane. That is, the position of the arbitrary plane is calculated from the obtained three-dimensional image of the reference plane by the conversion of the θ and h values in the specified plane and the specifying unit 95, and the calculation can be simply converted manually. Is possible. Similarly, the target image is subjected to the same processing through the θ, r designation unit (θ, h designation unit) 96, and a three-dimensional map is generated by the generation unit 93 via the target plane image extraction unit 97. It is.
FIG. 14 shows a plane extracted by the optical flow and an image of a three-dimensional map generated by three-dimensional transformation.
Further, the position and direction of the camera that acquires the video are detected and described on the reconstructed three-dimensional map by the processes of the θ and h detection units 85, the coordinate conversion unit 86, and the image plane development unit 87 described above. Can be plotted. That is, the angle between the optical axis of the camera and the target plane such as a road surface is given by the processing of the θ and h detection unit 85, the coordinate conversion unit 86, and the image plane development unit 87, and the coordinate origin is designated in the image. Then, the camera position and / or camera direction obtained by the calculation can be described in the image plane-developed by the conversion formula or in the coordinates of the target plane. The above formulas (1) and (2) are calculated by giving the camera focal length, the angle formed by the road surface and the optical axis of the camera, and the coordinate origin to the target plane such as the camera optical axis and the road surface. Thus, it is possible to obtain a plane development image of the road image acquired by the camera, and at this time, the camera position and the camera direction are obtained by calculation from the conditions of the conversion formula. Thereby, the camera position and camera direction can be detected in the converted image and plotted on a three-dimensional map or the like.
Also, a plurality of images taken by a moving camera and developed in a plane are combined to generate a single image, and the combined image that is expressed as a new common coordinate system, in the new coordinate system, The camera position and camera direction obtained by the above equations (1) and (2) can also be described one after another. For example, the image from the in-vehicle camera is developed in a plane, the corresponding points on the target plane in each frame image are searched automatically or manually, and the corresponding points are combined to match to generate a combined image of the target plane. , Integrated display in the same coordinate system. Then, the camera position and the camera direction can be detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.
FIG. 15 shows an image in a state where the locus of the camera position is described on the three-dimensional map.
Thus, by detecting the camera position and direction, the position of the object can be specified from the camera positions of a plurality of images, and a three-dimensional map or a three-dimensional image can be reconstructed from the planar development image. Accordingly, it is possible to automatically generate a three-dimensional map of the traveled range simply by photographing with an in-vehicle camera or the like. Further, since the camera position and direction can be detected in this way, it is possible to plot and describe the position and direction in which the camera moves on a three-dimensional map or a three-dimensional image reconstructed from the planar development image.
And since a three-dimensional map can be generated as described above, a plurality of images obtained by plane-decomposing perspective images and camera position information can be generated, and a desired three-dimensional image can be reconstructed from the information. By transmitting the plane decomposition image and the camera position information, it is possible to reconstruct the original moving image by reconstructing it on the receiving side.
Conventionally, as a compression method of moving images, as represented by the MPEG2 system, there is a compression method mainly for separating a moving part in a moving image, predicting the movement, and eliminating signal redundancy. Are known. However, this type of conventional method was effective in terms of movement partially when there is an object moving on a static background, but in the case of a moving image in which the camera itself moves, Since the entire image has a motion component and the moving speed is not the same, all of the moving images must be updated, and the compression effect is significantly reduced.
As described above, in the three-dimensional map generation according to the present invention, a moving image is analyzed and handled as an image composed of a three-dimensional plane. As a general property of an image, since the image space is surrounded by a plurality of planes, the image can be reconstructed by extracting each plane by the apparatus and reconstructing the planes. Since the plane is defined three-dimensionally, the reconstructed plane is arranged in the three-dimensional space, so that the final image is a three-dimensional image. Therefore, as long as the camera moves in a certain direction, the relative speed between the camera and the plane is constant, and movement speed reading is uniquely required for each plane. In the range where the camera moves at a constant speed, each plane has a unique velocity component, and it is only necessary to define the velocity corresponding to the number of planes.
In this way, the image is compressed, and the original moving image can be reproduced by moving each plane at a defined speed on the receiving side. In addition, since the image includes three-dimensional information, it can be expressed three-dimensionally.
As described above, in the present invention, it becomes possible to have a sufficient compression effect even for an image having a motion component of the entire image by moving the camera itself, and having a different motion speed component in each part in the image, Moreover, since the image is three-dimensionally analyzed during image compression, a three-dimensional image can be extracted from the moving image as a result.
FIG. 16 shows another embodiment. The same parts as those in the embodiment shown in FIG. 12 are denoted by the same reference numerals, and detailed description thereof is omitted.
Here, when a moving body such as a preceding vehicle or an oncoming vehicle is included in the input image, the optical flow of the reference plane is Opt. It takes a value different from the F value. Therefore, Opt. The Opt. In the F map, Opt. Is an abnormal value existing on the road surface (reference surface). If the F value is detected, the portion is an area of a moving body on the road surface. If the object is to delete the moving object, this moving object region is designated as the moving object Opt. Extraction and deletion may be performed by the F (optical flow) extraction unit 101 and the moving body part extraction unit 102, and the deletion region may be complemented from overlapping developed images before and after. In the figure, reference numeral 103 denotes a simple moving body relative speed extraction unit that simply extracts the relative speed of the moving body.
Furthermore, for the purpose of extracting the moving body itself, reproducing its three-dimensional shape, and measuring the speed, the process processing on the right side of FIG. 16 is performed through the moving body plane designating unit 105. However, the Opt. The entire F value remains unchanged as a Opt. This does not mean the F value, and the three-dimensional shape of the moving object and the Opt. In order to obtain the F value, each surface Opt. The moving object is decomposed into a plurality of planes by the F (optical flow) extracting unit 106, each moving object plane extracting unit 107, and the like, and the plane development of each plane constituting the moving object is calculated in the same process. The three-dimensional shape of the moving object can be obtained by the planar image forming unit 108, and the velocity vector of the moving object can be obtained by the moving object velocity vector extracting unit 109. Furthermore, it can be taken into the 3D map by the 3D map generation unit 110 including the moving body.
Next, with reference to FIG. 17, description will be given of a method of pasting the texture of an acquired image as a CG (computer graphics) image or the like as an embodiment in a 3D map generation application example.
In other words, the three-dimensional coordinates of the image developed on the plane have been acquired, and only the image within the target plane has been extracted. Since the image of the roadside tree can be obtained in a deleted form, it is possible to obtain not only the texture of the road surface, but also the other plane image by pasting the plane image constituting the target building wall surface in alignment with the CG image. The texture of the building wall surface of the video image, the roadside tree, the guardrail, etc. can be pasted on the CG image without overlapping.
In FIG. 17, the same parts in the embodiment shown in FIG. 12 are denoted by the same reference numerals, and detailed description thereof is omitted. Opt. The parallel plane image is extracted from the F (optical flow) map generation unit 89 by the parallel plane image extraction unit 111, while the target plane image extracted from the target plane image extraction unit 97 is textured through the texture signal generation unit 112. Get a signal. Further, the three-dimensional coordinates are acquired from the planar image construction unit 92 by the three-dimensional coordinate acquisition unit 113 and are adjusted by the CG coordinate matching unit 114 so as to match the coordinates with the CG image. A composite image in which texture is pasted together with the signal by the pasting unit 115 to the CG is obtained.
FIG. 18 shows an embodiment in which a three-dimensional map is constituted by recognized parts. The same reference numerals are given to the same components in the embodiments shown in FIGS. 12 to 17. The detailed description is omitted by being attached.
In other words, if attention is paid to the object on the plane developed in each plane and the virtual plane, the optical flow of the object in the image depends only on the relative speed between the camera and the object. Tracking is possible, and the speed can be extracted even for a moving object, so that tracking is easy.
If the target can be tracked, the position and shape change with respect to the time change of the target can be used as a clue for recognition, and image recognition can be easily performed, and the target in the image can be directly compared to the 3D CG model without tracking. Can be replaced. For this reason, the object selection tracking unit 121 selects and extracts a specific object existing in the screen from the image formed by the planar image configuration unit 92, and tracks it. The information is input to the attribute adding unit 123 for adding the attributes of the object through the object recognition unit 122 for recognizing the various types of information.
On the other hand, from the images formed by the moving object plane image construction unit 108 and the moving object velocity vector extraction unit 109, a moving object selection tracking unit 124 selects and extracts a specific moving object existing in the screen. The information is tracked, and is input to the attribute adding unit 126 for adding the attribute of the object through the recognition unit 125 of the moving object that recognizes the attribute of the moving object and other various information. A three-dimensional map is generated by the three-dimensional map generation unit 127 constituted by the recognition object from the attribute addition units 123 and 126.
Hereinafter, a specific example in which the object recognition and tracking processing using the flat developed image is applied to the image analysis of the traffic monitoring video camera will be described with reference to FIGS. 19 to 21. FIG.
FIG. 19 is a specific example of an image obtained by a traffic monitoring video camera. FIG. 19 (a) is a monitoring video camera image (far / near image), FIG. 19 (b) is a plane developed image converted by the present invention, ( c) is a region analysis display showing the object recognized by the present invention.
FIG. 20 is a flowchart showing the object recognition processing steps in the traffic monitoring video camera. FIG. 21 is a list summarizing the results obtained by the object recognition.
In the examples shown in these drawings, the vehicle type of the passing vehicle, the color of the vehicle, the traffic volume, the speed, the acceleration, and the vehicle trajectory in the monitoring video camera image are obtained from the traffic monitoring video camera image by image recognition. ing.
First, as shown in FIG. 19A, the monitoring video camera image obtained by the traffic monitoring video camera is a perspective image (an oblique image), and the size and speed of the target vehicle are not uniform.
This perspective image is digitized and developed on a plane to obtain a plane developed image of the present invention. The developed plan view is as shown in FIG. 19 (b). In this plane development image, the parameters f and θ are determined so that the road surface is developed into a plane. Therefore, on the road surface, any position in the image except the vehicle height direction such as the width and length of the vehicle is determined. But the scale is uniform and measurable. Therefore, image recognition for traffic volume monitoring can be performed by using the flat developed image.
Conventionally, it is common to perform vehicle recognition and recognition using a perspective image (see FIG. 19 (a)), limit the vehicle recognition area to a part of the screen, and detect and measure a vehicle passing there. there were. In the present invention, since the perspective image can be developed in a plane, the entire road surface of the image can be used as the vehicle recognition range by using the plane development image (see FIG. 19B).
In the example shown in FIG. 19 (b), the moving body (vehicle) captured at the upper part (upper side) of the planar development image moves to the lower part (lower side) of the image that is the traveling direction. At this time, since the size of the vehicle image does not change in the planar development image, a plurality of images can be acquired for the same vehicle. For example, a moving body image that moves from the upper part to the lower part of the flat developed image can be acquired as image data for about 30 frames.
By using the plurality of images, detailed image analysis (region analysis) can be performed. For example, the accuracy of identifying the vehicle type can be improved, and accurate image recognition can be performed. FIG. 19 (c) shows the area analysis display.
In a flat developed image, for example, in order to increase recognition accuracy, it is possible to perform processing such as addition averaging of images and addition averaging of contour images subjected to image processing, which is very effective in image processing. Moreover, since the scale is the same in any part of the road surface on the plane development image, the position information can be easily acquired, and the movement locus of the moving body can be traced together with the recognition. Since the scale is the same in this way, the position and moving speed of the vehicle (calculated from the number of video frames per second) can be measured anywhere on the road surface, and acceleration and speed can be easily calculated. .
With reference to FIG. 20, the flow of the traffic volume recognition process using the above-described plane development technique will be described in more detail.
First, the captured perspective image is digitized (201 in FIG. 20), and a flat developed image is created from the perspective image (202).
Then, the moving body region is detected in the flat developed image, and the candidate region is created by comparing the candidate region with the background image and the current image (step 203). The candidate area is subjected to image processing by image calculation with the background image, an area having a small independent area is removed for vehicle detection, and an image area of the vehicle candidate is specified by expansion coupling of the remaining areas. The background image is updated using a Kalman filter or the like (203).
The identified candidate area is subjected to processing such as changing the threshold value of image processing, and is analyzed in detail (step 204). In addition, since the vehicle candidate area can predict the movement amount by plane development, when acquiring the vehicle existence area image, the vehicle existence area is predicted and the shape correction such as the size of the image and the positional deviation, Adjustment is performed, and corresponding areas to be used are analyzed and determined (205).
The determined corresponding area is registered in the database together with the position information (206). Thereafter, an addition average image is created using the image registered in the database (207).
Then, using this added average image, the vehicle type is determined (208) and registered in the database together with the recognition result (209).
There are various types of information registered in the database. For example, as shown in FIG. 21, vehicle ID, passage time (Passed Time), average speed (Speed), acceleration (Acc), vehicle type (Type), Information on various items necessary for vehicle recognition such as color is registered. Of course, the information is not limited to the items shown in FIG. 21, and other information can be registered. For example, in addition to the above items, a vehicle addition average image, a vehicle movement locus, and the like can be registered.
In order to replace an object in an image with a three-dimensional CG model by direct comparison, for example, a method disclosed in Japanese Patent Application No. Hei 11-97562, Japanese Patent Application No. 2000-190725, Japanese Patent Application No. 2002-146874, etc. For example, an object can be recognized, specified, fixed, and tracked, and the object can be replaced with a three-dimensional CG model. An outline of the method and apparatus is an information code conversion device that converts object information acquired for an object into an information code registered in advance corresponding to the object, and transmits or outputs the information code. The apparatus includes a reproduction conversion device that receives or inputs an information code from the information code conversion device and converts the information code into reproduction object information registered in advance corresponding to the information code. In addition, an information input means for inputting information on a required object, information on various objects created in advance or their parts and their attributes, etc., and data obtained by coding those information, respectively, are stored in a database. Information code conversion for selecting and outputting data relating to corresponding information by comparing and comparing the formed first parts warehouse, information input to the information input means and information stored in the first parts warehouse Compare the data output from the information code conversion device with the data stored in the second component store, the second component store that forms a database in the same manner as the first component store. An information reproduction conversion device that selects information for reproducing a corresponding object and reproduces and outputs the object by a required output unit based on information for reproducing the object. Than it is. Furthermore, you can specify one or more objects at any position by name or attribute on the image display of the video image taken by one or more cameras from the outside world, or you can enclose it with a mouse or click a single point position. Or by specifying with a light pen or touch panel, the object including the change in the relative angle and direction between the camera and the object and the change in the distance is excluded while excluding the presence of the object. While tracking, the characteristics of each image frame of the target object or the characteristics of each component of the target object are sequentially detected in a time series, and the image data relating to a plurality of various characteristics is stored in the database. Search sequentially for image frames or image data of features that have a corresponding relationship with the characteristics of each image frame of the object or the continuous changes in the characteristics of each component. A reconstructed image including a two-dimensional or three-dimensional shape corresponding to the object for which pattern matching is obtained corresponding to the search result is sequentially configured for each time-series change, and each image frame including the reconstructed image Or, as a group of moving images or continuous still images, the image data of the feature is matched with the setting of a required size reference on a required area on the above image display or other image display via a communication line etc. In contrast, it is displayed sequentially, and if necessary, the name and attribute data accompanying or generated with the reproduced image are also displayed in a required area.
In this way, a 3D map can be constructed by combining and integrating the 3D CG models of the replaced objects by adding the attributes prepared in advance to the respective objects to the 3D CG model. Furthermore, by applying the texture, it is possible to apply a live-action texture to the three-dimensional CG model of the object.
Further, in the embodiment shown in FIGS. 22 to 27, for example, it is possible to continuously combine the moving directions so as to obtain surrounding images of moving objects such as various vehicles. That is, a moving direction image obtained by loading a camera on a moving vehicle, aircraft, ship, or the like and taking a picture is developed in a plane and continuously combined to form a single image. This is because a camera is mounted on a moving vehicle, aircraft, ship, etc., and then oblique images are taken sequentially, for example, a road image is taken along a road, it is developed on a plane, and one road image is joined together. Is to get.
Specifically, as shown in FIG. 22, for example, the situation of a camera that photographs a road around a bus 201 as a vehicle will be described. In this case, six cameras are used. May be increased. The first camera 201A for photographing the front of the bus 201 is at the front of the bus 201, the second camera 201B for photographing the rear is at the rear of the bus 201, and the third camera 201C for photographing the right front of the bus 201 is at the rear of the right side of the bus 201. A fourth camera 201D for photographing the left front of the bus 201 is taken at the rear left side of the bus 201, a fifth camera 201E for photographing the rear right side of the bus 201 is taken at the front right side of the bus 201, and a sixth camera for photographing the rear left side of the bus 201. 201F is provided in the front left side of the bus 201, and the range of images taken by these cameras 201A is shown in a fan shape. In this way, images of six types of road surfaces are taken in perspective, and are developed into a planar image such as a map by the above formulas (1) and (2). The upper coordinates (u, v) are converted into coordinates (x, y) converted on the map.
In this way, the perspective image obtained by photographing the area around the bus 201 is converted to a road image such as a map, as shown in FIG. It is possible to display a road surface such as Furthermore, if these plane development images are combined in the traveling direction of the bus 201 as much as possible, a map is completed.
Specific examples thereof are shown in FIGS. 24 and 25. In FIG. 24, there are nine oblique images of (1) to (9). FIG. 25 shows this as a map, which is developed into a planar image such as a map by the above formulas (1) and (2) and connected as a single road. This is an example of an image taken with a normal camera, that is, an example of plane development of an oblique image and an example of combination of plane development images.
At this time, if there is a moving body image that is not necessary for image formation, it can be deleted, and when a plurality of images that partially include overlapping objects are combined, By combining images while avoiding moving images, a combined image of only stationary objects is generated. For example, when a vehicle or the like is shown on the road, a picture of a long road in which only the road is shown can be obtained by combining and connecting images that are developed on a plane avoiding the vehicle or the like.
Next, some of application examples according to the present invention will be described as follows.
That is, as a flat development image, road surface, sea surface, lake surface, river surface, ground surface, vertical wall surface, vertical virtual plane created by objects arranged on the same plane, building wall floor surface, ship deck surface, runway It can handle airport facilities such as taxiways. This means that when an image taken with a normal camera is developed on a plane, the object on the plane development plane is an object arranged on the road surface, sea surface, lake surface, river surface, vertical wall surface, the same plane. It deals with airport facilities such as vertical virtual planes, ground surfaces, building wall floors, ship decks, runway taxiways, etc.
It can also be a vehicle as an example of applied equipment, such as a peripheral road surface, a building surface, a power pole array surface, a roadside tree array surface, a guardrail array surface, etc. in a land vehicle such as a bus. It is the sea surface of the vehicle, the deck of the ship, the wall surface, etc., the runway of the aircraft, etc., the ground surface, etc., thereby enabling the omnidirectional full surface display or the target area surface display.
That is, as shown in FIGS. 22 to 25, the surrounding road surface, the building surface, the electric pole arrangement surface, the roadside tree arrangement surface, the guardrail of the normal camera 201A attached to the land vehicle such as the bus 201, etc. Develop images such as array planes, sea surface of ships such as ships, decks and walls of ships, runways of airplanes, ground planes, etc. in plan view, surrounding road surface and building surface around each of them In other words, it is possible to display all directions such as a telephone pole arrangement surface, a roadside tree arrangement surface, and a guardrail arrangement surface. Or, although not shown in the figure, a full-scale omnidirectional display of a marine vehicle such as a ship, a full-scale omnidirectional display of a ship's deck, wall, etc. It can be done.
Furthermore, as another application example, it can be applied to a building structure. As shown in FIG. 26 and FIG. The display is made possible, the inside of the building is photographed with a normal camera, and the flat portions such as the floor surface and the wall surface are displayed in plane development and plane combination display.
That is, FIG. 26 shows oblique indoor images taken with a normal camera. For example, the above equations (1) and (2) are applied to 16 images (1) to (16). FIG. 27 shows that the image is converted into a planar image by connecting the images together and connected. In this way, an image that cannot be actually captured can be obtained as an image in which a floor surface and a wall surface in a room of a building are developed, and can be generated as an image in which a surrounding wall surface with respect to the floor surface is developed. It is.
Furthermore, 3D map creation is also possible. With multiple cameras, not only continuous shooting of the road surface, ground surface and water surface with moving vehicles, aircraft, ships, etc., but also vertical such as building wall surfaces, etc. By continuously shooting an object having a virtual vertical plane in which a plane or a plurality of utility poles, guard rails, etc. are regularly arranged in a plane, the image developed on the plane is coupled and extended in the moving direction, and at the same time A three-dimensional map is created by creating a wider plane vertical plane development including the plane.
That is, images of road surface, ground surface and water surface taken with a video camera mounted on a moving vehicle, such as a vehicle, an aircraft, a ship, etc. are developed in a plan view, and they are combined by an appropriate method to develop a combined plane A map is created by making a figure and then extending it in the direction of movement. Alternatively, by continuously shooting an object having a vertical surface such as a building wall surface or a virtual vertical plane in which a plurality of utility poles, guard rails, etc. are regularly arranged in a plane, the image developed on the plane is moved in the moving direction. A three-dimensional map is created by creating a wider range of plane vertical plane developments including vertical planes at the same time, while extending and coupling to.

Since the present invention is configured as described above, an image that cannot be actually captured can be converted using Equation (1) and Equation (2) by using the present method and apparatus. Since an oblique image can be made into a flat image, its application range is wide.
In addition, by generating a flat developed image from a plurality of videos taken from different points by using a plurality of cameras with overlapping viewpoints, parallax can be detected in the flat developed image of the overlapping portion. That is, conventionally, detection of three-dimensional data by detection of parallax has always been obtained from the original image, that is, the perspective image itself, but in the present invention, the parallax is detected after performing the flattened image processing. Because of the new method, three-dimensional data with good positional accuracy can be obtained.
In addition, this makes it possible to easily obtain direct three-dimensional shape data with higher accuracy than in the past, and the parallax obtained from multiple images with overlapping fields of view is often referred to as an optical flow in a stationary coordinate form in a video. Although it has the same meaning, it is particularly effective when the object changes with time or when the accuracy of the three-dimensional coordinates is increased.
Then, a desired moving image can be reconstructed by transmitting / receiving the image information and the position information of the camera which are developed in a plane, and moving image data can be transmitted / received at high speed while minimizing the data transmission amount. In particular, it is effective for video transmission using a telephone line, an Internet line, etc. with a narrow band.

Claims (36)

Road surface etc. characterized by reading information necessary for plane development from perspective images obtained from a normal camera, developing it into a plan view, combining it, joining them together, and making a single large development view Planar development image processing method for planar object video. Read the information necessary for plane development from perspective images obtained from a normal camera, develop it into a plan view using the following formulas (1) and (2), combine it, join it together, and develop a single large image A flat developed image processing method for a planar object image such as a road surface characterized by being made into a diagram.
y = v · 2 ^1/2 · h · cos (π / 4-θ) · cos (β−θ) / (f · sinβ) (1)
x = u · h · cos (β−θ) / (f · sin β) (2)
Where θ is the angle formed by the optical axis of the camera and the road surface, f is the focal length of the camera, h is the height of the camera, β is a distance h + y from directly below the camera, and the line segment connecting the camera and the road The angle formed by the plane, v is the vertical coordinate from the origin on the CCD plane, which is the projection plane of the camera, u is the horizontal coordinate from the origin on the CCD plane, and y is the point that has advanced h from directly below the camera on the road plane. Is the distance or coordinate that travels further in the optical axis direction from the origin, and x is the lateral distance or coordinate on the road surface. For an image of a real scene including a plane taken from an angle, a plane that is originally composed of a plane is developed and displayed on the plane as a plane image that is proportional to the plane of the real scene by mathematical calculation. A flat developed image processing method for a planar object image such as a road surface according to claim 1 or 2. The flat developed image processing method for a planar object image such as a road surface according to any one of claims 1 to 3, wherein a plurality of flat developed images are combined and expressed as a single large flat developed image. . Multiple input images are developed in a plane, and the images are combined to form a single image. The entire target area is displayed, and if necessary, the input image corresponding to the displayed location can also be directly displayed. The flat developed image processing method of a planar object image such as a road surface according to any one of claims 1 to 4, which is displayed simultaneously. The planar development of a planar object image such as a road surface according to any one of claims 1 to 5, wherein an image of a moving direction by a moving object is planarly developed and continuously combined into a single image. Image processing method. The road surface according to any one of claims 1 to 6, wherein a moving body image unnecessary for image formation is deleted, and the moving body image is avoided and the images are combined to generate a combined image of only a stationary object. Planar development image processing method for planar object video such as. In the process of obtaining the vanishing point in the acquired plane image, the image is moved and displayed so as to fix the position of the vanishing point of each image, thereby stabilizing the image that shakes due to camera shake or the like. Item 8. A flat developed image processing method for a planar object image such as a road surface according to any one of items 1 to 7. Obtaining the unit time movement amount of a minute area in a moving image composed of different planes obtained by plane development within the necessary range by using the optical flow method, and extracting the same component from the component distribution map The flat developed image processing method for a planar object image such as a road surface according to any one of claims 1 to 8, wherein a single planar image is separated by the above. In a plane moving image obtained from a plane converted moving image, an optical flow distribution map is generated, and unevenness of the plane is detected as a deviation from the plane from the minute difference, or from a different angle of view in the plane moving image. Parallax is detected by comparing and calculating the obtained plane image, the uneven component in the plane is detected from the component distribution, and the deviation from the plane of each point of the original plan view is included with the detected uneven value The flat developed image processing method for a planar object image such as a road surface according to any one of claims 1 to 9, wherein a corrected plan view is generated. By comparing and calculating a plurality of plane images developed in a plurality of plan views using a method such as a correlation method or a matching method, each small region on a plurality of plane images such as a road surface corresponds to each small area. The amount of movement of the area is obtained by parallax method or optical flow method, etc., and 3D data such as unevenness such as road surface is detected from the distribution of the component, or the plane of each point of the original plan view with the detected 3D unevenness value The flat developed image processing method for a planar object image such as a road surface according to any one of claims 1 to 9, wherein a corrected plan view including a deviation from a plane is generated. The average optical flow value of a continuous image developed on a plane or the movement distance of a matching-corresponding position is obtained, and the movement distance / movement speed / movement direction of the target plane or the movement distance / movement speed / movement direction of the photographed camera is obtained from the value. A plane development image processing method for a plane object image such as a road surface according to claim 10 or 11, wherein: By pasting the texture of the object plane in the separated single plane, which has been separated into planes, on the object plane in the CG (computer graphics) image or map image corresponding to the place, the actual image is captured on the CG image or map image. 13. The planar developed image processing method for a planar object image such as a road surface according to claim 1, wherein the image is captured and displayed as a plane or inversely converted and displayed as a perspective image. In the formulas (1) and (2) according to claim 2, first, f and h are given, and when the parallel line of the object is an intersection line forming an intersection point in the image, this intersection line The plane development image processing of a plane object image such as a road surface according to any one of claims 2 to 13, wherein θ is selected so that the plane becomes parallel when the plane is developed in a plane. Method. The planar developed image processing method for planar object images such as road surfaces according to claim 14, wherein the selected θ is finely adjusted in claim 14. A parallel line portion in a live-action image is extracted from the image, and a distance between a plane a that is a plane parallel to the target plane formed by the intersection and a plane b that is a plane parallel to the target plane including the optical axis point is calculated. The road according to any one of claims 2 to 15, wherein d is set to f, the virtual focal length is set to f, and θ is calculated from the ratio of d and f as θ = arcTan (d / f). Planar development image processing method for plane object image such as a plane. The parallax is detected by acquiring multiple simultaneous images of the same point with multiple ordinary cameras installed at different installation locations, and comparing the plane development images of the multiple images of the same point at the same point. The planar developed image processing method for a planar object image such as a road surface according to any one of claims 1 to 16, wherein a three-dimensional shape is generated. A plane developed view that is generated by converting a perspectively represented image including a plane image into a plan view, or a corresponding point after developing an image including a plurality of plane images taken from a plurality of directions on a plan view. The formulas (1) and (2) according to claim 2 based on a single large screen plane development view or a plan view-like CG (computer graphics) image or map generated by overlapping and combining. A virtual perspective image viewed from an arbitrary viewpoint is generated by an inverse transformation formula with respect to, or a moving image is generated from a virtual moving camera viewpoint by performing continuous processing. A plane development image processing method for a plane object image such as a road surface according to any one of the above. The flat developed image processing method for a planar object image such as a road surface according to claim 18, wherein the inverse transformation formula is the following formulas (3) and (4).
v = y · f · sin β / (2 ^1/2 · h · cos (π / 4-θ) · cos (β-θ))
………………… (3)
u = x · f · sin β / (h · cos (β−θ)) (4)
Where h is the height of the camera from the road surface, θ is the angle formed by the optical axis of the camera and the road surface, f is the focal length of the camera, and β is advanced y from the point h ahead of the camera. The angle formed by the line connecting the point and the camera lens and the road surface, x is the vertical coordinate from the line obtained by orthogonally projecting the optical axis of the camera onto the road surface, that is, the horizontal coordinate when viewed from the camera, y Is the coordinate in the optical axis direction when the origin is a point advanced from directly under the camera, v is the vertical coordinate on the CCD surface, which is the projection surface of the camera, and u is the horizontal coordinate on the CCD surface. . 20. A planar object such as a road surface according to any one of claims 1 to 19, which enables various types of recognition processing such as measurement processing and image recognition processing on the image by using a flatly developed image. A method for processing a flat image of an image. Plane development images include road surface, sea surface, lake surface, river surface, ground surface, vertical wall surface, vertical virtual plane created by objects arranged in the same plane, building wall floor surface, ship deck surface, runway guidance 21. A planar development image processing method for roads or the like according to any one of claims 1 to 20, which is an airport facility surface such as a road. Plane development images acquired by moving objects include the surrounding road surface, building surface, telephone pole array surface, roadside tree array surface, guardrail array surface, etc. of the land moving object itself. The planar developed image processing method for a road or the like according to any one of claims 1 to 21, which is an upper surface, a ship deck, a wall surface, and the like, and is a runway, a ground surface, and the like of an air moving object itself. The plane development image processing of a road or the like according to any one of claims 1 to 22, wherein the plane development image is a plane development display and a plane combination display of a plane portion such as a floor surface or a wall surface of a building. Method. Multiple video input devices continuously shoot moving road surface, ground surface and water surface, vertical surface such as building wall surface, or virtual vertical where multiple power poles, guard rails, etc. are regularly arranged in a plane Claims for creating a three-dimensional map by continuously capturing an object with a flat surface, and simultaneously extending and extending the flat image in the moving direction, and simultaneously creating a wide-area flat surface development view including the vertical surface. A planar development image processing method for a road or the like according to any one of items 1 to 23. Convert not only a plane in one direction but also a plane image in a plurality of directions in a video including a plane in a plurality of directions into a plurality of plan views, or display a plane image in the plurality of directions as a tertiary A three-dimensional plane development drawing obtained by combining the original and generating a perspective image viewed from an arbitrary viewpoint by inverse transformation, a reverse development image of a planar object image such as a road surface Conversion processing method. In a video input unit that acquires a perspective image, a video playback unit that plays back an oblique video shot by the video input unit, an image correction unit that corrects a shooting rotation angle by the video input device, and the video input device A spherical aberration correction unit that corrects spherical aberration and the like, a video development plane processing unit that converts a perspective image into a plane development view, a development image combination unit that combines videos that have undergone video development processing, and a combined image are displayed. A flat developed image processing apparatus for a planar object image such as a road surface, characterized by comprising a display unit. 27. An optical flow map generation unit that generates and illustrates an optical flow of a developed video, and an optical flow extraction unit that extracts only a target optical flow from the optical flow map. Planar development image processing apparatus for planar object images such as road surfaces. 28. The flat developed image processing apparatus for a planar object image such as a road surface according to claim 26 or 27, further comprising a parallax extracting unit that detects parallax from video at the same point from different positions. 29. A flat developed image processing apparatus for a planar object image such as a road surface according to any one of claims 26 to 28, further comprising a developed image comparing unit that compares a plurality of developed images at the same point. 30. A planar object image such as a road surface according to any one of claims 26 to 29, further comprising: an image comparison unit that extracts road surface unevenness by calculation; and a correction plane generation unit that takes into account the unevenness. Plane development image processing apparatus. A video input unit for generating video by a camera, an input image display unit for stabilizing and displaying the input image, a video recording unit for recording the input video, a video playback unit for reproducing the recorded image, and a lens such as a spherical aberration In order to correct the camera rotation angle by performing coordinate transformation to correct image distortion caused by the image, the image correction unit aligns the target plane image with the plane in the image, and mathematically calculates the plane image from the perspective image. An image development plane processing unit that generates a diagram, an optical flow map generation unit that generates and illustrates an optical flow of the expanded image, and an optical flow extraction unit that extracts only the target optical flow from the optical flow map And a parallax extraction unit that detects parallax from video at the same point from different positions, and deletes unnecessary images while leaving necessary objects, In addition, an object image processing unit for inserting a new video, a developed image combining unit for combining individual processed images that have been developed in a plane and generating a single continuous image, and a developed image for displaying them A display unit, a recording unit that records them, an arbitrary viewpoint image generation unit that performs reverse conversion to an arbitrary viewpoint, an arbitrary viewpoint image display unit that displays the image, and a plurality of developed images at the same point are compared. Planar development of a planar object image such as a road surface, which is composed of an unfolded image comparison unit, an image comparison unit that extracts road surface unevenness by calculation, and a modified plane generation unit that takes into account the unevenness as appropriate Image processing device. A flat developed image processing apparatus for a planar object image such as a road surface according to claims 26 to 31, wherein an arbitrary viewpoint image generating unit that performs reverse conversion to an arbitrary viewpoint and displays the image. A reverse development image conversion processing apparatus for a planar object video such as a road surface, characterized by comprising an arbitrary viewpoint image display unit. A video input unit that obtains a perspective image, a plane decomposition unit that decomposes the perspective image captured by the video input unit into one or more plane images constituting a three-dimensional space, and a three-dimensional video input unit A position detection unit for detecting a target position, and a display unit for reconstructing and displaying a three-dimensional image from a three-dimensional position of a plane image decomposed by the plane decomposition unit and a video input unit detected by the position detection unit A flat developed image processing apparatus for planar object images such as road surfaces. 34. A plane object such as a road surface according to claim 33, further comprising a position notation unit that indicates a three-dimensional position of the video input unit detected by the position detection unit in a plane image decomposed by the plane decomposition unit. Planar development image processing device for physical images. 35. The scope according to claim 34, wherein, when the video input unit moves, the position notation unit continuously describes the three-dimensional position of the moving video input unit in the planar image decomposed by the plane decomposition unit. Planar development image processing apparatus for planar object images such as road surfaces. When the display unit for reconstructing the three-dimensional image is disposed apart from the plane decomposition unit and the position detection unit, one or more plane image signals and video images from the plane decomposition unit and the position detection unit to the display unit. 36. The planar developed image processing apparatus for planar object images such as road surfaces according to claims 33 to 35, comprising transmission / reception means for transmitting a three-dimensional position signal of the input unit.

Images