WO2004008744A1

WO2004008744A1 - Road and other flat object video plan-view developing image processing method, reverse developing image conversion processing method, plan-view developing image processing device, and reverse developing image conversion processing device

Info

Publication number: WO2004008744A1
Application number: PCT/JP2003/008817
Authority: WO
Inventors: Waro Iwane
Original assignee: Iwane Laboratories, Ltd.
Priority date: 2002-07-12
Filing date: 2003-07-11
Publication date: 2004-01-22
Also published as: JP4273074B2; AU2003248269A1; WO2004008744A9; JPWO2004008744A1

Abstract

A method and a device for converting a perspective image obtained by the perspective of various monitor cameras so as to develop and display it on a plan view as expressed on a map. From the perspective video obtained by an ordinary camera, information required for a plan-view development is read in, developed into a plan view such as a map by a mathematical expression, combined/concatenated into a large road surface development diagram. For example, a plurality of ordinary video cameras are attached around a bus so as to constitute the entire field of view and pick up images of a road outside the bus with a certain angle, i.e., a dip. The images of the road surface are developed as images of a map, which are processed and combined to generate a road image around the bus. This is obtained by calculating a mathematical expression.

Description

Description: Plane developed image processing method for plane object image such as road surface, inverse developed image conversion processing method, planar developed image processing apparatus, inverse developed image conversion processing apparatus

The present invention provides a method of processing a plane developed image of a plane object image such as a road surface, which is photographed in a perspective view when photographed by a normal camera, and is developed into a plane like a map screen. The present invention relates to an inversely developed image conversion processing method, a planarly developed image processing apparatus, and an inversely developed image processing apparatus.

Specifically, images taken with a normal camera are developed, for example, images around the bus are taken with multiple cameras, each is developed as a flat road surface, and the road surface around the bus is taken as a bus. As shown on a map of the road surface from above, the developed images are combined so that they are displayed as a single planar road surface.By developing this planar image, it becomes possible to combine images by forming a linear scale, In addition, image recognition and measurement processing of the object can be performed within the developed plane image.Furthermore, using this image conversion principle in reverse, the road surface is developed into a plane, the viewpoint is moved, and the perspective By returning to the image, it is possible to obtain an image with a different viewpoint from the original perspective image, and to apply this principle of image development not only to the road surface but also to any plane in the image. It is intended.

In addition, an arbitrary plane is extracted from other planes by performing optical flow (or parallax, matching, etc.) processing on the plane-expanded image. Further, the unevenness of the plane is detected, and the deviation of the unevenness from the plane is detected. By using optical flow, we can extract the moving distance and speed direction, place an object consisting of a CG (computer graphics) image on a two-dimensional image, and paste the corresponding texture. By displaying the image as a plane or as a perspective image, or by parallelizing the line components that should become parallel lines in the screen when expanding the perspective image into a plane, the camera dip angle 0 is obtained, or the perspective angle By finding the point where the line component that should become a parallel line in the screen intersects from the image, that is, the vanishing point, the camera dip angle 0 can be obtained, By moving the image so that its dip 0 is constant, the camera The present invention relates to a method and an apparatus for developing the image on a plane which enables correction of blur, and an application thereof.

Furthermore, it is possible to reconstruct a desired moving image by transmitting and receiving plane-deployed image information and camera position information, so that moving image data can be transmitted and received at high speed while minimizing the data transmission amount. It is to be. Background art

In the past, CCTV (ClosedCircuitTelevisision) cameras and various monitor cameras naturally photographed in perspective, due to the nature of the lens. For example, a camera that captures the situation behind the bus arriving at the back of the bus will, of course, shoot a perspective image unless the camera is mounted vertically downward, and obtain an image of the road surface viewed from directly above. I couldn't do that. Furthermore, it was a reality that when viewed from the back of the bus, including the side, front, and back of both sides of the bus, they were partially visible, had blind spots, and were distorted. In other words, shooting with a normal camera alone would have resulted in a perspective shot, making it impossible to draw a two-dimensional image like a map.

As a technique for combining images, there is a technique for combining oblique images.However, in principle, this is a technique for combining multiple images taken from the same shooting position with different viewpoints. There is no technology that develops, combines, and displays multiple images that have been photographed in a similar manner to a real object. Not only that, there has been no method or apparatus for obtaining an image with a different viewpoint from a video image in the past, and a method of detecting unevenness of a plane by image processing technology from a plurality of perspective images has been used. There was no way to determine the deviation.

Also, there is no method or apparatus for extracting the moving distance and velocity direction from the perspective image of the video using the plane-expanded image, and furthermore, the CG image by deleting non-target images from the plane-expanded image There is no method or device for pasting textures into a plane or displaying it as a perspective image, and there are no precise methods and equations for extracting and calculating the variables of the equations necessary for plane expansion from within the image. It did not exist.

Moreover, even if you try to find a mathematical solution of the plane expansion, it will correspond to the actual coordinates Therefore, it is necessary to have 6 to 12 or more corresponding points that are actually measured, and it is virtually impossible to satisfy the condition especially for moving images.

Furthermore, in the case of moving images such as video images, when data is to be transmitted between a plurality of devices or the like, transmission data compression is performed. A method of separating and compressing only a certain moving part is adopted (for example, MPEG 2 method). In such a compression method, a moving image in which the camera itself moves and the whole image has a motion component is used. A sufficient compression effect could not be obtained, and as a result, data could not be transmitted overnight. Therefore, the present invention has been created in view of the above-described existing circumstances, and uses an image obtained by a camera perspective method without using a plurality of actual measurement points of a real object for each image. Is developed using only the shooting conditions of the camera and the information in the captured image, and is then combined and developed into a plan view that is represented on a map. .

For example, when viewing the outside situation from a bus, etc., use the video viewed from the front, the video viewed from the side, and the video viewed from the back, expand them, connect them, and view from the top of the bus. The road surface around it is displayed like a map, and the position of the vehicle is displayed in a plan view to make driving safer.

In other words, by converting oblique images obtained by perspective from various monitor cameras, they are to be developed and displayed in a plan view such as a map. Use the front, side, and rear views to unfold, join, and show the road surface around you like a map from above the bus. By developing such a planar image, it is possible to combine images by forming a linear scale, and furthermore, to develop a planar object image of a plane object such as a road surface so that measurement within the image becomes possible. It is an object of the present invention to provide a processing method, an inversely developed image conversion processing method, a planarly developed image processing apparatus, and an inversely developed image processing apparatus.

Also, as an application of the above principle, by expanding the road surface to a plane, moving the viewpoint, and returning to the perspective image again, it is possible to obtain an image with a different viewpoint from the first perspective image When extracting arbitrary planes other than roads, A technique to remove obstacle images and extract only the target image by using Luff Mouth (or parallax, matching, etc.), or to detect unevenness of a plane from multiple perspective images, and to shift the unevenness from the plane By extracting the moving distance and speed direction by using optical flows from multiple perspective images, placing an object consisting of a CG image on an image developed on a plane, and pasting the corresponding texture Display, as a plane or a perspective image, and so that the line components that should become parallel lines in the screen become parallel so that the measured values are not used as much as possible when developing the perspective image into a plane. To find the camera's dip angle 0, or the point in the perspective image where the line component that should become a parallel line in the screen intersects, that is, the vanishing point. In this way, the camera's dip S can be obtained, or the camera can be corrected by moving the image so that the dip 0 is constant. It is an object of the present invention to provide a planar developed image processing method for video, an inverse developed image conversion processing method, a planar developed image processing device thereof, and a reverse developed image processing device. Then, since the desired 3D image can be reconstructed from the multiple images obtained by expanding the perspective image on a plane and the position information of the camera, by transmitting only the plane developed image and the camera position information, the original moving image can be obtained on the receiving side. Images can be reproduced, transmission data can be made as small as possible, and even planes such as roads, etc., that can communicate moving images of desired objects overnight even with narrow-band lines, etc. It is an object of the present invention to provide a planar developed image processing method for an object video, an inversely developed image conversion processing method, a planarly developed image processing apparatus thereof, and an inversely developed image processing apparatus. Disclosure of the invention

In order to achieve the above-mentioned object, in the present invention, information necessary for plane development is read from a perspective image obtained from a normal camera, and the information is developed into a plan view such as a map by using a mathematical formula. Are combined into a single large road surface development map. For example, a normal video camera is mounted around the bus, and multiple video cameras are used to shoot the road surface outside the bus at an angle, that is, a dip, so as to compose the entire field of view. By calculation, the road surface image is developed like a map image, and the image developed like the map and the image of the building wall are In the same way, the same processing is performed, the images are joined together, and the road surface around the bus and the image of the building are created. Or, take an image of the floor and walls inside the building, develop the image taken at an angle into a plane, and open the wall of the room, for example, to create a drawing that resembles a room is there. This is obtained by calculation using mathematical formulas.

Then, a desired three-dimensional moving image can be reconstructed from a plurality of image data developed in a plane and position information data of the camera. Thus, for example, even when the camera for shooting and the display unit for the monitor are far apart, the original moving image can be reproduced on the monitor side by transmitting the flattened image and the camera position information from the camera side. It is possible to do. Planar images are still images, and their data volume is much smaller than that of moving images of oblique images.Therefore, it is possible to transmit data freely even between devices connected by a line with a narrow band, etc. Reconstruction at the receiving end enables data communication of a moving image of a desired object. Specifically, the oblique image plane developing device as the real-time processing includes, for example, a CC TV camera or a digital still camera as a video input device, a video reproducing unit, an image correcting unit, a spherical aberration correcting unit, a video developing plane processing unit, and a developing unit. It consists of an image combining unit, a display unit, and a recording unit.

Further, the oblique image plane developing device as the offline processing is composed of a video reproducing device having recorded an oblique image, an image correcting unit, a video expanding plane processing unit, a developed image combining unit, and a display unit.

As an expanse of a plane, an image of a real scene object including a plane is photographed obliquely, and a mathematical operation is used to calculate the plane originally composed of the plane in a proportional relationship (to a similar shape) with the plane of the real scene. It is displayed on a plane as a flat image. As a combination of the two-dimensional developments, a plurality of two-dimensional development images obtained by the above method are combined and expressed as one large two-dimensional development image.

In addition, for omnidirectional display and direct display using multiple CC TV images, multiple CC TV images are flattened by the above-mentioned device, and the images are combined into one image to display the entire target area. If necessary, the CCTV diagonal image corresponding to the displayed location is also displayed at the same time. As the continuous connection of the moving direction, the moving direction image obtained by loading and photographing a moving vehicle, an aircraft, a ship, etc. on the moving direction is developed on a plane, and it is connected continuously and one image You do it.

Also, when combining multiple images that partially include overlapping objects, if a moving object is captured, the images of the moving object are avoided and combined to form a combined image of only stationary objects. Is generated.

In developing the oblique image obtained by CCTV etc. into a two-dimensional image as the 0 method, the optical axis position 0 is read from the oblique image, the camera height h, the f value of the shooting lens, or the virtual on the monitor Read the f value and calculate the coordinates of the target location using the following formula: y = v-2 ¹ /2-h-cos (π / 4—-cos (j3-/ (f · sin β) (1) x = u · h · cos (β-θ) / (f-sin | 3) By using an expression such as (2) and an expression with the same meaning, the position coordinates and size of the object in the real world By giving information that can be read from the captured video and information on shooting conditions such as 0, h, and r without giving the information as known information, the coordinate system of the real world and the coordinate system on the image monitor can be obtained. In this case, coordinate transformation is performed, where 0 is the angle between the camera's optical axis and the road surface, f is the camera's focal length, h is the camera's height, and / 3 is the camera's height. The angle between the line connecting the camera and the road surface and the point at a distance of h + y from directly below the camera, and V is the vertical coordinate from the origin on the CCD (acquired image) plane, which is the projection plane of the camera. , U are the coordinates in the horizontal direction from the origin on the CCD (acquired image) surface, and y is the distance from the point on the road surface, which is h from just below the camera, to the origin, and further from the origin in the optical axis direction That is, the coordinates and X are the horizontal distances on the road surface, that is, the coordinates, and if the vertical wall surface is to be developed on a plane, the coordinates may be processed at an angle of 90 degrees. Instead, it may be a mathematical expression relating other similar perspectives to a plane.

Here, the numerical values required for the calculation, that is, the angle 0 between the target plane and the optical axis, the angle between the real point corresponding to any point in the image and the camera, etc., and the values such as (β — θ) are Since it is a physical quantity in the real world (live-action video), it can be obtained by actual measurement. However, as a practical matter, it is practically impossible to measure each moving image at multiple locations while the camera is moving, and the camera moves. Can be obtained from the image by the following method.

Determine the position of the optical axis in the image In many cases, the geometric center of the image is the optical axis position.However, to find it accurately, determine the optical center of the imaging equipment such as a camera by collimation etc. Once measured, the position can be obtained as a value unique to the camera system including the lens and the optical axis position as one point in the image. Next, an example of the most important measurement in the image of 0 in the above-mentioned conversion formula is as follows. A parallel line portion in the real world is empirically searched from the image, and the extension of the parallel line is defined as an intersection in the image. (The vanishing point, a point that intersects at a distant point when drawing in fiP or perspective drawing, and is a vanishing point in a perspective view, etc. For example, when a straight road is drawn in perspective, And the road becomes a single point, and that point is the vanishing point.) Since it is displayed as), the plane a, which is a plane parallel to the target plane containing the optical axis point, and the target plane containing the intersection point The distance between the plane b, which is a parallel plane, and the plane b is d, and 0 can be obtained as the ratio (arc T and / f) of this d to the virtual focal length f.

Here, an example of calculating the virtual focal length is as follows.The distance on the optical axis such that the angle at which an arbitrary object in the real space is seen in advance and the angle at which the same object in the displayed image is seen is the same. It is sufficient to determine it on the display image. If the unit at this time is expressed in pixels, it will be a value specific to the camera system including the lens, and it will be sufficient to determine it once. Furthermore, a parallel line portion of the object in the real world is searched for, and it is represented in the image as an intersection line having an intersection on the extension line, so that when this intersection line is expanded on a plane, it becomes parallel. Then, select に to obtain 0 or fine-tune S.

Note that 0 can also be obtained by actual measurement.For example, the degree of inclination of a camera attached to a vehicle such as a moving body facing downward as a dip can be determined by a simple method using a protractor. If it is desired to obtain the property, it can be obtained by measuring with a special angle measuring device.

In addition, in the process of finding the vanishing point in the acquired planar image, moving the image so that the position of the vanishing point of each image is fixed and displaying the image can stabilize the image that shakes due to camera shake or the like. . That is, this is the part of the parallel line component in the target plane in the image When the intersection is obtained by extending the minutes and the point becomes the above-mentioned vanishing point, the position of the vanishing point fluctuates due to the movement or shaking of the force camera, causing blur, but the position of the vanishing point By moving and displaying the entire B central image so that is fixed, the image that shakes due to camera shake or the like is stabilized. Optical flow refers to the amount of unit time movement of a minute area in a moving image composed of a plurality of different planes obtained by plane development (in this specification and drawings, abbreviated as “Opt.F”). It is possible to separate each single planar image by obtaining the same components from the necessary range by the method and extracting the same components from the component distribution map. When using this optical flow method, the optical flow of a perspective image generally takes the same value even in the same plane even in the moving direction and takes different values depending on the distance. The optical flow uses the property of taking the same value. In other words, even for an image in which a plurality of different planes obtained by plane expansion overlap, by extracting the same component from the overall diagram of the component distribution of the optical flow, it is possible to separate each single plane image. In the image developed on a plane, it is possible to separate the wall surface of a building such as a building that constitutes a parallel plane from the guardrail surface or streetlight surface of the road by the value of optical flow.

The optical flow is a flow that indicates how each corresponding point has moved in a plurality of images.If there is movement in a plurality of images, there is a flow of the optical flow. However, the movement can be displayed as a line, and if there is no movement, the flow of optical flow will not be represented by a line, and it is important to know whether the corresponding point has moved in multiple images.

Furthermore, in a plane moving image obtained from the plane-converted moving image, an optical flow distribution map is generated, and the unevenness of the plane is detected as a deviation from the plane from the small difference, or a different image in the plane moving image is detected. The parallax is detected by comparing the plane images obtained from the corners, and the unevenness component in the plane is detected from the component distribution, and the deviation of each point of the original plan view from the plane is determined by the detected unevenness value. Is generated by comparing and comparing a plurality of plane images developed in a plurality of plan views by a method such as a correlation method or a matching method. For each small area on the planar image, the amount of movement of the corresponding small area is determined by the parallax method or the optical flow method, etc. The data can be detected, or a corrected plan view including the deviation of each point of the original plan view from the plane can be generated by the detected three-dimensional unevenness values. In other words, a distribution map of the optical flow is generated, the unevenness of the plane is detected as a deviation from the plane from the small difference using the above principle, and the plane images obtained from different angles of view are compared and calculated. Then, a parallax component in the plane is detected from the component distribution, and a corrected plan view including deviation of each point of the original plan view from the plane is generated based on the detected concavo-convex value. In addition, if a small difference in the optical flow of the converted plane image can be detected when the plane has undulations or irregularities, it means a deviation from the plane. You can do it. In addition, since there are multiple images developed in the plan view of the road surface observed from different directions by moving images or images acquired by multiple cameras, they are compared and calculated using a method such as correlation method or matching. For each minute area of a plurality of planar images such as surfaces, the amount of movement is determined from the difference between the components of the minute area by calculating parallax, etc., and the corresponding points are combined and calculated to obtain the unevenness of the road surface. Is detected.

This means that by acquiring plane-deployed images from moving images with overlapping fields of view and calculating parallax or optical flow from them, all objects that overlap in the plane-deployed image are not limited to road surface irregularities. This means that three-dimensional data can be detected for objects.

All operations using optical flow can also be performed by parallax, and can also be performed by matching. Therefore, the term “optical flow” in the present invention means that any processing such as optical flow, parallax, and matching may be used. Also, the average optical flow value of the continuous image developed on the plane or the moving distance of the matching corresponding position is calculated, and the moving distance of the target plane · moving speed · moving direction, or the moving distance of the photographed camera · moving Speed · Moving direction can be obtained. That is, the optical flow on the same plane developed on a plane is constant Because of this property, the moving speed of the camera can be obtained from the optical flow of the target plane.Since this is the relative position and relative speed between the camera and the target plane, the stationary system and the moving system can be obtained. The same is true even if reversed. Here, the disparity has the same geometric meaning as the optical flow, and the optical flow or the disparity of a wide area of the continuous image developed on the plane is obtained, and the disparity is used to calculate the moving distance of the target plane. The moving speed, moving direction, or moving distance, moving speed, and moving direction of the camera that captured the image can be obtained.

By pasting the texture of an object plane in a single plane that has been separated and developed into a CG (computer graphics) image or a map object, the CG image or map image A real image can be captured and displayed as a plane image or inversely transformed and displayed as a perspective image. That is, since the same plane developed in the plane has the same optical flow, it is possible to cut out only one texture of the target plane from a plurality of mixed planes. By pasting the texture of the object plane in the isolated plane developed on the separated plane to the corresponding object plane in the CG image or map image, the actual image is imported into the CG image or map image, and Or, it is transformed and displayed as a perspective image. Further, in the above equations (1) and (2), first, f and h are given, and furthermore, when the parallel line of the object is an intersection line forming an intersection in the image, this intersection line is flattened. By selecting 0 so that it is sometimes parallel, 0 can be obtained and the selected Θ can be fine-tuned. This makes use of the property that, for a specific object such as a road, the object itself often has a portion that becomes a parallel line. In the perspective image, the extended line segment forms an intersection, and 0 is selected by selecting 0 so that the line segment intersecting at this intersection becomes a parallel line in the plane-transformed plane image. You can do it.

The parallel lines in the actual video are extracted from the image, and the distance between the plane a, which is the plane parallel to the target plane at the intersection, and the plane b, which is the plane parallel to the target plane including the optical axis point, is extracted. Is d and the virtual focal length is f. From the ratio of these d and f, 0 = arc 0 can be obtained as T an (d / f). In other words, the angle 0 between the target plane and the optical axis is a physical quantity in the real world (actual video) and should be measured, but it must be measured from the image and each frame image in the moving image. In order to find it, we empirically search for parallel lines in the real world (real-world video) from the image, and define the plane parallel to the target plane created by the intersection as the plane a, and the plane parallel to the target plane including the optical axis point. The plane is defined as plane b, the distance between plane a and plane b is defined as d, the virtual focal length is defined as f, and 0 is determined as 0 = arc Tan (d / f) from the ratio of d and f. It is. In order to find the angle 0 between the target plane, such as the road surface, and the optical axis of the camera, a parallel line part in the real world is empirically searched from the image, and the intersection point and the road created by the parallel line part Measure in advance the target plane, such as a plane, and the position of the optical axis in the image, or use the geometric center of the image as the approximate optical axis, and parallel to the target plane, such as a road, containing the optical axis point. The ratio of the distance to the plane to the virtual focal length is calculated, and the arc tangent is calculated to obtain 0.

In addition, parallax is detected by acquiring multiple simultaneous images of the same spot with multiple ordinary cameras installed at different installation locations, and comparing and calculating a planar development image of the multiple simultaneous images of the same spot to detect parallax. A three-dimensional shape of an object can be generated. In the above example, this is to obtain a two-dimensional image by flattening the image from one camera or by combining a plurality of force cameras with different viewpoints. By using a plurality of cameras and the like, an image of the same point taken from a different point is acquired as a plane developed image, and then the parallax is detected in the plane developed image of the overlapping portion. In other words, until now, the detection of 3D images by parallax detection has always been obtained from the original image, that is, the perspective image itself. However, here, a new method of detecting parallax after performing plane development image processing. As a result, it is possible to obtain three-dimensional data with high position accuracy, and thus it is possible to easily obtain data of a direct three-dimensional shape with higher accuracy than before. Here, the disparity obtained from a plurality of videos with overlapping visual fields often has the same meaning as the optical flow in the stationary coordinate form in the moving image, but when the object changes over time or when the three-dimensional coordinate In order to increase the accuracy, it is effective to acquire multiple simultaneous images of the same spot using such multiple force cameras. On the other hand, a map or plan is prepared from the beginning, and based on all of the plan, plan and CG (computer graphics) images, the inverse transformation is performed in the opposite direction to the previous transformation. In this way, it is possible to generate a perspective image from an arbitrary viewpoint different from the image of the previous viewpoint. In addition, by continuously inverting each frame image of a video image, viewpoint movement is repeated, and a video moving image from a virtual moving camera viewpoint that is not actually photographed can be generated.

Specifically, a planar development view generated by converting a perspective image including a two-dimensional image into a plan view, or an image including a plurality of two-dimensional images taken from a plurality of directions is a plan view. The expression (1) and the expression (1) described above based on a single large screen plane development view or plan view CG (computer graphics) image or map generated by combining (2) By using the inverse transformation formula, a virtual perspective image viewed from an arbitrary viewpoint can be generated, or by processing continuously, a moving image can be generated from a virtual moving camera viewpoint. After that, the specific inverse conversion formula by the inverse conversion method is based on the following formulas (3) and (4).

V = y · f · sin ^ / (2 ¹² · h ■ c 0 s (ττ / 4-Θ)-cos (β-θ))

(3) u = x · f · si η 3 / (h ■ c ο s (β-θ)) (4) where 'h is the height of the camera from the road surface and 0 is the optical axis of the camera and the road F is the focal length of the camera, | 3 is the angle between the line segment connecting the point advancing by y and the point advancing by y from the point directly below the camera to the road surface, X Is the coordinate in the vertical direction from the line segment obtained by orthogonally projecting the optical axis of the camera onto the road surface, that is, the coordinate in the horizontal direction when viewed from the camera, and y is the coordinate in the optical axis direction when the point advancing from directly below the camera to h is the origin. , V is the vertical coordinate on the CCD plane which is the projection plane of the camera, and u is the horizontal coordinate on the CCD plane which is the projection plane of the camera. It should be noted that the equation is not limited to this equation, and may be an equation relating other similar perspectives to a plane.

In addition, various types of recognition processing, such as measurement processing and image recognition processing, can be performed on the image using the plane-expanded image.The scale of the plane-expanded image becomes a linear scale, and measurement and Very easy image processing, image recognition, etc. It is. Since the optical flow is also obtained in a form proportional to the relative speed with respect to the camera, the relative speed of the object is expressed on a linear scale without depending on the distance, so not only in measurement but also in image processing recognition It is greatly simplified. An example of an application plane is a flat surface developed as road surface, sea surface, lake surface, lake surface, river surface, ground surface, vertical wall, vertical virtual plane created by objects arranged on the same plane, architectural wall floor, ship Deck surface ・ Airport facilities such as runway taxiways can be used.

Examples of vehicles as applied equipment include peripheral roads on buses and other land-based vehicles, buildings, telephone poles, street trees, guard rails, etc., and the sea surface of ships and other marine vehicles. It can be displayed on the deck, wall, etc. of a ship, on the runway of an aircraft, on the ground, etc., or on the target area.

Further, in a building structure as another application example, a plane portion such as a floor surface or a wall surface of a building is displayed in a plane-expanded manner and in a plane-bonded manner.

In addition, 3D map production as an application example is not only continuous shooting of the road surface, ground surface and water surface with moving vehicles, aircraft, ships, etc. with multiple cameras, but also vertical surfaces such as building walls etc. Alternatively, by continuously photographing an object having a virtual vertical plane in which a plurality of utility poles, guardrails, and the like are regularly arranged in a plane, the vertical plane can be simultaneously extended while the plane developed image is coupled and extended in the moving direction. A 3D map is created by creating an expanded view of a wider vertical plane including that. On the other hand, the planar development image processing apparatus and the reverse development image processing apparatus used directly in the planar development image processing method for the planar object image such as the road surface and the inverse development image conversion processing method described above convert perspective images into perspective images. A video input unit to be acquired, a video playback unit that plays back the oblique video captured by the video input unit, an image correction unit that corrects the shooting rotation angle, etc., of the video input device, spherical aberration in the video input device, etc. Spherical aberration corrector that corrects the image, an image expansion plane processor that converts the perspective image into a plane expansion view, a developed image combination unit that combines the images that have undergone the image expansion processing, and a display unit that displays the combined image It consists of:

An optical flow map generation unit that generates and illustrates an optical flow of the expanded video, and an optical flow extraction unit that extracts only a target optical lip from the optical lip map, are provided. An image comparison unit that detects a parallax from a video at the same point, includes a development image comparison unit that compares a plurality of development images at the same point, and extracts road surface unevenness by calculation; It can be configured to include a correction plane generation unit that takes into account unevenness.

Furthermore, a video input unit that generates video by a camera such as a CCTV camera or a digital still camera, an input image display unit that stabilizes and displays an input image, a video recording unit that records an input image, and a video recording unit that records the input image. An image that adjusts the direction of the target planar image to the plane in the image in order to perform coordinate transformation to correct the image distortion due to the lens, such as spherical aberration, etc., and to correct the camera rotation angle. A correction unit, an image development plane processing unit that generates a plan view from a perspective image by mathematical operation, and an optical flow map generation unit that generates and illustrates an optical lip of the developed image, The optical flow extraction unit extracts only the desired optical flow from the optical map and the parallax is detected from images of the same point from different positions. Combine the processed parallax extraction unit, the object image processing unit that deletes unnecessary images while leaving the necessary objects, and also introduces new images, and the processed individual images that have been developed in a plane. A developed image combining unit that generates one continuous image, a developed image display unit that displays them, a recording unit that records them, and an arbitrary viewpoint image generation unit that inversely converts the images to arbitrary viewpoints and displays them. An arbitrary viewpoint image display unit for displaying the image, a developed image comparison unit for comparing a plurality of developed images at the same point, an image comparison unit for extracting road surface unevenness by calculation, and a corrected plane generation unit considering the unevenness. Are appropriately combined.

Similarly, a decompressed image processing device directly used for the plane uncompressed image processing method and the uncompressed image conversion processing method for a plane object image such as a road surface is an arbitrary image that is inversely transformed to an arbitrary viewpoint and displayed. It can be configured to include a viewpoint image generation unit and an arbitrary viewpoint image display unit that displays the image.

Further, the planar developed image processing device and the decompressed image processing device include a video input unit for acquiring a perspective image, and one or two or more of a perspective image captured by the video input unit that constitute a three-dimensional space. A plane decomposition unit that decomposes the image into plane images, a position detection unit that detects the three-dimensional position of the image input unit, and a three-dimensional image that is decomposed by the plane decomposition unit and the image input unit that is detected by the position detection unit Table for reconstructing and displaying 3D images from positions And a display unit. A configuration may be provided that includes a position notation section that writes the three-dimensional position of the video input section detected by the position detection section in the plane image decomposed by the plane decomposition section. Further, when the image input unit moves, the position notation unit can be configured to continuously indicate the three-dimensional position of the moving image input unit in the plane image decomposed by the plane decomposition unit. .

When the display unit for reconstructing the three-dimensional image is disposed separately from the plane decomposition unit and the position detection unit, one or more plane images are displayed on the display unit from the plane decomposition unit and the position detection unit. It may be configured to include a transmission / reception unit that transmits a signal and a three-dimensional position signal of the video input unit. The method for processing a plane developed image of a plane object image such as a road surface according to the present invention configured as described above, the method of inversely developed image conversion and the planar developed image processing apparatus, and the image of the inversely developed image processing apparatus, The perspective image, which is an oblique image acquired by the input device, is converted into a flat developed view by equations (1) and (2) and displayed as a practical map-like image.

Combination of acquired and generated flat development images is displayed as a single expanded image, for example, on the map including the situation of the acquisition location and surrounding area, full display of the entire target area, direct display of the specific area Etc., and easily compare with the surrounding situation by the simultaneous direct display of the input video.

Also, when there are a plurality of images developed in a plan view of the road surface observed from different directions by moving images or images acquired by multiple cameras, they are compared and calculated using a method such as correlation method or matching. For each small area of a plurality of planar images such as a road surface, it is not possible to calculate the amount of movement from the difference between the components of the small area using a parallax method or an optical flow method, and to calculate the corresponding points in combination. Then, it detects irregularities on the road surface and generates a corrected plan view including the deviation from the detected irregularities in the case where there are undulations and irregularities on the plane.

This means that by acquiring plane development images from moving images with overlapping visual fields and calculating parallax or optical flow from them, it is possible to obtain not only irregularities on the road surface but also all objects overlapping in the planar development image 3D data is detected. Capture images of the same spot from different spots using multiple force cameras with overlapping viewpoints The parallax is detected in the plane developed image generated by the plurality of videos thus obtained in the plane developed image of the overlapping portion.

The developed plane development map expands various things such as land display, sea surface, airport facility surface, etc., as well as plane display on architectural structures, etc., and generates and obtains a plane development map by moving, the plane according to the moving direction A three-dimensional map can also be created by extension of expansion and vertical expansion.

Also, video input unit, input image display unit, video recording unit, video playback unit, image correction unit, video development plane processing unit, optical flow map generation unit, optical flow extraction unit, parallax extraction unit, object image processing unit , A developed image combining unit, a developed image display unit, a recording unit, an arbitrary viewpoint image generation unit, an arbitrary viewpoint image display unit, a developed image comparison unit, an image comparison unit, a corrected plane generation unit, etc. Depending on the purpose, for example, perspective imageDevelopment of road surface into planar image, development of perspective wall image displayed on planar image, separation of building wall surface and guardrail image, planar image After changing the viewpoint, change the viewpoint and invert again to the perspective image, display the road surface or building surface with the texture attached, or use the inverse transformation to perform the perspective image It can be applied to a wide range of applications, such as converting to a parallax image, combining parallax images from different viewpoints, and calculating road surface irregularities.

Then, a m-dimensional image, a three-dimensional map, etc. can be reconstructed from a plurality of plane-deployed image data and camera position information data, so that the camera for shooting and the display unit for mobile are separated. Even if the camera is installed in a remote location, the original moving image can be reproduced on the receiving side (monitor side) by transmitting the flattened image and camera position information from the image acquisition side (camera side). . The image data developed on a plane is a still image, and the amount of data is much smaller than that of a moving image of an oblique image. Therefore, by transmitting and receiving the plane developed image and the position information data for reconstructing the plane developed image, the data communication of the moving image with the data transmission amount as small as possible becomes possible. Brief description of the drawings,

FIG. 1 is a block diagram of an apparatus for developing a two-dimensional image of a flat object such as a road surface according to an embodiment of the present invention.

FIG. 2 is a block diagram showing another embodiment of the apparatus of the present invention. ―

2004/008744

FIG. 3 is a block diagram showing another embodiment of the apparatus of the present invention. FIG. 4 is a block diagram showing the image flattening device.

FIG. 5 is a block diagram showing an apparatus for detecting unevenness on a road.

FIG. 6 is a block diagram showing a viewpoint moving 'texture pasting device which also performs viewpoint moving and texture pasting by the road surface development method.

FIG. 7 is a block diagram of an embodiment in which an image is similarly developed on a plane by the optical flow method.

FIG. 8 is a schematic explanatory diagram of a case where a plane developed image is generated from the acquired image and an oblique image is further formed by moving the viewpoint.

FIG. 9 is a schematic explanatory diagram showing a case of detecting unevenness on a road surface. FIG. 10 is a schematic diagram for obtaining 0 from the vanishing point.

Fig. 11 is an example of the plane expansion when the 0 value is different when the plane is also expanded. (A) is the value before the plane expansion in perspective and (B) is the value different from the actual value. (C) is when the value of 0 is the same as the actual value, that is, when 0 is found correctly.

FIG. 12 is a block diagram of the three-dimensional diagram generation device.

FIG. 13 is an image diagram of a plane similarly extracted by the optical flow. Fig. 14 shows the plane extracted by the optical flow and the image of the three-dimensional map generated by the three-dimensional conversion.

FIG. 15 is an image diagram showing a state in which the locus of the camera position is similarly described on a three-dimensional map.

FIG. 16 is a block diagram of the same three-dimensional moving object detection apparatus.

FIG. 17 is a block diagram of a texture pasting device in the three-dimensional diagram generating device.

FIG. 18 is a block diagram of the object recognition device.

FIG. 19 is a specific example of an image obtained by a traffic surveillance video camera as an example of the object recognition device of FIG. 18, where (a) shows a surveillance video camera image (perspective image), (b) ) Is a plane developed image converted by the present invention, and (c) is an area analysis display showing the object recognized by the present invention.

Fig. 20 shows an example of the object recognition device shown in Fig. 18 as a traffic monitoring video camera difference sheet (Rule 26). 5 is a flowchart illustrating processing steps of object recognition in the first embodiment.

FIG. 21 is a table summarizing the results obtained by the object recognition processing shown in FIG.

FIG. 22 is a situation diagram in which a plurality of cameras for photographing a road around a bus are arranged.

Fig. 23 is a video image developed on the road like a map.

FIG. 24 shows nine road image diagrams of, for example, (1)... (9), which similarly photograph oblique images of the road.

Fig. 25 is a map-like planar image obtained by processing the same road image.

FIG. 26 shows another embodiment, in which, for example, (1) ... 16 (16) indoor images of plane parts such as floors and walls of a building structure taken as oblique images Figure Each.

FIG. 27 is a composite image diagram in which the image diagrams shown in FIG. 26 are combined in a plane. BEST MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

As shown in Fig. 1, a camera 1, such as a CC TV camera or a digital still camera, as an input device, an image reproducing unit 2 for reproducing oblique images captured by these cameras 1, an image correcting unit 3, and a spherical surface Aberration correction unit 4, image expansion plane processing unit 5 that expands an oblique image into a plan view according to equations (1) and (2) described later, and developed image combining unit that connects each developed image by an appropriate method 6, a developed image display unit 7 for displaying the joined images, and a recording unit 8 for recording the joined images on a recording medium. The oblique images are two-dimensionally developed as real-time processing. That is, the image correction unit 3 corrects the rotation angle and the like of the camera 1 while inputting the image from the camera 1 and playing back the video in real time so that it can be configured as an oblique image plane development device as real-time processing. The spherical aberration corrector 4 corrects the spherical aberration of the camera 1 to suit the purpose, and the image development plane processing unit 5 converts the perspective image into a plane development view like a map and displays the developed image. Display with 7. further If necessary, the obtained images are combined by the developed image combining unit 6 with the images obtained by performing the image developing process from the plurality of force cameras 1, displayed, and recorded in the recording unit 8. It is. Also, as shown in FIG. 2, in the oblique image plane developing apparatus that performs off-line processing, the oblique image reproducing section 11 having the oblique image recorded therein, the image correcting section 12, the image developing plane processing section 13 and the developed image combining section It consists of a unit 14 and a developed image display unit 15, and plays back the image from the oblique image playback unit 11 that records the image taken by the normal camera 1. The rotation angle, etc. are corrected to suit the purpose, and the image development plane processing unit 13 develops it into a flat image such as a map and displays it on the development image display unit 15. A plurality of images developed by the developed image combining unit 14 are connected by an appropriate method, and the developed image is displayed by the developed image display unit 15. Further, as shown in FIG. 3, the oblique image plane developing apparatus includes a camera side (transmitting side) for acquiring an image and a monitor side (displaying, recording, and reconstructing a planar image decomposed and expanded from the oblique image). (Receiving side) can be installed separately. In this case, as shown in FIG. 3, on the camera side (transmitting side), a camera 1 as an input device, an image reproducing unit 2, an image correcting unit 3, a spherical aberration correcting unit 4, and an image developing plane processing unit 5 are provided. On the monitor side (reception side), a developed image combining unit 6, a developed image display unit 7, and a recording unit 8 are provided. The camera is provided with a transmitter 5a for transmitting the developed and decomposed planar image signal to the monitor, and the monitor is provided with a receiver 6a for receiving the planar image signal transmitted from the camera. These transmission / reception units 5a and 6a are connected via a communication line so that they can communicate overnight. By transmitting the plane developed image of the moving image obtained on the camera side and predetermined information such as camera position information to the monitor side via a communication line, the monitor side can transmit the plane developed image to the received plane developed image. A moving image can be reconstructed on the basis of this, and it is possible to transmit and reproduce a desired moving image while minimizing the amount of transmission data.

In Fig. 3, the oblique image plane development device for real-time processing shown in Fig. 1 is installed separately, but if it is real-time processing, it is considered offline processing. It goes without saying that they can be installed separately regardless of As a plane development mode, a plane originally composed of planes is subjected to mathematical operations on an image of an object in a real scene including a plane that is obliquely photographed, and is proportional to the plane of the real scene (similar form). Is developed and displayed on a plane as a plane image. This is an image obtained by photographing a scene including a plane with an ordinary camera, that is, an image photographed from an oblique direction. For example, if it is a road surface, it is developed into a plane and developed like a map screen.

In addition, when combining the images, a plurality of planar developed images obtained by the above method are combined and expressed as a single large planar developed image. This means that a plurality of planar developed images are combined by an appropriate method and expressed as a single large planar developed image. In other words, when a plurality of road surfaces are developed and a planar development diagram such as a map is created, they are connected to form a single large planar development image. Of course, since the image is developed on a plane, it can be freely combined as much as possible, and it cannot be combined with a perspective image unless the camera position is the same.

Furthermore, it is possible to enable omnidirectional display and direct display based on images acquired by a plurality of CC TV cameras. As a single image, the entire area of the target area is displayed, and if necessary, an oblique image of the CCTV camera corresponding to the displayed location can be displayed at the same time.

That is, the images of a plurality of CCTV cameras are developed into planar images by the above-described device, and the respective images are combined by matching corresponding points to form a single image, and the entire target area is displayed. Furthermore, if necessary, by displaying the CCTV camera's corresponding image, that is, an oblique image, corresponding to the displayed location, it is possible to enhance the effects of monitoring and other purposes. FIG. 4 shows a processing block diagram when the image is developed on a plane. First, when a video image 21 or a still image image 22 is input to the device as an input image, the input image is input. At the time of force, the correction unit 23 performs spherical aberration correction, rotation correction, and the like. Next, the image plane expansion processing unit 24 performs the image plane expansion processing according to the following equations (1) and (2).

That is, in converting an oblique image obtained from the video image 21 or the still image image 22 into a flat image, the present invention obtains the image by a method referred to as a 0 method. In developing the oblique image into a two-dimensional image, read the optical axis position 0 from the oblique image, read the camera height h, the f-value of the taking lens, or the virtual f-value on the monitor, and set the coordinates of the target location as follows: It is to be obtained by equations (1) and (2).

y = v-2 ¹ /2--cos (π / 4-Θ)-cos (β ~ θ) / (f sin β) (1) x = u-h-cos (β-θ) / (f · Sin / 3) (2) By using the mathematical expressions expressed by the expressions (1) and (2) and the mathematical expression having the same meaning, the position coordinates and size of the object in the real world can be calculated. By giving information that can be read from the shot video and information about shooting conditions such as 0, h, and r without giving it as known information, the real world coordinate system and the coordinate system on the image monitor can be related. The coordinate conversion is performed in series.

Where x and y are the plane development coordinates, u and V are the coordinates in the image, 0 is the angle between the camera's optical axis and the road surface, f is the focal length of the camera, h is the height of the camera, and / 3 is the camera The point between h + y and the line connecting the camera and the road surface, V is the vertical coordinate from the origin on the CCD surface, which is the projection surface of the power camera, u is the CCD surface The coordinates are in the horizontal direction from the upper origin. Further, y is a distance or coordinates further advanced in the optical axis direction from the origin at a point advanced h from directly below the camera on the road surface, and X is a lateral distance or coordinates on the road surface.

Therefore, using the above equations (1) and (2), the perspective image obtained by photographing the road surface is converted into a map-like planar image and developed.

The plane image thus transformed and developed is displayed and recorded by the plane developed image display / recording unit 25. Next, a perspective image is developed on a plane, and the developed image combining unit 26 connects the planar developed images several times. The developed image combining unit 26 provides a planar expanded image of a large screen. Displaying and recording this is a combined expanded image display- The recording unit 27. Next, an arbitrary viewpoint is specified from the developed video and the arbitrary viewpoint is generated by the arbitrary viewpoint generator 28 in order to generate the arbitrary viewpoint. Then, the inverse expansion conversion processing unit 29 uses the following equations (3) and (4), which are the inverse conversion equations of the above equations (1) and (2), to obtain a perspective that is different from the original viewpoint. A normal image can be obtained, and the reversely developed image display / recording unit 30 displays and records this.

That is, the equation for the inverse transformation is as follows. .

v = y-f-sin i3 / (2 ¹ /2-h-cos (ττ / 4-Θ)-cos (β-θ))

(3) u = x-f-si η β / (cos (β-θ)) (4) where h is the height of the camera from the road surface and Θ is the angle between the optical axis of the camera and the road surface , F is the focal length of the camera, / 3 is the angle between the line that connects the point advancing by y and the point advancing by y from the point just below the camera to the camera lens, and the angle between the road surface and X is the camera light Y is the coordinate in the vertical direction from the line segment obtained by orthogonally projecting the road onto the road surface, that is, the horizontal direction when viewed from the camera. U is the vertical coordinate on the CCD plane, which is the projection plane of the camera, and u is the horizontal coordinate on the CCD plane.

Note that the inverse transformation equation may be not only these equations (3) and (4) but also an equation that relates other similar perspectives to a plane. In FIG. 5, a new block diagram is added to a part of the block diagram of FIG. That is, the video image 21 and the still image image 22 are input, and the perspective image is expanded into a plane image by the equations (1) and (2). This is performed by the image plane expansion processing unit 24. . The developed image comparison unit 31 compares the developed images. The developed image comparison unit 31 compares the images from different viewpoints to determine the unevenness of the road surface. The processing is performed by the unit 32. In Fig. 6, not only the road surface is deployed, but also the buildings, street trees, Guardrails etc. are also developed on a plane, and converted to arbitrary viewpoint images, or textures are pasted on buildings, and 3D CGs using actual textures are used to move viewpoints by road surface development method And a structure for performing texture pasting.

That is, first, a road surface plane development unit 42 obtains a road development plan from an image input from the video image input unit 41, while a horizontal elevation plane development unit 43 obtains a road side surface such as a building wall surface. Expand the part into a plan view. Thereafter, an optical flow map is generated by an optical flow map generation unit 44, and a target portion is selectively extracted by a next optical flow map selection and extraction unit 45. Thus, the curb surface extraction unit 46 extracts the curb of the road, and the building surface extraction unit 47 extracts the building wall and the like. The sidewalk development unit 48 performs sidewalk development, matches the data from the road surface development unit 42 described above, and combines the sidewalk and roadway parts with the road horizontal integration unit 49. On the other hand, from the optical flow map selection and extraction unit 45, the street tree is extracted by the street tree etc. plane extraction unit 50, and the data from the building surface extraction, the sidewalk surface development unit and the road vertical surface integration unit 51 are combined. Thus, the vertical component plane on the side of the road can be configured. When forming images in the road horizontal plane integration section 49 and the road vertical plane integration section 51, deletion and insertion of partial images according to the purpose are performed by passing through the image processing section 58. That is, various kinds of required images are created and displayed in an integrated manner.

Next, the 3DCG positioning unit 52 performs positioning, and the vertical surface texture such as a building pastes the texture of the vertical surface in the texture pasting unit 53 and pastes the actual shooting texture. It consists of part 54. In addition, by combining the data from the road horizontal plane integration unit 49 and the data from the road vertical plane integration unit 51 with the road vertical horizontal plane integration unit 55 and integrating them, a three-dimensional object having a horizontal plane and a vertical plane is obtained. Figure (solid map) is obtained. This is converted into an image from an arbitrary viewpoint by an arbitrary viewpoint perspective display unit 56, and a perspective image from the arbitrary viewpoint with the changed viewpoint is displayed.

The optical flow map generation unit 44 and the optical flow map selection extraction unit 45 allow the moving speed direction detection unit 57 to move the object (target plane) in the moving direction, moving speed, moving distance, and moving the camera. Detects direction and moving speed · Moving distance I have to do it. FIG. 7 shows an example of the configuration of the apparatus of the present invention. An image input section 61 is a section for inputting an image of a real photograph acquired by a camera such as a CCTV camera or a digital still camera.

The central image recording section 62 records the input video, the video playback section 63 reproduces the recorded image, and the image correction section 64 corrects the image distortion caused by the lens such as spherical aberration. In order to correct the camera rotation angle, the target plane image is oriented to the plane in the image. The image development plane processing unit 65 uses the equations (1) and (2) based on the above equations (1) and (2). The optical flow map generation unit 66 generates the optical flow of the expanded video by mathematical operation and generates the optical flow of the expanded video, and the optical flow selection and extraction unit. 6 7 is a part that extracts only the desired optical flow from the optical map, and the image processing unit 6 8 deletes unnecessary object images while leaving only the necessary objects from the image. There is a portion to insert the image. The developed image combining unit 69 combines the developed and processed individual images to generate one continuous image. The generated developed image is displayed on the display unit 70, and the recording unit 7 Recorded by one. Further, the arbitrary viewpoint image generating unit 72 is a part for inversely converting to an arbitrary viewpoint and displaying it as a perspective image. The developed image comparing unit 73 is a part for comparing developed images at a plurality of the same points. Is a portion for extracting road surface irregularities by calculation.

With these, for example, a perspective image road surface can be developed into a planar image according to the purpose, a building wall image displayed in perspective can be developed into a planar image, and a building wall image and a guardrail image can be displayed. After converting to a two-dimensional image, change the viewpoint and then reverse convert it back to a perspective image again, or display a road surface, building surface, etc. on which texture is pasted, or use a reverse transform to produce a perspective image It is also possible to obtain a parallax image by combining images from different viewpoints, and then perform processing such as calculating the unevenness of the road surface. Fig. 8 shows the procedure for moving the viewpoint, deleting the moving object, etc., and the top part of the figure shows the perspective image of the road surface viewed from a certain viewpoint. You. As indicated by the three types of arrows that expand the plane, the left plane, the road plane, and the right plane are developed in order from the left. Underneath it are several overlapping squares, which, as described above, develop perspective images into three different planes for many screens.

After developing many images into three planes, the system removes moving objects on the road surface, for example, the vehicles in front that would become invisible in perspective, and processes the vehicles from the road surface image. Can be obtained. Below the combined arrow, a left-side plane development image, a road-side plane development image, and a right-side plane development image developed on three planes are obtained. In the left-side plane development image and right-side plane development image, the optical tree is used to separate street trees, guardrails, and building walls.

As a result, the road surface is displayed as a map, and the two sides are displayed as flat surfaces with open walls, as shown in FIG. 8.

Finally, as shown by the arrow in the 逆 method inverse transformation, the perspective is changed by changing the viewpoint after the viewpoint is changed, as shown by the arrow at the bottom of Fig. 8. Will be obtained. The equations for the inverse transformation at this time are those shown in the above equations (3) and (4). FIG. 9 shows a procedure for detecting an uneven surface on a road surface.

That is, in image A, there is a hole on the road surface far in front of the right lane, and in central image B, there is also a hole in front of the right lane, but the camera is moving on the image In addition, the hole in image B is closer to the camera. When it is unfolded on the road, the positions of the holes appear differently, as shown in the images at the front of the view change 1 and change view 2 arrows. By using parallax from these two images, the three-dimensional appearance of this hole is synthesized. The same is true if two cameras are attached to the front and back, and the parallax or optical flow at the corresponding point of each image is extracted.

In this way, the unevenness of the road surface can be detected, and the depth of the hole and the like can be measured as shown in the lowermost figure in FIG.

Fig. 9 shows an example of the depth of a hole, for example.In addition to the depth of the hole, the asphalt swelling after road construction, or the rutting caused by driving a car, etc. Can be measured.

At present, there is an example of a device that measures the unevenness of the road surface in the world using a laser, but the price is considerably high. In the method of the present invention, as described above, the image processing is performed by software, so that the unevenness of the road surface can be measured by an inexpensive method. FIG. 10 shows a specific method for obtaining 0. The parallel line part in the real world (live-action video) is empirically searched from the image, and the extension of the parallel line is displayed as an intersection in the image, so it is a plane parallel to the target plane created by the intersection. Let d be the distance between plane a and plane b, which is a plane parallel to the target plane including the optical axis point, and let f be the virtual focal length.From the ratio of these d and f, 0 = arc T an (d / f) to find 0.

Here, an example of calculating the virtual focal length is as follows.The distance on the optical axis such that the angle at which an arbitrary object in the real space is seen in advance and the angle at which the same object in the displayed image is seen is the same. It is sufficient to find it on the display image. If the unit at this time is expressed in pixels, it will be a value specific to the camera system including the lens, and it will be sufficient to find it once. Furthermore, a parallel line portion of the object in the real world is searched for, and it is represented in the image as an intersection line having an intersection on the extension line, so that when this intersection line is expanded on a plane, it becomes parallel. By selecting 0 for で, can be obtained, and fine adjustment of 0 can be performed. FIG. 11 shows a method of measuring the actual value of 0 by developing a perspective image into a plane.

FIG. 11 (A) in FIG. 11 shows the perspective image as it is, and FIG. 11 (B) shows the state when the plane is developed with 0 being a certain value. At this time, the parallel lines in the real world, that is, the road surface, are not parallel lines in perspective but are line segments having intersections as shown in Fig. (A). Furthermore, if the value of 0 differs from the actual value when the plane is developed, the line segments on both sides of the road that should become parallel lines do not become parallel lines as shown in (B). On the other hand, when 0 is obtained correctly, the line segments on both sides of the road become parallel lines on the developed side view, as shown in (C). Since 0 was obtained, 0 can be obtained by this. Next, a case where a three-dimensional map is created will be described with reference to FIGS. 12 to 18. FIG. Here, a case will be described as an example in which a three-dimensional map is generated from a video taken in a substantially moving direction from a video camera mounted on a vehicle traveling on a road.

In FIG. 12, a moving image input unit 81 inputs an image taken in a direction substantially advancing from a bidet talent merchandise loaded on a vehicle running on a road. The image is decomposed into images by a multi-plane decomposition unit 82. Then, the reference plane designating section 83 interprets that the image is composed of a plurality of planes and sets a plurality of planes in the image. Set as the reference plane.

On the other hand, the arbitrary-purpose plane designating section 84 sets the street light plane assuming that a plurality of street lights and the like are in one plane because they are regularly installed, and similarly sets the curbstone plane and the street Multiple planes, such as a tree plane and the entire building plane, can be set. Fig. 13 shows an image of a plane extracted by optical flow. As shown in the figure, assuming that the vertical distance of the plane to which each object belongs from the standard position of the camera is D, all planes can be separated and extracted as a group of multiple parallel planes. At this time, for a curved object such as a street tree shown in the same figure, for a curved object having a point or surface that does not ride on one plane even if it is one object, By treating a curved surface as a group of multiple planes and giving a distance from a reference plane (street tree ① in the figure), it can be grasped as information on one object belonging to that plane.

The θ and h detection unit 85 reads the angle 0 between the road surface and the optical axis and the distance h between the camera system optical center and the road surface from the image.To read these automatically, The intersection lines by giving f and h in the above formulas (1) and (2) are set so as to be parallel when developed on a plane, or d and f related to the planes a and b This is due to 0 which is obtained from the ratio of, and this 0 may be read by actual measurement when it can be measured. The coordinate conversion unit 86 substitutes 0 and h into the plane expansion conversion formulas of the above-described equations (1) and (2) to perform an operation, and the image plane expansion unit 87 obtains a plane expansion image. . Opt. F (Optical flow) value calculator 8 8 divides the image into small areas because the video is a moving image, and calculates the optical flow of each part of the image by matching and correlation methods. The Opt.F (optical flow) map generator 89 displays the above calculation result as an image map.

The reference plane image extraction unit 90 obtains the reference plane image by extracting the reference plane image based only on the unique optical flow value indicating the road surface. In other words, the relative speed in the same plane is always constant on the road surface developed on a plane, so the optical flow is the same, and the road surface can be easily extracted. In addition, in a general perspective video, the relative speed changes even on the same plane in the image depending on the distance, so the reference plane cannot be extracted with the unique optical flow value. In addition, the comparison is not simple because the size changes depending on the distance. The parallel plane extracting unit 91 is configured to obtain an optical flow value different from that of the reference plane in the same manner as when extracting the above-mentioned reference plane. Since the plane parallel to the reference plane only has a different eigenvalue from the reference plane, the parallel plane can be obtained separately from the reference plane.

The plane image forming unit 92 treats the obtained planes as image planes as they are, so that an image within the set planes can be acquired. The three-dimensional map generation unit 93 generates a three-dimensional map having a configuration of a reference plane and a plane parallel thereto by assembling each parallel plane with three-dimensional coordinates. However, since not all objects can be represented using only the reference plane and a plane parallel to it, another plane different from the reference plane must be constructed in the same way.

In addition, one of the initially set planes can be handled in the same way as the reference plane, and can be generated in the same process as a three-dimensional map. Here, since 0 and h must be converted to their respective planes, 三 and h must be converted to their respective planes from the obtained three-dimensional coordinates of the reference plane. That is, the position of the arbitrary plane is calculated from the obtained three-dimensional image of the reference plane by the conversion of the S and h values on the specified plane and the specifying unit 95, and the calculation is simply manually converted. It is also possible. Similarly, the target image is subjected to the same processing through the 指定 r designation section (0, h designation section) 96, and a three-dimensional map is generated through the target plane image extraction section 97. It is generated by part 93.

Fig. 14 shows a plane extracted by optical flow and an image of a three-dimensional map generated by three-dimensional conversion. Further, the position and direction of the camera that acquires the video are detected by the processing of the θ and h detection unit 85, the coordinate conversion unit 86, and the image plane development unit 87 described above, and the reconstructed 3D map is displayed. And plotted. That is, by the processing of the detection unit 85, coordinate conversion unit 86, and image plane development unit 87 for Θ and h, the angle between the optical axis of the camera and the target plane such as the road surface is given, and the coordinates in the image are given. The camera position and the camera direction calculated by the calculation can be described in the image expanded on the plane by the conversion formula or the coordinates of the target plane by specifying the origin. The above equations (1) and (2) are calculated by giving the focal length of the camera, the angle between the road surface and the optical axis of the camera, and the coordinate origin to the target plane such as the camera optical axis and the road surface. Then, a plane developed image of the road image acquired by the camera can be obtained, and at that time, the camera position and the camera direction are obtained by calculation from the conditions of the conversion formula. As a result, the camera position and camera direction can be detected in the converted image and plotted on a three-dimensional map.

Also, a single image is generated by combining a plurality of images taken by a moving camera and developed in a plane, and the combined image represented is used as a new common coordinate system in the new coordinate system. The camera position and camera direction obtained by the above equations (1) and (2) can be described one after another. For example, a video from an on-board camera is flattened, the corresponding points on the target plane in each frame image are searched automatically or manually, and the corresponding points are combined so as to match, and a combined image of the target plane is generated. And display them in the same coordinate system. Then, the camera position and camera direction are detected one after another in the common coordinate system, and the position, direction, and locus can be plotted.

FIG. 15 shows an image of a state in which the locus of the camera position is described on a three-dimensional map. By detecting the camera position and direction in this way, the position of the object can be specified from the camera positions of a plurality of images, and a three-dimensional map or a three-dimensional image can be reconstructed from a two-dimensional image. Therefore, it is possible to automatically generate a three-dimensional map of the traveled range just by photographing with the on-vehicle camera or the like. In addition, since the camera position and direction can be detected in this way, a three-dimensional map or three-dimensional image reconstructed from a two-dimensional image Above, the position and direction in which the force camera moves can be plotted and described. Since the three-dimensional map can be generated as described above, a plurality of images obtained by planar decomposition of the perspective image and the position information of the camera can be generated, and a desired three-dimensional image can be reconstructed from the information. By transmitting the plane decomposition image and the camera position information, it becomes possible to reconstruct the original moving image by reconstructing it on the receiving side.

Conventionally, as a method of compressing moving images, as in the case of the MPEG-2 method, the main method is to separate moving parts in a moving image, predict the movement, and eliminate signal redundancy. It has been known. However, this type of conventional method is effective for partial movement, such as when there is a moving object on a still background, but for moving images where the camera itself moves, Since the whole image has a motion component and the moving speed is not the same, the entire video must be updated, and the compression effect is significantly reduced.

As described above, in the three-dimensional map generation according to the present invention, a moving image is analyzed and treated as an image composed of a three-dimensional plane. As a general property of images, since the image space is surrounded by multiple planes, the image can be reconstructed by extracting each plane with this device and reconstructing those planes . Since the plane is defined three-dimensionally, the reconstructed plane is placed in the three-dimensional space, so that the final image is a three-dimensional image. Therefore, as long as the camera moves in a certain direction, the relative speed between the camera and the plane is constant, and the speed reading is uniquely required for each plane. Each plane has its own velocity component in the range in which the force mea- sure moves at a constant speed.

In this way, the image is compressed and the receiving side can reproduce the original moving image by moving each plane at a defined speed. Moreover, since the image contains three-dimensional information, it can be expressed three-dimensionally. .

As described above, according to the present invention, it is possible to give a sufficient compression effect even to an image having a motion component in the entire image due to the movement of the camera itself, and having a different motion speed component in each part in the image. Moreover, since the image is three-dimensionally analyzed at the time of image compression, a three-dimensional image can be extracted from the moving image as a result. FIG. 16 shows another embodiment, and the same parts as those in the embodiment shown in FIG. 12 are denoted by the same reference numerals, and the detailed description thereof is omitted. Here, when a moving object such as a preceding vehicle or an oncoming vehicle is included in the input image, the optical flow of the reference plane takes a value different from the reference plane and an Opt. F value parallel thereto. Therefore, in the Opt.F map formed by the 〇pt.F (optical lip map) map generator 89, the 0 pt.F value which is an abnormal value existing on the road surface (reference surface) is detected. Then, that part will be the area of the moving object on the road surface. If the purpose is to delete the moving object, this moving object area is extracted and deleted by the moving object Opt. F (optical flow) extraction unit 101 and the moving object part extraction unit 102, and the deletion is performed. The region may be complemented from the overlapping developed images before and after. Reference numeral 103 in the figure denotes a simple moving body relative speed extraction unit that simply extracts the relative speed of the moving body.

Furthermore, for the purpose of extracting the moving object itself, reproducing its three-dimensional shape, and measuring the speed, the processing process on the right side of FIG. 16 is performed via the moving object plane designation unit 105. . However, the entire Opt.F value of the moving object area obtained on the flattened reference plane does not directly mean the Opt.F value specific to the moving object such as a vehicle. To determine the shape and the Opt.F value, the moving body is further decomposed into multiple planes by the Opt.F (optical flow) extraction unit 106 and the moving object plane extraction unit 107, etc. The three-dimensional shape of the moving object can be obtained by the moving object plane image forming unit 108 by calculating the plane development of each plane constituting the moving object in the same process. The speed vector of the moving object can be obtained by the vector extraction unit 109. Furthermore, it can be taken into a three-dimensional map by a three-dimensional map generation unit 110 that includes a moving object. Next, a method of pasting the texture of an acquired image to a CG (computer graphics) image or the like as an embodiment of an application example of the three-dimensional map generation will be described with reference to FIG.

That is, the image developed on the plane has acquired three-dimensional coordinates, and Because only the image of the road surface was extracted, the texture of the building wall, which was partially shaded by a street tree at the time of shooting, could be obtained by deleting the image of the street tree. Of course, by aligning the plane image that composes the desired building wall surface with the CG image and pasting it, the CG image can be used to add the texture of the building wall of the video image to the CG image and the street. Trees, guardrails, etc. can be attached.

In FIG. 17, the same parts in the embodiment shown in FIG. 12 are denoted by the same reference numerals, and their detailed description is omitted. Opt. F (Optical flow) While the parallel plane image is extracted from the map generation unit 89 by the parallel plane image extraction unit 111, the target plane image extracted from the target plane image extraction unit 97 Then, a texture signal is obtained through a texture signal generator 1 1 and 2. Also, three-dimensional coordinates are acquired by the three-dimensional coordinate acquisition unit 113 from the plane image forming unit 92, and the three-dimensional coordinates are matched by the CG coordinate matching unit 114 to match the coordinates with the CG image. The pasting section 1 15 to the CG together with the signal from the generating section 112 obtains a composite image on which the texture is pasted. FIG. 18 shows an embodiment in the case where a three-dimensional map is constituted by recognized parts, and shows the same configuration in the embodiment shown in FIGS. 12 to 17. Are denoted by the same reference numerals, and their detailed description is omitted.

In other words, if attention is paid to the object on each of the developed plane and the virtual plane, the optical flow of the object in the image depends only on the relative speed between the camera and the object. Objects can be tracked, and speed can be extracted even for moving objects, making it easy to track them.

If the object can be tracked, image recognition can be facilitated by using the position and shape change of the object with respect to time change as a clue for recognition, and the object in the image can be directly compared to the 3D CG model without using tracking. Can be replaced. Therefore, a specific object existing in the screen is selected and extracted from the image formed by the planar image forming section 92 by the object selection and tracking section 122, and the object is tracked. Through the object recognition section 1 2 2 Then, it is input to the attribute adding section 123 for adding the attribute of the object.

On the other hand, from among the images formed by the moving object plane image forming unit 108 and the moving object velocity vector extracting unit 109, a specific object existing in the screen is displayed by a moving object selection tracking unit 124. A mobile object is selected and extracted, and the mobile object is tracked. The attribute adding unit adds the attributes of the mobile object through the mobile object recognition unit 125 that recognizes the attributes of the mobile object and other various information. It is entered in 1 2 6. The three-dimensional map is generated by the three-dimensional map generation unit 127 composed of the recognition target object from the attribute addition units 123 and 126, respectively. Hereinafter, a specific example in which the object recognition and tracking processing using a planar developed image is applied to the image analysis of a traffic monitoring video camera will be described with reference to FIGS. 19 to 21.

FIG. 19 is a specific example of an image obtained by a traffic surveillance video camera. FIG. 19 (a) is a surveillance video camera image (perspective image), (b) is a plane developed image converted by the present invention, (C) is an area analysis display showing the object recognized according to the present invention.

FIG. 20 is a flowchart showing processing steps of object recognition in the traffic monitoring video camera. Fig. 21 is a list that summarizes the results obtained by object recognition.

In the examples shown in these figures, the types of passing vehicles, vehicle color, traffic volume, speed, acceleration, and the vehicle trajectory in the surveillance video camera images are obtained from the traffic monitoring video camera images by image recognition. Has become.

First, the surveillance video image obtained by the traffic surveillance video camera is a perspective image (oblique image) as shown in Fig. 19 (a), and the size and speed of the target vehicle are not uniform. Absent.

This perspective image is digitized and planarly developed to obtain a planar developed image of the present invention. The resulting plan view is as shown in Fig. 19 (b). In this plane-development image, the parameters f and 0 are determined so that the road surface is developed into a plane, so on the road surface, except for the vehicle height, such as the width and length of the vehicle, The scale is uniform and measurable even at the position. Therefore, the traffic volume Image recognition for monitoring can be performed.

Conventionally, it is common practice to perform vehicle recognition and recognition on perspective images (see Fig. 19 (a)), limit the vehicle recognition area to a part of the screen, and detect and measure vehicles passing through the area. It was a target. In the present invention, since the perspective image can be developed on a plane, the entire road surface of the image can be used as a vehicle recognition range by using the plane developed image (see FIG. 19 '(b)).

In the example shown in Fig. 19 (b), the moving object (vehicle) captured at the upper part (upper side) of the two-dimensional image moves to the lower part (lower side) of the image, which is the traveling direction. At this time, since the size of the vehicle image does not change in the two-dimensional image, a plurality of images can be obtained for the same vehicle. For example, a moving object image moving from the upper part to the lower part of the two-dimensional image can be acquired as image data for about 30 frames.

By using the plurality of images, a detailed image analysis (area analysis) can be performed. For example, the accuracy of identifying a vehicle type can be improved, and accurate image recognition can be performed. Figure 19 (c) shows the area analysis display.

In the case of a plane developed image, for example, in order to increase the recognition accuracy, it is possible to perform processing such as averaging of images and averaging of processed contour images, which is very effective in image processing. In addition, since the scale is the same on any part of the road surface on the planar developed image, position information can be easily obtained, and the movement trajectory of the moving object can be tracked along with recognition. In addition, since the scale is the same, the vehicle position and moving speed (calculated from the number of video frames per second) can be measured anywhere on the road surface, and acceleration and speed can be easily calculated. It is. With reference to FIG. 20, the flow of the traffic volume recognition process using the above-described plane development technology will be described in more detail.

First, the captured perspective image is digitized (201 in FIG. 20), and a plane developed image is created from the perspective image (202).

Then, the moving object region is detected in the plane developed image, and the candidate region is created by comparing the background image with the current image to generate a candidate region (203). The candidate region is subjected to image processing by image calculation with the background image, and a region having a small independent area is removed for vehicle detection, and the image region of the vehicle candidate is specified by expansion coupling of the remaining region. Ma In addition, the background image is updated using Kalman Phillips or the like (2003). The specified candidate area is subjected to processing such as changing the threshold value of image processing, and analyzed in detail (204). In addition, since the movement amount of the vehicle candidate region can be predicted by expanding the plane, when the vehicle presence region image is obtained, the vehicle presence region is predicted, and the shape correction such as the size of the image and the position shift is performed. Adjustments are made and the corresponding area to be used is analyzed and determined (Same as in 2005).

The determined corresponding area is registered on a data basis together with its location information (see

206). After that, an averaging image is created using the images registered in the database (207).

Then, the vehicle type is determined using the averaging image (2008), and is registered in the database together with the recognition result (2010).

There are various types of information registered in the database. For example, as shown in Fig. 21, vehicle ID, passing time (Passed Time), average speed (Speed), acceleration (Acc), vehicle type (Type) Information on various items required for vehicle recognition, such as) and color, is registered. Of course, the information is not limited to the items shown in FIG. 21 and other information can be registered.For example, in addition to the above items, a vehicle average image, a vehicle trajectory, and the like can be registered. . In order to replace the object in the image with a 3D CG model by direct comparison, refer to Japanese Patent Application No. Hei 11-975 / 62, Japanese Patent Application No. 2000-1990, Japanese Patent Application No. It is based on the method, equipment, etc., which are disclosed in the application such as 200 2-1 4 6 8 7 4 etc.For example, the object can be recognized, specified, fixed, tracked, and the object can be tracked. Can be replaced with a 3D CG model. An outline of the method and apparatus is an information code conversion device that converts object information acquired for an object into an information code registered in advance corresponding to the object, and transmits or outputs the information code. And a reproduction conversion device that receives or inputs an information code from the information code conversion device and converts the information code into reproduction object information registered in advance corresponding to the information code. . In addition, information input means for inputting information on required objects, information on various objects or their parts and their attributes and the like created in advance, and data obtained by encoding the information are stored. Form a database An information code conversion for comparing and comparing the information input to the information input means with the information stored in the first component storage, and selecting and outputting data relating to the corresponding information; An apparatus, a second parts warehouse that forms a database similarly to the first parts warehouse, and a data output from the information code converter is compared with data stored in the second parts warehouse. An information reproduction conversion device that selects information for reproducing the corresponding object and reproduces and outputs the object by required output means based on the information for reproducing the object. Furthermore, one or more objects at arbitrary positions can be specified by name or attribute on the image display of the video image taken by one or more cameras with the one or more cameras, or a single point position can be specified by enclosing the object with a mouse. By clicking or specifying with the light pen or sunset panel, the target object including the change in the relative angle and direction between the camera and the target object and the change in the distance while excluding the existence around the target object are excluded. While tracking the object, the features of each image frame of the object or the features of each component of the object are sequentially detected in chronological order, and a wealth of image data on various features is stored. From the database, image frames that have a correspondence with the feature of each image frame of the object or the continuous change of the feature of each component, or image data of the feature A search is performed sequentially, and a reproduction image including a two-dimensional or three-dimensional shape corresponding to the target object that has been subjected to pattern matching in accordance with the search result is sequentially formed for each time-series change. Each image frame including an image or the image data of the feature is matched with the required size standard setting on the required image on the above image display or other image display via a communication line, etc. Are displayed sequentially in contrast to each other as a group of still images, and the name and attribute data attached to or generated with the reproduced image are also displayed in a required area as necessary.

In this way, by adding the attributes previously prepared for each object to the 3D CG model, a 3D map is constructed by combining and accumulating the 3D CG models of the replaced objects In addition, the above-mentioned texture pasting makes it possible to paste the actual texture into the three-dimensional CG model of the object. In the embodiment shown in FIGS. 22 to 27, for example, This enables continuous connection in the moving direction so that a surrounding image of a moving object such as an object can be acquired. That is, images in the moving direction obtained by mounting a camera on a moving vehicle, an aircraft, a ship, or the like and capturing the images are developed in a plane, and continuously combined into one image. In this method, a camera is mounted on a moving vehicle, aircraft, ship, etc., which sequentially captures diagonal images, for example, captures a road image along a road, develops it on a plane, and joins a single road. It is to get a video.

Specifically, as shown in FIG. 22, for example, the situation of a camera for photographing a road around a bus 201 as a vehicle is described.In this case, six cameras are used. If necessary, the number may be further increased. The first camera 201, which captures the front of the bus 201, is at the front of the bus 201, the second camera 201, which captures the rear, is at the rear of the bus 201, and the bus 201 is at the rear. Pass the third camera 201 C to the right rear of the bus with the third camera 201 C to shoot the right front, and the fourth camera 201 D to the left left of the bus 201 to shoot the left front of the bus 201. A fifth camera 201 E for photographing the rear right is provided at the front right of the bus 201, and a sixth camera 201 F for photographing the rear left of the bus 201 is provided at the front left of the bus 201. The area photographed by these cameras 201 A... Is shown in a fan-shaped diagram. In this way, the images of the six types of road surfaces are captured in perspective, and are developed into a two-dimensional image such as a map by the above equations (1) and (2). The coordinates (u, V) above are converted to coordinates (X, y) converted on the map.

In this way, the perspective image obtained by shooting around the bus 201 is converted into an image of the road surface such as a map, as shown in FIG. 23. A road surface like a map around 0 1 can be displayed. Furthermore, if these plane-deployed images are combined as far as possible in the traveling direction of the bus 201, a map will be completed. FIG. 24 and FIG. 25 show specific examples, and FIG. 24 shows oblique images of nine roads (1) to (9). FIG. 25 shows this developed into a two-dimensional image such as a map by the above-mentioned equations (1) and (2), and then connected and displayed as a single road like a map. This is an example of plane development of an image taken by a normal camera, that is, an oblique image, and an example of combining plane development images. At this time, if there is a moving object image that is unnecessary for image formation, it can be deleted, and when combining multiple images that partially include overlapping objects, if the moving object is By combining the images while avoiding the image of the moving object, a combined image of only the stationary object is generated. For example, when a vehicle or the like is shown on a road, a combination of images that are developed on a plane while avoiding the vehicle or the like is combined and connected to obtain a long road photograph that shows only the road. Next, some of the application examples according to the present invention will be described as follows.

Road surface, sea surface, lake surface, river surface, ground surface, vertical wall surface, vertical virtual plane created by objects arranged on the same plane, architectural wall floor surface, ship deck surface Airport facilities such as taxiways can be handled. This is because when an image taken with a normal camera is developed on a plane, the objects on the plane development surface include road surface-sea surface · lake water surface · river surface · vertical wall surface · objects arranged on the same plane Vertical virtual planes to be created · Ground surface · Building wall floor · Ship deck · Airport facilities such as runway taxiways.

Vehicles can also be used as an example of applied equipment, such as the surrounding road surface of a land vehicle such as a bus, the surface of a building, the arrangement of telephone poles, the arrangement of street trees, the arrangement of guardrails, etc. It is the sea surface of a vehicle, such as the deck or wall surface of a ship, the runway of an aircraft, the ground surface, etc., which enables the display of all directions or the target area.

That is, as shown in FIGS. 22 to 25, the surrounding road surface, the building surface, the array of telephone poles, and the like from a normal camera 201A attached to a land vehicle such as a bus 201, etc. Develop images of the arrangement of street trees, guardrails, etc., the sea surface of vehicles such as ships, the decks and walls of ships, the runway of aircraft, and the ground surface, etc. in a plan view. It enables the omnidirectional display of the surrounding roads, buildings, telephone poles, street trees, guard rails, and so on. Alternatively, although not shown, the omnidirectional display of the marine vehicle and other vehicles on the sea, and the omnidirectional display of the ship's deck and wall surfaces, etc. Can be done. Further, as another application example, the present invention can be applied to a building structure, and as shown in FIGS. 26 and 27, for example, a plane portion such as a floor surface or a wall surface of a building is displayed in a plane-expanded manner, and This enables plane combination display, in which the inside of the building is photographed with a normal force camera, and flat parts such as the floor surface and wall surface are displayed in plane development and plane combination display.

That is, FIG. 26 shows an oblique image of a room taken by a normal camera. For example, for the 16 images (1) to (16), the above equation (1) is used. The two-dimensional images are converted by (2) and (2), and they are connected as shown in FIG. In this way, an image that cannot be actually captured can be obtained as an image in which the floor and walls in a building room are developed, and an image in which the surrounding wall is developed with respect to the floor can be generated. It is.

In addition, it enables the creation of three-dimensional maps, as well as the continuous shooting of road, ground and water surfaces with moving cameras, aircraft, ships, etc. By continuously photographing a surface or an object having a virtual vertical plane in which a plurality of telephone poles, guardrails, etc. are regularly arranged in a plane, the plane-expanded image is simultaneously extended while being coupled in the moving direction. A 3D map is created by creating a wider view of the vertical plane, including the vertical plane.

That is, images of the road surface, ground surface, and water surface captured by a video camera mounted on a moving vehicle, such as a vehicle, aircraft, or ship, are developed into a plan view, and they are combined by an appropriate method. A map is created by drawing a figure and extending it in the direction of movement. Alternatively, by continuously photographing an object having a vertical surface such as a building wall surface, or a virtual vertical plane in which a plurality of utility poles, guardrails, and the like are regularly arranged in a plane, the image developed in the plane can be obtained. A three-dimensional map is created by creating a wider vertical plane development map, including the vertical plane, while extending the connection in the movement direction. Industrial applicability

Since the present invention is configured as described above, an image that cannot be actually photographed is converted by using the equation (1) and the equation (2) by using the present method and apparatus. By doing so, the oblique image can be converted to a planar image. The application range ffl is also broad.

In addition, it is possible to detect parallax in the plane developed image of the overlapping part by generating a plane developed image from multiple images of the same point taken from different points by multiple cameras with overlapping viewpoints. . That is, conventionally, the detection of the three-dimensional data by the detection of the parallax has always been obtained from the original image, that is, the perspective image itself, but in the present invention, the parallax is detected after performing the planar development image processing. With this new method, three-dimensional data with high positional accuracy can be obtained. In addition, this makes it possible to easily obtain data of a direct three-dimensional shape with higher accuracy than in the past, and the parallax obtained from a plurality of videos having overlapping visual fields is not suitable for the static coordinate form in the video. It often has the same meaning as the physical flow, but is particularly effective when the object changes over time or when the accuracy of three-dimensional coordinates is increased.

The desired moving image can be reconstructed by transmitting and receiving the planar image information and the camera position information, and moving image data can be transmitted and received at high speed while minimizing the data transmission amount. This is particularly effective for video transmission using narrow-band telephone lines and Internet lines.

Claims

The scope of the claims

1. It specializes in reading the information necessary for plane development from perspective images obtained from a normal camera, developing it into a plan view, combining and joining it together, and creating a single large development view. A planar development image processing method for planar elephant images such as road surfaces.

2. Read the information necessary for plane development from the perspective image obtained from a normal force camera, develop it into a plan view using the following equations (1) and (2), combine them, and connect them. A planar development image processing method for planar target images such as road surfaces, which is combined into a single large development view.

y = V · 2 ¹² · · cos (π / 4-Θ)-cos (ι3-θ) / (f · si η 3) (1) x = u · · cos (β-θ) / (f · sin i3) (2) where 0 is the angle between the camera's optical axis and the road surface: f is the focal length of the camera, h is the height of the camera, and i3 is the point at h + y from directly below the camera. , The angle between the line connecting the camera and the road surface, V is the vertical coordinate from the origin on the CCD surface, which is the projection surface of the camera, u is the horizontal coordinate from the origin on the CCD surface, and y is the road surface In this case, the point that is h from directly below the camera at the origin is set as the origin, and the distance or coordinate further advanced in the optical axis direction from that point, and X is the lateral distance or coordinate on the road surface.

3. With respect to an image of a real scene object including a plane that is photographed obliquely, a plane originally composed of a plane is developed into a plane as a plane image that is proportional to the plane of the real scene by mathematical operation. 3. The method of processing a planar object image such as a road surface according to claim 1 or 2, wherein the image is processed and displayed.

4. Combining a plurality of planar developed images and expressing them as one large planar developed image Planar development of a plane object image such as a road surface according to any one of claims 1 to 3 above. Image processing method.

5. Develop multiple input videos on a plane and combine them into a single image. The road surface or the like according to any one of claims 1 to 4, wherein an entire area of the area is displayed, and, if necessary, an input image corresponding to the displayed location is also simultaneously displayed directly. A flat object image processing method for a two-dimensional object image.

6. The image of a plane object such as a road surface according to any one of claims 1 to 5, wherein the image of the moving direction of the moving object is developed in a plane and continuously combined to form one image. Plane development image processing method.

7. The method according to any one of claims 1 to 6, wherein a moving object image unnecessary for image formation is deleted, and the moving object image is combined to avoid the moving object image to generate a combined image of only a stationary object. A flat developed image processing method for a plane object image such as a road surface. '

8. In the process of finding the vanishing point in the acquired plane image, the image is moved and displayed so that the position of the vanishing point of each image is fixed, thereby stabilizing the image that shakes due to camera shake or the like. 8. The planar development image processing method for a planar object image such as a road surface according to any one of Items 1 to 7.

9. In the moving image composed of a plurality of different planes obtained by plane expansion, the amount of unit time movement of a minute area in the moving image is determined within the necessary range by the optical flow method, and the same component is determined from the component distribution diagram. 9. The plane developed image processing method for a plane object image such as a road surface according to any one of claims 1 to 8, wherein each of the plane images is separated by extracting the plane image.

10. In a plane moving image obtained from the plane-converted moving image, a distribution map of optical flows is generated, and the unevenness of the plane is detected as a deviation from the plane from the small difference, or in the plane moving image, Parallax is detected by comparing and calculating plane images obtained from different angles of view, and unevenness components within the plane are detected from the component distribution. 10. The method for processing a planar developed image of a planar object image such as a road surface according to claim 1, wherein the modified planar view including the deviation is generated.

11 1. By performing a comparison operation on a plurality of plane images developed on a plurality of plan views by a method such as a correlation method or a matching method, each of the small areas on the plurality of plane images such as a road surface can be obtained. The amount of movement of the corresponding small area is calculated by the parallax method or optical method, and three-dimensional data such as unevenness on the road surface is detected from the distribution of the components. 10. The plane development image processing method for a plane object image such as a road surface according to claim 1, wherein a corrected plane view including a deviation of each point from a plane is generated.

1 2. Calculate the average optical flow value of the continuous image developed on the plane or the moving distance of the matching position, and calculate the moving distance, moving speed, moving direction of the target plane, or the moving distance of the photographed camera, moving from the value. The method for processing a plane developed image of a plane object image such as a road surface according to claim 10 or 11, wherein the speed and the moving direction are obtained.

1 3. Paste the texture of the object plane in the isolated plane developed on the separated plane into the corresponding CG (computer graphics) image of the place or the object plane in the map image. A plane object image such as a road surface according to any one of claims 1 to 12, wherein a real image is taken into an image or a map image, and displayed as a plane or inversely transformed and displayed as a perspective image. 2D image processing method.

1 4. In formulas (1) and (2) described in claim 2, when f and h are given first, and the parallel line of the object is an intersection that forms an intersection in the image A plane such as a road surface according to any one of claims 2 to 13, wherein 0 is selected so that the intersection line is parallel when the plane is developed in a plane. A planar development image processing method for an object image.

15. The planar development image processing method for a planar object image such as a road surface according to claim 14, wherein a selected color is finely adjusted in claim 14.

1 6. The parallel lines in the actual video are extracted from the image, and the plane a that is parallel to the target plane created by the intersection and the plane b that is parallel to the target plane including the optical axis point The distance between and is assumed to be d, the virtual focal length is assumed to be f, and 0 is obtained as 0 == arcT an (d / f) from the ratio between these d and f. A planar development image processing method for a planar object image such as a road surface according to any one of the five items. ·

1 7. Simultaneous images at the same point are acquired by multiple cameras installed at different locations, and parallax is detected by comparing and calculating the two-dimensional images of the simultaneous images at the same point. 17. The plane development image processing method for a plane object image such as a road surface according to claim 1, wherein a three-dimensional shape of the object is generated from parallax.

1 8. Either a flat developed view generated by converting a perspective image including a two-dimensional image into a plan view, or an image containing a plurality of two-dimensional images shot from multiple directions into a plan view. The expression (1) described in Claim 2 is based on a single large-screen flat-panel view or a CG (computer graphics) image or map generated by combining and superimposing corresponding points later. ) And (2) to generate a virtual perspective image viewed from an arbitrary viewpoint by the inverse transformation formula, or to generate a moving image from the moving camera viewpoint by processing continuously. 20. The planar development image processing method for a planar object image such as a road surface according to any one of Items 2 to 17.

1 9. The plane development image processing method for a plane object image such as a road surface according to claim 18, wherein the inverse transformation equation is the following equation (3) or (4).

V = y · f · sin / 3 / (2 ¹² · h · cos (π / 4-θ)-cos (β — θ))

(3) u = x-f-si β / (-cos (β-θ)) (4) where h is the height of the camera from the road surface, 0 is the angle between the optical axis of the camera and the road surface, f is the focal length of the camera, / 3 is the angle formed by the line connecting the camera lens with the point advancing y from the point h advanced from just below the camera to the road surface, and X is the camera V is the coordinate in the vertical direction from the line segment obtained by orthogonally projecting the optical axis of the road surface, that is, the horizontal direction when viewed from the camera, y is the coordinate in the optical axis direction with the point advanced h from directly below the camera as the origin, V Is the vertical coordinate on the CCD plane, which is the projection plane of the camera, and u is the horizontal coordinate on the CCD plane.

20. The road surface according to any one of claims 1 to 19, wherein various types of recognition processing, such as measurement processing and image recognition processing, can be performed on the image using the image developed on a plane. And the like.

2 1. Planned images are road surface, sea surface, lake water surface, river surface, ground surface, vertical wall surface, vertical virtual plane created by objects arranged on the same plane, building wall floor surface, ship deck surface- 22. The method for processing a planar developed image of a road or the like according to any one of claims 1 to 20, wherein the method is a plane on an airport facility such as a runway taxiway.

2 2. The plane development image acquired by the moving object is the road surface, pill surface, array of utility poles, array of street trees, array of guardrails, etc. on the land moving object itself. The surface of a road or the like according to any one of claims 1 to 21 which is a surrounding sea surface, a deck of a ship, a wall surface, or the like, and is a runway, a ground surface, or the like of the moving object in the air itself. Expanded image processing method.

23. The plane development image is a plane development display of a plane portion such as a floor surface or a wall surface of a building, and a plane combination display, such as a road or the like according to any one of claims 1 to 22. Plane development image processing method.

2 4. Continuously photograph the moving road surface, ground surface, and water surface with multiple video input devices, and a vertical surface such as a building wall, or a plurality of telephone poles, guardrails, etc. are regularly arranged in a plane. A 3D map is created by continuously photographing an object with a virtual vertical plane, and by combining and extending the flattened image in the direction of movement, and simultaneously creating a broad vertical plane development map including the vertical plane. The planar development image processing method for roads and the like according to any one of claims 1 to 23, wherein

2 5 .—Convert not only a plane in one direction but also a plane image in a plurality of directions in an image including planes in a plurality of directions into a plurality of plan views, or display the plane in the plurality of directions. Generates a three-dimensional planar development drawing obtained by combining images three-dimensionally, and generates a perspective image viewed from any viewpoint by inverse transformation. Reverse expansion image conversion processing method.

26. A video input unit for acquiring a perspective image, a video playback unit for playing back oblique video shot by this video input unit, an image correction unit for correcting the shooting rotation angle and the like by the video input device, Spherical aberration corrector for correcting spherical aberration etc. in an image input device, image expansion plane processor for converting a perspective image into a plane expansion view, and developed image connection for combining images that have undergone B image expansion processing And a display unit for displaying a combined image.

27. An optical flow map generation unit that generates and illustrates an optical flow of a developed video, and an optical flow extraction unit that extracts only a target optical flow from an optical flow map 26. A plane development image processing apparatus for a plane object image such as a road surface according to Item 26. '

28. The plane of a plane object image such as a road surface according to claim 26 or 27, further comprising a parallax extraction unit that detects parallax from images of the same point from different positions.

29. The plane of a plane object image such as a road surface according to any one of claims 26 to 28, further comprising a developed image comparison unit that compares developed images at a plurality of same points.

30. The road surface according to any one of claims 26 to 29, further comprising: an image comparison unit that extracts road surface unevenness by calculation, and a corrected plane surface generation unit that considers the unevenness. And a plane development image processing apparatus for plane object images.

3 1. An image input unit that generates an image with a camera, an input image display unit that stabilizes and displays an input image, a video recording unit that records an input image, and a video playback unit that plays back a recorded image. Image correction unit that performs coordinate transformation to correct image distortion due to lens such as spherical aberration, and adjusts the direction of the target flat image to the plane in the image to correct the camera rotation angle, and mathematical operation An image expansion plane processing unit that generates a plan view from a perspective image, an optical flow map generation unit that generates and illustrates an optical flow of the expanded image, and an objective optical from those optical flow maps An optical flow extractor that extracts only the flow, a parallax extractor that detects parallax from the video of the same point from different positions, and an An object image processing unit that deletes a new image and further introduces a new image; a developed image combining unit that combines the processed individual images that have been flattened to generate one continuous image; A developed image display unit for displaying them, a recording unit for recording them, an arbitrary viewpoint image generation unit for inversely converting to an arbitrary viewpoint for display, and an arbitrary viewpoint image display unit for displaying the image. A road surface or the like characterized by appropriately combining a developed image comparison unit that compares developed images of points, an image comparison unit that extracts road surface irregularities by calculation, and a corrected plane generation unit that considers the irregularities. Planar image processing system for planar object images.

32. An arbitrary viewpoint image generating unit for performing an inverse conversion to an arbitrary viewpoint and displaying the image, wherein the plane development image processing apparatus for a plane object image such as a road surface according to any one of claims 26 to 31 is provided. And an arbitrary viewpoint image display unit for displaying the image. A reverse-developed image conversion processing device for a plane object image such as a road surface, characterized in that the device is provided with an arbitrary viewpoint image display unit.

33. A video input unit for acquiring a perspective image, a plane decomposition unit for decomposing a perspective image captured by the video input unit into one or more plane images forming a three-dimensional space, and a video input unit A three-dimensional image is reconstructed and displayed from the position detector that detects the three-dimensional position of the image, and the three-dimensional position of the image input unit detected by the plane image decomposed by the plane decomposer and the position detector. A flat developed image processing apparatus for a plane object image such as a road surface, comprising:

3. The road surface according to claim 33, further comprising a position notifying unit that indicates a three-dimensional position of the video input unit detected by the position detecting unit in the plane image decomposed by the plane decomposing unit. And a plane development image processing apparatus for plane object images.

3 5. When the video input unit moves, the position notation unit continuously writes the three-dimensional position of the moving video input unit in the plane image decomposed by the plane decomposition unit. Item 4. A plane development image processing apparatus for a plane object image such as a road surface according to item 4.

36. When the display unit for reconstructing the three-dimensional image is disposed separately from the plane separation unit and the position detection unit, one or more plane images are displayed on the display unit from the plane separation unit and the position detection unit. A plane developed image of a plane object image such as a road surface according to any one of claims 33 to 35, further comprising transmission / reception means for transmitting a signal and a three-dimensional position signal of the image input unit.