JP3282332B2 - Image input system - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention generates image data showing the shape of an object from an object having a relatively complicated shape,
The present invention relates to an image input system that performs image bonding based on the image data.

[0002]

2. Description of the Related Art Conventionally, for example, among the means for recognizing a three-dimensional shape of an object, a light cutting method is often used as the most practical means. The light cutting method irradiates a slit-shaped laser beam S to a target object as shown in FIG.
Is captured on the imaging surface of the camera. Then, the spatial coordinates of the point p on the target object corresponding to a certain point p ′ on the slit are represented by a plane S formed by the slit light and a straight line L connecting the point p ′ and the center point O of the lens of the imaging device. Is obtained as the coordinates of the intersection of in this way,
The spatial coordinates of a point group on the object surface corresponding to each point on the slit image are obtained from one image, and three-dimensional information of the entire target object is obtained by repeating horizontal movement of the slit light and image input. be able to.

[0003]

In the conventional method, in order to improve the measurement accuracy, it is necessary to increase the base line length, increase the focal length of the lens, or use a high-precision CCD. However, there are drawbacks such as the apparatuses becoming too large, a large subject being unable to be taken, and an excessively large data amount.

[0004] The present invention eliminates the drawbacks of the prior art, inputs a plurality of image data with an accuracy adapted to the shape of the object by judging the shape of the object, and then synthesizes the plurality of image data. The purpose of this is to generate an integrated image.

[0005]

In order to achieve the above object, an image input system according to the present invention comprises: an image input system for inputting image data indicating a shape of an object through an optical system set to a first focal length; First input means, second input means for inputting image data indicating a shape of the object via an optical system set to a second focal length shorter than the first focal length, and A combining unit for combining a plurality of pieces of image data input by the first and second input units. Further, the image input system of the present invention comprises a first input means for inputting image data indicating the shape of the object at a first magnification, and a second input means which is lower than the first magnification.
A second input means for inputting image data indicating the shape of the object at a magnification of; and a bonding means for bonding a plurality of image data input by the first and second input means. It is characterized by.

[0006]

According to the structure of the present invention, image data is selectively input with high precision only to a complicated portion of the shape of an object, and is bonded to image data with low precision.

[0007]

Embodiments of the present invention will be described below with reference to the drawings. First, FIG. 2 is a schematic block diagram of the entire apparatus according to the present invention. This apparatus can be roughly divided into a light projecting optical system 2 for irradiating a laser beam output from a semiconductor laser 5 as a slit-shaped light beam to a target object 1, and a light receiving optical system for guiding the irradiated laser beam to image sensors 24 and 12. There is a system 3, and the light projecting optical system and the light receiving optical system are arranged on the same rotating base 4. In addition to the optical system, the optical system includes a signal processing system that processes a signal output from the sensor to generate a pitch shift image and a color image, and a recording device that records the generated image. The solid arrows shown in FIG. 2 indicate the flow of electrical signals of the image signal and the control signal, and the broken arrows indicate the flow of projected light. A detailed description of these optical systems will be described later.

The outline of the signal processing system will be described. The image obtained by the distance image sensor 12 is obtained by subtracting the image 18a when the slit light is projected and the image 18b when the slit light is not projected, and subtracts the image. Incident light centroid position calculation processing 1
9. Pitch shift information calculation processing 20 and pitch shift image generation processing 21 are performed. The pitch shift image obtained is NTS
The data is output to the output terminal 50 by the C conversion process 27 or transferred to the SCSI terminal 49 or the built-in recording device 22 as digital data for use. The image obtained by the color image sensor 24 is converted into an analog
, A color image generation process 26 is performed. The obtained color image is subjected to NTSC conversion processing 28 and output terminal 51
To the SCSI terminal 49 or the recording device 22 as digital.

FIG. 3 is a perspective view showing a schematic configuration of the entire apparatus. In this embodiment, the length of the slit light is 2 in the length direction.
A 256 × 324 distance image generation system having distance information of 56 points and 324 points in the slit scanning direction will be described as an example. The LCD monitor 41 displays a color image picked up by the color image sensor 24, three-dimensional data recorded in a recording device inside or outside the apparatus, various kinds of information, a selection menu, and the like. The cursor key 42, the select key 43, and the cancel key 44 are operation members for selecting an image, setting various modes from a menu, and the like.
A zoom button 45 changes the focal length of the light projecting / receiving optical system, and an MF button 46 performs manual focusing. A shutter button 47 captures a distance image when turned on in a shutter mode described later. As a recording device of the captured image, a magneto-optical disk (hereinafter, referred to as MO) or a mini disk (hereinafter, referred to as MD) built in the device.
). Terminal 4
9 is a terminal for digitally inputting / outputting a signal such as an image.
SI or the like. A pitch shift image output terminal 50 and a color image output terminal 51 are terminals for outputting an image as an analog signal, and output a video signal such as NTSC.

The light projecting optical system scans slit light long in the horizontal direction in the vertical direction. Light from the semiconductor laser 5 is rotated by a polygon mirror 7, a condenser lens 10, and the like.
The light is projected onto the target object via the light projecting zoom lens 11 and the like. The light receiving optical system picks up an image with a distance image sensor 12 and a color image sensor 24 arranged on a light receiving and imaging surface via a light receiving zoom lens 14, a beam splitter 15, and the like. The optical system and the imaging system will be described later in detail.

While the distance image sensor 12 accumulates one image, the slit light from the light projecting system is shifted downward by one pixel pitch of the distance image sensor 12 by the polygon mirror 7 which is rotating normally. Scanned. The range image sensor scans and outputs the stored image information and stores the next image. The distance information of 256 points in the length direction of the slit light can be calculated from the image output by this one output. Further, by repeating mirror scanning and image capturing 324 times, a distance image of 256 × 324 points is generated.

The range of the distance to a target object measured with respect to one slit light is limited to the closest distance to be measured and the longest distance to be measured. Therefore, the slit light is reflected by the object and enters the image pickup device. The range is limited to a certain range. This is because the light projecting system and the light receiving system are set apart from each other by the base line length (length 1). This is shown in FIG. 4, in which the direction perpendicular to the image sensor surface for the distance image is the Z axis. The position of the broken line indicated by d is the measurement reference plane, and the distance from the element surface is d.

The shortest distance measured is Df, and the shortest distance measured is Dn. Now, when the cutting plane by the slit light emitted from the light projecting system is the slit A (SA), the range of the imaging element surface that receives the slit light reflected on the object surface is the distance between the nearest distance Dn measured and the slit A. The point at which the three-dimensional position of the intersection point Pan is projected on the image sensor is defined as the lowest point in the figure, and the three-dimensional position of the intersection point PAf between the measured farthest distance Df and the slit A is centered on the principal point of the imaging system lens. Is limited to a closed section Ar on the image sensor having a point projected on the image sensor as the uppermost point in the drawing. Similarly, in the case of the slit light B (SB), with the positional relationship between the light projecting system and the light receiving system unchanged, the projected point of the intersection PBn between the measured closest distance Dn and the slit B is defined as the lowest point in the figure. The point where the intersection point PBf of the farthest distance Df and the slit B is projected is limited to a closed section Br on the image sensor whose uppermost point is shown in the drawing.

For this reason, first, in the measuring device, the position of the center of gravity of the laser light received in 256 lines from the input image is determined by the object distance output from the autofocus unit and the azimuth of the slit light to be projected, ie, the scanning start. The calculation is performed as the amount of deviation from the measurement reference plane determined based on the time from the measurement reference plane. The calculation of the pitch shift amount will be described with reference to FIG. 5. First, FIG. 5 shows a light amount distribution generated by slit light projected on the target object surface. The squares shown in the lower part of the figure indicate the area that each element of the range image sensor looks at, and the squares are numbered from 1, 2, 3, 4,... Since the slit light having an extremely narrow slit width is scanned by the rotation of the polygon mirror 7 by one pitch of the distance image sensor during one image accumulation, the light quantity distribution upon input of one image is equivalent to one pitch of the distance image sensor. Is a rectangular light quantity distribution having a width of

In order to calculate the distance information in the Z-axis direction for each pixel of the range image sensor, it is desirable that the rectangular light amount distribution has such a one-pitch width. When the width of the light quantity distribution is equal to or more than one pitch, the measured distance information is obtained as a load average of the received light intensity over the adjacent area, and accurate distance information cannot be obtained.

It is assumed that a stepped object surface indicated by a halftone dot in FIG. 5 exists under such a light amount distribution, and slit light is projected from a direction perpendicular to the object surface. An elongated rectangular parallelepiped indicates a slit light amount distribution, and a hatched area indicates a light projection slit image. Then, assuming that the light receiving system optical axis Oxp is provided in a direction inclined to the left from the light projecting system optical axis Oxa, the light receiving slit light amount distribution on the light receiving surface is the distribution shown in FIG. Become. This received light quantity does not include the steady light component,
It is desirable to remove the stationary light component other than the laser light component. For that purpose, images in a state where the laser light is not irradiated and an image in the irradiated state are input, and the difference between the two is used. The cells shown below indicate the respective element regions of the range image sensor.

On the front surface of the range image sensor, there is provided an optical filter having anisotropy for reducing the resolution in the width direction of the slit light without decreasing the resolution in the length direction of the received slit light. This filter produces a Gaussian light quantity distribution as shown in FIG.
The light receiving position can be calculated at a resolution finer than the pixel pitch by obtaining the center of the light amount distribution from each sensor in each of the columns 1, 2, 3, 4,. . As described above, when detecting the slit light incident position, the width of the slit light incident on the sensor is not narrowed but a distribution having a width of about 5 to 6 pixels is obtained by using a filter. This is because if the width is smaller than the width of one pixel, only a position detection resolution equivalent to the pixel pitch can be obtained.

The first column center of gravity G1 is determined from the light quantity distribution D1 incident on the first column, and similarly the second, third, fourth,.
, The center of gravity is calculated for each column by calculating the center of gravity position G2, G3, G4,. As shown in the figure, the optical axis of the light projecting system is a direction perpendicular to the object plane, but the optical axis of the light receiving system is a direction inclined to the left. Therefore, in the case of a target object having a step as shown in FIG. In the portion (third and fourth rows) higher than the center of gravity of the portion (first and second rows), the center of gravity is located at a position shifted to the right. FIG. 6 shows only two types of distributions, a distribution D1 in the first column and a distribution D4 in the fourth column. However, the distribution D2 in the second column is the same as the distribution D1 in the first column. The distribution D3 in the column is the same as the distribution D4 in the fourth column. FIG. 7 is a plan view showing the relationship between the light amount distribution and the position of the center of gravity. Since the distributions of the first and second columns are the same, the centroids G1 and G2 obtained are detected as the same position, and the third and fourth columns have the same distribution, so that the centroids G3 and G4 are detected as the same position.

As described above, 256 incident light positions are obtained from the strip image for one slit, and 324
324 images are obtained by performing the same calculation for the slit projected in the azimuth direction, and a pitch shift image composed of 256 × 324 points is obtained. The obtained pitch shift image is an image of only the position information of the slit light, and calibration (correction) from a table of detailed data such as correction of lens aberration is required to obtain an accurate distance image from the image. In this method, the lens aberration estimated from the focal length f and the focus position d of the photographing lens is calculated and corrected, and the vertical and horizontal distortions of the camera are corrected. The same processing is performed for a color image. The data required at this time is information on various measurement lenses, that is, the focal length f and the focus position d. In the system of the present embodiment, calibration is performed on a computer system, and the measurement apparatus (shown in FIG. 3) can be connected via a terminal such as SCSI or shared with a recording medium such as MO. To

As described above, a color image and a pitch shift image are output as digital signals from a terminal such as SCSI or an analog video signal from an output terminal such as NTSC from the main body of the measuring apparatus. Output to the computer as a digital signal. Also, when recording on a recording medium using a drive device 48 such as an MO or MD built in the main body, an image and various data are recorded.

The captured pitch shift image and color image are transferred to a computer connected to a measuring device together with various types of photographing lens information, and the computer calculates information on the distance to the target object from the transferred pitch shift image and photographing lens information. Calibration and conversion are performed on the held distance image. After the calibration, for the pitch shift image, a conversion curve between the shift distance and the measurement distance stored for each of the focal length f and the focus position d, the vertical and horizontal positions in the screen, and the XY positions is derived, and the conversion curve is obtained. The pitch shift image is converted into a distance image on the basis of.

Conversion to a distance image is well known,
For details, see the IEICE Technical Meeting, PRU91-113.
[Geometric correction of images that do not require camera positioning] Onodera and Kanaya, IEICE Transactions D-II vol. J74
-D-II No.9 pp.1227-1235, '91 / 9 [High-precision calibration method of range finder based on three-dimensional model of optical system] is disclosed in Ueshiba, Yoshimi, Oshima and others.

Hereinafter, the photographing method and data processing according to the present invention will be described in detail. First, a description will be given of high-accuracy input by divisional capture using a camera platform in the three-dimensional shape measuring apparatus. When the distance between the light projecting system and the light receiving system, that is, the base line length l and the focal length f, and the distance d to the measurement object are determined, the three-dimensional resolution and accuracy are determined. Therefore, high-precision measurement is achieved by setting the focal length f large and measuring. In other words, the measurement accuracy increases as the distance increases. However, 3
Although a two-dimensional image can be obtained, the visual field region has a focal length f
As it extends.

Therefore, the resolution at which the focal length f is desired to be measured,
The value is set to a value corresponding to the accuracy, and the rotary gantry 4 such as an electric camera platform is operated to divide the field of view into a plurality of regions, and each of the divided regions is measured. This is to reconstruct a single image. With such a function, it is possible to realize a three-dimensional shape measuring apparatus with variable resolution. By making use of this function, the environment can be measured by performing three-dimensional measurement on the entire surrounding space. A specific example will be described below, and the operation will be described.
The example shown in FIG. 8 is a simplified explanatory diagram, and the light projecting system 2 and the light receiving system 3 are arranged in a horizontal positional relationship, which is different from the example shown in FIG. In this arrangement, the slit light has a length in the vertical direction and needs to scan in the left-right direction.

FIG. 8 shows a state when the image combining function is used, and FIG. 9 is a flowchart showing an operation in the image combining function. FIG. 10 shows a display state when this function is used, and there is a display section indicating measurement accuracy below the image display section.

First, as shown in FIG. 8A, the zoom drive system 1 is set to a wide angle and wide state (focal length f0) in which the target object 1 can be imaged within the field of view by an operation by the user.
6 is set to set the field of view (step # 101).
At this time, the assumed resolution in the Z-axis direction (see FIG. 4, the uneven direction of the object) is represented by a bar display below the image as shown in FIG. When the base line length is fixed as in the present system, the Z-axis direction resolution ΔZ simply has the following relationship between the distance d to the measurement target and the focal length f at the time of measurement. ΔZ = K × d × (df) / f (1) where K is a coefficient for estimating the resolution in the Z-axis direction,
It is determined by the sensor pitch and the like. In addition, the above zooming operation is performed by the system computer from the SCSI.
A command can be transmitted via a terminal, and operation settings such as a zoom operation and a release operation by remote control can be performed.

If the user determines that the measurement is performed with the accuracy and resolution that are satisfied by the above setting operation (NO in step # 102), the measurement is started by the user's release operation (step # 103). ), And the result is displayed on the display (step # 104). In this display, as shown in FIG. 10A, the input pitch shift image or color image and the measurement resolution in the Z-axis direction obtained by capturing the image are displayed as bars below the image.
As a result, when measurement with higher measurement accuracy is not required (NO in step # 105), the user is asked to determine whether or not to complete the measurement and write data to the storage medium. Perform the processing and complete the operation.

When the user determines that the measurement is not performed with satisfactory accuracy (YE in the determination in step # 102)
S), or the pitch shift image captured by the first release operation, or the display of the Z-axis direction measurement resolution, allows the user to change the accuracy by key operation to set the desired Z-axis direction resolution and accuracy, and instruct re-measurement. Can be performed (YES in step # 105).

When the accuracy setting key is input, the system determines the current state, that is, the focal length f0 in a state where the entire view of the measurement target is captured, and the approximate distance d to the measurement target obtained from the AF sensor. The memory is stored to store the field of view (step # 106). Further, the focal length f1 to be set is calculated from the input desired resolution in the Z-axis direction and the approximate distance d using the above equation (1) (step # 10).
7).

When the focal length f1 is calculated, zooming is automatically performed on the focal length f1 (step # 10).
8) The number of frames to be divided and input based on the stored visual field range to be measured, the approximate distance d, and the focal length f1, the calculation of the pan and tilt angles corresponding thereto, and the pan and tilt of the pan / tilt rotation base to determine the visual field position Make settings (Step # 1)
09), measurement is performed for each divided input frame (step # 110). The image to be divided and input at the time of the image stitching function is to be stitched later and reconstructed into one image.
It is set to include the overlap that should be the margin.

Information indicating the obtained pitch shift image, color image, and captured view directions in the X and Y directions (for example,
The pan / tilt decode angle values or the order of capture in the X and Y directions, the lens focal length, and the measured distance information are stored in an internal MO storage device (step # 11).
1). At this time, the directory information such as the file name and the file size is not written into the memory, and the directory information can be written after the user's confirmation at the end to temporarily store the information.

Next, in the field of view slightly overlapped with the above field of view, the field of view is controlled by the pan and tilt operations according to the calculated pan and tilt angles, and an image of an adjacent area is input. By repeating the above, the input of the entire area is performed (NO in the determination in step # 112, see FIG. 8B).

When the input of all areas is completed (step #
In the case of YES in the determination of 112), the camera is returned to the initial camera posture and the focal length before the measurement accuracy is increased (step # 11).
3) Complete the operation, wait for the user to judge the writing, write the directory information in the case of the write instruction, and terminate without writing the directory information in the case of the instruction not to write. The information stored in the memory up to the next is erased.

When the measurement is performed in advance and the measurement is performed again as in the above operation, the distance measurement to the target object and the measurement of the distance distribution within the measurement angle of view are completed by the first measurement. are doing. Therefore, re-measurement is not performed for a region having a large difference in the distance from the target object, that is, a divided input frame including only a peripheral region (background) different from the measurement target because of the bonding. It is also possible to perform re-measurement of only the frame. FIG.
In the example shown in (b), the halftone dot region including the target object is a region where the re-measurement is performed, and the other region is a region where the target object is not included and the re-measurement is not performed.

As described above, high-speed three-dimensional measurement can be performed, and three-dimensional shape measurement in which the resolution can be freely set by repeating a partial input based on the three-dimensional measurement and performing a bonding operation. Becomes possible.

In such a bonding measurement, the resolution of the entire screen is measured at a uniform resolution, but the shape, color information is complicated for the eyes, mouth, and nose like a human face. Although high-resolution data is required, a measurement target such as a cheek or a forehead, which needs sufficient measurement at a low resolution, may be used. For such a measurement target, efficient data input can be realized by performing data bonding by a partial zooming operation. This partial zooming bonding function is realized by the following operation.

FIG. 11 is a flowchart showing the operation of the partial zooming and bonding function. First, as in the case of the uniform resolution bonding in step # 201, a visual field setting for capturing the entire area of the measurement target is performed, and step # 20 is performed.
In step 2, a partial zooming input mode is selected. When the selection is made, the currently set focal length f0 and pan / tilt decode angle values are stored in memory (step # 20).
3), measurement is started in the state of the focal length f0, and image input is performed as general image data (step # 204). The resulting pitch shift image, color image, and information indicating the view direction of the captured X and Y directions of the image (for example,
Pan and tilt decode angle values), lens focal length, and measured distance information are stored in an internal storage device (step #).
205). Subsequently, in step # 206, the maximum focal length fma
After zooming to x, the above-described schematic image data is analyzed, and it is determined whether or not to perform re-measurement for each divided input frame input after zooming.

When the zooming is performed and the measurement is performed at the maximum focal length fmax, the rough data is divided into inputtable frame sizes. In step # 207, the pan / tilt positions X and Y are set to the initial start positions Xs and Ys. The pan / tilt is controlled to the X and Y positions set in step # 208. Next, in step # 209, X ± ΔX,
The color information R of the initial input color image in the area of Y ± ΔY,
Statistical processing is performed on the G and B values to calculate standard deviations σR, σG, and σB for each area. In step # 210, it is determined whether all the calculated standard deviations σR, σG, and σB are less than the set predetermined values. Is a uniform area, and the process proceeds to step # 211 without performing the zooming measurement. Conversely, standard deviation σ that is equal to or greater than a predetermined value
If there is R, σG, σB, it is determined that the area has complex color information, and zooming measurement is performed (step #).
213).

At step # 211 X ± ΔX, Y ± ΔY
The standard deviation σd is calculated from the information of the initial input distance value d in the area of, and in step # 212, it is determined whether the calculated value of the standard deviation σd is smaller than a predetermined value. Since the small area is a flat area with little change in shape, zooming measurement is not performed, and step # 215 is performed.
Proceed to. Conversely, if the value is equal to or more than the predetermined value, it is determined that the region has a complicated shape (distance information) and zooming measurement is performed (step # 213).

After the zooming measurement in step # 213, the obtained pitch shift image, color image, information indicating the viewing direction in the X and Y directions from which the image has been captured (eg, pan and tilt decode angle values), Information such as lens focal length and measured distance information is stored in a storage device such as an internal MO (step # 214). Thereafter, the process proceeds to step # 215.

Next, in step # 215, the pan / tilt position X is changed by 2ΔX. X in step # 216
It is determined whether the directional scanning has been completed, and if not, the process returns to step # 208. Step # if completed
At 217, the pan / tilt position Y is changed by 2ΔY.
In step # 218, it is determined whether or not all the scans have been completed.
If not completed, the process returns to step # 208. If completed, the process proceeds to step # 219, and this routine ends.

As described above, the general image data and the partial detailed image information whose position can be determined can be input, and the partial detailed image data at the position can be attached to the general image data to obtain the shape and color information. Efficient three-dimensional input according to complexity can be realized.

In the embodiment described above, the focal length of the lens is changed by performing zooming, and photographing is performed with an accuracy suitable for the shape and color of the object. However, two lenses having different focal lengths (long focal length lens) are used. And a short focal length lens), it is also possible to take an image of a region where the shape and color of the object are determined to be complicated through a long focal length lens, and to photograph other regions through a short focal length lens.

Next, data processing for bonding will be described. This is to convert a plurality of pieces of image data into one piece of image data by performing coordinate transformation of a plurality of pieces of image data input using the above-described camera device into one coordinate system.

First, a description will be given of the bonding in the case where the camera is mounted on the camera platform for photographing. This is a camera mounted on a pan / tilt pan head that shoots a fixed object multiple times while panning and tilting, transforms the captured image data into one coordinate system, and pastes them together. It is for obtaining an image.

When the camera is panned or tilted, if the rotation angle can be controlled with high precision, high-precision bonding can be performed without any problem. However, since a high-precision camera platform is generally very expensive, it is desired to take an image with an inexpensive camera platform that includes many errors in controlling the rotation angle.

In this case, a camera model as shown in FIG. 12 is prepared. This represents a panning / tilting camera on three-dimensional coordinates. In FIG. 12, C is a camera, Θ is a camera rotation axis (pan), and Φ is a camera rotation axis (tilt).

The model parameters (the direction and position of the pan rotation axis and the direction and position of the tilt rotation axis) are determined in advance by performing calibration. The search for a junction point (a point where two pieces of image data are joined) performed below is performed by changing the model parameters Θ (pan angle) and Φ (tilt angle).

This operation will be described with reference to the flowchart of FIG. First, the photographed data (as mentioned above,
Stored in a storage device inside the camera device)
A three-dimensional color image, three-dimensional data, a focal length, and a reference plane distance are captured (step # 301). Next, a search for a joining point is performed from the two-dimensional color image (step # 302, which will be described in detail later).

However, the measurement points of the two images shot by panning and tilting do not always coincide (even when shot at the same shooting distance and focal length, they are shifted by a maximum of half a pixel). Therefore, a color image (two-dimensional data)
It is assumed that the search for the junction point has been completed up to one pixel.
The following search is performed using the three-dimensional data.

First, the camera rotation angle (pan angle Θ, tilt angle Φ) is calculated from the two-dimensional image joining point, the focal length, and the reference plane distance (step # 303, details will be described later). Next, a coordinate conversion parameter in a three-dimensional space is calculated based on the calculated camera rotation angle (step # 304, details of which will be described later).

Next, a search is made to minimize the sum of the squares of the angles formed by the normals of the two planes passing through the connection point (step # 305). The method of calculating the evaluation value will be described later in detail.

Since a rough search is performed in a two-dimensional image, a search in a very narrow range can be completed for three-dimensional data. As a whole, the amount of calculation is smaller than when searching only with three-dimensional data, and high-speed bonding can be performed.

Next, it is determined whether or not the calculated evaluation value is equal to or smaller than a predetermined value (step # 306). If the calculated evaluation value is equal to or smaller than the predetermined value, two 3
The two-dimensional images are converted into the same coordinate system and are bonded (Step # 307), and the operation is terminated (Step # 31).
0). The method of coordinate conversion will be described later.

If it is determined in step # 306 that the evaluation value is not equal to or smaller than the predetermined value, it is next determined whether the number of repetitions of the continuity evaluation is equal to or greater than a predetermined value (step # 308). Perform alignment processing. This is because the evaluation value converges when the number of repetitions exceeds a certain value, and it is not necessary to perform any more repetitions.

If it is determined in step # 308 that the number of repetitions is not equal to or greater than the predetermined value, then the rotation angle of the camera is slightly changed in a direction in which the evaluation value obtained in step # 305 becomes smaller (step # 309). Go to # 304,
Repeat the continuity evaluation of the patch at the connection.

Next, each step will be described in detail. First, a method of searching for a joint point from a two-dimensional color image in step # 302 will be described with reference to FIGS. As shown in FIG. 14, it is assumed that the two images to be combined have a marginal portion (width T pixels). FIG.
As shown in FIG. 15A, a reference window is set at the center of the margin of one image (the dotted line in FIG. 15A is the center line of the margin). FIG. 15B is an enlarged view of the reference window portion of FIG. 15A. The reference window is further divided into small windows of about 8 × 8 (pixels). A window having a complicated shape or pattern (large variance) among the small windows is set as a comparison window. This is because the use of a portion having a clear edge or a portion having a complicated pattern or shape enables highly reliable evaluation.

In the other image, a search window is set which has the same size as the reference window and moves so as to cover all the margins (FIG. 16).

Also in this search window, a small window is set at the same position as the comparison window in the reference window, and the sum of squares of the luminance difference between this small window and the comparison window is used as an evaluation value. To explore.

Next, a method of calculating the camera rotation angle from the two-dimensional image joining point, the focal length, and the reference plane distance in step # 303 will be described.

Assuming that the pixel size is PS, the camera plane size is 2 × S, the focal length is f, and the number of shifted pixels is t, the rotation axis coincides with the camera position (the rotation axis intersects with the optical axis of the camera). ), The camera rotation angle θ is obtained by the following equation (FIG. 17). θ = π−arctan (S / f) −arctan ((S−PS × t) / f) When the rotation axis and the camera position do not match (the rotation axis and the optical axis of the camera are displaced), When the radius of rotation (distance between the rotation axis and the optical axis of the camera) is r and the reference plane distance is D, the following equation is obtained. t × PS × D / f = 2S × D / f− (D + r × sin θ) / t
an (π-arctan (f / S) −θ) −S × D / fr × cos θ When the rotation axis does not match the camera position, the calculation becomes complicated and the rotation angle cannot be simply obtained. Therefore, if an angle corresponding to the number of shifted pixels (t) is obtained in advance by a search method and a table is created, the rotation angle can be easily obtained.

Next, the method of calculating the coordinate conversion parameters of the camera in step # 304 and the method of coordinate conversion in step # 307 will be described. Let the coordinate systems of the two cameras be C1 (X1, Y1, Z1) and C2 (X2, Y2,
Z2), the position of the camera rotation axis is T (t1, t2, t
3) If the rotation direction of the camera is (1, 0, 0) (rotation about the X axis in the coordinate system of C1), θ around the rotation axis
To convert C2 when rotated to the coordinate system of C1, it is as follows. C2−T = R (θ) · (C1−T) where R (θ) is obtained from the camera rotation angle θ by the following equation.

[0063]

(Equation 1)

Is as follows. Therefore, the conversion from the C2 coordinate system to the C1 coordinate system is represented by the following equation using the parameters R (θ) and T. C1 = R (θ) ~ ¹ · (C2−T) + T That is, the point C1 (C1 coordinate system) is translated to the rotation axis, and the coordinate is converted to the C2 coordinate system on the rotation axis (θ rotation), This is performed by performing parallel movement from the rotation axis to the point C2.

Although the above description is for the pan angle, the coordinate conversion can be similarly performed for the tilt angle only by setting the rotation axis to the Y axis.

Next, the method of calculating the continuity evaluation value of the patch at the connection portion in step # 305 will be described in detail with reference to the flowchart of FIG. 18 and FIG.

First, three-dimensional data of two images including the junction point searched from the two-dimensional color image is fetched (step # 401), and the first image (the image represented by a white circle in FIG. 19) and the second image 19 (the image represented by a black circle in FIG. 19), the normal at the center of the plane is calculated for the planes 1 to 12 at the connection point (step # 402).

Next, for the first image, e1 (1) = (angle between normals of 1 and 2-1) − (1 and 2-12
(The angle formed by the normals of the planes), and similarly, a set of 4 and subsequent n faces is calculated to obtain the sum of squares (E1) (step # 40).
3).

Similarly, for the second image, e2 (1) = (the angle between the normals of 3 and 2-2)-(3 and 2-12
(The angle formed by the normal line of), and similarly, a set of n and subsequent n faces is calculated to obtain the sum of squares (E2) (step # 40).
4).

Then, it is evaluated whether or not a smooth joint is made by using (E1 + E2) / n (step # 405), and the evaluation value is returned to the main routine (step # 406).

Next, a description will be given of the bonding of data shot by a plurality of cameras. When shooting with multiple cameras, shoot each other's cameras to determine their relative position,
Since the orientation can be measured, the position of the object (corresponding to the position of the rotation axis) and the angle between the cameras viewed from the object (corresponding to the rotation angle) are determined based on the data, and the coordinate conversion parameters are determined based on the angle. The two three-dimensional images are converted into the same coordinate system and are bonded. For details of the bonding operation,
Since this is the same as the above-described shooting using the camera platform, the description is omitted here.

Also, if a long focal length lens is used for photographing between the cameras when photographing the objects, the camera position is obtained with higher precision than the object data, and the object data is not adversely affected when the images are bonded. I'm sorry.

Next, a description will be given of the bonding of data in the case where an object is photographed while mounted on a rotating stage.
1 will be described. FIG. 20 shows a rotary stage on which a subject is placed. The periphery of the rotary stage is composed of a polyhedron, and the normal of each plane is perpendicular to the rotation axis and is arranged at an equal distance from the rotation axis. Therefore, if each plane can be measured, the direction and position of the rotation axis of the rotary stage can be obtained.

For example, as shown in FIG. 22 (a model obtained by taking a picture of the rotating stage), the rotating stage is rotated by 90 ° so that the rotating stage is within the measurement range, and four pieces of three-dimensional and two-dimensional data are obtained. Shooting (step #
502). Next, the data of the subject and the rotary stage section are separated from each of the four pieces of captured data, and a plane group that is a part of the lower rotary stage is extracted (step # 503). At this time, if a specific color is given to the plane portion of the rotary stage, it is possible to more easily extract the color image.

Next, among the data separated in step # 503, the position and orientation of the rotary stage are calculated using the data of the rotary stage (step # 504). This method will be described later in detail.

From the position and orientation of the rotary stage and the rotation angle obtained in the subroutine of step # 504, a coordinate conversion (around the rotation axis) parameter for each photographing data is calculated (step # 505). Is performed to unify each photographing data into one coordinate system (step # 506). Note that the method of calculating parameters from the rotation angle and the method of coordinate conversion are the same as the above-described parameter calculation method and coordinate conversion method when using the camera platform, if the rotation angle of the camera is replaced with the rotation angle of the rotation stage. is there.

Next, a connection section is set (the method will be described later), and data outside the boundary is deleted from each photographed data.
The surface is reconstructed at the joint (Step # 507), and the bonding of the three-dimensional data is completed (Step # 508).

As a result, as shown in FIG. 23, the first image (black point) and the second image (white point) are pasted together via the boundary to form one image.

The method of calculating the position and orientation of the rotary stage in step # 504 will be described with reference to the flowchart of FIG. First, the rotation stage three-dimensional data and the rotation stage color image are captured (step # 601), and the data is separated for each plane (step # 60).
2). Next, a normal vector of the plane is calculated for each plane (step # 603), and a straight line perpendicular to the normal vector and equidistant from each plane is set as a rotation axis (step # 6).
04), this rotation axis is returned to the main routine as the position and orientation of the rotation stage (step # 605).

Next, a method for setting the joint will be described in detail. First, the case where the actual data that has been shot is not changed will be described with reference to the flowchart in FIG. First, only data sandwiched between two image boundaries out of the data cut at the image boundary (four orthogonal planes including the axis of the rotary stage) is deleted, and other data is deleted (step # 701). The correspondence between the end points of the photographing data is determined (corresponding to those having a short distance between the two points), and the surfaces are sequentially pasted (step # 702), and the process returns to the main routine (step # 703).

In this case, when data is deleted, if a margin is provided on the two surfaces to be joined, a joint point search can be performed as described in the attachment using the camera platform. More smooth bonding is possible.

Next, a case where the photographed data is changed to perform smooth bonding will be described with reference to the flowchart of FIG. 26 and FIG. A point that is approximately the same distance from the image boundary (four orthogonal planes including the axis of the rotating stage) for the two pieces of photographing data is determined as a corresponding point (1 in FIG.
-1, 2-1, 1-2 and 2-2, ..., 1-n and 2-
n) (Step # 801), a new one point (the point marked X in FIG. 27) is generated from two points corresponding to the data up to a certain distance from the boundary surface (Step # 802). The point for generating this new point is determined as follows according to the distance from the boundary surface.

A predetermined range (a range for generating a new point) from the boundary surface is D, a point of one photographing data is X1, a point of the other photographing data is X2, and two points (X1, X2) from the boundary surface.
X3 = ((D + d) × X1 + (D−d) × X2) / (2 × D) where d is the average distance to X, and X3 is the newly generated point.

The surface is reconstructed by using newly generated data in the vicinity of the boundary (the range from the boundary surface up to D), and by using actual data in the other regions (the range in which the distance from the boundary surface exceeds D). (Step # 803), the process returns to the main routine (Step # 804).

Further, as shown in FIG. 20 (b), if the rotary stage is provided with a concave / convex at every 90 °, the rotation angle can be made highly accurate at a very low cost. By performing coordinate transformation on the image based on the rotation axis obtained in advance, the entire circumference data can be obtained. When such a rotary stage is used, the subject can be set at an arbitrary position within the measurement range, and photographing becomes easy.

Next, a description will be given, with reference to the flowchart of FIG. 28, of a method of zooming together when the above-mentioned zooming input is performed.

First, photographic data having different magnifications are fetched by the method described for the zooming input (step # 9).
01). Next, by using the same method as the flowchart of FIG. 13 described above (the last bonding is not performed), bonding is performed using a camera platform, calculation of coordinate conversion parameters and data extraction of a boundary portion are performed (step # 902). . However,
Before searching for a junction from the two-dimensional color image (step # 302 in FIG. 13), resampling is performed on each of the two-dimensional image and the three-dimensional image (the method will be described later).

Next, coordinate conversion is performed on the high-magnification data, the high-magnification data is adjusted to the coordinate system of the low-magnification data (step # 903), and the surface is reconstructed at the boundary (step # 903). # 904), the bonding is completed (step # 905).

FIG. 29 shows a model of this zoom bonding. Data of FIG. 29B (magnification N2)
After resampling is performed so that the magnification becomes N1 as shown in FIG. 29C, the data (magnification N1) in FIG. 29A is pasted together, and the portion where the magnification is N2 is replaced with the original magnification (N
Returning to 2), as a result, a bonded image as shown in FIG.

Next, a method of resampling a two-dimensional image and a three-dimensional image will be described in detail. First, a two-dimensional image will be described with reference to FIG. In FIG. 30, the solid line shows an image at the magnification N1, and the dotted line shows an image at the magnification N1.
2 (the minimum square is one pixel, N1 <N
2).

The resampling is performed on the image of the magnification N2, and the phase is adjusted so that the upper left pixel of the image of the magnification N2 exactly matches the sampling point of the image of the magnification N1.

The resampling value (average luminance) is calculated by using a weighted average value based on the area of the pixels of the image of the magnification N2 included in the pixels of the image of the magnification N1. This is the magnification N1
Is obtained by dividing the sum of the product of the area of the image of the magnification N2 and the luminance included in one pixel of the image of the image by the area of one pixel of the image of the magnification N1.

Next, a three-dimensional image will be described with reference to FIG. FIG. 31 shows an expression in a camera line of sight. In FIG. 31, a solid line and a white circle represent an image at the magnification N1, and a dotted line and a black circle represent an image at the magnification N2 (both are minimum pixels of one pixel and N1 <N2).

The resampling is performed on the image of the magnification N2, and the phase is adjusted so that the upper left pixel of the image of the magnification N2 exactly matches the sampling point of the image of the magnification N1.

The resampling value is calculated by using the intersection of the camera's line of sight passing through a point of the image of magnification N1 and a quadratic surface consisting of four points of the image of magnification N2 surrounding the point.

In the embodiment described above, the calculation of the coordinate conversion parameters is performed using both the two-dimensional color image and the three-dimensional data. However, the joint point search from the two-dimensional color image is not performed and only the three-dimensional data is used. It is also possible to determine coordinate transformation parameters. Further, in the present embodiment, a description has been given of a three-dimensional input. However, it is needless to say that the present invention can be applied to a two-dimensional image input based on the same principle as the present invention.

[0097]

As described above, the image input camera according to the present invention can acquire lean image data adapted to the shape of an object, and perform high-speed display and image recognition.

[Brief description of the drawings]

FIG. 1 is an explanatory diagram showing the principle of the light section method.

FIG. 2 is a schematic block diagram of the entire apparatus according to the present invention.

FIG. 3 is a perspective view showing a schematic configuration of the entire apparatus according to the present invention.

FIG. 4 is an explanatory diagram of an incident range and a scanning range of reflected light incident on an image sensor.

FIG. 5 is an explanatory diagram of a light amount distribution generated on a target object plane.

FIG. 6 is an explanatory diagram of a light amount distribution generated on a light receiving surface of an image sensor.

FIG. 7 is an explanatory diagram of a light amount distribution generated on a light receiving surface of an image sensor.

FIG. 8 is an explanatory diagram of an image pasting function.

FIG. 9 is a flowchart showing the operation of an image combining function.

FIG. 10 is an explanatory diagram of a display state of an image pasting function.

FIG. 11 is a flowchart showing an operation of a partial zooming and bonding function;

FIG. 12 is an explanatory diagram showing a camera model when shooting using a camera platform.

FIG. 13 is a flowchart showing an operation of pasting three-dimensional data photographed using a camera platform;

FIG. 14 is an explanatory diagram of a margin in two-dimensional image bonding.

FIG. 15 is an explanatory diagram of a reference window in pasting two-dimensional images.

FIG. 16 is an explanatory diagram of a search window in pasting two-dimensional images.

FIG. 17 is an explanatory diagram of a method of calculating a camera rotation angle.

FIG. 18 is a flowchart illustrating an operation of evaluating the continuity of patches at a joint portion;

FIG. 19 is an explanatory diagram of pasting of photographing data by a camera head.

FIG. 20 is a perspective view showing the appearance of a rotary stage.

FIG. 21 is a flowchart illustrating an operation of attaching three-dimensional data captured using a rotating stage.

FIG. 22 is an explanatory diagram of photographing and image pasting using a rotating stage.

FIG. 23 is an explanatory view of pasting of photographing data by a rotating stage.

FIG. 24 is a flowchart showing the operation of a method for calculating the position and orientation of the rotary stage.

FIG. 25 is a flowchart showing an operation of a connection section setting method (when actual data is not changed);

FIG. 26 is a flowchart showing an operation of a connection section setting method (when actual data is changed);

FIG. 27 is an explanatory diagram of a data generation point in a connection section setting method (when actual data is changed).

FIG. 28 is a flowchart showing an operation of combining three-dimensional data obtained by zooming using a camera platform;

FIG. 29 is an explanatory diagram of combining photographing data by zoom photographing;

FIG. 30 is an explanatory diagram of resampling of a two-dimensional image.

FIG. 31 is an explanatory diagram of resampling of a three-dimensional image.

[Explanation of symbols]

 $ 210: Judgment unit # 212: Judgment unit # 213: Input unit $ 902: Bonding unit $ 903: Bonding unit $ 904: Bonding unit

Continuation of the front page (51) Int.Cl. ⁷ identification code FIG06F 15/62 415 (56) References JP-A-4-57173 (JP, A) JP-A-5-199404 (JP, A) (58) Surveyed field (Int.Cl. ⁷ , DB name) G01B 11/00-11/30 102 G06T 1/00 G06T 7/00 G06T 7/60 H04N 5/265

Claims (5)

(57) [Claims] 1. A first input means for inputting image data indicating a shape of an object via an optical system set to a first focal length, and a second focal point shorter than the first focal length. A second input unit for inputting image data indicating the shape of the object through an optical system set to a distance, and a plurality of image data input by the first and second input units; An image input system comprising a matching unit. 2. The image input system according to claim 1, further comprising a determination unit configured to determine a shape of the target object, wherein a portion determined to have a complicated shape by the determination unit inputs image data using the first input unit, 2. The image input system according to claim 1, wherein the other part inputs image data by said second input means. 3. The image input system according to claim 2, wherein said judging means judges the shape of the object based on an edge of the shape of the object. 4. The bonding means samples the image data obtained by the first input means so as to have the same magnification as the image data obtained by the second input means, and then outputs the image data. The image input system according to claim 1, wherein the image input system is configured to perform bonding. 5. A first input means for inputting image data indicating a shape of an object at a first magnification, and image data indicating a shape of the object at a second magnification lower than the first magnification. An image input system comprising: a second input unit for inputting a plurality of image data; and a bonding unit for bonding a plurality of pieces of image data input by the first and second input units.