JPH09305796A - Image information processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily obtain an object image at an optional visual point by inputting an image while moving it around the object via a monocullar or ommateal pickup system and synthesizing the image data into an image. SOLUTION: A three-dimensional image data calculation part 6 calculates the three-dimensional image data on an object that is photographed based on the image data stored in a memory 5 and on the position information on a camera head 1 corresponding to the image data. A two-dimensional image data calculation means 7 calculates the two-dimensional image data viewed at a position where the object is first photographed and in an image form that is selected from the three-dimensional image data on the object by an operation means 11. The two-dimensional image data are displayed on a monitor 8. When a user operates the means 11 under such conditions, the means 7 performs the corresponding operation and changes the object image shown on the monitor 8 into an image that is viewed at an optional visual point.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention uses an image pickup device having a display means for displaying a picked up image or an image being picked up, and is capable of being used in CG, CAD, etc. The present invention relates to an image processing device capable of extracting dimensional information or synthesizing it with a text file.

[0002]

2. Description of the Related Art Conventionally, techniques for obtaining a three-dimensional shape of an object are roughly classified into a passive technique and an active technique. A typical passive method is a stereo image method in which triangulation is performed using two cameras. With this method, search for the same object in the left and right images,
The three-dimensional position of the subject is measured from the amount of displacement of the position.

Further, as a typical active method,
Measuring the time it takes to project light, reflect it, and return to determine the distance Optical radar type range finder and slit-shaped light patterns are projected to measure the three-dimensional recording from the displacement of the pattern shape reflected on the subject There is a slit light projection method.

From the obtained three-dimensional data of the subject, a subject image when the subject is viewed from an arbitrary viewpoint is reproduced on a two-dimensional display or the like.

With the spread of personal computers, it has become possible to capture and edit images taken by electronic cameras. For example, suppose that an electronic camera divides a landscape into a plurality of images and shoots the landscape. At this time, the captured images are captured so as to exist in the overlapping portion.

Then, the picked-up image is taken into a personal computer and subjected to a combining process by application software. Specifically, the images are combined so that the overlapping areas overlap. As a result, it is possible to obtain an image with an angle of view that is much wider than the angle of view that can be captured with an electronic camera.

[0007]

However, in the above-mentioned stereo image method, the main purpose is to calculate distance information from a specific position where the camera is installed, and not to measure the stereoscopic shape itself of a certain subject. .

Further, in the active method, since an object is irradiated with a laser or the like, it is complicated to use. Furthermore, in any of the methods, camera control that can flexibly deal with a dynamic imaging method in which an image is taken while moving around a certain object is not performed.

Further, in the conventional passive method using two cameras, it is necessary to provide two viewfinders, and there are problems in terms of cost and operability, such as taking images while comparing them.

For example, there are problems in that it takes time during framing, and when a three-dimensional shape is to be obtained, the overlapping area is too small to be implemented.

Further, an image generally handled in an office or the like is finally output on paper in many cases, and the image form used may be a natural image or a line image in which a subject is represented only by a contour line. is there. However, the above-mentioned conventional example is not used in the office because the main purpose is to obtain the stereoscopic shape data of the subject and display it faithfully on the two-dimensional display.

Therefore, a first object of the present invention is to obtain an image of the obtained shape data of an object viewed from an arbitrary viewpoint in an apparatus for capturing an object by a single or a plurality of imaging systems to obtain stereoscopic shape and pixel information. The purpose is to make it available in an image format according to the purpose.

A second object of the present invention is to provide an apparatus for obtaining a stereoscopic shape and pixel information by capturing an image of a subject with a single or a plurality of image capturing systems, and including position detecting means for detecting a relative positional change of the image capturing means. A method of integrating a plurality of image data and calculating a three-dimensional shape data of a subject, and making the obtained shape data of the subject available from an arbitrary viewpoint in an image form according to the purpose. is there.

A third object of the present application is to automatically perform focal length and focusing in an apparatus for obtaining a three-dimensional shape and pixel information by picking up an image of a subject with a single or a plurality of image pickup systems. A shape corresponding to the currently captured image area is displayed on the finder as a monitor for the captured area, and the image is captured while easily recognizing the overlapping area between the already captured image and the currently captured image. It is possible to realize accurate shape extraction, reduce the load on the photographer, and use the obtained shape data of the subject in an image viewed from an arbitrary viewpoint in an image format according to the purpose. Is. That is, it is possible to combine with other files.

The fourth object of the present application is to use the obtained shape data of the object as a two-dimensional image data so that the image viewed from an arbitrary viewpoint can be used in an image form according to the purpose. Is calculated and made available.

A fifth object of the present application is to combine image data with a text file so that image data can be used.

A sixth object of the present application is to hold information regarding a plurality of pads and to detect a relative positional change of the image pickup means.

A seventh object of the present application is to provide an image information processing apparatus having a stereoscopic function in an image pickup system.

An eighth object of the present application is that the image pickup apparatus has an optical finder, and the apparatus can be constructed at low cost.

A ninth object of the present application is to use a device for picking up an image of a subject by a plurality of image pickup optical systems and outputting pixel information of the subject, and software in a computer calculates three-dimensional shape data of the subject. An object is to provide an image information processing device that makes things possible.

[0021]

In order to achieve the first object, the invention according to claim 1 of the present application is to capture an image of a subject by a single or a plurality of image pickup means, and to obtain a stereoscopic shape and pixels of the subject. An apparatus for outputting information, which is an image information processing apparatus including means for forming a file by converting the stereoscopic shape data into a predetermined image form when a subject is viewed from an arbitrary position.

In order to achieve the above-mentioned second object, the invention according to claim 2 of the present application is an apparatus for picking up an image of a subject by a single or a plurality of image pickup means and outputting the three-dimensional shape of the subject and pixel information. And a plurality of image data obtained by the image pickup unit and the image pickup obtained by the position detection unit corresponding to each image data. The image data includes a means for calculating three-dimensional shape data of the subject from the relative positional relationship of the means, and a calculation means for converting the three-dimensional shape data into a predetermined image form when the subject is viewed from an arbitrary position. The image information processing apparatus is provided with means for synthesizing the above file and another file.

In order to achieve the above-mentioned third object, the invention according to claim 3 of the present application captures an image of a subject by an image pickup optical system of a single or a plurality of image pickup means, and obtains the three-dimensional shape and pixel information of the subject. In the output device, there is an image information processing apparatus including a stereoscopic shape extraction device that automatically adjusts imaging parameters so that the main subject fits within the overlapping area of each imaging system and the main subject fits within the depth of focus.

In order to achieve the fourth object, the invention according to claim 4 of the present application is a calculation means for converting from the three-dimensional shape data into a predetermined image form when a subject is viewed from an arbitrary position. Is in an image information processing apparatus configured to calculate image data as two-dimensional image data.

In order to achieve the fifth object, the invention according to claim 5 of the present application is that the means for synthesizing the image data file and the other file is that the other file is a text file. The image information processing apparatus is characterized by

In order to achieve the sixth object, the invention according to claim 6 of the present application is, in claims 1 and 2, a position detecting means for detecting a relative position change of the image pickup means,
An image information processing device is configured to hold information regarding a plurality of pads.

In order to achieve the seventh subject, the invention according to claim 7 of the present application is an apparatus for capturing an image of a subject by a single or a plurality of image pickup optical systems, and outputting a stereoscopic shape of the subject and pixel information. A plurality of image data obtained by the image pickup means and the image pickup obtained by the position detection means corresponding to each of the image data. The image data includes a means for calculating three-dimensional shape data of the subject from the relative positional relationship of the means, and a calculation means for converting the three-dimensional shape data into a predetermined image form when the subject is viewed from an arbitrary position. The image information processing apparatus has a means for synthesizing the above file and another file and further has a stereoscopic function.

In order to achieve the above-mentioned eighth object, the invention according to claim 8 of the present application is an apparatus for picking up an image of a subject by a single or a plurality of image pickup optical systems and outputting the stereoscopic shape of the subject and pixel information. The image pickup device includes an optical finder, position detection means for detecting a relative position change of the image pickup means, and a plurality of image data obtained by the image pickup means and the image data corresponding to each of the image data. The stereoscopic shape data of the subject is calculated from the relative positional relationship of the image pickup means obtained by the position detection means, and the stereoscopic shape data is converted into a predetermined image form when the subject is viewed from an arbitrary position. An image information processing apparatus having a calculation means and a means for synthesizing the image data file and another file.

In order to achieve the ninth subject, the invention according to claim 9 of the present application is a device for picking up an image of a subject by a single or plural image pickup optical systems and outputting pixel information. A plurality of image data obtained by the image pickup means and relative positions of the image pickup means obtained by the position detection means corresponding to the respective image data. A software for calculating the three-dimensional shape data of the subject based on the relationship is provided, and the three-dimensional shape data has a calculation means for converting the three-dimensional shape data into a predetermined image form when the subject is viewed from an arbitrary position. An image information processing apparatus having a means for synthesizing a data file and another file.

[0030]

BEST MODE FOR CARRYING OUT THE INVENTION

(First Embodiment) An image processing apparatus of the present invention will be described in detail below with reference to the drawings.

The outline of the image pickup system, the structure of the finder at the time of image capturing, the three-dimensional information extraction, the image pickup parameter correction method, the three-dimensional information, the combination of the image texture and the text file, and the output will be described in this order. Is performed in the order of the configuration of the imaging system, the sequence of capturing images, the configuration and operation of the three-dimensional shape extraction block.

1. Outline of imaging system Configuration of imaging system Sequence of image capture processing, etc. Configuration and operation details of 3D shape extraction block 2. Finder configuration when capturing images 3. Stereoscopic information extraction Extraction of range image from stereo image Time series integration of range information 4. Imaging parameter correction method 5. 3D information, composition of image texture and text file,
output

(Outline of Imaging System) The configuration of the imaging system and the sequence of image capturing will be described.

FIG. 1 is a schematic diagram showing a three-dimensional shape extraction apparatus according to the present invention and an environment in which it is used.

In the figure, reference numeral 1 is an image pickup head section (camera head section) of the three-dimensional shape extraction apparatus according to the present invention, and this image pickup head section constitutes a so-called compound eye image pickup system.
0L and 100R are the left and right imaging lenses, 10L,
10R are left and right imaging lenses 100L and 100R, respectively.
Are shown. Further, in order to capture a stereoscopic image, it is necessary that these imaging ranges overlap each other.

Reference numeral 2 is a detected object, that is, a subject, and 3 is a subject 2.
The back surface, which is the background of the, and an illumination unit 200 for illuminating the subject. The illumination unit emits illumination light according to the imaging environment.

Reference numeral 4 denotes a position detecting section as a position detecting means for detecting the image pickup position of the image pickup head 1, which detects the position of the image pickup head 1 by image processing based on the information obtained from the back surface 3 and outputs the position information. And means for physically detecting the position of the imaging head 1 by a sensor such as a gyro and outputting position information.

The image pickup head unit 1 of the three-dimensional shape extraction apparatus picks up an image of the subject 2 while moving from the image pickup start position A0 to the image pickup end position An. At this time, the position of the image pickup head unit 1 of the three-dimensional shape extraction device at each image pickup point between A0 and An is detected by the position detection unit described above, and the position information is output.

Reference numeral 5 is a memory for storing the image data obtained by the camera head 1 and the position information of the camera head 1 obtained by the position detecting means 4.

Reference numeral 6 is a three-dimensional image data calculation unit for calculating the three-dimensional shape of the subject based on the image data stored in the memory 5 and the positional information of the camera head 1 corresponding thereto.

Reference numeral 7 calculates the two-dimensional image data of the subject when viewed from an arbitrary viewpoint in the image form designated by the user from the three-dimensional image data of the subject obtained by the three-dimensional image data computing unit 2 It is a three-dimensional image data calculation unit.

In the present embodiment, as shown in FIG. 2, the user has 12 natural images of the object, or a line drawing (13 in the drawing) in which the edges of the object are represented by lines, or the surface of the object has a predetermined size. Polygon image expressed as a continuum of the plane of Sasano (1 in the figure
It is assumed that the three image forms of 4) can be selected by the operation means 11 described later.

Reference numeral 1001 is a text data creating means for creating document data such as text. Reference numeral 1000 is processing for synthesizing or editing the document data and subject data obtained by the two-dimensional image data calculating means. Is a data synthesizing means capable of

Reference numeral 8 is a monitor for displaying two-dimensional image data and document data of the subject.

A printer 9 prints two-dimensional image data of a subject and document data on paper or the like.

11 is for moving the viewpoint for seeing the subject,
The image form of the subject can be changed, and the data synthesizing means 1 can be used.
000 is an operating means for performing various operations for synthesizing and editing each data.

As described above, when the user points the camera head 1 at the subject 2 and operates a release button (not shown), the subject is photographed and the first image data is stored in the memory 5. Next, the user moves the camera head 1 from A0 to An centering on the subject.

During the movement from A0 to An, when the position detecting means 4 detects that the position and direction have changed by a predetermined amount with respect to the first position A0, the second photographing is performed, and the following sequence is performed. Imaging is performed up to the nth time. At this time, the image data and the amount of displacement of the camera head 1 obtained by the position detection means with respect to the position and direction in which the subject 2 is first photographed are recorded in the memory 5.

At this time, when at least one of the position and the moving direction of the camera head 1 is displaced by the position detecting section 4 by a larger amount than a predetermined value, a warning means to be described later gives a warning. Thereafter, this operation is repeated several times, and when the image data sufficient for calculating the three-dimensional image data of the subject is obtained, the photographing end notifying means (not shown) notifies the user of the fact and the photographing is ended.

Next, the three-dimensional image data calculation unit 6 calculates the three-dimensional image data of the photographed subject from the image data stored in the memory 5 and the position information of the camera head 1 corresponding to each image data.

The two-dimensional image data calculating means calculates the two-dimensional image data in the image form selected by the operating means 11 from the three-dimensional image data of the subject (see FIG. 2) and seen from the position where the subject is first photographed. Then, it is displayed on the monitor 8.

At this time, when the user operates the operation means 11, the two-dimensional image data operation means 7 performs an operation according to the operation, and the subject image 15 displayed on the monitor 8 as shown in FIG. It is possible to change to a subject image viewed from an arbitrary viewpoint. Further, by operating the operating means 11, the subject displayed on the monitor 8 can be changed to an image form as shown in FIG.

Then, the user can change the viewpoint and the image form of the photographed object according to the purpose and then output the image to the printer 8. In addition, the user uses the data synthesizing means 1000.
It is also possible to perform composition and editing work while displaying on the monitor 8 the document data created in advance and the subject data calculated by the two-dimensional image data calculation unit. Also at this time, if it is desired to change the image form or the viewpoint of the subject, it is possible by operating the operating means 11.

Next, the details of the configuration and operation of the three-dimensional shape extraction block will be described.

FIG. 4 shows a block diagram of a solid shape extraction according to the present invention. This is a detail of the head and the processing unit in FIG.

In FIG. 4, reference numerals 100L and 100R denote left and right image pickup lenses, which are zoom lenses. 101L and 101R are diaphragms, 102L and 102R are image sensors, and image pickup devices such as CCDs are used.

Image sensors 103L and 103R are image sensors 1.
An A / D converter that converts the image signals output from the 02L and 102R into digital signals.
Is a video signal processing unit that converts the image signal converted into a digital signal by the A / D conversion units 103L and 103R into a standardized video signal.

105L and 105R are video signal processing units 10
The subject separation unit separates the image of the object (subject) from which the stereoscopic information is to be extracted from the image signals output from the 4L and 104R units, and the image on the back side.

Reference numerals 106L and 106R denote zoom control units for adjusting the focal length of the zoom lens.
L and 107R are focus control units for adjusting the focus position. Further, 108L and 108R are diaphragm control units, which adjust the diaphragm amount.

Reference numeral 4 denotes a camera posture detecting section corresponding to a position detecting section for detecting the position of the image pickup head 1.

Reference numeral 210 denotes a system controller for integrally controlling each process of the present three-dimensional shape extraction apparatus, which is composed of a microcomputer.

As shown in FIG. 5, the system controller 210 includes a microcomputer 900 and a memory 910.
And an image calculation processing unit 920.

Reference numeral 220 denotes an image processing unit, which realizes 5, 6 and 7 shown in the conceptual diagram of FIG. Each stereoscopic information of the subject is extracted from the video signal corresponding to the subject image captured by each of the imaging lenses 100L and 100R, and the stereoscopic information of the subject at each of the captured image points is obtained by the camera attitude detection unit 4. It is integrated and output based on the information on the posture of the camera head at the position.

The detailed operation of the camera portion of the three-dimensional shape extracting apparatus of the present invention will be described below with reference to FIG.

The subject images are image pickup lenses 100R and 100R, respectively.
It is input through 00L. The input subject image is converted into an electric signal by the image sensors 102R and 102L. The converted signals are output to the A / D conversion unit 103, respectively.
The analog signals are converted into digital signals in R and 103L and supplied to the video signal processing units 104R and 104L.

Each of the video signal processing units 104R and 104
In L, the signal of the digitized subject is converted into a luminance signal and a chrominance signal of appropriate forms. In the subject separation units 105R and 105L, based on the signals obtained from the video signal processing units 104R and 104L, respectively,
In the imaged subject, the main subject whose stereoscopic shape is desired to be measured and the back face are separated.

As a separating method, for example, a back image is picked up in advance, the image is held in a memory, and then a main subject to be measured is placed and picked up. A method of performing matching and difference processing between the imaged image and the image of the rear face held in the memory in advance to separate the rear face region is used. The separation method is not limited to this, and separation may be performed based on color or texture information.

The image of the separated main subject is the image processing unit 2.
20 and the image processing unit 220 performs a three-dimensional shape extraction process based on each parameter at the time of imaging.

Next, the processing sequence of the camera unit of the three-dimensional shape extraction device according to the present invention will be described with reference to the flowchart of FIG.

When the power is turned on and the video signal is input, the controller 210 converts the video signal obtained by each of the subject separation units 105R and 105L into the image calculation processing unit 9
The luminance level of the main subject 2 shown in FIG. 1 is calculated through integration processing by 20, and when it is determined that the luminance is insufficient and the three-dimensional shape extraction is insufficient, the illumination 200 is turned on. At this time, the illumination intensity level may be variable according to the calculated brightness level.

Next, the focus position is adjusted using each video signal set to an appropriate brightness level. As shown in FIG. 7, the focus position is first adjusted on the main subject A and then on the back surface B. The focus state is detected by the focus detection unit 270. As a detection method, a well-known method can be used in which the sharpness or blur amount of an edge is detected and the photographing lens is driven so that the sharpness is maximum and the blur amount is minimum.

The distance at which focusing is performed in each case is X.
If the values are 1 and X2, the focus detection unit 270 outputs this value to the controller 210. The controller 210 determines the distance X for focusing at the time of measurement based on this value, and outputs a control signal to each of the focus control units 107R and 107L. The distance X may be, for example, an intermediate position X = (X1 + X2) between X1 and X2, or may be determined as X = (mX1 + nX2) by appropriate weighting. m and n are constants for determining weighting.

The range within the depth of focus when the focus position is adjusted to the distance X can be expressed as follows in the range from the distance X1 'to X2'.

X1 ′ = Xf ² / (f ² + δF (X−f) (Equation 1) X2 ′ = Xf ² / (f ² −δF (X−f) where δ represents a circle of least confusion and represents, for example, an image. The pixel size of the sensor may be used.

The controller 210 uses X1 and X2 and X
In order to set the F number so that 1'and X2 'substantially match, a control signal is given to the aperture control units 108R and 108L. When the brightness level is changed to a certain extent or more by this operation, the intensity of the illumination 200 is changed to handle it. Alternatively, an AGC (auto gain control) circuit may be incorporated to perform level correction electrically.

After performing the focus position adjustment, the zoom is adjusted.

FIG. 8 shows an outline of zoom adjustment. In a state where the main subject is substantially within the depth of focus, the images obtained from the respective imaging systems 1R and 1L are held in the memory 910 of the controller 210, and the image calculation processing unit 920 detects the overlap area.
The detection method uses correlation calculation processing, template matching processing, or the like.

As shown in FIG. 8, the overlap area 500 is detected in the initial state, and then the controller 210 sets the zoom value in the direction in which the area of this area becomes larger on the screen, and the respective zoom control units 106R. And 106L to output control signals.

FIG. 9 shows an outline of changes in the overlap area in the screen due to a series of zoom adjustments. In FIG. 9, the image calculation processing unit 920 of the controller 210 calculates f at which the area P of the overlap region has a peak, and gives a control signal to each of the zoom control units 106R and 106L.

When the focal length f is changed by this operation and the range of the depth of focus is changed to some extent or more as a result, X1 and X2 and X1 'and X are obtained by the equation (1).
In order to set the F number so that 2'substantially match each other, a control signal is given to the aperture control units 108R and 108L according to step S2 of the parameter readjustment in FIG.

After the series of adjustment steps S1 described above, a step S2 of parameter readjustment and left / right difference correction is performed. The left / right difference correction is performed by the left / right difference detection unit 260.
The focus position and zoom value are detected from the video signal. Based on the detected signals, the controller 210 controls the zoom control units 106R and 106L and the focus control unit 107R.
And 107L and diaphragm control units 108R and 108L.

Further, the distance information can be expressed by the following equation.

Z = fb / d (Equation 2) Here, Z: distance, f: focal length, b: base line length, and d: parallax.

Therefore, the focal length is as follows with the distance resolution determined by parallax as a parameter.

∂Z / ∂d = −fb / d ² (Equation 3) f = (− d ² / b) · (∂Z / ∂d) (Equation 4)

Therefore, it is also possible to set the resolution from a computer or the like through the external input I / F 760 and set the focal length based on this value.

When the imaging parameters are adjusted in steps S1 and S2, the controller 210 causes the display unit 2 to
A signal is given to 40 to notify the photographer of the end of parameter setting. The display unit may be a display such as a CRT or LCD, or may be a simple display such as an LED. It goes without saying that a sound may be generated together with the display.

The photographer confirms the display of the LED or the like and starts the work of extracting the three-dimensional shape.

When the photographer presses an input start button (not shown), the detection signal of the camera posture detection unit 201 is initialized.

Next, the structure of the finder at the time of capturing an image will be described.

The block diagram of FIG. 4 is used to show the structure of the finder at the time of image capturing. In FIG. 4, 73,
75, 83 and 85 are image memories, 104L and 104R are video signal processing units of the left and right imaging systems, 240 is a display unit, and 92.
Is a left and right image overlap detector, and 97 is a sounding body.

For example, one object is photographed from a plurality of viewpoints, and the three-dimensional shape of the object is obtained based on the images obtained there. At this time, the photographed images are related. It is recorded as an image.

At this time, the most recently captured image is stored in the image memories 75 and 85, and the image stored therein and the currently captured image, that is, the first image memories 73 and 83 are stored. The overlap detection unit 92 detects the overlapped portion with the displayed image as the left and right images. The output is input to the system control 210, and the overlap portion of the previous image and the current image is displayed on the display unit (EVF) 240 according to the output result.

When the external input I / F 760 is used as a three-dimensional shape input and an object is photographed from a certain viewpoint, only a portion corresponding to the right side imaging system of the overlapping portion is displayed on the EVF 240, and a portion other than the overlapping portion is black. Is displayed. Then, one image determined by looking through the EVF 240 is photographed. At this time, only the left and right overlapping portions are respectively compressed and recorded in the recording unit 250 by, for example, JPEG which is one of the compression methods. This is because the three-dimensional shape is required only in the overlapping portion, and unnecessary information is omitted.

Next, when performing the next framing, E
On the VF 240, an overlap portion between the current image and the most recently captured image is displayed as shown in FIG. In the figure, a black display is shown in a place where the left and right images do not overlap, and a place where the first shot is picked up is displayed in a frame.

Then, the release button 230 is pressed with an angle of view such that the overlap portion is displayed on the screen. At this time, release button 2 with the overlap part not in the angle of view.
When the user presses 30, the warning sound is emitted from the sounding body 97 because there is no correlation between the previous image and the current image. As a result, the photographer notices that there is no correlation between the previous image and the current image, but if that does not matter, shooting is possible by pressing the release button 230 all the way.

In this way, after recording a plurality of images of the object from various viewpoints and without omission, the recording unit 250
Is removed from the main body of the imaging device and connected to a personal computer. It can be used with application software on a personal computer.

Further, since the imaging system itself has a function of constructing a three-dimensional shape, it is possible to use the obtained three-dimensional shape data in a computer.

When used in the computer, the grouping flag is automatically selected from the supplementary information of the image, the image is automatically selected and displayed on the personal computer, and the left and right images are used. .

As described above, by displaying on the EVF 240 so that the overlap portion of the current image with respect to the previously recorded image can be known, it is possible to prevent the photographing failure.

Further, when an attempt is made to take a picture without the overlapping portion, a warning is issued to prevent further failure of the picture taking.

Further, when it is desired to associate photographing as one group, the grouping information is recorded as the supplementary information of the image, so that the processing on the personal computer can be facilitated.

Further, since only the overlapping portions of the left and right image pickup areas are displayed on the EVF, it is possible to make the portions capable of stereoscopic representation obvious. In addition, since only the left and right overlapping portions are recorded, it is possible to suppress unnecessary consumption of memory.

As another example of the present embodiment, the left and right overlapping portions can be obtained from the parameters of the image pickup apparatus, that is, the focal length, the subject distance, the base line length, and the vergence angle, instead of being obtained from the image correlation. By displaying it, the accuracy is lower than that performed by the correlation of images, but the image memory can be saved and the cost can be reduced.

Next, the stereoscopic information extraction will be described. First, extraction of a distance image from a stereo image will be described.

FIG. 11 shows a processing procedure for extracting a distance image from a stereo image. In FIG. 11, 110 is a stereo image stored in the image memory. Each is called a left image and a right image. An edge extraction processing unit 111 generates an image in which each edge is extracted from the stereo image. A corresponding point extraction processing unit 113 is a processing unit that extracts what correspondence relationship each pixel of the stereo image has.

Reference numeral 112 is also an edge image corresponding point extraction processing section. However, this is a processing unit that extracts the correspondence relationship between the respective pixels in the two images formed into the edge images by the edge extraction processing unit 111.

Reference numeral 114 is a processing unit that determines whether or not there is a contradiction in the correspondence relationship obtained by the corresponding point extraction processing unit 113 and the edge image corresponding point extraction processing unit 112, and removes the contradiction.

Reference numeral 115 is a processing unit for judging the occlusion area based on the obtained corresponding point portion and an index indicating the degree of correlation used in the process of obtaining the corresponding point, for example, the residual.

Reference numeral 116 denotes a processing unit for calculating a distance distribution based on the principle of triangulation from the corresponding point relation, 117 denotes a processing unit for identifying a characteristic point on the back side, and 118 uses a characteristic point on the back side.
It is a processing unit that acquires imaging parameters, postures, and movement relationships.

First, two processes are performed from the stereo image stored in the image memory. This is a process in which the corresponding point extraction processing unit 113 uses a processing method described later to extract from the luminance value of the stereo image what correspondence each pixel has.

Further, the edge image corresponding point extraction processing unit 112 uses the edge extraction processing unit 11 by using the processing method described later.
This is a process of extracting the correspondence relationship between the respective pixels in the two images formed into the edge image in 1. Regarding the generation of the edge image, it is assumed that the edge image is generated through the edge extraction processing unit 111 using the processing method described later.

Next, in the inconsistent portion, the removal processing unit 114 determines the inconsistency or the like in the correspondence relationship from the output from each corresponding point processing unit. When the correspondence relationship from the luminance part and the correspondence relationship from the edge part do not match, it is appropriate to exclude the correspondence relationship because the reliability is low. Alternatively, it is also possible to make a determination by weighting each relationship.

The next step is to find the corresponding points and
This is a process of determining an occlusion area based on an index indicating the degree of correlation used in the process of obtaining corresponding points, for example, a residual.

This is a process in which the corresponding point process gives a tentative result, but its reliability is added. A correlation coefficient or a residual is used as an index indicating the degree of correlation, and when the residual is extremely large or when the correlation coefficient is low, the reliability of the correspondence is determined to be low. This low place is treated as an occlusion area or an uncorresponding area.

Through the above steps, the distance information of the object is calculated using the principle of triangulation using the obtained correspondence. Triangulation is as described in Equation 2.

The corresponding point extraction method will be described. A representative method thereof is the template matching method. As shown in FIG. 17, a template image of N * N pixels is cut out from an image obtained from the left imaging system, for example. This is moved on the search area range (M−N + 1) ² in the input image of M * M pixels of the image obtained from the right imaging system,

[0118]

[Outside 1] The position of the template image that minimizes the residual R (a, b) of is calculated, and the central pixel of the N * N template image is calculated as the matching point.

However, (a, b) indicates the upper left position of the template image in the image, I _{R (a, b)} (i, j) is a partial image of the right image, and T _L (i, j) is the left image. It is a template image cut out from an image.

As the edge extraction method, a method such as a Robert filter is used. Alternatively, a Sobel filter can be used.

The Robert filter uses the input image f (i,
j) and output image g (i, j) g (i, j) = sqrt ({f (i, j) -f (i + 1, j + l)} ² ) + {f (i + 1, j) -f ( i, j + l)} ² ) (Equation 6) or g (i, j) = abs {f (i, j) -f (i + 1, j + i)} + abs {f (i + l, j) -f (i, j + l) } To obtain.

(Equation 7) In the case of a Sobel filter,

[0123]

[Outside 2] is calculated by _{^{θ = tan -1 (f y /}} f x) ( Equation 10).

The image in which the edge portion is emphasized in this way is binarized to extract the edge component. Binarization is performed using an appropriate threshold.

Next, the time series integration of the distance information obtained by the above means will be described.

FIG. 12 shows a procedure for integrating the distance information from the stereo image obtained by the above means moment by moment.

In FIG. 12, 120 is distance information from the pair of stereo images obtained by the above means.

Reference numeral 121 is means for converting the distance information 120 from a pair of stereo images into a unified coordinate system.

Reference numeral 122 is a means for integrating the respective distance information converted into a unified coordinate system. The integration here means that the same points are the same, that the coordinates of the obtained points are interpolated, and the reliability of the coordinates of the points is judged based on the flag from the depth information of the imaging system. This includes selection based on the occlusion part detection information.

Reference numeral 123 is a means for transmitting occlusion information. Reference numeral 124 is a means for displaying integrated distance information. Reference numeral 125 is a means for detecting the movement amount, direction, etc. of the imaging head according to the present invention.

First, the distance information 120 from the stereo image obtained by the above-mentioned explanation means is generated every moment.

On the other hand, the information is sent from the means 125 for detecting the moving amount and direction of the image pickup head. The obtained distance information is converted into an integrated coordinate system by a processing method described later using these. By converting to the unified coordinate system in this way, it is easy to integrate the information obtained moment by moment.

Next, the distance information converted into the same coordinate system is integrated. In the case where the same point is shown in each of the distance information, only one of them is used to reduce the amount of information. If they are judged to be the same, the criteria are (x ₀ −x ₁ ) ² + (y ₀ −y ₁ ) ² + (z ₀ −z ₁ ) ² <ε 1 (equation 11) or a (x ₀ −x ₁ ) ² _{+ b (y 0 -y 1)} 2 + c (z 0 -z 1) 2 <ε2 used (equation 12), and the like.

However, ε1 and ε2 are reference values, a, b, c and d.
Is an appropriate coefficient.

For example, it is possible to make the determination more sensitive to the difference in distance by setting a = b = 1 and c = 2.

Interpolation is performed from the coordinates of the obtained points.
As the interpolation, for example, an intermediate point shown in FIG. 13 is obtained.

In FIG. 13, ◯ and ● are the extracted data, and □ are the data obtained by the intermediate point interpolation.

As the interpolation method, linear interpolation, spline interpolation or the like can be used.

Next, the reliability of the coordinates of the point is judged based on the flag from the depth information of the image pickup system. This is to use the depth of focus information of the optical system of the image pickup system to eliminate places of low reliability, and to select the information based on the occlusion part detection information.

A method for converting distance information into an integrated coordinate system is shown in FIGS. 18 (a) and 18 (b). In FIG. 18, 2 is a subject and 3 is a pad. The pad corresponds to the back surface as the background of the subject.

Reference numerals 1800 to 1804 are virtual projection planes for registering the distance information of the image pickup head with respect to the subject 2.

1810 to 1814 are the central axes (optical axes) of the virtual projection plane.

The unified coordinate system here means five sets of reference coordinates (x, y, z axes). That is, an axis for forming the virtual projection plane shown above, for example, x,
It shows that there are five y and z axes from 1800 to 1804.

First, the distance information Z ^t obtained by the above method
_ij is projected on each projection surface (5 surfaces). Transformation such as rotation and translation is performed so that the projection follows each reference coordinate. This is shown in FIG. 18 (b). Although this is a schematic diagram regarding the projection surface 1803, the same is applied to the other projection surfaces. The next distance information Z ^{t + δ t} _ij is similarly _processed , but when projecting, the objects at the previously written time points are sequentially overwritten.

Therefore, the distance information along the five reference axes is obtained for a certain subject. For example, one point is (x
0, y0, z0), (x1, y1, z1), (x2, y
2, z2), (x3, y3, z3), (x4, y4, z
It is expressed by 5 of 4).

Next, the image pickup parameter correction method will be described.

FIG. 14 shows an image pickup parameter correction method.

The correction means obtains from the information from the sensor and the information from the image processing means. First, the method from the image processing means will be described.

Assuming that the coordinate system of the pad is X, Y, Z, the coordinate axes of the compound-eye image pickup system at any place are as follows.

The imaging system on the left is

[0151]

[Outside 3] The same applies to A _R , B _R and C _R.

For example, the image pickup system has a base line length B along the X axis,
If they are far apart,

[0153]

[Outside 4]

Now, the image pickup point p ₁ (x) of P ₀ (x, y, z)
Supposing that the relations of ₁ y ₁ ) and p _r (x _r y _r ) are extracted and obtained by the feature point extraction processing, (uv) = (x ₁ y ₁ ) − (x _r y _r ) (Equation 20) ( x ₁ y ₁ ) = f (X _L / Z _L Y _L / Z _L ) (Formula 21) From (x _r y _r ) = f (X _R / Z _R Y _R / Z _R ) (Formula 22) (uv ) = F (X _L / Z _L Y _L / Z _L ) −f (X _R / Z _R Y _R / Z _R ) (Equation 23) For the sake of simplicity, the compound-eye imaging system has no convergence angle and A _L = _BL = C
_L = E (identity matrix), A _R = _BR = _CR = E (identity matrix),
If U = B and V = W = 0,

[0155]

[Outside 5] From (uv) = (f / Z _L ) · (B 0) (Equation 25) Z _L = Z _R = fB / u is obtained. (Equation 26)

In this way, the relationship between the pad and the image pickup system is obtained.

The above description is a simplified one, but the generality is not lost. In addition, Japanese Patent Application No. 4-34 by the present applicant
Further details are described in Japanese Patent No. 4556 (Japanese Patent Laid-Open No. 6-195446).

Next, it is necessary to take consistency with the output from the sensor system. It is preferable to simply take the average of the output from the image processing system and the output from the sensor system.

Alternatively, it is possible to shorten the processing time from the image processing by setting the output from the sensor system as an initial value.

Next, the stereoscopic information, the combination of the image texture and the text file, and the output processing will be described.

FIG. 16 shows a means for synthesizing the stereoscopic information and the image texture obtained by the above-described means with a text file.

In FIG. 16, 160 is a fitting processing means for correcting the extracted distance information 1600 using the distance template 162.

Reference numeral 161 is a texture sticking means, which is means for pasting the image texture 1602 to the corrected distance information 1601.

Reference numeral 163 denotes a file synthesizing processing unit which synthesizes the corrected distance information 1601, the image texture 1602 and the text file 1603 to generate one file. Reference numeral 1604 denotes a 2D (two-dimensional) / 3D (three-dimensional) file output, which indicates the generated file.

First, the extracted distance information 1600 obtained by the above-mentioned means is compared with the distance template 162 held in advance, and the fitting processing means 160 makes a fine correction. FIG. 20 shows the flow of the fitting process. In FIG. 20, reference numeral 2001 is an extracted image, 2000 is a model image, and 2002 and 2003 are differences between the extracted image 2001 and the model image 2000.

In the fitting process, two images are displayed as shown in FIG. 20, and correction is performed with, for example, an input pen. Alternatively, the difference 2002 of the two images is taken and automatically corrected so as to become a uniform difference 2003.

Then, as shown in FIG. 16, the image texture 1602 is pasted to the corrected distance information 1601 by the texture pasting means. This is the same as the method used in CG and the like.

Now that the image side is ready,
By combining with the text file 1603, a file for presentation, for example, is formed.

As the forming method, the information designated by the area where the image in the sentence is inserted and the information of the other party are added to each of the files to be combined.

An example of this is shown in FIG. 19. In FIG. 19, 1901 is a text file, and 1902 and 1903 are image files.

Reference numerals 19001 and 19002 are link information.
The connection relation between a text file and an image file is shown. 190
03, 19005, 19007 are text data. 1
Reference numerals 9004 and 19006 are image insertion flags, which indicate the positions where the image is inserted, and indicate the image file in combination with the link information. 19008 and 19009 are image coordinate information.

The structure of the text file is composed of link information, an image insertion flag, and text data. The image file is inserted at the position of the image insertion flag. Since the image file contains image coordinate information, it is actually inserted after being converted into a figure from an arbitrary viewpoint based on that information. In other words, the image coordinate information is information indicating the conversion relationship for the original image. The file created in this way is finally used.

(Second Embodiment) Next, a second embodiment according to the present invention will be described.

FIG. 21 shows the outline of the second embodiment of the three-dimensional shape extraction device according to the present invention.

In FIG. 21, reference numeral 2101 is the main subject,
2100 is the present stereoscopic extraction device, 100 is an imaging lens, 20
Reference numeral 0 is an illumination unit.

Reference numeral 2102 is a calibration pad, and the three-dimensional extraction device detects the posture based on the image of this pad.

The characters A, B, C, on the pad 2102
D is a characteristic part for detecting the posture of the device 2100, and the posture is calculated from the direction, distortion and the like of these characteristic portions.

FIG. 22 is a block diagram of the three-dimensional shape extraction device 2100 according to this embodiment.

It is to be noted that, in FIG. 22, those having the same numbers as those in the first embodiment except for the symbol portions of L and R have the same functions and operations, and therefore the description thereof will be omitted.

In FIG. 22, reference numeral 100 is an image pickup lens, and a zoom lens or the like is used.

Reference numeral 101 is an aperture, and 102 is an image sensor such as a CCD.

Reference numeral 103 is an A / D converter which converts an analog signal obtained from the image sensor 102 into a digital signal.

Reference numeral 104 is a video signal processing section, and 105 is a subject separation section.

Reference numeral 4 denotes a camera posture position detection unit which detects the posture of the apparatus 2100 from the direction and distortion of the characteristic portion of the pad 2102. An image processing unit 220 extracts the three-dimensional shape of the subject from the video signal and the posture information.

Further, the function and operation are the same as those in the first embodiment except that the focus detection unit 270 is a monocular.

A system controller 210 controls the entire apparatus.

Next, the operation will be described. FIG.
It is a flow which shows the processing of the solid shape extraction device concerning the present invention.

The present embodiment differs from the first embodiment in the zoom adjusting method. The device of this example has a pad 21
Since the posture detection is performed in combination with 02, it is necessary to obtain the pad 2102 in an appropriate range at the time of imaging.

Therefore, first, the subject separating section 105 performs a correlation calculation between the characteristic portions (A, B, C, D at the four corners in FIG. 21) of the pad 2102 held in advance and the image currently obtained, or Template matching processing is performed and a detection signal is output to the system controller 210. The system controller 210 sets the focal length so that the pad 2102 falls within an appropriate range within the field of view. At the same time, the information on the focal length is held in the memory 910 in the system controller 210.

As a result, the entire pad can be kept in the visual field at all times, so that the posture can always be detected from the shape of the characteristic portion.

As shown in the flow chart of FIG. 23, when the parameters of the optical system are set, the LED of the display section 240 is turned on to inform the photographer that the input is possible.

Upon receipt of this display, the photographer starts the input and, while moving the device 2100, releases the shutter 230 at appropriate intervals to input the image. At this time, based on the information of the subject separation unit 105, the system controller 210
Sets the focal length so that the characteristic portion of the pad 2102 including the main subject always fits within the visual field within an appropriate range. At the same time, the imaging parameter information including the focal length at each imaging position is held in the memory 910. As a result, the posture detection unit 4 detects the posture from the state of the characteristic portion.

The image processing section 220 has the image memories 73, 7
5 is read out, the image is converted based on the imaging parameter information stored in the memory 910 in the system controller, and corrected to an image with the same focal length. Further, the image processing unit 220 extracts the three-dimensional shape of the subject from the corrected video signal and the posture signal obtained by 4, and supplies it to the recording unit 250. Recording unit 250
Converts the obtained signal into an appropriate format and records it on a recording medium.

(Third Embodiment) FIG. 24 is a block diagram showing a third embodiment of the three-dimensional shape extraction device according to the present invention. In FIG. 24, components having the same numbers as those in the first and second embodiments have the same functions and operations, and the description thereof will be omitted.

In FIG. 24, 2400 is a memory,
It holds information about the pad. The present embodiment is characterized by having a plurality of pieces of information regarding this pad.

Reference numeral 760 denotes an external input I / F section, which is connected to, for example, a computer to input information.

The type of pad and the like can be selected by the external input I / F section.

The recording section 250 records the three-dimensional shape and the imaging parameter at the same time. It also has a function of reading the recorded information as needed. Reference numeral 210 denotes a controller, which performs overall control.

A matching processing unit 2401 performs matching processing with the currently picked-up image on the basis of the three-dimensional shape and pixel information recorded in advance by the recording unit 250.

The operation will be described below. Also in FIG.
The flow of this embodiment is shown in FIG.

In the apparatus of this embodiment, the photographer selects the type of pad at the start of input. The system controller 210 operates the pad memory 2 based on the selected information.
Information is read from 400 and used for controlling parameters such as focal length and focus, and for posture detection.

As shown in the flow below, the same control as in the second embodiment is performed to start the input and extract the three-dimensional shape. Here, if it is desired to perform the retake during input, the retake mode is selected by the external input I / F section.

At this time, the recording section 250 reads out the signals recorded so far, and carries out a matching calculation process with the image currently being imaged.

When a correspondence between the currently imaged area and the read signal is obtained by the matching calculation processing, the LED or the like of the display 240 is turned on to notify the photographer that the input preparation is completed.

In the case where this re-taking is performed, the pad 210
It is also possible to change the arrangement of the subject 2101 on the second object, and in this case as well, matching processing is performed with the previously recorded signal, and input is started based on the area where the correspondence has been achieved.

When the input work is ended and the process is redone, the recording unit 250 reads out the imaging parameters in addition to the previously recorded three-dimensional shape and pixel signal, and the imaging parameters are the same as those previously captured. Set and enter.

Further, when it is desired to use the pad 2102 which is not registered in the memory 2400 in advance, the external input I
Setting is performed from a computer or the like through / F760.

(Fourth Embodiment) By using the image pickup system described in the first embodiment, not only the stereoscopic shape input but also the stereoscopic photographing mode can be performed. That is, it is possible to provide a stereoscopic system by using a plurality of imaging systems.

For selection, either the stereoscopic photographing mode or the stereoscopic shape input mode can be selected by using the external input I / F 760.

Next, the operation of the compound-eye image pickup device in the stereoscopic photographing mode will be described.

When the stereoscopic photographing mode is selected by the external input I / F 760, the images picked up by the left and right image pickup systems are output as a moving image at the output terminal.

Further, since the EVF 240 can find the overlapped portion of the left and right images by obtaining the correlation between the first image memories 73 and 83, the portion corresponding to the right side image pickup system of the overlapped portion can be seen. As shown in FIG. 26, the brightness of the part which is not hungry is displayed low. In the figure, when a subject on the table is imaged, the photographer understands that both ends of the screen are low-intensity parts and the ball on the left side in the figure is in the low-intensity part, so stereoscopic representation is not possible.

On the other hand, it can be seen that the tree and the plate in the center can be three-dimensionally expressed. You can see at a glance where the three-dimensional representation is possible. Each time the release button 230 is pressed, the left and right images are compressed in JPEG and the recording unit 25
Recorded as 0.

(Fifth Embodiment) In order to provide the image pickup apparatus of the present invention at low cost, it is conceivable that the EVF 240 will be an optical type. Then, if it is optical, it is impossible to represent a previously captured image on the viewfinder.

Also, unless it is the TTL viewfinder, the imaging area and the observation area in the viewfinder are deviated from each other, so that there is a possibility that the photographing with the overlapping area may fail.

Therefore, as a fifth embodiment of the present invention, the functions of other examples in this specification are provided at low cost. Specifically, an LED is provided in the visual field of the optical finder or in the vicinity thereof, and the LED is turned on and off according to the output of the correlation detection circuit. That is, the LED is turned on when there is an overlapping portion, and the LED is blinked when there is no overlapping portion. Thereby, the imaging device can be provided at low cost.

Further, a plurality of LEDs are provided in the X and Y directions of the display section, and the overlapping section lights the LEDs. This makes it possible to recognize not only the presence or absence of the overlapping portion but also the ratio of the overlapping portion to the display area and other positions, which is easier to understand.

[0218] Further, the viewfinder is an electronic type LED
Even if the above is provided, cost reduction of the video signal processing circuit can be realized.

Furthermore, by forming the visual field frame of the optical finder from liquid crystal, the overlapping portion can be expressed more finely than the LED.

(Sixth Embodiment) The concept and basic configuration of three-dimensional shape extraction in the tertiary shape extraction apparatus of the sixth embodiment are the same as in FIG.

However, the function of inputting a three-dimensional shape is performed by activating an image input program of a computer, for example, not in the image pickup system. At this time, the user first places the measured object on the correction pad.

Then, when the image input program of the computer is started by a command from the input device, a window corresponding to the finder of the camera (hereinafter referred to as finder window) is generated on the display device of the computer, and the window is opened when the camera is switched on. An image captured by the camera is displayed inside.

The user views the displayed image, performs framing so that the object to be measured is almost in the center of the screen, and presses the shutter. Then, the object to be measured is directly scanned, and the image data of the object to be measured is acquired. This image data is processed by the camera-dedicated processing device and the processing device to obtain three-dimensional data of the measured object.

[0224]

As described above, according to the image processing apparatus of claim 1 of the present application, an image is input while moving around a subject using a monocular or compound eye imaging system, and each of them is input. By subjecting the image data to image synthesis, a subject image at an arbitrary viewpoint can be easily obtained.

Further, according to the image processing apparatus of the second aspect of the present application, each image pickup position of the image pickup means is further stored together with the picked-up image, and a plurality of image data are integrated, so that a stereoscopic image at an arbitrary viewpoint is obtained. The shapes can be easily combined, and the image data file and the other data files can be combined and managed to be used for image generation.

According to the image processing apparatus of the third aspect of the present invention, in the apparatus for obtaining the shape of the subject in the image processing by the finder, the appropriate state image for obtaining the three-dimensional shape can be input. By making it possible to automatically adjust the imaging parameters and displaying the shape corresponding to the imaging area currently being imaged with respect to the previously captured imaging area on the monitor, the already captured image and the current imaging Since it is possible to perform imaging while easily recognizing the overlapping area with the image being processed, it is possible to realize accurate shape extraction and easily and reliably combine images from any viewpoint. Becomes

Further, according to the invention of claim 4 in the present application, the image of the subject shape data viewed from an arbitrary viewpoint is calculated as the two-dimensional image data.
It becomes possible to use the image format according to the purpose.

According to the invention described in claim 5 of the present application, since the image data can be used by combining with the text file, the image data can be displayed and processed in association with various additional information. The image database can be easily constructed and various usage forms are possible.

According to the invention of claim 6 in the present application, by holding a plurality of pieces of image information of the pad which is the background of the subject, the position information of the image pickup means can be known from the image information of the pad. The processing is easy, and the position change of the image pickup means can be easily and surely detected.

According to the invention described in claim 7 of the present application, each image pickup position of the image pickup means is stored together with the picked-up image, and a plurality of image data are integrated, so that a three-dimensional shape can be easily obtained from an arbitrary viewpoint. The image data file and the other data files can be combined and managed and used for image generation.

According to the invention of claim 8 of the present application, each image pickup position of the image pickup means is stored together with the picked-up image, and a plurality of image data are integrated, so that the three-dimensional shape at any arbitrary viewpoint can be easily obtained. The image data file and other data files can be combined and managed to be used for image generation, and the imaging device is equipped with an optical finder, so the device can be manufactured at low cost. Can be realized.

According to the invention described in claim 9 of the present application, each image pickup position of the image pickup means is stored together with the picked-up image, and a plurality of image data are integrated, so that the three-dimensional shape can be easily obtained from an arbitrary viewpoint. It can be used for image generation by synthesizing and managing image data files and other data files as well as calculating the three-dimensional shape data of the subject by software in the calculator. Is possible.

As described above, when the three-dimensional shape of the subject is obtained from a plurality of images by automatically adjusting the imaging parameters so that the image is input in an appropriate state to obtain the three-dimensional shape, It is possible to easily provide overlapping areas between images, and it is possible to convert the obtained shape into an image from an arbitrary viewpoint and combine it with a text file, which is also effective in an image database.

[Brief description of drawings]

FIG. 1 is a block diagram showing a first embodiment of an image information processing apparatus of the invention.

FIG. 2 is a diagram showing an image form of a subject.

FIG. 3 is a diagram showing a state in which a viewpoint for viewing a subject is changed.

FIG. 4 is a block diagram of an apparatus according to the present invention.

FIG. 5 is a diagram showing a configuration of a controller according to the first embodiment of the present invention.

FIG. 6 is a diagram showing a control flow of the device according to the present invention.

FIG. 7 is a diagram showing an outline of focusing of an apparatus according to the present invention.

FIG. 8 is a diagram showing an outline of zoom adjustment of the apparatus according to the present invention.

FIG. 9 is a diagram showing an outline of zoom control of the apparatus according to the present invention.

FIG. 10 is a diagram showing a state of a finder according to the present invention.

FIG. 11 is a diagram showing an outline of calculation of distance information from a stereo image according to the present invention.

FIG. 12 is a diagram showing an outline of integration of distance information according to the present invention.

FIG. 13 is a diagram showing an outline of an intermediate point interpolation method according to the present invention.

FIG. 14 is a diagram showing an outline of a coordinate system of an image pickup system according to the present invention.

FIG. 15 is a diagram showing an outline when the imaging system according to the present invention is rotated.

FIG. 16 is a diagram showing an outline of a flow of synthesizing a text file, image information, and distance information according to the present invention.

FIG. 17 is a diagram showing template matching according to the present invention.

FIG. 18 is a diagram showing a method of converting distance information into a unified coordinate system according to the present invention.

FIG. 19 is a diagram showing that a text file and an image information file according to the present invention are combined.

FIG. 20 is a diagram showing a flow in which image information according to the present invention is fitted to a model image.

FIG. 21 is a diagram showing an outline of a three-dimensional shape extraction device according to a second embodiment of the present invention.

FIG. 22 is a block schematic diagram of a three-dimensional shape extraction device according to a second embodiment of the present invention.

FIG. 23 is a diagram showing an outline of a flow of a three-dimensional shape extraction device according to a second embodiment of the present invention.

FIG. 24 is a block schematic diagram of a three-dimensional shape extraction device according to a third embodiment of the present invention.

FIG. 25 is a diagram showing an outline of a flow of a three-dimensional shape extraction device of a third embodiment according to the present invention.

FIG. 26 is a diagram showing a state of a finder according to a fourth embodiment of the present invention.

[Explanation of symbols]

1 Imaging Head Unit (Camera Head Unit) of 3D Shape Extraction Device 2 Main Subject 4 Position Detection Unit (Position Detection Unit) 5 Memory 6 3D Image Data Calculation Unit 7 2D Image Data Calculation Unit 8 Monitor 9 Printer 1000 Data Synthesis Unit 1001 Text data creating means 11 Operating means 10R, 10L Imaging optical system 100R, 100L Shooting lens 101R, 101L Aperture 102R, 102L Image sensor 103R, 103L A / D converter 104R, 104L Video signal processor 105R, 105L Subject separation unit 106R , 106L Zoom control unit 107R, 107L Focus control unit 108R, 108L Aperture control unit 200 Illumination 201 Camera attitude detection unit 210 Controller 220 Image processing unit 230 Shutter 240 Display unit 250 Recording unit 260 Right difference detection unit 270 Focus detection unit 760 External input I / F 900 Microcomputer 910 Memory 920 Image calculation processing unit 110 Stereo image 111 Edge extraction processing unit 112 Edge image corresponding point extraction processing unit 113 Corresponding point extraction processing unit 114 Contradiction point Is a processing unit to be removed 115 A processing unit to determine an occlusion area 116 A processing unit to calculate a distance distribution 117 A processing unit to identify a feature point on the back surface 118 A processing unit to acquire imaging parameters, postures and movement relationships 120 Distance information 121 Coordinate System conversion 122 Integration of distance information 123 Occlusion area information 120 Display 160 Fitting processing means 161 Texture pasting 162 Distance template 163 File combining means 1600 Extracted distance information 1601 Corrected distance image 1602 Texture Image 1603 Text file 1604 2D / 3D file output 1800 to 1804 Virtual projection plane 1810 to 1814 for registering distance information Central axis (optical axis) of virtual projection plane 1901 Text file 1902, 1903 Image file 19001, 19002 Link information 19003, 19005, 19007 Text data 19004, 19006 Image insertion flag 19008, 19909 Image coordinate information 2001 Extracted image 2000 Model image 2002, 2003 Difference between extracted image and model image 2101 Main subject 2100 Stereoscopic extraction device 2102 Calibration Pads A, B, C, D Characteristic for posture detection with characters on the calibration pad

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Masayoshi Sekine, Inventor Masayoshi Sekine, 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor, Tatsutsugu Katayama 3-30-2 Shimomaruko, Ota-ku, Tokyo (72) Inventor Kotaro Yano 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Nobuo Fukushima 3-30-2 Shimomaruko, Ota-ku, Tokyo Canon Inc. (72) Inventor Masatsugu Yuwa Canon Inc. 3-30-2 Shimomaruko, Ota-ku, Tokyo

Claims (9)

[Claims] 1. A device for picking up an image of a subject by a single or a plurality of image pickup means, and outputting a stereoscopic shape and image information of the subject, the predetermined image when the subject is viewed from an arbitrary position from the stereoscopic shape data. An image information processing apparatus comprising means for converting into a form and forming a file. 2. An apparatus for picking up an image of a subject by a single or a plurality of image pickup means, and outputting a stereoscopic shape and image information of the subject, comprising position detection means for detecting a relative positional change of the image pickup means, The stereoscopic shape data of the subject is calculated from the relative positional relationship between the plurality of image data obtained by the image pickup means and the image pickup means obtained by the position means corresponding to the respective image data. Image information processing characterized by having arithmetic means for converting data into a predetermined image formation when a subject is viewed from an arbitrary position, and having means for synthesizing the image data file and another file apparatus. 3. The image pickup means according to claim 1 or 2, wherein the one or more image pickup means includes an image pickup optical system, and the image pickup optical system is configured such that the main subject is set in an overlapping region of the respective image pickup systems and the main subject is included. An image information processing apparatus comprising: a unit that automatically adjusts an imaging parameter so that the image is within the depth of focus; and a monitor that can display an imaging region of the imaging unit. 4. The arithmetic unit according to claim 1 or 2, wherein the arithmetic means converts the three-dimensional shape data into a predetermined image form when a subject is viewed from an arbitrary position.
An image information processing device characterized by being calculated as three-dimensional image data. 5. The image information processing apparatus according to claim 2, wherein the unit for synthesizing the image data file and the other file is a text file. 6. The image information input / output device according to claim 1, wherein the position detection unit that detects a relative position change of the image pickup unit holds information regarding a plurality of pads. 7. An apparatus for capturing an image of a subject by a single or a plurality of image pickup means, and outputting a stereoscopic shape of the subject and pixel information, comprising position detection means for detecting a relative positional change of the image pickup means, A plurality of image data obtained by the image pickup means and a means for calculating three-dimensional shape data of the subject from the relative positional relationship of the image pickup means obtained by the position detection means corresponding to each image data, A calculation means for converting the subject into a predetermined image formation when viewed from an arbitrary position, and a synthesizing means for synthesizing the image data file and another file, based on the output of the synthesizing means. An image information processing apparatus including means for stereoscopically viewing image data. 8. An apparatus for picking up an image of a subject by a single or a plurality of image pickup optical systems and outputting the stereoscopic shape and pixel information of the subject, wherein the image pickup apparatus includes an optical finder, and a relative position change of the image pickup means. A plurality of image data obtained by the image pickup means and the relative positional relationship of the image pickup means obtained by the position detection means corresponding to each image data And a means for synthesizing the file of the image data with another file, which has a means for computing from the stereoscopic shape data into a predetermined image formation when the subject is viewed from an arbitrary position. An image information processing apparatus characterized by having. 9. An apparatus for picking up an image of a subject by a single or a plurality of image pickup optical systems and outputting pixel information, comprising: position detecting means for detecting a relative positional change of the image pickup means; A plurality of image data and means for calculating three-dimensional shape data of the subject separately from the image pickup optical system from the relative positional relationship of the image pickup means obtained by the position detection means corresponding to each image data; Image information processing, comprising: arithmetic means for converting the shape data into a predetermined image form when the subject is viewed from an arbitrary position; and means for synthesizing the image data file and another file. apparatus.