JPH1198485A - Image input device and recording medium recording image processing program of the image input device and read by computer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image input device that receives an image photographed in a wide range by a camera and conducts display composition processing in response to a photographed object automatically. SOLUTION: A camera 12 is directed to an object and photographs the object, and an instruction device 13 instructs a processing type. An object decision device 20 decides the object based on information received from a camera direction detector 14 and denoting a photographing direction of the camera and the instruction content of the instruction device, and a mode identification device 21 decides the processing type of a photographed image by the information from the camera direction detector and the instruction device. In the composite processing, an image converter 18 converts an input image so that a geometrical shape of an object in the input image is correct, an image composition device 15 composes pluralities of images whose geometrical shape is correctly decoded and a display image converter 19 converts the image so that the shape and the direction of the image or the composes image and allows the resulting image to be displayed.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image input apparatus used in a TV conference apparatus, a personal computer, and the like. The present invention relates to an image input device having a function of combining and inputting, and a computer-readable recording medium that stores an image processing program of the image input device.

[0002]

2. Description of the Related Art Conventionally, an image input device used in a TV conference device, a personal computer, or the like is a CCD (Character).
Generally, an imaging device such as a video camera using a solid-state imaging device such as a Ge Coupled Device is used. With this type of image input device, the resolution and the imaging range are determined according to the number of pixels of the CCD area sensor, and when it is necessary to input a wide range of images, the required resolution and the imaging range are made compatible. There was a problem that could not be read. Further, since the shape of the obtained image is fixed, there is a problem that it is difficult to obtain an image including a long range in, for example, the horizontal direction, the vertical direction, and the oblique direction.

[0003] In order to solve the problem of the conventional image input device, there has been proposed a method of synthesizing a plurality of images taken by one camera. The invention described in Japanese Patent Application Laid-Open No. 8-65457 is known as a prior art especially for calligraphy. According to this conventional technique, only a required portion of a document is acquired at a required resolution by a plurality of divided images, and a desired image is input by applying a combining process to the acquired image. The advantage is that you can selectively input only the necessary parts of the original at high resolution, but in order to generate images of a wide range of documents, it is necessary to move the camera while keeping the camera parallel on the object, In order to determine the combining position of the images, it is necessary to take the trouble of capturing the entire image in advance.

[0004] In addition, there are various objects to be imaged by the same video camera, and it is necessary for the user to switch and use different image input devices for each application by complicated procedures. As a conventional technique considering switching between a person and a document mode according to the posture of a camera, Japanese Patent Laid-Open No.
The invention described in Japanese Patent No. 8029 is known.
This prior art is intended to switch between a low-resolution mode and a high-resolution mode depending on the direction of the camera, but the image acquisition range is always constant, and image acquisition over a wide range has not been considered.

[0005]

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems which have not been solved in the prior art. More specifically, an object of the present invention is to provide an image input device capable of inputting an image captured over a wide range without moving an imaging device (hereinafter, referred to as a camera) in parallel with an imaging target. Provided is an image input device that automatically performs display and composition processing according to an imaging target simply by performing a simple operation of pointing a camera at the imaging target without being conscious of switching of an imaging target and an operation method. That is.

[0006]

According to the present invention, in order to solve the above-mentioned problems, an image input device includes a camera for imaging an object, an instruction device for a user to instruct processing contents, and a camera direction. The camera, the pointing device, and the camera direction detection device are integrated so that the user can freely change the imaging direction of the camera. A camera fixing mechanism that holds the camera at a certain height, and an object determination device that determines an imaging target from information of the camera direction detection device,
The mode determining device that determines the type of processing on the captured image from the information on the pointing device and the target determining device, and the information on the target determining device and the camera direction detecting device determines that the geometric shape of the capturing target included in the image captured by the camera is correct. Applying a geometric transformation so that the image conversion device stores the image in the image memory, and the image captured by the camera and the image stored in the image memory from the information of the mode determination device, the object determination device, and the camera direction detection device. A display image conversion device that converts the image into a shape suitable for display, a display device that displays the image converted by the display conversion device, and an image stored in the image memory from information of the mode determination device and the target determination device. An image synthesizing device that stores the image in the image memory, an image converted by the image converting device, and an image synthesized by the synthesizing device. An image memory for storing the formed image,
It consists of.

[0007]

Embodiments of the present invention will now be described in detail with reference to the drawings.

FIG. 1 is a block diagram showing a configuration example of an embodiment of the present invention.

In FIG. 1, an image input device 100 according to the present invention includes a camera 12 for imaging an object to be imaged, an instruction device 13 for a user to instruct the image input device 100 on image processing contents, and a camera 12. A camera direction detection device 14 for detecting a direction in which an image can be captured and a posture of the camera (hereinafter, referred to as a camera direction), and a camera 12 and a pointing device 13 so that a user can freely change the imaging direction of the camera 12. And a camera direction detecting device 14 integrated with the camera fixing mechanism 11 for holding a certain point or a certain axis as a center of rotation and holding the position of the camera 12 at a fixed height, and a camera direction of the camera direction detecting device 14. Is identified from the information indicating the type, and the identified information (hereinafter, referred to as an imaging target type) is output. Elephant determination unit 20, instruction information of the user obtained from the instruction unit 13 and the object determining device 20
A mode determining device 21 that determines the type of image processing for a captured image from the imaging target type
Based on the information indicating the type of imaging target and the direction of the camera of the camera direction detection device 14, a geometric transformation is applied so that the geometric shape of the imaging target included in the image captured by the camera 12 is correct, and stored in the image memory 17. The image conversion device 18, the information that determines the type of image processing from the mode determination device 21, the imaging target type from the target determination device 20, and the information indicating the direction of the camera 12 from the camera direction detection device 14 A display image converter 19 for converting an image captured by the camera 12 and an image stored in the image memory 17 into a shape suitable for display;
The image stored in the image memory 17 is obtained from the display device 16 that displays the image converted by the above, the information that determines the type of image processing from the mode determining device 21, and the imaging target type from the target determining device 20. An image synthesizing device 15 for synthesizing the image and storing it in an image memory 17;
And an image memory 17 for storing a combined image combined by the image combining device 15.

Next, an outline of the operation of the embodiment of the present invention will be described.

The camera fixing mechanism 11 holds the camera 12 at a certain height, and also holds the camera 12 so that the camera can freely rotate about a certain point or a certain axis.
2. The pointing device 13 and the camera direction detecting device 14 are integrated.

The direction of the camera rotatably supported by the camera fixing mechanism 11 is detected by a camera direction detecting device 14.

The pointing device 13 transmits the user's intention to the control device 10.
Is used to indicate the processing type for the image captured by the camera 12.

The user points the camera 12 at the object to be imaged so that the object to be imaged is imaged.

The object determining device 20 specifies the type of the imaging target based on the information on the direction of the camera from the camera direction detecting device 14 and the information on the processing type for the image to be captured from the pointing device 13. The imaging target type specifies the type of the imaging target of the camera 12. For example, the characteristics of the respective imaging targets are different between a case where a document is imaged and a case where a person who is participating in a conference such as a TV conference is imaged. The information is used to perform image processing according to the characteristics of each imaging target.

The mode determining device 21 determines a processing type for an image to be captured based on information on a camera direction from the camera direction detecting device 14 and information on a processing type for the image to be captured from the pointing device 13. For example, when the user wants to combine a plurality of images captured by the camera 12, the processing type is “synthesis”, and the user inputs that the user wants to perform the combination processing using the instruction device 13. Also,
It is also conceivable to determine the processing type for the image to be captured from information indicating the direction of the camera of the camera direction detecting device 14, and similarly, the pointing device 13, the camera direction detecting device 1
4 may be used to determine the processing type for the image to be captured.

The control device 10 instructs the object determining device 20 to determine the type of the imaging target, and the mode determining device 21 to determine the processing type for the captured image. Further, the control device 10 instructs the display image conversion device 19 to convert the image captured by the camera 12 according to the type of imaging target and display the converted image on the display device 16.

The display image conversion device 19 is determined by the mode determination device 21 according to the instruction of the control device 10, information indicating the direction and attitude of the camera of the camera direction detection device 14, the type of imaging target of the target determination device 20, and the mode determination device 21. From the processing type
Conversion is performed so that the shape and direction of the image captured by the camera 12 are correctly displayed on the display device 16.

The display device 16 displays the image converted by the display image conversion device 19.

The control device 10 includes a camera direction detecting device 1
4. The image conversion device 18, the display image conversion device 19, and the image synthesis device 15 are controlled based on information obtained from each of the target determination device 20 and the mode determination device 21. When the control device 10 determines from the processing type determined by the mode determination device 21 that it is necessary to execute the combining process,
First, the image conversion device 18 is instructed to correct geometric distortion included in an image captured by the camera 12 according to an imaging target.

In accordance with an instruction from the control device 10, the image conversion device 18 obtains an image captured by the camera 12 from information indicating the direction of the camera from the camera direction detection device 14 and the type of imaging target from the target determination device 20. Is converted so that the geometrical shape becomes correct, and the converted image is stored in the image memory 17. At the same time, the control device 10 instructs the image composition device 15 to perform the composition process.

The image synthesizing device 15 synthesizes a plurality of images whose geometrical shapes stored in the image memory 17 have been correctly restored in accordance with an instruction from the control device 10, and stores the images in the image memory 17. The control device 10 instructs the display image conversion device 19 to display the synthesized image on the display device 16.

The display image conversion device 19 converts the composite image stored in the image memory 17 so that the shape and the direction are suitable for the display device in accordance with an instruction from the control device 10. The display device 16 displays the composite image converted by the display image conversion device 19.

The composite image stored in the image memory 17 can be used by the external device 50.

[0025]

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a block diagram showing the configuration of an image input apparatus according to an embodiment of the present invention.

FIG. 2 is a diagram for explaining the configuration of the present embodiment. Each configuration when an image is input to the document 102 or the like as an object to be imaged will be described with reference to FIG.

In FIG. 2, a housing 101 includes a control device 10, an object determination device 20,
Mode determining device 21, image synthesizing device 15, image memory 1
7, an image conversion device 18 and a display image conversion device 19 are housed, and are connected to other components by cables.

The pointing device 13 is integrated with the camera 12, and the pointing device 13, the camera 12, and the camera direction detecting device 14 are integrated with the camera fixing mechanism 11.

The three-dimensional coordinate system 300 is a coordinate system temporarily set for the sake of simplicity. The origin is set to the mounting portion of the axis on which the camera 12 is mounted to the camera fixing mechanism 11, the front is the Y axis, This is a coordinate system in which the right direction is the X axis and the upper direction is the Z axis.

The camera 12 is held so as to freely rotate around the X axis, and so that the top, bottom, left and right of the image coincide with the physical top, bottom, left and right when facing the Y axis direction. The direction in which the camera 12 can capture an image is detected by the camera direction detection device 14. The camera fixing mechanism 11 is attached to the display device 16, and the rotation center of the camera is located at a certain position above the document surface.

In the following description, for the sake of simplicity, it is assumed that the rotation center of the camera 12 matches the viewpoint. If the distance to the object to be imaged is sufficiently longer than the distance between the viewpoint of the camera 12 and the center of rotation, a slight difference between the two has only a negligible effect. Also, of course,
The present invention can be applied on the assumption that the rotation center and the viewpoint do not match.

FIG. 9 is a diagram for explaining a camera model in this embodiment.

In FIG. 9, half angles of the horizontal and vertical viewing angles of the camera 12 are α and β (α, β>
0), the focal length is f (f> 0), and the captured image is a rectangular area within the range (± U, ± V) centered on the image center (0, 0). Each value is determined at the time of designing a camera or obtained by measuring actual camera characteristics. Since the characteristics do not change before and after the processing, it can be regarded as known. Therefore, it is assumed that the information is stored in advance in the apparatus. Regarding the technology for measuring camera characteristics,
“Exit“ Recent Trends in Camera Calibration Techniques ”, Jikken Kenho Vol.93, No.25, CV821,199
3 ”, so it is omitted here.

FIG. 8 is a diagram showing a model for explaining a coordinate system when an image of an original is captured by the camera 12 in this embodiment.

In the three-dimensional coordinate system 300 shown in FIG. 8, the rotation center of the camera 12, that is, the viewpoint R301 is (r,
(0,0). In the three-dimensional coordinate system 300, it is assumed that the document exists in an area (± L + r, ± M, −H) (L, M> 0) on the document surface 305 at Z = −H (H> 0). . The direction of the optical axis 303 of the camera 12 detected by the camera direction detection device 14 is defined as θ. θ
Is positive when rotated counterclockwise when viewing the X axis in the positive direction, and negative when rotated clockwise, and the camera 12 is oriented in the same direction as the Y axis. 0 for case
As the degree, the camera direction θ is detected within the range of −180 degrees <θ ≦ 180 degrees. Also, four corner points a410, b411, c412, d413, on the image plane of the camera 12,
And the image center e414 are points A310, B311, C312, D313, and E314 on the document surface 305, respectively.
Shall correspond to

Here, from the original surface 305 to the rotation center 301
The height up to is a value determined by the camera fixing mechanism 11 and the display device 16, and is a constant in the image input device of the present invention. Also, the document existence range is determined by the maximum size of the target document, and both are stored in advance in the apparatus.
These values may be configured to be set by the user when using the image input device according to the present invention.

The camera direction detecting device 14 in the present embodiment
Is constituted by a small rotary encoder, and returns the direction of the camera 12 in units of angles. In the present embodiment, the camera direction detecting device 14 using the small rotary encoder will be described, but this does not limit the application of the present invention. As shown in FIG. 5, a contact switch is provided for each fixed angle, and other mechanisms such as a structure in which an angle is identified by the number of a switch that is energized can be used.

Hereinafter, the flow of processing in this embodiment will be described in detail with reference to the drawings.

FIG. 7 is a diagram for explaining the flow of processing in the control device 10.

The user directly operates the camera 12 to point the camera 12 at an object to be imaged. At this time, the mode determining device 2
In step A1, it is determined whether the mode is the combination mode or the display mode by referring to the information on whether the pointing device 13 is pressed or not (step A1). That is, when the user changes the direction of the camera 12 while pressing the pointing device 13, the combination mode is regarded as the combination mode, and when the direction of the camera 12 is merely changed, the display mode is regarded. With this configuration, when the user wants to acquire a wide range of images, the pointing device 1 is turned to the object to be imaged by the camera 12.
A desired composite image can be obtained only by an intuitive operation of changing the direction of the camera 12 so as to lick the object while pressing 3.

In the display mode, the control device 10 displays the image captured by the camera 12 on the display device 16 by the processing of steps A2 to A5.

The control device 10 specifies an object to be imaged by the object determining device 20 (step A2).

The target determining device 20 determines from the information of the camera direction detecting device 14 that the type of image capturing target is a document when the camera 12 captures a range corresponding to the document on the document surface 305, If an image is captured in a range other than the range, it is determined that the type of imaging target is a landscape or a general landscape such as a person. That is, the vertices a41 of the four corners of the captured image
Imaging area 10 on original surface 305 corresponding to 0 to d413
If the Y coordinates of the points A310 to D313 forming the third point fall within the document area, the image capturing target type is determined to be a document, and if not, it is determined to be a general landscape. However, this imaging target type is merely an example, and the present invention is not limited to this example.

The points a 410 to 413 correspond to the three-dimensional coordinate system 30
0, when θ = 0, a (-U + r, f, V) (Equation 1-1) b (-U + r, f, -V) (Equation 1-2) c (U + r, f, -V) ( (Equation 1-3) d (U + r, f, V) (Equation 1-4) This is rotated about the X axis by θ, and the points projected on the plane Z = −H are points A310 to D3.
13 The point a410 by θ rotation around the X axis
d413 is a point ar, br, c on the three-dimensional coordinate system 300
r, dr, ar = (ax, ay, az) = (− U + r, f · cos θ + V sin θ, −f · sin θ + V · cos θ) (equation 2-1) br = (bx, by, bz) = (− U + r, f · Cos θ-V sin θ,-f · sin θ-V · cos θ) (Equation 2-2) cr = (cx, cy, cz) = (U + r, f · cos θ-V sin θ,-f · sin θ-V · cos θ) (Equation 2) 2-3) dr = (dx, dy, dz) = (U + r, f · cos θ + Vsin θ, −f · sin θ + V · cos θ) (Equation 2-3) From this point, the coordinates of the points A 310 to D 313 are A ( ax · (−H / az), ay · (−H / az), −H) (Equation 3-1) B (bx · (−H / bz), by · (−H / bz), −H) (Equation 3-2) C (cx · (−H / cz), cy · (−H / cz) , -H) (Formula 3-3) D (dx · (-H / dz), dy · (-H / dz), - H) become (Equation 3-4). However, the points A310 to D313 are defined as follows: (−H / az)> 0 (Equation 4-1) (−H / bz)> 0 (Equation 4-2) (−H / cz) > 0 (Equation 4-3) (−H / dz)> 0 (Equation 4-4) The control device 103 determines that the Y coordinates of the points A310 to D313 forming the imaging region 103 fall within the document region, that is, all the expressions 4-1 to 4-4 are satisfied, and ay · (−H / Az) ≦ M (Formula 5-1) by · (−H / bz) ≧ −M (Formula 5-2) cy · (−H / cz) ≧ −M (Formula 5-3) dy · (−H / Dz) ≦ M (Equation 5-4) is satisfied, it is assumed that the imaging target type is a document, and otherwise, it is assumed that a general landscape is being imaged. As described above, U, V, r, H, M, and f are known, and values stored in advance in the apparatus can be used. Therefore, whether or not the above condition is satisfied depends only on θ. . That is, the target determination device 20 determines the imaging target based only on the information θ obtained from the camera direction detection device 14.

It should be noted that the method of identifying the type of imaging target described here does not limit the scope of the present invention. The criterion can be changed according to the size and shape of the document and the characteristics of the camera. In the above-described example, the imaging target is determined using only the Y coordinate, but it is possible to limit the imaging target including the X coordinate. It is also possible to adopt a configuration in which the presence or absence of a document is read by an optical sensor or the like on a desk, and only when the document is present and the above conditions are satisfied, the imaging target type is regarded as a document. In this case, a more accurate processing target can be determined.

Next, the control device 10 captures an image with the camera 12 (step A3), and converts the captured image into a form suitable for display in accordance with the identification result of the type of imaging target in step A2. Image conversion device 1
9 to instruct to perform a display image conversion process (step A4).

FIG. 3 is a diagram for explaining a display image conversion process when the type of imaging subject is a general landscape.

FIG. 3A shows a state in which a general scene is imaged when the camera is facing in the positive direction of the Y axis. At this time, an image taken by the camera 12 and
The physical relationship between up, down, left, and right from the viewpoint is the same. However, when the camera 12 is in front, that is, when the camera 12 is oriented in the negative direction of the Y axis (FIG. 3B), the vertical and horizontal relations of the image obtained by the camera 12 are physically different from the viewpoint. Does not match, and the top, bottom, left and right are reversed. Therefore, when the target is a general landscape and the front is imaged, the control device 10 inverts the top, bottom, left, and right of the image picked up by the camera 12 by the display image conversion device 19, and The display image conversion device 19 is controlled so that an image is displayed in a state where the physical positional relationship is correct at 16. Specifically, when the imaging target is a general landscape and the optical axis 302 faces the negative direction of the Y axis, the image is inverted vertically and horizontally. here,
When θ = 0, the image center e414 exists in e (r, f, 0) (Equation 6), and by rotating θ around the X axis, er = (ex, ey, ez) = (r, f · cos θ) , −f · sin θ) (Equation 7). At this time, if ey <0, it corresponds to the optical axis pointing in the negative direction of the Y axis, and the corresponding condition is cos θ <0 (Equation 8). That is, the display image conversion device 19 inverts the top and bottom and left and right of the camera image when the imaging target type is a general landscape and Expression 8 is satisfied, and uses the captured image as it is when the conditions are not satisfied. In addition, TV
When it is preferable that the left and right directions of the image coincide with the left and right as seen from the user, such as when capturing the image of the user himself / herself in a conference, it is also possible to adopt a configuration in which only upside down is applied. In this case, when the mode determining device 21 determines that the user's own image is to be captured from the camera direction, the mode is set to, for example, “TV conference mode”, and only the upside-down inversion is applied by the display image converter 19. Alternatively, the user may input an instruction that the TV conference mode is set by the instruction device 13, and when the input is made, the display image conversion device 19 applies only the upside down operation. You may comprise.

The image converted in step A4 is displayed on the display device 16 (step A5). Then, control device 10 repeats the processing of step A1 and subsequent steps again.

If it is determined in step A1 that the mode is the combination mode, the processing of steps B1 to B10 is performed to combine the captured images.

The control device 10 repeats the processing of steps B2 to B7 until the combining mode is released, that is, until the pointing device 13 is released (step B1).

First, similarly to the processing in step A2, the control device 10 specifies the current imaging target by the target determination device 20 based on the information of the camera direction detection device 14 (step B2). If the imaging target type changes from the previous imaging target type, that is, if the imaging target type changes from the original to the general landscape and from the general landscape to the original and it becomes necessary to apply a different geometric transformation, the processing is performed. The operation is suspended, and the process proceeds to step B8 (step B3).

The control device 10 repeats the processing of steps B4 to B7 while the combination mode is not canceled and the same imaging target type is imaged.

First, a target divided image is obtained by the camera 12 (step B4). Imaging is automatically performed when the direction of the camera changes from the angle at the time of the immediately preceding imaging by a threshold Th or more. The threshold Th is set to a vertical viewing angle of 2 ×
By making it smaller than β, it becomes possible to acquire an image including all the imaging targets. When a camera direction detection device 14 having a simpler structure and a larger detection angle as shown in FIG. 5 is used, an image can be taken every time a new contact switch is turned on. In addition to the method of performing imaging using the angle as a threshold, imaging can be performed using a fixed time interval. At this time, the product including the maximum angular velocity and the time interval when the user moves in the direction of the camera 12 is set to be smaller than the vertical viewing angle 2 × β, so that the image including all the imaging targets can be obtained. Can be obtained.

Next, based on the information of the camera direction detecting device 14, a geometric transformation according to the type of the imaging object is applied, and the transformed image is stored in the image memory 17 (step B5).

FIG. 4 is a diagram for explaining the geometric transformation in this embodiment.

The image conversion device 18 stores the image in the image memory 17 together with the information of the camera direction detection device 14 without applying the geometric conversion particularly when the type of the imaging target is the general scenery. Hereinafter, the geometric conversion processing for the document will be described.

In FIG. 4B, an imaging range 103 is a range where the camera 12 captures an image of the document 102. Point A31
Areas surrounded by 0 to D313 are points a410 to d on the image.
413, and an image as shown in FIG. 4A is generated. Geometric distortion occurs in the image generated by this projection process, and the geometric shape on the document 102 cannot be correctly maintained in the image. The image conversion device 18 corrects the distortion due to the projection process based on the information of the camera direction detection device 14.

FIG. 6 shows the original 1 in the image conversion device 18.
FIG. 11 is a diagram for describing a flow of a geometric transformation process for the image data 02.

First, the image conversion device 18 executes step A
2, and 3 of points A310 to D313 in accordance with Equations 1 to 3 in the same manner as the imaging target determination method in step B2.
Find coordinates in the three-dimensional coordinate system 300 (step C
1). The area surrounded by the points A310 to D313 corresponds to a correct geometric shape. By projecting the image back into this region, distortion caused by the projection assumption is removed.
For this purpose, first, an element I in an area surrounded by points A310 to D313 is selected without duplication (step C2).
The element I is a point on the document surface 305 and is represented by I (x, y, -H) (Equation 9). Next, the image conversion device 18 obtains coordinates on the image plane corresponding to the element I in order to obtain a pixel value corresponding to the element I (step C3). For that, the original side 3
05 is rotated about the X axis by -θ, and a projection point on the image plane of Y = f is obtained. Considering a coordinate Ir obtained by rotating the element I by -θ around the X axis and using the center of rotation as a reference, I
Moves to Ir (Ix, Iy, Iz) = (x−r, y · cos θ + H · sin θ, −y · sin θ−H · cos θ) (Equation 10). Coordinates of Ir projected onto the Y = f plane, (Ix · (f / Iy), f, Iz · (f / Iy)) (Equation 11) From the equation (11), the coordinate i on the image plane is i (Ix · (f / Iy), Iz · (f / Iy)) (Equation 12). At this time, since the coordinate i is not always an integer, the pixel value of the element I is determined from the value on the image by bilinear interpolation (step C4). When the pixel values are obtained for all the elements I, the area surrounded by the points A310 to D313 is stored as a new image in the image memory 17 together with the coordinates of the point E314 which is the projection point of the image center e414 (step C5). The processing in the conversion device 18 ends.

Note that the image conversion device 18 in this embodiment is
Does not limit an application example of the present invention. For the geometric transformation performed by the image transformation device 18, a method used as a process accompanying the projective transformation can be applied. Such processing is described in “Exit“ Image and space ”, p.
p. 104-118, Shokodo, 1991 "and" Takagi et al., "Image Analysis Handbook", pp. 582-591, University of Tokyo Press, 1992 ", and are omitted here.

At the same time as the geometric conversion processing in the image conversion device 18, the control device 10
After applying the same image conversion processing by the display image conversion device 19 as in Step 4 (Step B6), the display device 16 displays the captured image so that the physical vertical relationship is maintained correctly (Step B7). It returns to step B1.

When the synthesizing mode is canceled in step B1 or B3, the control device 10 generates a synthetic image from the group of geometrically transformed images stored in the image memory 17 by the image synthesizing device 15 (step B).
8).

FIG. 10A is a diagram for explaining the composition when the object to be imaged is a general landscape. Each image in the image memory 17 has a number j (j ≧ 1), Further, angle information θj obtained from the camera direction detecting device 14 at the time of imaging is added. The image synthesizing device 15 arranges images according to the angle information θj of each image, and
To generate a composite image by projecting the respective images.

That is, when considering the point p (u, v) on the image j, the coordinates pr on the three-dimensional coordinate system 300 when θ is rotated about the X-axis of P are expressed by pr = (px, py, pz) = (u, f · cos θ + v · sin θ, −f · sin θ + v · cos θ) (Equation 13), which is expressed as Y = Q · cosρ, Z = Q · sinρ, −180 degrees <Ρ ≦ 180 degrees, Q> 0 A projected image is projected onto the cylindrical surface 501 represented by the following equation (14), and a composite image is generated and stored in the image memory 17. Such projection processing is described in “Takagi et al.,“ Image Analysis Handbook ”, pp. 276-
292, The University of Tokyo Press, 1992 ", so details are omitted. In addition to the method described in the present embodiment, a method introduced in the reference can be applied.

FIG. 10B is a view for explaining a process of generating a composite image when the object to be imaged is a document.
Each image in the image memory 17 has a number j (j ≧ 1) in the order in which the images were captured, and a geometric conversion process (step B).
The position (xi, yi, zi) of the projection point E414 at the center of the image on the image plane 305 added to 5) is added. The image synthesizing device 15 generates a synthesized image by sequentially projecting each image on the same plane according to the position information, and stores the synthesized image in the image memory 17.

After the image synthesizing process by the image synthesizing device 18 is completed, the display image converting device 19 causes the image memory 1
7 is enlarged or reduced to a size suitable for the display device 16 (step B9) and displayed on the display device 16 (step B10). Further, the composite image stored in the image memory 17 is taken out by the external device 50 and used.

As described above, the image input apparatus according to the present invention automatically determines the object to be imaged and performs conversion in accordance with the object to be imaged and the imaging direction only by pointing the camera 12 at the object to be imaged. An appropriate image is displayed on the display device 16. At the same time, by scanning the imaging target with the camera 12 while pressing down the pointing device 13, a plurality of images are automatically acquired, and a geometric transformation according to the imaging target is applied. Generate a composite image with a shape. In other words, the user can easily input an image in which the geometric shape is correct and a wide range is projected simply by pointing the camera 12 at the imaging target and scanning the imaging target while pressing the pointing device 13. It becomes possible.

Next, a second embodiment of the present invention will be described with reference to the drawings.

FIG. 11 is a diagram for explaining the second embodiment.

The image input device according to the second embodiment of the present invention differs from the image input device according to the first embodiment in the following points.

The camera 12 is held by the camera fixing mechanism 11 so as to rotate also around the Y axis, and the camera direction detecting device 14 determines not only the rotation amount θ about the X axis but also the rotation amount φ about the Y axis. Can also be detected. Camera 12
Is rotated θ around the X axis first, then φ around the Y axis.
It is expressed by considering it as rotated. The control device 10
The object to be imaged is identified using the rotation amount θ around the X axis and the rotation amount φ around the Y axis obtained by the camera direction detection device 14. Further, when controlling each component device, the control device 10 sends information of the camera direction detection device 14, that is, the rotation amount θ around the X axis and the rotation amount φ around the Y axis together with an instruction. Each device is configured to perform display image conversion, geometric conversion, and synthesis processing using the rotation amount θ around the X axis and the rotation amount φ around the Y axis.

In the following description, it is assumed that the viewpoint coincides with the center of rotation of the camera 12 around the X axis, as in the first embodiment. The camera model and the coordinate system at the time of imaging of the original by the camera 12 are the same as in the first embodiment. However,
The inclination around the Y axis detected by the camera direction detection device 14 is defined as φ. φ is positive when the Y axis is rotated counterclockwise toward the positive direction, and is negative when the Y axis is rotated clockwise, and the camera 12 is oriented in the positive direction of the X axis. Taking the case as 0 degree, the camera inclination φ is detected within the range of −180 degrees <φ ≦ 180 degrees. The original surface 30
Assume that the height from 5 to the center of rotation 301 is known,
The setting method of the document existence range is the same as in the first embodiment.

The camera direction detecting device 14 in the second embodiment is different from the camera direction detecting device 14 in the first embodiment.
And a small rotary encoder for detecting the amount of rotation about the Y axis is added, and the amount of rotation θ about the X axis and the amount of rotation φ about the Y axis are returned in angle units.
In this embodiment, this small rotary encoder is
A description will be given of the camera direction detecting device 14 which utilizes one of the embodiments, but the application of the present invention is not limited to the same as in the first embodiment. It is needless to say that a structure in which contact switches are provided at every predetermined angle as shown in FIG. 5 and the angle is identified by the energized switch can be used for detecting the rotation amount around the Y axis.

The processing flow in the second embodiment is the same as the processing flow in the first embodiment. Hereinafter, based on the processing flow in FIG. 7, the operation of the second embodiment will be described focusing on the differences from the first embodiment.

In the second embodiment, step A1 is the same as in the first embodiment, but the user determines the rotation amount φ around the X-axis and then changes only θ to obtain the target image. Shall be entered.

In the imaging target specifying process in step A2, when the camera 12 is imaging an area corresponding to the original on the original surface 305, it is determined that the imaging target is the original, and the other areas are imaged. In this case, it is determined that the imaging target is a landscape or a general landscape such as a person, but the vertices a410 to d413 of the four corners of the captured image are the same.
Are different in the coordinates of points A310 to D313 that form the imaging region 103 on the document surface 305 corresponding to

The points a 410 to 413 have the coordinates expressed by Equation 1 when θ = 0 and φ = 0. This is θ around the X axis.
Point a410 that has been rotated and further rotated φ around the Y axis.
The transformation for moving d413 to points ar to dr on the three-dimensional coordinate system 300 is as follows.

[0079]

(Equation 1)

Is obtained. The procedure for obtaining points A310 to D313 on the plane Z = -H from ar to dr, the conditions for determining the imaging target, and the like are all the same as in the first embodiment. Further, the imaging process in step A3 is the same as in the first embodiment.

Also in step A4, the display image conversion device 19 performs a display image conversion process in consideration of the rotation amount φ about the Y axis. That is, since the u axis of the image obtained by the camera 12 is inclined by φ with respect to the X axis,
The obtained image is rotated by φ with the center of the image plane as the origin. At the same time, similarly to step A4 in the first embodiment, when the camera 12 is in the foreground, that is, when the target is a general landscape and the foreground is imaged,
The display image conversion device 19 rotates the image captured by the camera 12 by φ and then flips the image left and right so that the display device 16 displays the image in a state where the physical positional relationship is correct. The conversion device 19 is controlled. In addition,
When it is preferable that the left and right directions of the image coincide with the left and right as seen from the user, such as when capturing the image of the user himself at a TV conference, it is possible to configure so that only the rotation processing is applied. This is the same as the first embodiment. Here, considering the direction of the optical axis, the image center e414 is rotated by θ around the X axis and φ around the Y axis, so that er = (ex, ey, ez) = (r · cos φ + f · sin φ · sin θ, f · cos θ, r · sin φ−f · cos φ · sin θ) (Equation 16) The condition for determining the case where the optical axis is oriented in the negative direction of the Y axis is cos θ < 0 (Equation 17). The converted image is displayed on the display device 16 in step A5.

Steps B1 to B7 are the same as in the first embodiment. However, in the geometric transformation of step B5, step C1 is based on Equations 3 and 15,
Point A31 corresponding to points a410 to d413 on the image plane
0 to D313 are obtained. In step C3, instead of equation 10,

[0083]

(Equation 2)

The coordinates Ir are obtained based on the following equation (11) and equation (1).
2, coordinates (u, v) on the image plane are obtained.

In the synthesizing process of step B8, when the object to be imaged is a general scene, first, of the rotation amount of the camera 12, φ =
Assuming 0, the composite coordinates are formed on the cylindrical surface on the basis of only θ in the same manner as in the first embodiment. After that, the obtained composite image is rotated by φ around the Y axis to generate a composite image in which the physical vertical relationship matches the vertical relationship in the image. When the object to be imaged is a document, a composite image is generated by arranging each image on a plane based on the image center E314 of each image, as in the first embodiment.

As described above as the second embodiment of the present invention, the image input apparatus according to the present invention automatically determines the object to be imaged only by pointing the camera 12 at the object to be imaged, and Then, conversion according to the imaging direction is performed, and an appropriate image is displayed on the display device 16. At this time, since the camera is configured to be rotatable around the Y axis,
Horizontal and vertical pan operations can be easily realized. At the same time, as in the first embodiment, the user can turn the camera 12 toward the imaging target, scan the imaging target while pressing the pointing device 13, and obtain the correct geometric shape, horizontal, Panoramic images taken in vertical and oblique directions can be easily input.

In the description of the second embodiment, the amount of rotation φ around the Y axis is variable, but may be fixed at a certain angle. For example, when the camera 12 is configured to be fixed at φ = 10 degrees, the imaging range of the camera 12 moves in the X-axis direction more than when φ = 0 degrees.
6 and can be installed closer to the housing, and the entire device can be made more compact.

Further, it has been described that when φ = 0 and θ = 0, the top, bottom, left and right of the image picked up by the camera coincide with the top, bottom, left, and right of the three-dimensional space. It is not limited. For example, the upper, lower, left, and right sides of the image may correspond to the physical right, left, upper, and lower sides, that is, when φ = 90 degrees, the upper, lower, left, and right sides of the image and the three-dimensional space may be configured to match.

Further, in the above description of the first and second embodiments, an example was described in which the device was configured as a dedicated device. However, the display device 16, the display image conversion device 19, the image memory 17, the image synthesis device 15, The image conversion device 18, the object determination device 20, the mode determination device 21, and the control device 10 can be realized by a normal personal computer. At this time, the camera fixing mechanism 11 may be configured to be fixed to the upper part of the display of the personal computer.
In particular, if the camera fixing mechanism 11 is fixed to the upper part of the liquid crystal display unit of the portable personal computer, a single camera can be used for capturing a person such as a TV conference and for inputting a document, so that portability can be improved. Is greatly enhanced.

In the above description of the first and second embodiments, the processing is performed on the assumption that the angle information from the camera direction detecting device 14 does not include an error. May be included, but in such a case, it is also possible to adopt a configuration in which an error is eliminated by applying a positioning process between images. For example, the θ and φ added to each image are varied within a range in which an error can be eliminated, a geometric transformation is applied to each image by the image conversion device 18, and when the images are combined by the image synthesis device 15, overlapping between the images is performed. The similarity of the parts is evaluated using the cross-correlation coefficient and the residual, and θ and φ with the highest similarity are searched for, so that the error included in the angle information is eliminated. A composite image with high accuracy can be generated.

Further, the image input device according to the present invention has the configuration shown in FIG.
As shown in FIG. 1, a computer generally called a personal computer includes a camera 12 and a camera direction detecting device 1.
4 is mounted on a personal computer by a camera fixing mechanism 11 capable of at least integrating the control device 10, the image synthesizing device 15, the image memory 17, the image converting device 18, and the display image converting device 1 on a personal computer.
9, a mechanism corresponding to the object determining device 20 and the mode determining device 21 may be provided. In this case, a computer-readable program for realizing the above mechanism is created on a personal computer, and the program is stored on a CD-ROM.
The program can be easily realized by recording the program on a recording medium such as a floppy disk or a floppy disk and reading the program from the recording medium on a personal computer.

The pointing device 13 may be attached to the camera 12 as in the previous embodiment,
Generally, a keyboard, a mouse, or the like, which is widely used as an input device of a personal computer, may be used.

[0093]

As described above, according to the present invention, the following effects can be obtained.

The first effect is that the user simply points the camera at the object to be picked up, the object to be picked up is automatically determined, a conversion is performed according to the object to be picked up and the image pickup direction, and an appropriate image is displayed on the display device. It is. It is possible to display an appropriate image without complicated processing such as the user instructing the apparatus on the imaging target.

The second effect is to reduce the amount of operation performed by the user in the synthesis processing. The user can easily input a panoramic image with correct geometrical shape, horizontal, vertical, and diagonal directions by simply pointing the camera at the imaging target and scanning the imaging target while pressing the pointing device. It becomes possible.

The third effect is that if the present invention is realized by a configuration in which the camera fixing mechanism is fixed on the upper part of the liquid crystal display unit of the portable personal computer, one person can be used for a TV conference or the like and a document can be input. Because it can be configured with a camera, portability is greatly improved. this is,
This is an effect derived from the first and second effects.

The fourth effect is that, as described in the second embodiment, by fixing the rotation axis of the camera at a certain angle, the size of the entire device can be further reduced. This is advantageous in that the imaging range of the camera moves in the horizontal direction, so that the camera can be installed closer to the display device or the housing.

[Brief description of the drawings]

FIG. 1 is a block diagram showing an embodiment of the configuration of an image input device according to the present invention.

FIG. 2 is a view for explaining a situation in which a document is input in the first embodiment of the image input apparatus of the present invention.

FIG. 3 is a diagram for explaining a situation where scenery is input in the first embodiment of the image input device of the present invention.

FIG. 4 is a diagram for explaining geometric distortion correction in a document image in the first embodiment of the image input apparatus of the present invention.

FIG. 5 is a diagram illustrating a configuration example of a camera direction detection device according to the present invention.

FIG. 6 is a flowchart for explaining an embodiment of a processing flow of the present invention.

FIG. 7 is a flowchart for explaining an embodiment of a processing flow of the present invention.

FIG. 8 is a diagram for explaining a coordinate system in the embodiment of the present invention.

FIG. 9 is a diagram for explaining a camera model in the embodiment of the present invention.

FIG. 10 is a diagram for explaining a combining process in the embodiment of the present invention.

FIG. 11 is a diagram for explaining a configuration of an apparatus according to a second embodiment of the present invention.

DESCRIPTION OF SYMBOLS 10 control device 11 camera fixing mechanism 12 camera 13 pointing device 14 camera direction detecting device 15 image synthesizing device 16 display device 17 image memory 18 image converting device 19 display image converting device 20 target determining device 21 mode determining device 50 External device 101 Housing 102 Document 103 Imaging range 300 Three-dimensional coordinate system 301 Rotation center R 305 Document surface 310 Point A on document surface 311 Point B on document surface 312 Point C on document surface 313 Point D on document surface 314 Point E 310 on the document surface Point A 410 on the document surface Point a 411 on the image plane b 412 Point on the image plane c 412 Point c 413 on the image plane d 414 Point e 501 on the image plane Cylindrical surface

Claims (9)

[Claims] A camera for imaging an object to be imaged; a camera direction detection device for detecting a direction of the camera; and a user capable of freely changing an imaging direction of the camera with respect to a rotation axis fixed in a horizontal direction. As described above, at least the camera and the camera direction detection device are integrated, a camera fixing mechanism that holds the position of the rotation center of the camera at a fixed height, and information on the camera direction of the camera direction detection device, Applying a geometric transformation so that the geometric shape of the imaging target included in the image captured by the camera is correct, storing the image in the image memory, and performing a synthesis process of the image stored in the image memory, and performing synthesis. And an image synthesizing device for storing the obtained image in the image memory. 2. The image input apparatus according to claim 1, wherein the camera fixing mechanism allows a user to freely change an imaging direction of the camera on a plane perpendicular to a rotation axis and rotate the camera. An image input device, wherein the camera and the camera direction detection device are integrated so that a user can freely set the angle of the axis with respect to the horizontal. 3. The image input device according to claim 1, wherein an object determination device that determines an imaging object type indicating an object to be imaged by the camera from information on a camera direction of the camera direction detection device; The apparatus further includes: an instruction device for inputting an instruction of a processing content by a user; and a mode determination device for determining a processing type for an image captured by the camera by referring to at least the processing content input by the instruction device. The image input device, wherein the image synthesizing device performs an image synthesizing process when the processing type determined by the mode determining device is a synthesizing process. 4. The image input device according to claim 3, wherein the pointing device is attached to the camera integrated by the camera fixing mechanism, so that a user can change an imaging direction at the same time. An image input device capable of instructing the processing content. 5. The image input device according to claim 3, wherein the image conversion device applies the geometric distortion correction of the imaging target to a captured image according to an imaging target type of the target determination device. The geometric distortion correction is performed based on information on the camera direction of the camera direction detection device so that the geometric shape of the imaging target is correctly reproduced, and the result is stored in the image memory. Image input device. 6. The image input device according to claim 3, wherein the display device displays an image captured by the camera or an image stored in the image memory. A display image conversion device for converting an image to be displayed on the display device in order to perform a display suitable for the display device according to information on a camera direction of the camera direction detection device and an imaging target type of the target determination device. An image input device, further comprising: 7. The image input device according to claim 6, wherein the display image conversion device refers to information on a process type from the mode determination device to perform display suitable for the display device. An image input device for converting an image to be displayed on the display device. 8. A camera for imaging an object to be imaged, a camera direction detecting device for detecting the direction of the camera, an instruction device for inputting an instruction of image processing contents by a user, and an image capturing direction of the camera fixed in a horizontal direction. The camera, the pointing device, and the camera direction detection device are integrated so that the user can freely change the rotation axis that has been set, and the camera is fixed so that the position of the rotation center of the camera is maintained at a constant height. Mechanism and
A computer-readable recording medium storing an image processing program of at least an image input device, comprising: an image capturing target type indicating a target to be captured by the camera from information on a camera direction of the camera direction detecting device. A mode determination for determining a processing type for an image captured by the camera by referring to a target determination function to be determined, and a process content input by the pointing device and information on a camera direction of the camera direction detection device. The function and the information on the direction of the camera of the camera direction detecting device are subjected to a geometric transformation so that the geometric shape of the imaging target included in the image captured by the camera is correct, and stored in a storage device of the computer. When the processing type determined by the image conversion function and the mode determination function is a combination processing An image combining function of performing a combining process of the images stored in the storage device and storing the combined image in the storage device; information regarding a camera direction of the camera direction detection device and the target determination function are determined. A display image conversion function of converting an image to be displayed on the display device in order to perform a display suitable for the display device, according to the type of imaging target that has been obtained. A computer-readable recording medium that records the recorded information. 9. The image input device according to claim 8, wherein the camera fixing mechanism allows a user to freely change an imaging direction of the camera on a plane perpendicular to a rotation axis and rotate the camera. A computer that records an image processing program of an image input device, wherein the camera and the camera direction detection device are configured to be integrated so that the user can freely set the angle of the axis with respect to the horizontal. A readable recording medium.