JPH118845A - Panoramic image generation device and its method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a panoramic image that has high picture quality and has no distortion caused at each connection part of the image by mapping plural images which are photographed by controlling the photographing direction of an image pickup means via a drive means onto a virtual spherical surface that is viewed from the rotational center of the drive means. SOLUTION: A computer 1 photographs a scenery 4 by controlling a pan tilter camera 3 located at a remote place and displays the image of a photographed screen 5 in an operation area 6A of a monitor 2. When a panoramic screen production instruction is produced via a panorama production button 6E, a pan tilter and a zoom lens are driven to each prescribed position to acquire each image. These acquired images are connected together to a virtual spherical surface defining a mobile axis of the pan tilter as an original point, and a plane image which is normalized in both latitude and longitude directions of the virtual spherical surface is shown in a panoramic image display area 6B. In such a way, a panoramic image is easily produced with no use of an expensive super-wide-angle lens nor fisheye lens.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a panoramic image generating apparatus and method capable of easily obtaining a high-resolution and high-quality panoramic image with an inexpensive apparatus.

[0002]

2. Description of the Related Art Conventionally, when images are connected to each other, a correlated position (pixel) of the images to be connected is found, and the images are simply superimposed or the connected image is enlarged / reduced for one image. , And then superimposed after deformation.

[0003]

However, enlargement / reduction and deformation of an image have a problem of deteriorating image quality such as deterioration of resolution of the image. Further, when a plurality of images are simply connected, there is a problem that distortion occurs. For example, when connecting ten images as shown in FIG. 21A, the ten images obtained from the imaging device map a plane group perpendicular to the optical axis of the lens. Therefore, according to the method of simply connecting the ten images, when a subject in the depth direction is projected between the connected images,
As shown in FIG. 21B, there is a problem that distortion occurs in the connection portion.

Accordingly, it is an object of the present invention to provide a panoramic image generating apparatus and method capable of generating a panoramic image having high image quality and having no distortion in a connected portion.

[0005]

According to a first aspect of the present invention, there is provided an image capturing means for capturing an image, a driving means for controlling a direction of the image capturing means, and a plurality of images obtained by the image capturing means. A panoramic image generating apparatus characterized by comprising image connecting means for connecting without distortion by mapping to a virtual spherical surface viewed from the rotation center of the driving means.

According to an eleventh aspect of the present invention, a step of photographing an image, a step of controlling a direction in which the image is photographed, and a method of viewing a plurality of images obtained by photographing from the rotation center of the photographing position. And forming a panoramic image by mapping onto a virtual sphere.

In the imaging means (for example, a video camera),
The image of the subject imaged through the zoom lens is converted into an electric signal, output as an image signal, and the lens is driven with respect to the zoom lens position coordinates specified by the image connecting means. Further, the current position coordinates of the zoom lens are supplied to the image connecting means.

The driving means (for example, pan tilter) is capable of driving the imaging means in the rotating direction to change the direction in which the image is taken by the imaging device. And supplies the current position coordinates of the driving means to the image connecting means. The image linking means obtains the current zoom position of the lens from the imaging means, calculates the magnification thereof, obtains the current pan / tilt position data of the pan / tilt from the driving means, and obtains the home position (for example, Calculate the angle coordinates of the pan tilter with the origin of latitude and longitude as the center of the movable range). When an instruction to create a panoramic image is issued from inside the apparatus (for example, a periodic creation instruction) or from outside the apparatus (for example, an instruction from an operator), the driving unit generates a panoramic image. Instruct the position coordinates to acquire the image, acquire the image after reaching the target position, map the acquired image to the virtual sphere, compress the image in the horizontal and vertical directions, to the previously captured image And outputs a connected image to the display means. Then, the images are sequentially connected in such a procedure, and the finally obtained panoramic image is output to the display means, and the image is stored. When setting the target position for the driving means, the image connecting means gives an instruction to the driving means to enter the target position from the same direction so that the absolute position of the driving means is not disturbed by the influence of the backlash.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Some embodiments of the present invention will be described below with reference to the drawings. FIG. 1-1 shows a schematic configuration of an embodiment of the present invention. The computer 1 to which the monitor 2 and the mouse 8 are connected controls the driving of the pan tilter camera 3 connected to a remote place.
That is, the computer 1 forms a controller of the imaging device.

The pan tilter camera 3 means a pan tilter unit and a camera unit integrally formed. This figure 1
Then, as an example, the pan tilter camera 3 is installed in an actual scene in an environment as shown in FIG. 5 shows a screen shot by the pan tilter camera 3 installed in the actual scene 4 of the environment (hereinafter, referred to as a shooting screen). This photographing screen is a screen in which photographing is actually performed. When the zoom lens is operated on the telephoto side, the angle of view is reduced, and when the zoom lens is operated on the wide side, the angle of view is increased.

An image of the photographing screen 5 captured by the pan tilter camera 3 is supplied to the computer 1 via a video cable or the like, and the video data supplied to the computer 1 is decoded and displayed on the operation area 6A of the monitor 2. You. The monitor 2 has a panorama image display area 6B. When a panorama screen generation instruction is given by a panorama generation button 6E, the pan tilter and the zoom lens are driven to predetermined positions, and an image is displayed at each location. At the same time, the obtained images are connected to the virtual sphere with the origin of the movable axis of the pan tilter camera, and the plane image is normalized with respect to the latitude and longitude directions of the virtual sphere. indicate. The panoramic operation area 6B is an arbitrary point or an arbitrary area 6C in a video or a panoramic image in which an arrow-shaped cursor 7 is drawn at the position of the mouse 8.
Is operated with the mouse 8 to operate the pan tilter camera 3. Further, a pan / tilt limiter display 6D is displayed in the panorama operation area 6B as indicated by a dotted line. Pantil Tarita display 6D
Indicates the limit of the movable range of the pan tilter camera 3.

As shown in FIG. 2, an operation area 6A and a panorama operation area 6B are displayed on the screen of the monitor 2. By operating the mouse 8, the cursor moves, and it is possible to indicate an arbitrary point in the panoramic operation area 6B or an arbitrary point generated from the arbitrary area 6C. The pan tilter is driven so that the designated one point is located at the center of the operation area 6A, and the imaging screen 5 moves. That is, a result to be displayed is input in advance, and the subject selected in accordance with the input is displayed so as to be at the center of the operation area 6A.

FIG. 3 is a block diagram of the entire system to which an embodiment of the present invention is applied. The system includes a camera unit 11, a pan tilter unit 12, a TV monitor 13, a computer 1, and a pointing device 1 such as a mouse 8.
4. It consists of a monitor 2. The above-described pan-tilt camera 3 includes a camera unit 11 and a pan-tilt unit 12, and the camera unit 11 is installed on the pan-tilt unit 12 as an example. The camera unit 11 includes a lens block unit 15,
It comprises a zoom lens 16, a zoom unit 17, a motor 18 for the zoom lens 16, a solid-state imaging device 19, a signal separation / automatic gain adjustment circuit (SH / AGC) 20, an A / D converter 21, and a signal processing circuit 22, A video camera is shown as a whole.

The pan tilter section 12 includes a mode controller 23, a camera controller 24, a pan tilter controller 25, a pan motor 26, a tilt motor 27,
It is composed of a pan tilter 28. The computer 1 includes a control unit 31, a video capture unit 29 including a video capture board, and a storage unit 30.

The image pickup light arriving from the subject passes through the lens group of the lens block section 15 and the diaphragm, and then passes through the solid-state image pickup device 19.
Is imaged. As an example of the solid-state imaging device 19, a CCD
(Charge Coupled Device). The formed imaging light is supplied to a signal separation / automatic gain adjustment circuit 20 after converting a visual field image into a video signal. In the signal separation / automatic gain adjustment circuit 20, the video output signal is sampled and held, and the gain is controlled so as to have a predetermined gain by an auto iris (AE) control signal. The video output signal thus obtained is supplied to the signal processing circuit 22 via the A / D converter 21. Signal processing circuit 2
2, the input signal is converted into signals such as luminance (Y), color (C), and video signals,
The data is supplied to the video capture unit 29 of the computer 1 via the monitor 13.

The lens block unit 1 of the camera unit 11
Reference numeral 5 indicates that the angle of view to be imaged can be changed by driving the zoom lens 16. The lens block unit 15 drives the zoom lens 16 by rotating a motor 18, for example, a stepping motor, according to a driving command of the camera controller 24 of the pan / tilt unit 12. The camera controller 24 controls the lens of the camera unit 11 (for example, focus, zoom, etc.)
The controller normally performs exposure control (for example, aperture, gain, electronic shutter speed, and the like), white balance control, image quality control, and the like, and also interfaces with the mode controller 23. As an interface control related to the control of the zoom lens 16, a control signal is sent to the motor driver so that the zoom lens 16 is driven to the commanded position in response to the drive command of the zoom lens 16 sent from the mode controller 23. At the same time, the current position information of the zoom lens 16 is communicated to the mode controller 23 at all times.

The camera unit 11 is installed on a pan / tilt unit 12 which is a device having two degrees of freedom in a rotational direction such as pan and tilt. The pan tilter unit 12
The pan motor 26 and the tilt motor 27 are rotated by the drive command of the pan tilt controller 25,
The pan head of the pan tilter 28 is driven. An example of these motors 26 and 27 is a stepping motor. The pan / tilt controller 25 responds to a drive command in each of the pan and tilt directions sent from the mode controller 23 so that the pan and tilt heads are driven to the commanded positions. The current position information of the pan and tilt of the pan / tilt 28 is output to the motor driver and the current
3 is communicated.

The mode controller 23 includes the camera unit 1
1. According to the internal state of the pan / tilt unit 12 and interface information from outside the pan / tilt camera 3, the entire system is controlled as described later. The mode controller 23 is connected to the computer 1 by, for example, an RS-232C, and responds to a driving command from the computer 1 to the pan tilter 28 and the zoom lens 16
The command is distributed to the pan tilter controller 25 and the camera controller 24 so as to drive the camera, and the current position information sent from the pan tilter controller 25 and the camera controller 24 is transmitted to the computer 1.

In this embodiment, the computer 1 is used to select an image projected by the pan / tilt camera 3. Then, the operation area 6A on the screen of the monitor 2
The communication data to the mode controller 23 is determined by processing information obtained from the graphic display displayed on the panorama operation area 6B and the operation such as the pointing position of the pointing device 14 (mouse 8) and the click. . In addition, a video capture unit 29 is used to display an image of the camera unit 11 on the monitor 2. The video capture unit 29 is capable of displaying a video signal input from the camera unit 11 on the monitor 2 with an arbitrary quality, and is configured to use an arbitrary image format (for example, a bitmap format, a J
PEG format still image, JPEG format moving image, etc.) at any quality and stored in the storage unit 30 of the computer 1.
(Eg, a hard disk).

Here, referring to FIG. 4, a brief description will be given for creating a panoramic image in this embodiment. First, the surrounding environment where the pan tilter camera 3 is set is assumed to be a spherical surface. This is called a virtual spherical surface. In FIG. 4, two adjacent images on the virtual spherical surface are connected to create one panoramic image. First, in order to create a panoramic image, as shown in FIG. 4A, the pan tilter camera 3 located at the center captures two adjacent images on the virtual spherical surface. The pan tilter camera 3 images a plane orthogonal to the optical axis of the lens. FIG. 4D shows two adjacent pixels on the virtual spherical surface.
A state in which these two images are mapped on a plane orthogonal to the optical axis by capturing one image with the pan tilter camera 3 is shown. When two adjacent images are simply connected, there is a portion where a joint overlaps or a distortion occurs.

As shown in FIG. 4B, two images adjacent to the virtual spherical surface are mapped to eliminate the overlap and distortion of the joint. FIG. 4E shows a state where two captured images, which are planes orthogonal to the optical axis, are mapped onto a virtual spherical surface.
In this manner, a plane orthogonal to the optical axis, that is, a captured image is mapped onto a virtual spherical surface, the mapped images are connected, and an overlap image,
Image connection processing for deleting unnecessary images is performed. Then, by normalizing the image mapped on the virtual spherical surface by latitude and longitude, a panoramic image can be generated as shown in FIGS. 4C and 4D.

Next, one method of creating a panoramic image according to the present invention will be described. In this method, as shown in FIG. 5, ten images are connected to form one panoramic image. First, as shown in FIG. 5A, ten images are taken from a pan tilter camera 3 (not shown) arranged at the center of the virtual spherical surface. At this time, as shown in the figure, the optical axis of the lens of the pan tilter camera 3 is adjusted to the position shown by a circle for each image area, so that the pan tilter camera 3
Can be obtained. Pantilta camera 3
5B is an image on a plane orthogonal to the optical axis of the lens, as shown in FIG. 5B. After the acquired images are respectively developed on the virtual spherical surface, they are normalized by latitude and longitude as shown in FIG. 5C. The images of the ten images developed on the virtual spherical surface are acquired at positions where the images overlap each other so that there is no gap in the joint at the time of the connection processing. Then, after the overlapping image and the unnecessary image are deleted, the ten images are connected, and a panoramic image is generated as shown in FIG. 5D.

Next, another method of creating a panoramic image according to the present invention will be described with reference to FIG. It is calculated which pixel of the image acquired by the pan tilter camera 3 is assigned to the pixel of the panoramic image normalized by the latitude and longitude, that is, each coordinate (s, t). When pixels of an image acquired by the pan tilter camera 3 are made to correspond to pixels of a panoramic image as in the method of FIG. 5, pixels of a panoramic image without corresponding pixels may occur, and all pixels of the panoramic image may be generated. This is for making the pixels of the acquired image correspond. Thus, a panorama image is realized by the pixels calculated for each coordinate. As a procedure of the processing,
First, the angle coordinates (α, β) (FIG. 6B) on the virtual sphere corresponding to the coordinates (s, t) (FIG. 6A) of the panoramic image are calculated using the equation (1).

(Α, β) = (a (s), b (t)) (1) The details of the expression (1) will be described later with reference to FIG.

Using these coordinates (s, t), the angular coordinates (θ, φ) of the pan tilter 28 from which the image was obtained, and the imaging magnification γ with the wide end of the imaging device being 1, the image is shown in FIG. 6C. Next, the coordinate data (ξ, η) on the acquired image is calculated using the equation (2).

(Ξ, η) = (f (α, β, θ, φ, γ), g (α, β, θ, φ, γ)) (2) Details of the equation (2) will be described later. 8 will be described.

Using the above equation, a connected image, that is, a panoramic image is generated by associating each pixel of the panoramic image with the acquired image.

Here, a method of converting the coordinates (s, t) of the panoramic image into angular coordinates (α, β) on the virtual spherical surface will be described with reference to FIG. First, as shown in FIG. 7A,
PragMin sets the home position of the pan tilter 28 to 0 (r
ag) is the angle data at the left end when PragMax is
This is right angle data when the home position of the pan tilter 28 is set to 0 (rag). Further, Ny ₂ is a horizontal coordinate of the panorama operation area 6B, -Ny _2/2
Is coordinate data of the left end of the panorama operation area 6B,
Ny _2/2 is the right edge of the coordinate data of the panorama operation area 6B.

Then, in order to obtain the pan angle α from the coordinate data s, (PragMax−α) :( PragMax−PragMin) = (Ny ₂ / 2−)
s): Ny ₂ next, than this pan angle α, α = PragMax - (PragMax -PragMin) × (Ny 2/2-s) /
Ny ₂ .

In FIG. 7B, TragMin is the angle data of the upper end when the home position of the pan tilter 28 is set to 0 (rag), and TragMax is when the home position of the pan tilter 28 is set to 0 (rag). Is the angle data of the lower end of. Further, Nz ₂ represents a vertical coordinate of the panorama operation area 6B, -Nz _2/2 is the coordinate data of the upper end of the panorama operation area 6B, Nz _2/2, the lower edge of the panorama operation area 6B It is coordinate data.

Then, in order to obtain the tilt angle β from the coordinate data t, (TragMax−β) :( TragMax−TragMin) = (Nz ₂ / 2−
t): Nz ₂ , from which the tilt angle β is β = TragMax− (TragMax−TragMin) × (Nz ₂ / 2−t) /
Nz ₂ .

Referring to FIG. 8, the process of plane spherical surface conversion will be described. As shown in FIG. 8A, the home position (latitude,
The coordinates in space of the point (ξ, η) on the camera image facing the origin of the longitude) can be expressed as follows.

[0033]

(Equation 1)

At this time, k ₁ = tan (λ / 2γ) / (Ny / 2) k ₂ = tan (μ / 2γ) / (Nz / 2), and (Ny, Nz) is the pointing device 1
4 (mouse 8) in the drive range (y direction, z direction)
(Λ, μ) is the horizontal and vertical angle of view at the wide end, and γ is the current zoom relative magnification (magnification information) with the wide end being 1 ×.

As shown in FIG. 8B, it is generally known that a three-dimensional rotation matrix is represented by the following equation.

[0036]

(Equation 2)

Angle information (θ, φ) from home position
Only one point (ξ, η) on the camera image panned and tilted
And one point of (α, β) is in the same direction from the home position, the following relationship holds.

[0038] Solving for _{R z (θ) R y (} φ) p = lR z (α) R y (β) e x This p,

[0039]

(Equation 3)

Here,

[0041]

(Equation 4)

お く, η are obtained as follows.

L = 1 / a ξ = -lb / k ₁ = -b / k _1a η = lc / k ₂ = c / k _{2a From the} above equation, coordinate data at an angle of (α, β) from the home position. From the imaged coordinates (ξ,
η) can be obtained.

Ξ = (− sin (α−θ) cosβ) / (k ₁ (cos (α−
θ) cosφ cosβ + sinφ sinβ)) η = (-cos (α−θ) sinφ cosβ + cosφ sinβ) / (k ₂ (c
os (α−θ) cosφcosβ + sinφsinβ)) In this way, the image acquired by the pan tilter camera 3 from the angular coordinates (α, β) on the virtual sphere corresponding to the coordinates (s, t) of the panoramic image By obtaining the coordinate data (ξ, η), a panoramic image can be generated.

Conversely, coordinate data at an angle of (α, β) can be obtained from (ξ, η) reflected on the imaging coordinates by the following equation.

[0046] First of all, l = | p | because it is, a = 1 / √ (1 + k 1 2 ξ 2 + k 2 2 η 2) b = -k 1 ξ / √ (1 + k 1 2 ξ 2 + k 2 2 η 2 ) becomes _{c = k 2 η / √ (} 1 + k 1 2 ξ 2 + k 2 2 η 2). However, √ () performs square root processing on the calculation result in ().

From equation (3), a = cos (α−θ) cosφcosβ + sinφsinβb = sin (α−θ) cosβc = −cos (α−θ) sinφcosβ + cosφsinβ

[0048] From the above formula a sinφ + c sinθ = sinβ tan (α-θ) = b / because it is (a cosφ-c sinθ), β = sin -1 (sinφ / √ (1 + k 1 2 ξ 2 + k 2 2 η ² ) +
_{sinθk 2 η / √ (1 +} k 1 2 ξ 2 + k 2 2 η 2) α = tan -1 (-k 1 ξ / (cosφ-k 2 η sinθ)) + θ become.

Therefore, (α, β) = (f (ξ, η, θ, φ, γ), g (ξ, η, θ, φ, γ)) (4) can be obtained.

If the error is somewhat tolerable, (α, β) can be obtained as follows.

Α = θ + (λ / γ) × (ξ / Ny) β = φ + (μ / γ) × (η / Nz) That is, the equation (4) can be expressed as follows: (α, β) = (f (ξ, θ, γ), g (η, φ, γ)) (5), which can be simplified.

The angle information (α, β) of the pan tilter 28 shown in the above equations (4) and (5) is stored in the operation area 6A.
The process of calculating from the position coordinates (ξ, η) will be described with reference to FIG. First, an example of a method of directly specifying an arbitrary point in the operation area 6A will be described. As shown in FIG. 9A, relative coordinates are set with the center of the operation area 6A being (0, 0), and the position coordinates (ξ, η) of the mouse 8 on the operation area 6A are obtained.

Next, an example of another method for designating an arbitrary point generated from an arbitrary area in the operation area 6A will be described. First, as shown in FIG. 9A, the start point (m
1, n1) is specified, and then the end point (m2,
n2) is designated, and as a central coordinate of a rectangle generated from these two points, an arbitrary point (ξ) is given by (ξ, η) = ((m1, n1) + (m2, n2)) / 2 (6) , Η) are obtained.

FIG. 9A shows the mouse 8 on the operation area 6A.
The coordinates of the pointing device (14), and movable range (y-direction, z-direction) of the mouse 8 in the operation area 6A and the (Ny _1, Nz _1). The position coordinates (ξ, η) of this arbitrary point (mouse 8), angle information (θ, φ) to which the pan tilter 28 faces, and the current zoom relative magnification with the wide end of the zoom lens 16 being 1 × The angle coordinates (α, β) of the pan tilter 28 are obtained from the equation (4) or (5) using the magnification information (γ) to be calculated.

The angle coordinates (α, β) shown in FIG. 9B are for projecting the location specified by the pointing device in the center of the image pickup screen when the home position of the pan tilter 28 is the origin of latitude and longitude. .

In FIG. 9, the coordinates obtained may be absolute coordinates on the screen of the monitor 2 or relative coordinates with the center of the operation area 6A being (0, 0). The coordinates in the pan direction are indicated by ξ, m1, m2, θ, α, and the coordinates in the tilt direction are indicated by η, n1, n2, φ, β.

As described above, when the mouse 8 is in the operation area 6A, the current angle information (θ, φ) of the pan tilter 28 obtained from the received data, the zoom magnification information (γ), and the mouse 8 are designated. Using the position information (ξ, η) of the mouse 8 in the region, the angle information (α, β) of the pan tilter 28 such that the subject on the designated operation region 6A comes to the center of the operation region 6A is calculated by Expression (4). ) Or Equation (5). The angle coordinates (α, β) of the pan tilter 28 thus obtained are converted into internal position information (PNew, TNew) of the pan tilter 28 according to FIG. 11 and stored in the transmission buffer together with the absolute position driving command of the pan tilter 28. Also, as described later, the data transmission request flag (FlagSo)
Is set so that data is transmitted on a timer event.

The process of converting the position coordinates (ξ, η) of the mouse 8 in the panorama operation area 6B on which a panoramic image is displayed into angle coordinates (α, β) will be described with reference to FIG.
The method of directly specifying an arbitrary point in the panorama operation area 6B is the same as the method of directly specifying an arbitrary point in the operation area 6A described above, and as shown in FIG. η, η) can be obtained.

Next, an example of another method for specifying an arbitrary point generated from an arbitrary area in the panorama operation area 6B will be described. First, as shown in FIG. 10A, after the start point (m1, n1) of an arbitrary area is specified, the end point (m2, n2) of an arbitrary area is specified, and an arbitrary point (ξ) is obtained by Expression (6). , Η) are obtained.

FIG. 10A shows the coordinates of the mouse 8 (pointing device 14) on the panoramic operation area 6B, and the movable range (y) of the mouse 8 in the panoramic operation area 6B.
(Direction, z direction) is (Ny ₂ , Nz ₂ ). This movable range is limited by a pan / tilt limiter display 6D indicated by a dotted line in the panoramic operation area 6B. The pan tilter limiter display 6D indicates the movable range of the optical axis of the lens of the pan tilter camera 3. That is, it is impossible to instruct beyond the pan / tilt limiter display 6D. The position coordinates (ξ, η) of this arbitrary point, angle information (θ, φ) at which the pan tilter 28 is facing, and magnification information (current zoom relative magnification with the wide end of the zoom lens 16 being 1 ×) γ), the equations (7), (8) and (9)
From the position coordinates (x, y) of the panorama operation area 6B, the angle of view information (s, t), and the angle coordinates (α,
β) is required.

(X, y) = (f ₀ (θ), g ₀ (φ)) (7) (s, t) = (f ₁ (γ), g ₁ (γ)) (8) (α, β) = (f (ξ), g (η)) (9) The position coordinates (x, y) shown in FIG.
Is the current orientation of the pan tilter 28 and the angle of view information (s, t)
Is the angle of view currently displayed in the operation area 6A. FIG. 10B shows the state of zoom and pan / tilt on the panorama operation area 6B.

The angle coordinates (α, β) shown in FIG. 10C are for projecting the location specified by the pointing device at the center of the image pickup screen when the home position of the pan tilter 28 is the origin of latitude and longitude. . (PragMa
x, TragMax) to (PragMin, TragMin) are the drive range of the pan tilter, that is, the range indicated by the pan tilter limiter display 6D. FIG. 10C shows a drive target value in the pan tilter movable range.

In FIG. 10, the coordinates obtained are:
The coordinates may be absolute coordinates on the screen of the monitor 2 or relative coordinates with the center of the panorama operation area 6B being (0, 0). These coordinates are represented by 座標, m1, m2, x, s, α in the pan direction.
And the coordinates in the tilt direction are η, n1, n2, y, t,
Indicated by β.

As described above, when the mouse 8 is in the panoramic operation area 6B, the mouse 8 in the area designated by the mouse 8
Using the position information (ξ, η) of the
The angle information (α, β) of the pan tilter 28 such that the subject on A comes to the center of the operation area 6A is calculated by using Expression (9). The angle coordinates (α, β) of the pan tilter 28 thus obtained are converted into internal position information (PNew, TNew) of the pan tilter 28 according to FIG. 11 and stored in the transmission buffer together with the absolute position driving command of the pan tilter 28. In addition, as described later, the data transmission request flag (FlagSo)
Is set so that data is transmitted on a timer event.

Here, the method of converting the above-mentioned position information (p, t) inside the pan tilter 28 into angle information (θ, φ) and the angle coordinate (α, β) into the position information (PNew, TNew) inside the pan tilter 28 ) Will be described with reference to FIG. First, PragMin shown in FIG. 11A is the left end angle data when the home position of the pan tilter 28 is set to 0 (rag), and PragMax is the right end angle data when the home position of the pan tilter 28 is set to 0 (rag). This is angle data. Also, PdatMin is a pan tilter controller 25
Is the internal count data at the left end, and PdatMax is the internal count data at the right end of the pan tilter controller 25.

Then, in order to obtain the pan angle θ from the pan data p, (PragMax−θ): (PragMax−PragMin) = (PdatMax−p):
(PdatMax−PdatMin), from which the pan angle θ is: θ = PragMax− (PragMax−PragMin) × (PdatMax−p) / (P
datMax-PdatMin).

From this, the pan data p is obtained by the following equation: p = PdatMax− (PragMax−θ) × (PdatMax−PdatMin) /
(PragMax-PragMin).

In order to determine the pan data PNew from the pan angle α, (PragMax−α) :( PragMax−PragMin) = (PdatMax−p−ne)
w): (PdatMax-PdatMin), from which the pan data PNew is PNew = PragMax− (PragMax−α) × (PdatMax−PdatMin)
/ (PragMax-PragMin).

Also, TragMin shown in FIG. 11B is angle data of the upper end when the home position of the pan tilter 28 is set to 0 (rag), and TragMax is obtained when the home position of the pan tilter 28 is set to 0 (rag). Is the angle data of the lower end of. TdatMin is internal count data at the upper end of the pan tilter controller 25, and TdatMax is
This is internal count data at the lower end of the pan tilter controller 25.

Then, the tilt angle φ is obtained from the tilt data t.
(TragMax−φ) :( TragMax−TragMin) = (TdatMax−t):
(TdatMax−TdatMin), from which the tilt angle φ is: φ = TragMax− (TragMax−TragMin) × (TdatMax−t) / (T
datMax-TdatMin).

From this, the tilt data t is calculated as follows: t = TdatMax− (TragMax−φ) × (TdatMax−TdatMin) /
(TragMax-TragMin).

Further, the tilt data TNew is obtained from the tilt angle β.
(TragMax−β) :( TragMax−TragMin) = (TdatMax−TNe
w): (TdatMax−TdatMin). From this, the tilt data TNew is obtained as follows: TNew = TragMax− (TragMax−β) × (TdatMax−TdatMin)
/ (TragMax-TragMin).

Next, the position coordinates (ξ, η) in the panorama operation area 6B are converted to the angle coordinates (α,
β) and the angle information (θ, φ) of the pan tilter 28 is converted into position coordinates (x,
The method of conversion to y) will be described with reference to FIG. First,
PragMin shown in FIG.
Is the left end angle data when the home position of the pan tilter 28 is 0 (rag), and PragMax is the right end angle data when the home position of the pan tilter 28 is 0 (rag).
Further, Ny ₂ is a horizontal coordinate of the panorama operation area 6B, -Ny _2/2 is the left end of the coordinate data of the panorama operation area 6B, Ny _2/2 is the right end of the panorama operation area 6B It is coordinate data.

Then, in order to obtain the pan angle α from the coordinate data ξ, (PragMax−α) :( PragMax−PragMin) = (Ny ₂ / 2−
xi]): Ny ₂ next, than this pan angle α, α = PragMax - (PragMax -PragMin) × (Ny 2/2-ξ) /
Ny ₂ .

In order to obtain the coordinate data x from the pan angle θ, (PragMax−θ): (PragMax−PragMin) = (Ny ₂ / 2−)
x): Ny ₂ , and the coordinate data x is x = Ny ₂ / 2− (PragMax−θ) × Ny ₂ / (PragMax−PragMi)
n).

In FIG. 12B, TragMin is the angle data of the upper end when the home position of the pan tilter 28 is set to 0 (rag), and TragMax is obtained when the home position of the pan tilter 28 is set to 0 (rag). Is the angle data of the lower end of. Further, Nz ₂ represents a vertical coordinate of the panorama operation area 6B, -Nz _2/2 is the coordinate data of the upper end of the panorama operation area 6B, Nz _2/2, the lower edge of the panorama operation area 6B It is coordinate data.

Then, in order to obtain the tilt angle β from the coordinate data η, (TragMax−β) :( TragMax−TragMin) = (Nz ₂ / 2−
η): Nz ₂ and the tilt angle β is β = TragMax− (TragMax−TragMin) × (Nz ₂ / 2−η) /
Nz ₂ .

To find the coordinate data y from the tilt angle φ, (TragMax−φ) :( TragMax−TragMin) = (Nz ₂ / 2−
y): Nz ₂ next, than this coordinate data y _{is, y = Nz 2 / 2- (} TragMax-θ) × Nz 2 / (TragMax-TragMi
n).

A method of converting the angle-of-view information (ψ, ω) cut out by the pan tilter 28 into the angle-of-view information (s, t) of the image frame 6C in the panoramic operation area 6B will be described with reference to FIG. First, FIG. 13A shows current view angle information (ψ, ω) of the pan tilter 28. This angle-of-view information (ψ, ω) is obtained by (ψ, ω) = 1 / γ × (ψ0, ω0). At this time, (ψ0, ω0) indicates the horizontal angle of view and the vertical angle of view at the wide end, and γ indicates 1 at the wide end.
This shows the lens magnification when the magnification is set to "2".

As shown in FIG. 13B, PragMin is the left end angle data when the home position of the pan tilter 28 is set to 0 (rag), and PragMax is when the home position of the pan tilter 28 is set to 0 (rag). Is the right end angle data. Further, Ny ₂ is a horizontal coordinate of the panorama operation area 6B, -Ny _2/2 is the left end of the coordinate data of the panorama operation area 6B, Ny _2/2 is the right end of the panorama operation area 6B It is coordinate data.

Then, in order to obtain the horizontal angle of view s from the horizontal angle of view ψ, ψ: (PragMax−PragMin) = s: Ny _{2. From} this, the horizontal angle of view s is given by s = × Ny ₂ / (PragMax −PragMin).

In FIG. 13C, TragMin is angle data of the lower end when the home position of the pan tilter 28 is set to 0 (rag), and TragMax is obtained when the home position of the pan tilter 28 is set to 0 (rag). Is the angle data at the upper end of. Further, Nz ₂ represents a vertical coordinate of the panorama operation area 6B, -Nz _2/2 is the coordinate data of the lower end of the panorama operation area 6B, Nz _2/2, the upper edge of the panorama operation area 6B It is coordinate data.

Then, in order to obtain the vertical angle of view t from the vertical angle of view ω, ω: (TragMax−TragMin) = t: Nz _2. Thus, the vertical angle of view t is t = ω × Nz ₂ / (TragMax −TragMin).

Thus, the view angle information (s, t) shown in FIG. 13D is displayed as a picture frame 6C in the panoramic operation area 6B.

Next, a method for converting the position information (z) of the zoom lens 16 into magnification information (γ) will be described with reference to FIG. In FIG. 14, the vertical axis represents lens magnification information, and the horizontal axis represents internal information of the zoom lens. The acquired position information (z) of the zoom lens 16 is converted into magnification information (γ) on the computer 1 by referring to a conversion graph shown in FIG. As an example, the position information (z) is converted into magnification information (γ) by a ROM table or a mathematical expression.

Next, an example of a control algorithm in the computer 1 will be described with reference to FIG. First, step S
In FIG. 1, when the program is started, the operation area 6A and the panorama operation area 6 are displayed on the monitor 2 as shown in FIG.
B, the cursor 7, and an initialization for setting the pan / tilt limiter shown in the pan / tilt limiter display 6D are performed. The range of the pantilter limiter may be a fixed value,
The range may be freely changed when it is desired to change the range. Then, in step S2, a timer is set so that the computer 1 and the mode controller 23 perform communication at a predetermined cycle. When these initialization operations are completed, the control shifts to an event waiting state in step S3 where various events occur, and the control shifts from step S3 in response to the event that has occurred. The events that occur include the timer event (step S4) set previously and a panorama creation request event (step S5).

The details of the algorithm of the panorama creation request event will be described with reference to the flowchart of FIG. When a panorama creation request event occurs, in step S8, a panorama creation request (FlagPa) is set (Tr
ue).

The details of the timer event algorithm will be described with reference to the flowchart of FIG. The timer event of this example is an event that occurs to periodically perform communication between the computer 1 and the mode controller 23. This timer event is, for example, 50
It occurs at msec intervals. When the timer event occurs, in step S11, it is determined whether the setting of the communication port has been completed. If it is determined that the communication port setting has been completed (completed), the control moves to step S12. If it is determined that the communication port setting has not been completed (not yet), the control moves to step S18. Here, only in the first time when the setting of the communication port is not completed, step S1
Then, control is transferred to step 8 to open a communication port. Specifically, in step S18, an RS-232C port on the computer 1 is opened.

In the subsequent timer events, confirmation of received data, analysis processing, transmission processing of data stored in the transmission buffer such as a driving command of the pan / tilt 28, or communication for requesting a state confirmation of the pan / tilt 28 and the zoom lens 16 are performed. Data transmission processing is performed. In this algorithm, the control shifts from step S11 to step S12. In step S12, the presence or absence of data in the reception buffer is confirmed.
When the control moves to 3 and there is no received data, the control moves to step S14. In step S13, the reception data existing in the reception buffer is analyzed, and the position information (p, p,
t) and position information (z) of the zoom lens 16 are acquired. These data are converted into the angle information (θ, φ) of the pan tilter 28 and the magnification information (γ) of the zoom lens 16 in accordance with the methods shown in FIGS.

Next, in step S14, the presence or absence of a data transmission request is confirmed. If there is a data transmission request (FlagSo == True), the control shifts to step S19. In step S19, after the transmission processing of the data stored in the transmission buffer is performed, the transmission request flag (Fl
agSo) is reset (False). As an example of the data stored in the transmission buffer, there is data of a driving command of the pan tilter 28 set by the mouse 8 and the like. And
If there is no transmission request (FlagSo == False), step S1
The control moves to 5. In step S15, it is determined whether the internal counter (ReqCnt) of the transmission request is 0. If the internal counter of the transmission request is 0 (ReqCnt = 0), step S1 is performed.
When the internal counter of the transmission request is not 0 (ReqCnt ≠ 0), the control shifts to step S20.

In step S16, a position request command for the pan tilter 28 and the zoom lens 16 is transmitted to the mode controller 23. Then, in step S17, the internal counter (ReqCnt) of the transmission request is incremented.

In step S20, it is determined whether or not there is a panorama creation request. If there is a panorama creation request (FlagPa == True), control is transferred to step S21, and if there is no panorama creation request (FlagPa). == Fa
lse), control transfers to step S22. In step S21, a panorama creation process described later is executed. And
In step S22, an internal counter (ReqCn
t) is set to 0.

Next, the algorithm of the above-described panorama creation processing will be described with reference to the flowchart of FIG. The panorama creation is set (FlagPa = True) by a panorama creation request event. When the panorama creation request event occurs in advance, the panorama creation process (step S21) is executed at the time of the timer event as described above. In this flowchart, the procedure of the panorama creation processing is performed according to the panorama counter (PanoCnt).

First, when this process is started, a process for setting the angle of view of the imaging device to the widest angle is performed. That is,
In step S31, the panorama counter (PanoCnt)
Is determined to be 1 or not, and if it is determined that the panorama counter is 1 (PanoCnt == 1), step S
43, the panorama counter is not 1 (Pano
If it is determined that Cnt ≠ 1), the control moves to step S32. In step S43, a drive command for moving the zoom to the wide end is transmitted, and the panorama counter is incremented. This is for performing a panoramic image of a wider range with a small number of image acquisitions.

In step S32, it is determined whether or not the panorama counter is 2, and if it is determined that the panorama counter is 2 (PanoCnt == 2), step S4 is performed.
The control moves to 4 and the panorama counter is not 2 (PanoCn
If it is determined that t ≠ 2), the control moves to step S33. In step S44, a drive command to move to the position (POS (PanoCnt-2)) where the pan tilter 28 is first photographed is transmitted, and the panorama counter is incremented. That is, a process of moving to the first position (the position indicated by 1 in FIG. 5) for acquiring the angle of view of the pan tilter 28 indicated by POS (PanoCnt-2) is performed.

When the panorama counter (PanoCnt) is 3 or more, it is confirmed that the imaging apparatus has moved to the set zoom and pan / tilt positions, and when the panorama counter reaches the designated position, an image is captured. In step S33, the position information (z) of the zoom lens 16 is confirmed. If the zoom position is present, the control proceeds to step S34. If the zoom position is not present, this flowchart ends. Further, in step S34, the position information (p, t) of the pan tilter 28 is confirmed, and if the pan tilter position is present (== POS (PanoCnt-3)), the control proceeds to step S35, where the pan tilter position is changed. If not (≠ POS (PanoCnt-3)), this flowchart ends. Then, in step S35, an image at the current pan tilter position is acquired. The acquired image is
It is stored in the memory as a bitmap image file.

In step S36, a drive command to move the pan tilter 28 to the next position for photographing (POS (PanoCnt-2)) is transmitted. In step S37, as described above, the acquired image is subjected to data conversion into a virtual spherical surface, and a process of linking images in which a duplicate image and an unnecessary image are deleted is performed. If the computer 1 does not have the processing capability, in step S37, compression is also performed in the horizontal and vertical directions. In step S38, the connected images are normalized by latitude and longitude and displayed. Then, in step S39, the panorama counter (PanoCnt) is incremented.

That is, a process of moving the pan tilter 28 to the next image acquisition position is performed, followed by a process of mapping the previously captured image onto the virtual sphere, a process of compressing the image in the horizontal and vertical directions, an overlapping image, and an unnecessary image. And the progress of panorama image creation is displayed. The above operation is performed until the panoramic image is completed.

In step S40, it is determined whether or not the panorama image has been completely completed. If it is determined that the panorama image has been completed, control is transferred to step S41. If it is determined that the panorama image has not been completed, this flowchart ends. The above control is repeated until the panoramic image is completed. Step S41
Then, since the panorama image is completed, the panorama creation request (FlagPa) is reset (False). Then, in step S42, the completed panoramic image is stored.

In this flowchart, the image 1 shown in FIG. 5 can be obtained when the panorama counter (PanoCnt) is 3, the image 2 shown in FIG. 5 can be obtained when the panorama counter is 4, and the panorama counter is obtained. When 10 images shown in FIG. 5 acquired at 12 are acquired, all images for generating a panoramic image are acquired, and a panoramic image is completed.

The drive device operates so as to always enter the target position designated here from the same position direction, and play of the drive system mechanism (for example, the (Backlash) to correct the displacement of the connected image.

According to one embodiment of the present invention described above,
A panoramic image as shown in FIG. 21C can be obtained from ten images as shown in FIG. 21A.

In the above-described embodiment, all the controls are performed using one computer.
In another embodiment of the present invention, as shown in FIG. 19, roles are divided between a server computer and a client computer, and a pan tilter camera is controlled even in a form having a limited communication capacity such as a network line. It is like that. The computer 1 to which the monitor 2 and the mouse 8 are connected controls the driving of the pan tilter camera 3 installed at a remote place via the transmission line and the server 9. That is, the computer 1 forms an imaging device controller. Various existing transmission lines such as a communication line (wireless or wired) and a network can be used as the transmission line. The computer 1 has a client relationship with the server 9, and a plurality of computers 1 can be connected to the server 9.

The pan tilter camera 3 and the server 9
It is installed in the actual scene in the environment as shown in. A screen shot by the pan tilter camera 3 installed in the actual scenery 4 of this environment (hereinafter referred to as a shooting screen) is denoted by 5.
Shown in The photographing screen 5 is a screen for actually photographing. When the zoom lens is operated on the telephoto side, the angle of view is reduced, and when the zoom lens is operated on the wide side, the angle of view is increased.

The video of the photographing screen 5 captured by the pan tilter camera 3 is converted into video data via the server 9. This video data is supplied to the computer 1 via the transmission path. The video data supplied to the computer 1 is displayed on the monitor 2. Monitor 2
Then, the supplied shooting screen 5 is displayed in the operation area 6A on the screen of the monitor 2.

On the monitor 2, the panorama operation area 6
B and a panorama generation button 6E. A panorama image creation instruction is transmitted to the server 9 by the panorama generation button 6E, and the server 9 drives the pan tilter and the zoom to a predetermined position, acquires images at each location, and displays a plurality of acquired images, Images are connected to a virtual spherical surface whose origin is the movable axis of the pan tilter, and a plane image is created by normalizing the latitude and longitude directions of the virtual spherical surface. The created panoramic image is displayed on the panoramic operation area 6B of the monitor 2 by the computer 1 from the server 9 via the transmission path.

As in the case of the above-described embodiment, an arrow-shaped cursor 7 is drawn at the position of the mouse 8 in the panorama operation area 6B. The pan tilter camera 3 is operated by pointing an arbitrary point or area 6C in the video or the panoramic image with the mouse 8. Further, a pan / tilt limiter display 6D is displayed in the panorama operation area 6B as indicated by a dotted line. The pan / tilt display 6D indicates the limit of the movable range of the pan / tilt camera 3.

By operating the mouse 8, the cursor moves, and can point to an arbitrary point in the panoramic operation area 6B or an arbitrary point generated from the arbitrary area 6C. The pan tilter is driven via the server 9 and the transmission path so that the designated one point is located at the center of the operation area 6A, and the imaging screen 5 moves.
That is, the selected subject is displayed via the server 9 and the transmission path so as to be at the center of the panorama operation area 6B.

FIG. 20 is a block diagram of the entire system according to another embodiment of the present invention. However, the camera unit 11
Since the configuration and function of the pan tilter unit 12 are the same as those of the above-described embodiment (FIG. 3), the detailed configuration thereof is omitted in FIG. The server 9 includes a control unit 13
1. A video capture unit 129 comprising a video capture board and a storage unit 30. The computer 1 is connected to the transmission line 132 via a network, and includes the control unit 31 and the like as in the embodiment shown in FIG. In addition, also about the detailed algorithm in each computer, since the content overlaps with the one embodiment,
The description is omitted.

As in the case of the first embodiment, the image pickup light arriving from the subject is signal-processed by the camera section 11 and converted into signals such as luminance (Y), color (C), and video signals. And a video capture unit 129 of the server 9. As in the embodiment, the pan / tilt camera includes a mode controller 23, a camera controller 24, and a pan / tilt controller 25, and controls the camera unit 11 and the pan / tilt unit 28. The mode controller 23 controls the entire system in the same manner as in the embodiment according to the internal states of the camera unit 11 and the pan / tilt unit 12 and external commands.

The mode controller 23 is connected to the server 9 via a communication path (specifically, using RS232C), and sends a command sent directly from the server 9 or a command sent from the computer 1 via the server 9. In response to the received command, the received command is distributed to the pan tilter controller 25 and the camera controller 24 so as to drive the pan tilter 28 and the zoom lens 16 of the lens block unit 15. Further, the mode controller 23 always obtains information from the pan / tilt controller 25 and the camera controller 24 in order to send the internal state of the pan / tilt camera to the outside via the server 9.

The server 9 periodically acquires the internal state of the pan / tilt camera (for example, current position information of the pan / tilt, zoom lens, etc.) from the mode controller 23 of the pan / tilt unit 12. In addition, a video capture unit 129 is used to transmit the video of the camera unit 11 to the transmission path 132.
The video information input from the transmission path 1 with an arbitrary quality
Digital image data that can be easily sent to
JPEG format still image or bitmap format still image). In addition, digital images of the same format
It can be stored on the storage unit 130 (for example, a hard disk).

When a connection request is made from the computer 1 to the server 9, the server 9 sends out GUI (graphical interface) panel information to be displayed on the monitor 2 connected to the computer 1. The panel information includes an arrangement of the panel, an operation program in the computer 1 when a mouse operation is performed on the panel, and the like, and specifically, an HTML or JAVA program is used. In addition, image data captured by the pan / tilt camera, the state of the pan / tilt camera, and the like are periodically transmitted to the computer 1 via the transmission line 132.

In another embodiment, the Internet is used for the transmission path 132, and exchanges are performed on the transmission path 132 using the HTTP protocol. In the computer 1, the GUI panel information, image information,
The state of the pan-tilt camera is displayed on the monitor 2. On the GUI panel displayed on the screen of the monitor 2, an operation area 6A and a panorama operation area 6B, zoom operation buttons, a cursor of the pointing device 14 (mouse 8), and the like are displayed. Then, in the operation area 6A, the image data sent from the server is decoded and displayed, and the video is rewritten together with the update of the image data. Further, the panorama image data sent from the server 9 is decoded, the panorama image is displayed in the panorama operation area 6B, and the operation range of the pan / tilt camera, the current position of the pan / tilt, the angle of view of the zoom, etc. It is displayed in the same manner as in the embodiment. Then, the computer 1 executes the operation program sent from the server 9 when the GUI panel is operated.

The operation program of another embodiment generates a drive command to the pan / tilt camera and an operation command on the server by clicking the mouse. When the mouse is clicked on the operation area 6A, the operation area 6A is determined based on the view angle information, the current pan tilter position information, and the mouse position information when the mouse is clicked, as in the embodiment.
The position where the mouse is clicked on the image displayed on A is
A command (an absolute position driving command or a relative position driving command) is transmitted to the server 9 so that the pan tilter is driven to be at the center of the operation area 6A (video display).

When the server 9 acquires the command, the command is relayed and transmitted to the pan tilter camera, and the pan tilter is driven to a desired position. As described above, since the driving target of the pan tilter is set on the video, it is possible to easily operate the pan tilter without being conscious of the driving command on the network, the delay of the video, and the like.

In the computer 1, the transmitted panoramic image is displayed so as to be superimposed on the panoramic operation area 6B of the monitor 2, and the environment in which the pan tilter camera is currently set can be displayed at a glance. . Also, as described above, the panorama operation area 6B includes:
Since the position information of the pan tilter, the angle of view of the zoom lens, and the driving range of the pan tilter are displayed, it is possible to easily confirm the current state of the pan tilter on the panoramic image. That is, if the computer 1 or the server 9 issues a command to drive the pan tilter or the zoom, the situation of the pan tilter camera corresponding to the command can be confirmed on the panoramic image.

When the mouse is clicked on the panorama operation area 6B, the position of the mouse click on the panorama image is changed based on the position information of the mouse at the time of the click, as in the embodiment. A command (absolute position driving command) is transmitted to the server 9 so as to be driven to be at the center of the area 6A (image). When the server 9 acquires the command, the command is relayed and transmitted to the pan tilter camera, and the pan tilter is driven to a desired position. As described above, since the drive target of the pan tilter is set on the panoramic image, it is possible to easily operate the pan tilter without being conscious of the drive command on the network, the delay of the video, and the like.

In the embodiment of the present invention, the panoramic images are connected in the computer 1 each time an image is supplied from the pan / tilt camera 3 to the computer 1, and the images are displayed in the panoramic operation area 6B each time they are connected. However, the image may be displayed in the panoramic operation area 6B after all the images are connected.

In the embodiment of the present invention, the computer 1
Although the operation area 6A and the panorama operation area 6B are displayed on the screen of the monitor 2 connected to the monitor 2, the operation area 6A and / or the panorama operation area 6B may be displayed on another display different from the monitor 2. good.

In the embodiment of the present invention, the range that can be photographed by the pan tilter camera 3 may be the maximum range in which the pan tilter camera 3 can move, or the range that can be photographed by the limiter may be limited. Further, the function of limiting the photographing range by the limiter may be provided in the pan tilter camera 3 or the computer 1.

In the embodiment of the present invention, an arbitrary point generated from an arbitrary area is set as the center of the area. However, the present invention is not limited to this, and the outer center, inner center, center of gravity, or vertical center of an arbitrary area may be set as an arbitrary point. good.

In another embodiment of the present invention, the computer 1 creates the panoramic image. However, the server 9 creates the panoramic image and transfers the data of the panoramic image to a network line having a small line capacity. Transfer to the computer 1 and the panorama operation area 6B of the monitor 2
May be displayed.

In another embodiment of the present invention, for ease of explanation, one computer 1 is used for the server 9 and the pan tilter camera 3 installed in a remote place. However, the server 9 and the pan tilter camera are used. 3 are located around the world, for example, from a plurality of computers via the Internet.
One pan tilter camera 3 may be controlled.

[0125]

According to the present invention, since a method of mapping a captured image on a virtual spherical surface and connecting the images is employed, it is possible to provide an image having no distortion at the connecting portion. This makes it possible to combine high-resolution (high-quality, high-quality) images that can only be obtained using high-resolution devices and image formats, even if images from devices using low-cost devices and image formats are connected. Obtainable. In addition, since a panoramic image that can be created using an ultra-wide-angle lens or a fisheye lens, which is generally expensive, can be easily created, an application used in a device that can change the imaging direction of an imaging device (for example, a camera with a built-in pan tilter) The effect can be expected.

Further, according to the present invention, for example, by displaying a zoom angle of view, a pan tilter position, and the like on a panoramic image, it is possible to easily grasp the state of the imaging apparatus and to obtain a panoramic image. In the above, a place to be driven and a subject to be projected can be designated, so that a desired image can be easily obtained. In particular, by using panoramic images for such applications, better operability and visibility can be realized for control applications such as monitoring, observation, guidance, and introduction with a remote video camera using a line. be able to.

[Brief description of the drawings]

FIG. 1 is a schematic diagram illustrating a schematic configuration of a system according to an embodiment of the present invention.

FIG. 2 is a schematic diagram illustrating a screen according to an embodiment of the present invention.

FIG. 3 is a block diagram illustrating a system configuration according to an embodiment of the present invention.

FIG. 4 is a schematic diagram illustrating generation of a panoramic image according to an embodiment of the present invention.

FIG. 5 is a schematic diagram for explaining generation of a panoramic image according to an embodiment of the present invention.

FIG. 6 is a schematic diagram for explaining generation of a panoramic image according to an embodiment of the present invention.

FIG. 7 is a schematic diagram used for explaining generation of angle information of a pan tilter camera from position coordinates on a panorama operation area according to an embodiment of the present invention.

FIG. 8 is a schematic diagram for explaining plane spherical conversion in one embodiment of the present invention.

FIG. 9 is a schematic diagram illustrating coordinate conversion in an operation area according to an embodiment of the present invention.

FIG. 10 is a schematic diagram for explaining coordinate conversion in a panorama operation area according to an embodiment of the present invention.

FIG. 11 is a schematic diagram for explaining position information and angle information inside the pan tilter camera in one embodiment of the present invention.

FIG. 12 is a schematic diagram illustrating angle coordinates of a pan tilter camera and position coordinates of a panorama operation area according to an embodiment of the present invention.

FIG. 13 is a schematic diagram illustrating an angle of view of a pan tilter camera and a frame in a panorama operation area according to an embodiment of the present invention.

FIG. 14 is a schematic diagram illustrating conversion of zoom data and magnification data according to an embodiment of the present invention.

FIG. 15 is a flowchart illustrating an example of overall processing according to an embodiment of the present invention.

FIG. 16 is a flowchart illustrating an example of processing of a panorama creation request event according to an embodiment of the present invention.

FIG. 17 is a flowchart illustrating an example of processing of a timer event according to an embodiment of the present invention.

FIG. 18 is a flowchart illustrating an example of a panorama creation process according to an embodiment of the present invention.

FIG. 19 is a schematic diagram illustrating a schematic configuration of a system according to another embodiment of the present invention.

FIG. 20 is a block diagram showing a system configuration according to another embodiment of the present invention.

FIG. 21 is a schematic diagram for explaining creation of a panoramic image.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Computer, 2 ... Monitor, 6A ... Operation area, 6B ... Panorama operation area, 7 ... Cursor, 9 ... Server, 11 ... Camera part, 12 ...
Pan tilter unit, 13 TV monitor, 14 pointing device, 15 lens block unit, 1
6 zoom lens, 17 zoom unit, 18, 2
6, 27 ... motor, 19 ... solid-state image sensor, 20
... Signal separation / automatic gain adjustment circuit, 21 ... A / D
Converter, 22 ... Signal processing circuit, 23 ... Mode controller, 24 ... Camera controller, 25 ...
・ Pan tilter controller, 28 ・ ・ ・ Pan tilter, 2
9: video capture, 30: storage unit, 31
... Control unit

 ────────────────────────────────────────────────── ─── Continuing on the front page (72) Inventor Naoyasu Hosunuma 6-7-35 Kita-Shinagawa, Shinagawa-ku, Tokyo Inside Sony Corporation (72) Inventor Ken Tamayama 6-35, Kita-Shinagawa, Shinagawa-ku, Tokyo No. Sony Corporation

Claims (11)

[Claims] An imaging unit that captures an image; a driving unit that controls a direction in which the imaging unit captures an image; and a plurality of images obtained by the imaging unit, a virtual spherical surface viewed from a rotation center of the driving unit. A panoramic image generating apparatus, comprising: image linking means for linking without distortion by mapping to the image. 2. The method according to claim 1, wherein the image linking means maps the image mapped to the virtual sphere to a plane by normalizing the image with the latitude and longitude of the virtual sphere. A panoramic image generation device characterized by doing the following. 3. The driving device according to claim 1, wherein the driving unit has a mechanism with play in a driving system, and when connecting the images obtained by the imaging unit, the driving unit is moved in the same direction. A panoramic image generation device, wherein the effect of backlash is corrected using an image acquired by driving. 4. The method according to claim 1, wherein, when a plurality of images are linked by the image linking means, the images are linked each time one image is acquired, and the result is displayed on a display means. A panoramic image generating apparatus, wherein the panoramic image generating apparatus displays the progress of the panoramic image. 5. The method according to claim 1, wherein when connecting a plurality of images by the image connecting means, every time one image is obtained, the result of the connection is not displayed on the display means, but all images are obtained. A panoramic image generating device, wherein the panoramic image generating device is displayed on the display unit later. 6. The method according to claim 1, wherein, when a plurality of images are connected by the image connecting means, the images are sequentially connected each time one image is obtained, but the result is not displayed on the display means. A panoramic image generating apparatus, wherein when all images are captured and the connection is completed, the result is displayed on a display means. 7. The panoramic image generating apparatus according to claim 1, wherein an image of a movable range of the driving unit is captured by the imaging unit, and a panoramic image is automatically generated. 8. The panoramic image generating apparatus according to claim 7, wherein the panoramic image is displayed in an operation area for instructing a driving direction of the driving unit. 9. The method according to claim 8, wherein by designating an arbitrary point in the operation area, a subject at a position corresponding to the designated arbitrary point is moved to arbitrary position coordinates of the driving unit. A panoramic image generation device characterized by doing the following. 10. The object according to claim 8, wherein by specifying an arbitrary area of the operation area, an object at a position corresponding to an arbitrary point generated from the arbitrary area is set to an arbitrary position coordinate of the driving means. A panoramic image generation device, wherein 11. A step of photographing an image, a step of controlling a direction in which the image is photographed, and a step of dividing a plurality of images obtained by photographing with respect to a virtual spherical surface viewed from a rotation center of the photographing position. A step of connecting without distortion by mapping.