AU3227899A - Camera control user interface, camera control method and computer program product - Google Patents

Description

S F Ref: 465352

AUSTRALIA

PATENTS ACT 1990 COMPLETE SPECIFICATION FOR A STANDARD PATENT 0* 4* 4 .*u

S.

S

.5.5

ORIGINAL

Name and Address of Applicant: NEC Corporation 7-1, Shiba Minato-ku Tokyo

JAPAN

Actual Inventor(s): Fujie Mitsuhashi, Naoto Okano and Akira Yajima Yoshimura, Hideo Ohno, Hisashi Address for Service: Invention Title: Spruson Ferguson, Patent Attorneys Level 33 St Martins Tower, 31 Market Street Sydney, New South Wales, 2000, Australia Camera Control User Interface, Camera Control Method and Computer Program Product The following statement is a full description of this invention, including the best method of performing it known to me/us:- 5845 CAMERA CONTROL USER INTERFACE, CAMERA CONTROL METHOD AND COMPUTER PROGRAM PRODUCT This application is based on Japanese Patent Application No. H10-188006 filed May 29, 1998, which is incorporated herein by reference in its entirety.

BACKGROUND OF THE INVENTION Field of the Invention The present invention relates to a user interface for remotely controlling the S.rotating direction, etc. of a situated rotatable camera.

"•'•"Description of the Related Art 10 Some conventional user interfaces for controlling the orientation of a camera are of the type in which one of arrow buttons provided on an operation panel is depressed, while some conventional user interfaces for controlling the camera orientation are of the type in which the rotating direction of a camera is determined using a scroll bar provided on an operation screen, and other conventional user interfaces for controlling the camera orientation are of the type in which the value of a rotating angle is input directly from a keyboard or the like.

However, the above-described user interfaces have the following drawbacks when the user intends to photograph a predetermined target: In the case of the interfaces of the type in which the rotating direction of a camera is determined in accordance with the operation of the arrow buttons, the scroll bar or the like, for example, the user if unskilled in the camera operation cannot easily understand which arrow button and by what amount he/she needs to depress in order to photograph the target, or by what amount and in which direction he/she needs to move the scroll bar in order to photograph the target.

On the other hand, the interfaces of the type wherein a value is input from the keyboard or the like are advantageous when the user precisely knows the position, photographing direction, etc. of the situated camera. However, unless the user knows such information, it may be difficult for him/her to determine the camera angle so that the camera orients to the target.

Meanwhile, some conventional user interfaces for setting the photographing magnification of a camera are the type wherein one of the buttons corresponding to different magnifications is depressed, while some conventional user interfaces for setting the photographing magnification are the type wherein the magnification is input directly from a keyboard or the like. However, these operations for setting the magnification are independent from the rotating direction determining operation. Therefore, when the user desires to perform .i S. 10 both the rotating direction determining operation and the magnification setting operation, he/she has to complete one operation first and then perform the other operation.

Unexamined Japanese Patent Application KOKAI Publication No. H8- 336166 discloses, as a system for enabling the user to view the image of a remote place, a system which determines the rotating direction of a photograph camera through use of a sensor that senses the position and direction of the 0 head of a user.

According to the above system which varies the rotating direction of the camera in accordance with the movement of the user's head, when the user desires to further view an image on the left side of a currently displayed image, the isr tl irns to the left qnd thp svstem then senses the movement of the user's head and turns the camera to the left accordingly.

The above-described system is suitable for the case where there is no specific photographing target and where the camera need not be operated precisely. However, in the case of directing the camera to a specific target, the user cannot understand by what amount he/she needs to move his/her head in order to adjust the focal point to the target. Furthermore, in order to zoom in on the target, the user has to finish the operation of adjusting the focal point to the target first and thereafter has to perform the operation of inclining a joystick or the like forward, for example.

Realizing a user friendly function, such as the function of supplying the user with photographing target information or the like when he/she desires to know it, is also preferred.

SUMMARY OF THE INVENTION It is accordingly an object of the present invention to provide a camera control user interface apparatus, a camera control method and a computer S 10 program product, all facilitating camera operations for photographing a target located in a remote place.

It is another object of the present invention to provide a camera control user interface apparatus, a camera control method and a computer program product, all being capable of supplying a user with information concerning an article, a place or the like which a camera is photographing.

According to the first aspect of the present invention having the above- :described objects, there is provided a camera control user interface apparatus for controlling a camera situated in a remote place, comprising: panorama image display means 3, 6) which displays a panorama image which covers a range that is photographable with the camera; inn ut mann (1 which inni it rnnrdinin~t nn qnirl nqnnrqmq imnge sending means which sends, to the camera, values of the coordinates which have been input from said input means; and photographing result display means 3) which displays a photographing result received from the camera.

According to the above structure, a user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward and leftward arrow buttons, levers or the like, and speeds up the execution of a decision made by the user.

The camera control user interface apparatus may further comprise menu display means 3, 6) which displays a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; and .o S* 10 the sending means may comprise means which sends the magnification selected on said menu to the camera.

S"Thus, since the user can select the zoom magnification of the camera on the displayed menu, he/she can determine the position and the magnification with ease by performing a series of operations.

The menu, displayed on said menu display means 3, may be a pop-up menu displayed in response to an input of the coordinates from said input means.

According to the second aspect of the present invention having the abovedescribed objects, there is provided a camera control user interface apparatus for controlling a camera situated in a remote place, comprising: a display device an inpi u rAeice I/1) \which innn nte nnnrincta nn nn imagn ricnla\lr nn airl display device; a storage device which stores a panorama image which covers a range that is photographable with the camera; and a control device which reads out said panorama image from said storage device causes said display device to display the readout panorama image, sends, to the camera, values of the coordinates which have been input from said input device and causes said display device to display a photographed image received from the camera.

According to the above structure, the user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward and leftward arrow buttons, levers or the like, and speeds up the execution of a decision made by the user.

The control device may cause said display device to display a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image, and said control device may send, to the camera, the magnification which has been input from said input device Thus, since the user can select the zoom magnification of the camera on the displayed menu, he/she can determine the position and the magnification with ease by performing a series of operations.

,The control device may divide a screen of said display device into a plurality of areas and displays said panorama image and the photographed image received from the camera in the divided areas of said screen.

The storage device may further store photographing target information -I Ir InI I ItL%-ranhinn tarignt, in accnritinn with vnlll Ps of onordinates, on IL.#UI Ix d l I C lVLV k ui n> *i i ii 11 ly ^vL, u11 v said panorama image; in addition to the magnification, a display of the photographing target information may be selectable on said menu displayed on said display device (3) by said control device and when the display of the photographing target information is selected on said menu, said control device may read out from said storage device the 6 photographing target information associated with the values of the coordinates which have been input from said input device and may cause said display device to display the readout photographing target information.

The menu, displayed on said display device by said control device may be a pop-up menu displayed in response to an input of coordinates on said panorama image from said input device According to the third aspect of the present invention, there is provided a camera control method for controlling a camera situated in a remote place, comprising: 10 displaying a panorama image which covers a range that is photographable with the camera; sending values of coordinates of a point designated on said panorama image S" to the camera; and receiving a photographed image from the camera and displaying the 15 photographed image as received.

According to the above structure, the user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward anrd InftiLardc arrIoA hi ttons ie'/ers onr r the lil k, anrl oiJ mrl I u th nvl ti~ i f M decision made by the user.

The camera control method may further comprise: displaying a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; wherein the sending of the values of the coordinates of the point designated on said panorama image further comprises sending the magnification selected on said menu to the camera.

Thus, since the user can select the zoom magnification of the camera on the displayed menu, he/she can determine the position and the magnification with ease by performing a series of operations.

The panorama image and said photographed image may be displayed in a plurality of areas into which a single display screen has been divided.

In addition to the magnification, a display of photographing target information concerning a photographing target may be selectable on the displayed menu; and o 10 said camera control method may further comprise storing the photographing target information concerning the photographing So target, in association with values of coordinates on said panorama image as o displayed, and reading out the stored photographing target information associated with values of coordinates of the point designated on said panorama image, when the display of the photographing target information is selected on said menu.

According to the above structure, the user can obtain various information concerning the photographing target, such as information as to what the user himself/herself has photographed what building the user has photographed).

The dinsplayed meni may be a pop-up n menu displayed in response to the designation of the point on said panorama image.

According to the fourth aspect of the present invention, there is provided a computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer to work as a camera control user interface apparatus used to control a camera situated in a remote place, the computer readable program comprising instructions for causing the computer to serve as: means for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; means for sending values of input coordinates to the camera; and means for displaying, in another predetermined area on the display screen, a photographed image received from the camera.

According to the above structure, the user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The 10 designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward and leftward arrow buttons, levers or the like, and speeds up the execution of a decision made by the user.

According to the fifth aspect of the present invention, there is provided a computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer to work as camera control user interface apparatus used to control a camera situated in a remote place, the computer readable program comprising instructions for causing the computer to serve as: means for displaying, in a predetermined area on a display screen, a nannrama imrana Wnhirh rn\l-re a ranno tha i2 nhrntrnrnhahl irith thna rammrameans for displaying a menu to select a magnification of the camera, in a state wherein said menu is superimposed on said panorama image; means for sending values of input coordinates and the magnification to the camera; and means for displaying, in another predetermined area on the display screen, a photographed image received from the camera.

9 According to the above structure, the user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward and leftward arrow buttons, levers or the like, and speeds up the execution of a decision made by the user.

Moreover, since the user can select the zoom magnification of the camera on the displayed menu, he/she can determine the position and the magnification 10 with ease by performing a series of operations.

According to the sixth aspect of the present invention, there is provided a computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer, which stores photographing target information concerning a photographing target in association with values of coordinates on a panorama image, to work as a camera control user interface apparatus used to control a camera situated in a remote place, the panorama image covering a range that is photographable with the camera, and the computer readable program comprising instructions for causing the computer to serve as: means for displaying said panorama image on a display screen; means for diqnlavinn a me.nu to select on of a mannificntinn of the camer and a display of information, in a state in which said menu is superimposed on said panorama image; means for sending values of input coordinates and the magnification to the camera when the magnification is selected on said menu; means for sending the values of the input coordinates and a predetermined magnification to the camera, reading the photographing target information associated with the values of the input coordinates, and displaying the readout photographing target information on the display screen, when the display of information is selected on said menu; and means for displaying a photographed image received from the camera.

According to the above structure, the user can rotate the photograph camera relative to the target only by designating the target's position on the panorama image with an input device such as a pointing device or the like. The designating operation in this case is quicker and easier than the operation of designating the direction of rotation through use of upward, downward, rightward .9 10 and leftward arrow buttons, levers or the like, and speeds up the execution of a decision made by the user.

According to the seventh aspect of the present invention, there is provided a carrier wave including a program signal embedded therein for causing a computer to work as a camera control user interface apparatus used to control a camera situated in a remote place, the program signal representing a program which comprises: segment for displaying, in a predetermined area on a display screen, a S .i panorama image which covers a range that is photographable with the camera; a segment for sending values of input coordinates to the camera; and a segment for displaying, in another predetermined area on the display scran nhnnnrnh imnne receive.d frnm the cqmprq_ U r J According to the eighth aspect of the present invention, there is provided a program signal embedded in a carrier wave, for causing a computer to work as a camera control user interface apparatus used to control a camera situated in a remote place, the program signal representing a program which comprises: a segment for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; 11 a segment for displaying a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; a segment for sending values of input coordinates and the magnification to the camera; and a segment for displaying, in another predetermined area on the display screen, a photographed image received from the camera.

According to the ninth aspect of the present invention, there is provided a ::program signal embedded in a carrier wave, for causing a computer, which stores photographing target information concerning a photographing target in association with values of coordinates on a panorama image, to work as a camera control user interface apparatus used to control a camera situated in a remote place, the panorama image covering a range that is photographable with the camera, and the program signal representing a program which comprises: a segment for displaying said panorama image on a display screen;

S

a segment for displaying a menu to select one of a magnification of the camera and a display of information, in a state in which said menu is superimposed on said panorama image; a segment for sending values of input coordinates and the magnification to the camera when the magnification is selected on said menu; a segment for sending the values of the input coordinates and a n m mn tn thp tnmprn rinrfini th nhntnnrnnhinn trnmit rSU IIII i III I IIII.L4I'J* *V rll lllu *u ul** *l rl ru I J l information associated with the values of the input coordinates, and displaying the readout photographing target information on the display screen, when the display of information is selected on said menu; and a segment for displaying a photographed image received from the camera on the display screen.

BRIEF DESCRIPTION OF THE DRAWINGS

I

FIG. 1 is a diagram illustrating the structure of a camera system according to an embodiment of the present invention; FIG. 2 is a diagram exemplifying photographing target information stored in the camera system illustrated in FIG. 1; FIG. 3 is a diagram exemplifying the display screen of the display device of the camera system illustrated in FIG. 1; FIG. 4 is a diagram for explaining an example of a method for transforming S. coordinates on a panorama image into the rotating FIGS. 5 and 6 are diagrams for explaining another example of the method for cc 10 transforming coordinates on the panorama image into the rotating coordinates of the camera; FIG.7 is a flowchart for explaining a camera remote operation which a user performs in the camera system of FIG. 1; and FIG. 8 is a flowchart for explaining the operation of a control section included in the camera system of FIG. 1.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT A camera system according to an embodiment of the present invention will now be described with reference to the drawings.

According to this system, by operating a personal computer or a workstation, a user can remotely operate a rotatable camera which is connected to a network, +kril i unsk o ic f n r[ II /(rapr hi-al I Icor Interfar'-n LIII.JUJII I J JI lk C 4n %IAVJI 1 Il4l v IIil m.vv FIG. 1 is a diagram illustrating the hardware structure of the camera system according to an embodiment of the present invention. This camera system comprises: an input device 1; a control section 2; a display device 3; a network 4 which connects the control section 2 and a camera 5; the camera 5 which is rotatable up, down, clockwise and counterclockwise, or either up-down or clockwise-counterclockwise only; and a storage device 6.

13 The input device 1 comprises a pointing device such as a mouse, and has such a function as can designate points on the display device 3. If, however, the input device 1 is designed so that it can designate points on the display device 3 upon depression of a specific key on a keyboard, then the keyboard can be used as the input device 1.

The control section 2 comprises a personal computer, a workstation or the like. The control section 2 causes the display device 3 to display a predetermined operation screen which will be described later, and controls the °"-photographing position and photographing magnification of the camera 5 in 10 accordance with a user's input operation at the operation screen. In the case where photographing target information associated with a point designated on the operation screen by a user is stored in the storage section 6, the control section 2 displays the photographing target information on the operation screen.

The control section 2 also controls the operation of an input/output device, communications with the camera, the receipt of a photographing result from the camera and an output to the display.

The camera 5, which is rotatable up, down, clockwise and counterclockwise, or either up-down or clockwise-counterclockwise only, is connected to the network 4 directly or via a computer, for example. Further, the camera 5 and the control section 2 communicate with each other through the network 4. The cram 5 nhntnnrnhsa nn imrni in nrnrdrnc. with inqtriutinnq frnm the rnntrnl section 2, for example, and sends the photographed image to the control section 2 through the network 4.

The storage section 6 stores a panorama image, the photographing target information, etc. in advance, the panorama image being displayed on the operation screen as will be explained later. The panorama image is an image by reference to which the user can determine the position of a photographing 14 target. The photographing target information is information concerning the photographing target. Furthermore, an image which the camera 5 has photographed while being remotely operated by the user, is temporarily stored in the storage section 6. The image as photographed is displayed on a part of the operation screen which will be explained later.

The photographing target information contains data "designated area" and "information contents." When a point designated on the panorama image by the user is within a rectangular area whose diagonal extends between the point (Xl, Y1) and the point (X2, Y2), the photographing target information R1 shown in FIG.

2, for example, serves as data for displaying characters "this is a cottage on X mountain." The storage section 6, which stores the photographing target information, may be provided in a computer system including the control section 2, or may be provided in another computer system which is connectable to the control section 2 via the network.

The operation screen, which is displayed on the display device 3, will now be described with reference to FIG. 3. A panorama image 12, a photographing result 14 which is a still or moving image photographed with the camera 5, are laid out on the operation screen 11. The photographing result 14 is an image which the camera 5 has photographed in accordance with a user's input operation The panorama image 12 is an image by reference to which the user can determine the photographing position, and contains all images present within the range that the user can photograph rotating the camera 5. The panorama image 12 needs to be prepared in advance by a system manager. For example, in the case of photographing an outdoor scene with the camera 5, the outdoor panorama image 12 is stored in advance in the storage section 6 or the like, and is displayed on the display device 3 as a wide-range landscape picture, as shown in FIG. 3, for example.

The panorama image 12 cannot be photographed at a time with the camera because the panorama image 12 contains all images present within the range of rotation of the camera. In consideration of this, the entire panoramic scene is photographed part by part, and the images as photographed are combined and processed into the panorama image 12. At the time of this processing, an 4 4image size reducing process may also be performed.

:goo. Furthermore, at the time of processing of the photographed images into the 10 panorama image 12, it is preferred that the following conditions be satisfied in order to obtain correct coordinates (rotating coordinates) specifying a position to which the camera is to be rotated: he camera 5 is designed so as to be rotatable up, down, clockwise, and counterclockwise on the axis thereof. Let it be assumed that the camera 5 is rotatable up, down, clockwise and counterclockwise through 3600 when the camera 5 thus designed is focused to all points equidistant from its rotational axis, 09** 0 a set of focal points define a sphere. In other words, when the photographed images are simply combined, the combination thereof forms an image like one represented on the inner surface of a sphere. However, since the panorama image 12 is rectangular, the inner surface of the sphere needs to be developed intn n ninnp Tn thi. and i irh a mpthnd q.q i nrniptinn m.thod fnr mans is employed. More specifically, the function (matrix) f shown below is adopted to transform the coordinates y) of points on the photographed images into those for use after the development of the inner surface of the sphere into a plane: f y) By applying the photographed images to the function f shown above, the coordinates for use after the development of the inner surface of the 16 sphere into a plane can be obtained, and the photographed images are combined into one image on the basis of the obtained coordinates. By so doing, the panorama image 12 is created so that points thereon and the rotating coordinates of the camera correspond to each other. Employing the panorama image 12 thus created, the rotating coordinates to adjust the focal point of the camera to an arbitrary point, when designated on the panorama image 12 by the user, can be obtained.

*The use of the inverse function f 1 of the function f is practical in actually :°"°"transforming the coordinates on the panorama image into the rotating 10 coordinates y) of the camera.

C*

y) f y') Adopting the inverse function the rotating coordinates corresponding to all points forming the panorama image 12 as obtained can be calculated.

An example of the function used to transform the coordinates of points on 15 the panorama image into the rotating coordinates of the camera 5 will now be o described.

FIG. 4 shows a plane F on which a photographed image is projected, a locus R of horizontal photographing points of the camera 5, and a predetermined starting point P on the locus R. For example, when the camera 5 photographs a point P1 which is separated clockwise (or counterclockwise) from the starting point P by an angular distance of e 1 and when a nnpoint P1'. at which the image of the point P1 is projected on the plane F, is a point where a line extended from the point P1 meets the plane F, the distance "xa" between the starting point P and the point P1' is expressed as follows: xa d x tan 8 (d is the distance between the camera 5 and the focal point) Similarly, when the camera 5 photographs a point P2 which is vertically separated upward (or downward) from the starting point P by an angular distance of 6 2, the distance "ya" between the starting point P and a point P2' at which the image of the point P2 is projected on the plane F, is expressed as follows: ya d x tan 8 (d is the distance between the camera 5 and the focal point) In consequence, the relationship between the photographing coordinates (6 1, 6 2) and the projection points (xa, ya), from which the rotating coordinates can be derived, is expressed as follows: 1,8 2) (tan-' (xa tan-' (ya d)) The above-described method is advantageous when using an image of a specific single spot as the panorama image.

10 However, when using an image of a considerably wide view as the panorama S image, a plurality of images photographed by the camera can be combined into the panorama image by the following method: As shown in FIG. 5, for example, an imaginary plane F coordinate plane) of an image, which the camera attains through one photographing action, 15 is determined in an X-Y-Z coordinate system that adopts the location of the camera as the origin. The angle (vertical angle) which is defined between an X-Y plane and the optical axis extending from the camera up to the origin on the coordinate plane, is expressed as P. The angle (horizontal angle) which is defined between the X axis and a line attained by projecting the optical axis on the X-Y plane, is expressed as The vertical angle which is defined between the X-YV nlane and line extendinn from the orinin of the X-Y-Z coordinate system up to a given pixel on the coordinate plane, is expressed while the horizontal angle which is defined between the X axis and a line attained by projecting, on the X-Y plane, the line extending from the origin of the X-Y-Z coordinate system up to the given pixel on the coordinate plane, is expressed as 8 Furthermore, when the lateral and vertical dimensions of the image shown in FIG. 6, for example, are expressed as Wdip and Hdsp, respectively, the x and y coordinates of an arbitrary point y) on the image are given as follows: Xdisp a x (Wdip 2 tan hfizt 2)) Yd,sp b x (Hdip 2 tan veruc, 2)) where hoonta, represents the horizontal angle of view of the camera, while I veruic represents the vertical angle of view of the camera.

a (-(cosQ' x sin 8 k cos x sin e- b x sinP x sin 8) cos e (sin k-sin(P) cos k cost xcost' x cos (8 sinQP x sinV' 10 A rectangular image is attained by repeating the above calculations. Color data pertaining to all pixels on the coordinate plane are incarnated on the rectangular image.

A plurality of rectangular images such as that described above are created and combined into the panorama image. Points on the panorama image thus 00 15 created correspond to the angles through which the camera can be rotated.

When one point on the panorama image is designated, the camera is rotated S" through the corresponding angle.

In these cases, a calculated value may further be rounded up, down, off, etc.

to the appropriate value whose unit is the minimum angle of rotation of the camera The ,panornama imagn which cnntains all imnnps npresent within the ranae that the user can photograph rotating the camera 5, may be displayed at a time in its entirety, or may be displayed part by part while being scrolled in accordance with an up-down or right-left scrolling operation.

A pop-up menu 13 will now be described. The pop-up menu 13 is superimposed on the panorama image 12 when the user designates a point on the panorama image 12. On the pop-up menu 13, the user can select either the 19 setting of the zoom magnification or the display of information. When the user selects the setting of the zoom magnification, available zoom magnification values, "X "X "X "x "X 8" and "X 10", are displayed on the popup menu 13 as illustrated in FIG. 3, and the zoom magnification which the camera 5 is to employ at the time of photographing is determined when the user selects one magnification value with the input device 1. The function of determining the magnification in accordance with such an operation is called "pop-up zoom.

Thus, the angle and zoom magnification of the camera 5 can be determined S" 10 by two simple operations, the operations of designating coordinates on the panorama image 12 and selecting the magnification.

Meanwhile, when the user selects the display of information from the pop-up menu 13 the item "info" on the pop-up menu illustrated in FIG. the photographing target information corresponding to the coordinates designated beforehand on the panorama image 12 is read out from the storage section 6 and is displayed. The photographing target information may be displayed in the form of characters, images or any other form which can be output to the display or peripheral terminal of the computer. For example, when the photographing target information is described in HTML (HyperText Markup Language), data such as images, moving pictures and voices can be handled. By virtue of the nrar-znn nf the nhntnnrqnhinn tarnt infnrmatinn_ the iqser c~n not only photograph an image with the camera, but also obtain information concerning the photographed target with efficiency.

Areas associated with the information contents of the photographing target information may be areas (circles, polygons, etc.) defined by a plurality of coordinates (Xl, Y1) (Xn, Yn) on the panorama image 12. In the case of searching the information in order to specify a polygon containing a designated

I

point, the search is easiest if the polygon is rectangular. Further, points can also be associated with the information contents.

Explanations will now be made as to what operation the user performs and how the above-described camera system operates. First of all, operations including the determination of a user's intention will be described, and the operations of the present apparatus will be described thereafter.

FIG. 7 shows the flow of operations which the user performs. Let it be assumed that the camera 5 is situated to photograph an outdoor scene. Let it be further assumed that the outdoor panorama image 12 prepared beforehand in 10 that case is displayed on the operation screen 11, and the photographing target information associated with the panorama image 12 is stored in the storage section 6.

First, the user views the prepared panorama image 12 (a step U01). Since the camera 5 is situated to photograph an outdoor scene, the panorama image 12 shows a wide-range landscape. Next, the user determines a photographing target point on the landscape represented by the panorama image 12 (a step U02).

Following the above, the user operates the input device 1 while viewing the panorama image 12, in order to send position information specifying the position of the point determined on the panorama image 12 by the step U02 to the controrl sction 2 The iilr .n Pqsilv dn son with q nnintinn device such as a mouse. In accordance with this operation, the control section 2 displays the pop-up menu 13 on the panorama image 12, and then the user views it (a step U04).

Next, the user determines whether to select, on the pop-up menu 13, the setting of the magnification or the display of information (a step When the user determines to select the setting of the magnification in this

M

step, then he/she selects the desired magnification value on the pop-up menu 13 by using the input device 1 (a step U06, a step U07). When the user determines to select the display of information, then he/she selects the display of information on the pop-up menu 13 by using the input device 1 (a step U08).

The user's operations are thus completed. The user can view an image photographed in accordance with the designated position and magnification.

Furthermore, in the case where the user selects the display of information, he/she can also view information concerning the designated position. The user is led from one operation to another in a dialogic manner, which prevents the 10 user from being perplexed as to which operation he/she is to perform.

The operations of the control section 2 according to this embodiment will now be explained in sequence, with reference to FIG. 8.

First, the control section 2 conducts a predetermined initial process such as reading out data representing the panorama image from the storage section 6 and displaying the readout data on the operation screen, etc. (a step AO).

Then, the control section 2 receives the position information specifying the position of the point which has been designated on the panorama image 12 with the input device 1 (a step A01). The control section 2 transforms the position information specifying the position of the point designated on the panorama image 12 into rotating position information (such as the rotating coordinates renre.qentinn the hnrinzontal anular rdistannrce nr vertical angular distance frorm h e reference position) used to rotate the camera 5 (a step A02). This transformation may be performed before a step Al 11 after the completion of the steps A03 to A10. Alternatively, an intervening program may be provided between the camera 5 and the present apparatus, and the present apparatus sends the position information to the intervening program so that the intervening program transforms the position information into the rotating position information to be sent to the camera 5. In this case, there is no need to transform the information during the process shown in FIG. 8.

Having received an input of the point on the panorama image 12, the control section 2 displays the pop-up menu 13 (a step A03) and waits for the user selecting either the setting of the magnification or the display of information.

Having received an input on the pop-up menu 13 (a step A04), the control section 2 determines which of the setting of the magnification and the display of information the user has selected (a step When the user has selected the setting of the magnification on the pop-up menu 13, the magnification selected using the input device 1 is set as the magnification information to be sent to the camera 5 (a step A06).

S". On the other hand, when the user has selected the display of information on the pop-up menu 13, the control section 2 searches the storage section 6 for the :photographing target information, while using as a search key the position information y) specifying the position designated with the input device 1 (a step A07). If there is the photographing target information corresponding to the search key ("yes" in a step A08), then the control section 2 displays the detected photographing target information on the display device 3 (a step A09), whereas if there is no photographing target information corresponding to the search key in the step A08), then the control section 2 advances to the next step.

MlNYt th rnntrml oe'tinn 0 otsoe -ae th mannifi-tion information to be+ sent to I- 11, -l -D w i L l .l l IIII %C041I1 I I f IIIIUIIIII LIWI I LU W OV IIL I the camera, the magnification which has been input by a prior operation or the magnification which has been set as the initial value (the step A10). This step may be performed before the step A07.

When the step A06 or the step A10 is completed, the control section 2 in the step Al 11 sends the rotating position information, the magnification information and a photographing instruction to the remote camera 5 via the network 4. In 23 response to this, the camera 5 performs the setting pertaining to both a rotation and the zoom magnification, conducts photographing, and sends the photographing result to the control section 2 via the network 4. The control -section 2 displays the received photographing result on the display device 3.

The rotating position information, into which the position information is transformed, is not limited to the rotating coordinates, and may include various data. For example, the rotating position information may include motor rotation information (such as the rotary speed, the number of steps, the number of pulses, etc.).

10 In the case where the camera 5 has the control section 5, a driving section, etc., the camera itself may perform the above-described transformation.

Explained in the above description is the case where the photographing result is displayed on the single display device 3, together with the panorama image 12. However, the photographing result may be output to a different output device. For example, the photographing result may be displayed on another display, or may be output to a photograph output device. Similarly, the photographing target information may also be output to a different output device.

In the above description, the computer is used to control an input, an output and communications. However, an information terminal into which a computer and an input/output device have been integrated may be adopted.

Furthermore. the photooranhing targiet informition can he dilnved withnl t entailing a photographing operation, in such cases as when the transmission of a photographed image is slow.

Moreover, in the above description, an explanation is made to a mode in which the photographing result and the photographing target information are attained in real time. For example, however, the present invention is also applicable to another mode in which the camera 5 conducts photographing at the 24 time appointed by reservations, and the user can view the photographing result and the photographing target information after the photographing.

Further, in the above description, the setting of the magnification or the display of information is selected on the pop-up menu. For example, however, only the magnification may be selectable on the pop-up menu, and another menu on which it is selectable whether the display of information is necessary or not, may be further displayed when the magnification is selected. In this case, the designation of a point as the photographing target, the setting of the °°magnification and a request for the display of information can be simultaneously .o performed by a series of operations.

Moreover, in the above description, the menu on which the setting of the magnification or the display of information is selected is realized as the pop-up menu. However, the menu is not limited thereto, and may be the initial menu to select the setting of the magnification or the display of information.

a Further, the apparatus according to the present invention can be realized using not only a dedicated system, but also a computer system for general use.

For example, the apparatus which carries out the above-described steps can be attained by installing a program for executing the above-described operations into a computer from a medium (such as a floppy disk, a CD-ROM or the like) containing the program stored therein. In addition, when an OS undertakes the execution of one or some of the hnove-desricrihed ft unttinns or when th n OS and an application realize the above-described functions in cooperation with each other, steps other than those to be executed by the OS may be stored in the medium.

Further, the medium from which the program is supplied to the computer may be a communication medium (which temporarily stores the program as in the case of a communication line, a communication network and a communication system). For example, the program may be posted on a bulletin board (BBS) for the communication network, and may be distributed through the network.

The above-described steps can be carried out by running and executing the program under the control of the OS in the same manner as in the case of other application programs.

Finally, the network 4 between the present apparatus and the camera 5 may be the Internet, for example, and the user may remotely operate the camera .through the Internet by operating the computer having the above-described .t o• a a a

Claims (20)

1. A camera control user interface apparatus for controlling a camera situated in a remote place, comprising: panorama image display means 3, 6) which displays a panorama image which covers a range that is photographable with the camera; input means 2) which inputs coordinates on said panorama image; sending means which sends, to the camera, values of the coordinates which have been input from said input means; and photographing result display means 3) which displays a photographing S. result received from the camera.

2. The camera control user interface apparatus according to claim 1, wherein: said camera control user interface apparatus further comprises menu display means 3, 6) which displays a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; and said sending means comprises means which sends the magnification selected on said menu to the camera.

3. The camera control user interface apparatus according to 1, wherein said menu, displayed on said menu display means 3, is a pop-up menu displayed in response to an input of the coordinates from said input means. 4 A cameran control user interface annaratusi for rcontrolinn a camera 1- situated in a remote place, comprising: a display device an input device which inputs coordinates on an image displayed on said display device; a storage device which stores a panorama image which covers a range that is photographable with the camera; and 27 a control device which reads out said panorama image from said storage device causes said display device to display the readout panorama image, sends, to the camera, values of the coordinates which have been input from said input device and causes said display device to display a photographed image received from the camera. The camera control user interface apparatus according to claim 4, wherein said control device causes said display device to display a menu a .o oto select a magnification of the camera, in a state in which said menu is superimposed on said panorama image, and said control device sends, to the S 5 camera, the magnification which has been input from said input device

6. The camera control user interface apparatus according to claim 4, wherein said control device divides a screen of said display device into a plurality of areas and displays said panorama image and the photographed image received from the camera in the divided areas of said screen.

7. The camera control user interface apparatus according to claim 4, wherein: said storage device further stores photographing target information concerning a photographing target, in association with values of coordinates on said panorama image; in addition to the magnification, a display of the photographing target informatinn is selectable nn said menu disnlayoed r l n soirl rliriniay Ailice IN\ t\ said control device and when the display of the photographing target information is selected on said menu, said control device reads out from said storage device the photographing target information associated with the values of the coordinates which have been input from said input device and causes said display device to display the readout photographing target information.

8. The camera control user interface apparatus according to claim 4, wherein said menu, displayed on said display device by said control device is a pop-up menu displayed in response to an input of coordinates on said panorama image from said input device

9. A camera control method for controlling a camera situated in a remote place, comprising: displaying a panorama image which covers a range that is photographable with the camera; 5 sending values of coordinates of a point designated on said panorama image to the camera; and 9 receiving a photographed image from the camera and displaying the •photographed image as received. 99• The camera control method according to claim 9, further comprising: displaying a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; ~wherein the sending of the values of the coordinates of the point designated 9 S* 5 on said panorama image further comprises sending the magnification selected on said menu to the camera.

11. The camera control method according to claim 9, wherein said panorama image and said photographed image are displayed in a plurality of areas into which q -innIe dispnl screan han hoon rlivirl-,,

12. The camera control method according to claim 10, wherein: in addition to the magnification, a display of photographing target information concerning a photographing target is selectable on the displayed menu; and said camera control method further comprises storing the photographing target information concerning the photographing target, in association with values of coordinates on said panorama image as displayed, and reading out the stored photographing target information associated with values of coordinates of the point designated on said panorama image, when the display of the photographing target information is selected on said menu.

13. The camera control method according to claim 10, wherein the displayed menu is a pop-up menu displayed in response to the designation of the point on said panorama image.

14. A computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer to S. work as a camera control user interface apparatus used to control a camera situated in a remote place, the computer readable program comprising instructions for causing the computer to serve as: means for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; S means for sending values of input coordinates to the camera; and means for displaying, in another predetermined area on the display screen, a 10 photographed image received from the camera. A computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer to work as a camera control user interface apparatus used to control a camera situated i n a remnlo +p'c ek rtr m ntr m'rl'klei ror comrpr;ioing ,'ILUOCLIGI I I A I I I IIL% iJI«^L LI 1, I IF^UI. I I Vi_ ULLJ' I WJI l I I Fll lJ I 1 0 Id instructions for causing the computer to serve as: means for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; means for displaying a menu to select a magnification of the camera, in a state wherein said menu is superimposed on said panorama image; means for sending values of input coordinates and the magnification to the camera; and means for displaying, in another predetermined area on the display screen, a photographed image received from the camera.

16. A computer program product, including a computer usable medium having a computer readable program embodied therein for causing a computer, which stores photographing target information concerning a photographing target *in association with values of coordinates on a panorama image, to work as a 5 camera control user interface apparatus used to control a camera situated in a a. t9 remote place, the panorama image covering a range that is photographable with S .i the camera, and the computer readable program comprising instructions for causing the computer to serve as: means for displaying said panorama image on a display screen; 9 means for displaying a menu to select one of a magnification of the camera and a display of information, in a state in which said menu is superimposed on 9999 said panorama image; :•'"°means for sending values of input coordinates and the magnification to the .i camera when the magnification is selected on said menu; means for sending the values of the input coordinates and a predetermined magnification to the camera, reading the photographing target information associated with the values of the input coordinates, and displaying the readout jjolgy aplinigy taiye InfuuIaIu, u L Iue uisply scleen, when the disp U ay Ut information is selected on said menu; and means for displaying a photographed image received from the camera.

17. A carrier wave including a program signal embedded therein for causing a computer to work as a camera control user interface apparatus used to control a camera situated in a remote place, the program signal representing a program which comprises: 31 a segment for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; a segment for sending values of input coordinates to the camera; and a segment for displaying, in another predetermined area on the display screen, a photographed image received from the camera.

18. A program signal embedded in a carrier wave, for causing a computer to work as a camera control user interface apparatus used to control a camera ;i'oo• situated in a remote place, the program signal representing a program which comprises: a segment for displaying, in a predetermined area on a display screen, a panorama image which covers a range that is photographable with the camera; o* *a segment for displaying a menu to select a magnification of the camera, in a state in which said menu is superimposed on said panorama image; a segment for sending values of input coordinates and the magnification to the camera; and a segment for displaying, in another predetermined area on the display t screen, a photographed image received from the camera.

19. A program signal embedded in a carrier wave, for causing a computer, which stores photographing target information concerning a photographing target in association with values of coordinates on a panorama image, to work as a amra nntrl user interfcr annarat'c 'ieri to crntroyl a cra i+ uae ;n rr vv ww ILl l %W I 1 -1 JILUL U II I remote place, the panorama image covering a range that is photographable with the camera, and the program signal representing a program which comprises: a segment for displaying said panorama image on a display screen; a segment for displaying a menu to select one of a magnification of the camera and a display of information, in a state in which said menu is superimposed on said panorama image; -32- a segment for sending values of input coordinates and the magnification to the camera when the magnification is selected on said menu; a segment for sending the values of the input coordinates and a predetermined magnification to the camera, reading the photographing target information associated with s the values of the input coordinates, and displaying the readout photographing target information on the display screen, when the display of information is selected on said menu; and a segment for displaying a photographed image received from the camera on the display screen. S

20. A camera control user interface apparatus substantially as herein S. S S•described with reference to any one of the embodiments as that embodiment is illustrated in the drawings.

21. A camera control method substantially as herein described with reference to any one of the embodiments as that embodiment is illustrated in the drawings. °o• S

22. A computer program product, including a computer usable medium o 20 having a computer readable program embodied therein for causing a computer to work as a camera control user interface apparatus, said computer program product being substantially as herein described with reference to any one of the embodiments as that embodiment is illustrated in the drawings.

23. A carrier wave including a program signal embedded therein for causing a computer to work as a camera control user interface apparatus, said carrier wave being substantially as herein described with reference to any one of the embodiments as that embodiment is illustrated in the drawings.

24. A program signal embedded in a carrier wave, for causing a computer to work as a camera control user interface apparatus, said program signal being substantially as herein described with reference to any one of the embodiments as that embodiment is illustrated in the drawings. [R:\LIBL]00163.doc:BFD -33 A program signal embedded in a carrier wave, for causing a computer, which stores photographing target information concerning a photographing target in association with values of coordinates on a panorama image, to work as a camera control user interface apparatus, said program signal being substantially as herein described with reference to any one of the embodiments as that embodiment is illustrated in the drawings. DATED this Twenty-sixth Day of May, 1999 NEC Corporation 10 Patent Attorneys for the Applicant SPRUSON FERGUSON 0* 00 0* 0o S..o p. S S. i [R:\LIBL]00163.doc:BFD