JP2017017446A - Information processing device and control method thereof, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable camera control using a preview frame by displaying the preview frame not only on a rectangular omnidirectional image but also on a circular omnidirectional image.SOLUTION: An information processing device includes acquisition means for acquiring an omnidirectional image photographed by a camera device, display processing means for displaying the omnidirectional image on a display screen of display means, and control means for switching a preview frame in the omnidirectional image and an image corresponding to the preview frame according to the detection of a user operation to the preview frame in the omnidirectional image displayed on the display screen. The preview frame displayed in the omnidirectional image is acquired by projecting a visual field range displayed on a spherical model surrounding the camera device to the omnidirectional image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing apparatus, a control method thereof, and a program, and more particularly to a preview frame display method using an omnidirectional image photographed by an imaging apparatus.

  Some network cameras that have become widespread in recent years can capture remote images via a network, display the images on a monitor, and remotely control cameras such as pan, tilt, and zoom. There is also a network camera that is equipped with a fisheye lens and can be cut out and displayed after converting the coordinate system.

  A GUI (Graphical User Interface) for performing these operations includes a GUI for specifying pan, tilt, zoom, and cutout position using a rectangular or circular omnidirectional image. In the GUI using a rectangular omnidirectional image, a rectangular preview frame is displayed, the center of the preview frame is moved by clicking the mouse, the size of the preview frame is changed by dragging, and the preview frame is redrawn. By doing so, there is one that can perform pan, tilt, and zoom control. Some GUIs using circular omnidirectional images can change pan and tilt control and cutout positions by clicking an arbitrary position on the omnidirectional image with a mouse.

  Patent Document 1 discloses that pan, tilt, and zoom control is performed by operating a preview frame on a rectangular captured image that indicates a shootable range by changing pan, tilt, and zoom. Yes. Patent Document 2 discloses a GUI for camera control by clicking a circular panoramic image.

Japanese Patent No. 3315554 Japanese Patent No. 5464290

  In the technique of Patent Document 1, a captured image is displayed on a rectangle indicating a shootable range. For this reason, there is a possibility that the distortion of the subject increases when it is desired to look directly below the camera installed on the ceiling. In the technique of Patent Document 2, there is a demand for camera control by an intuitive preview frame operation as disclosed in Patent Document 1 using a circular panoramic image.

  In addition, in both rectangular and circular omnidirectional images, if the corresponding portion is accurately surrounded by a preview frame on the omnidirectional image of the currently visible video, a complicated shape surrounded by a curve is obtained. For this reason, it is difficult to smoothly change the position and size of the preview frame by dragging the mouse.

  An object of the present invention is to display a preview frame on a circular panoramic image as well as on a rectangular omnidirectional image, and to enable camera control using the preview frame.

  In order to solve the above problems, the present invention has the following configuration. That is, an information processing apparatus connected to a camera device via a network, an acquisition unit that acquires an omnidirectional image captured by the camera device, and a display unit that displays the omnidirectional image acquired by the acquisition unit Display processing means to be displayed on the screen, and the detection of the preview frame in the omnidirectional image and the image corresponding to the preview frame in response to detection of a user operation on the preview frame in the omnidirectional image displayed on the display screen A preview frame displayed in the omnidirectional image is obtained by projecting a visual field range represented by a spherical model surrounding the camera device onto the omnidirectional image. Features.

  According to the present invention, it is possible to display a preview frame on a circular omnidirectional image and improve user convenience in camera control using the preview frame.

The figure which shows a system structural example. The figure which shows the hardware structural example of a viewer terminal. The figure which shows the example of installation of a camera apparatus. The figure which shows the structural example of a viewer screen. The figure showing the correspondence of a spherical model, a circular omnidirectional image, and a preview frame. The figure showing the difference of the preview frame by the inscribed and circumscribed to a spherical model. The figure for demonstrating the change of the screen at the time of the click outside a preview frame. The figure for demonstrating the change of the screen at the time of dragging outside a preview frame. The flowchart of the process at the time of a click and dragging out of a preview frame. The figure for demonstrating the change of the screen at the time of dragging in a preview frame. The flowchart of the process at the time of dragging in a preview frame. The figure for demonstrating the change of the screen at the time of dragging on a preview frame. The flowchart of the process at the time of dragging on a preview frame. The figure showing the correspondence of a spherical model, a rectangular omnidirectional image, and a preview frame. The figure for demonstrating the change of the screen at the time of the preview frame operation on a circular omnidirectional image. The figure for demonstrating the change of the screen at the time of the preview frame operation on a rectangular omnidirectional image. The figure for demonstrating the time of the automatic switching of a rectangular omnidirectional image and a circular omnidirectional image. The flowchart of the process at the time of automatic switching of a rectangular omnidirectional image and a circular omnidirectional image.

  Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.

<First embodiment>
In the first embodiment, an embodiment in which panning, tilting, zooming control, and image cutout control can be performed using a preview frame on a circular omnidirectional image will be described.

[System configuration]
FIG. 1 is a diagram illustrating a system configuration example according to the present embodiment. The system according to the present embodiment includes a camera device 1, a camera device 2, and a viewer terminal 3, and each device is connected via a network 4.

  The camera device 1 includes a zoom lens 101, an imaging unit 102, an image compression unit 103, a movable camera platform 104, a camera control unit 105, a communication unit 106, and an omnidirectional image storage unit 107. Further, it is assumed that the camera device 1 can perform pan, tilt, and zoom control according to various camera control commands.

  When a camera control command of any one of pan, tilt, and zoom is transmitted from the viewer terminal 3 via the network 4, the camera apparatus 1 receives the camera control command by the communication unit 106. Further, the communication unit 106 passes the received camera control command to the camera control unit 105. The camera control unit 105 executes camera control by sending a pan / tilt control signal to the movable camera platform 104 and a zoom control signal to the zoom lens 101 in response to the camera control command. In parallel with this, the image compression unit 103 compresses the image received from the imaging unit 102 into image data that can be distributed over the network. Finally, the camera control unit 105 passes the camera control result to the communication unit 106, and the image compression unit 103 passes the compressed image data to the communication unit 106 as main image data. The communication unit 106 transmits the acquired information to the external device.

  Although not shown in FIG. 1, an omnidirectional image obtained by synthesizing and compressing an image taken while panning and tilting the camera device 1 with an omnidirectional image creation terminal (not shown). Is stored in the omnidirectional image storage unit 107.

  The camera device 2 includes a fisheye lens 201, an imaging unit 202, an image compression unit 203, a dewarp processing unit 204, a camera control unit 205, and a communication unit 206. Further, it is assumed that the camera device 2 can perform virtual pan, tilt, and zoom control by image cutout processing.

  When a camera control command for panning, tilting, or zooming is transmitted from the viewer terminal 3 via the network 4, the camera apparatus 2 receives the camera control command by the communication unit 206. Further, the communication unit 206 passes the received camera control command to the camera control unit 205. The camera control unit 105 generates an image cutout parameter corresponding to the camera control command and sends it to the dewarp processing unit 204. Next, the dewarping processing unit 204 performs dewarping processing (coordinate system conversion and clipping processing) for reducing distortion of the omnidirectional image acquired from the imaging unit 202 in accordance with the acquired image clipping parameter.

  Finally, the image compression unit 203 compresses the dewarped image into image data that can be distributed over the network, and passes the compressed image data to the communication unit 206 as main image data. In parallel with this, the imaging unit 202 directly passes the omnidirectional image to the image compression unit 203, and the image compression unit 203 compresses the omnidirectional image into image data that can be distributed over the network to obtain omnidirectional image data. The data is passed to the communication unit 206. The communication unit 206 transmits the acquired image data to the external device.

  These camera devices themselves may use the same configuration as a conventional camera system, but the configuration of the viewer terminal 3 connected to the camera device 1 and the camera device 2 via the network 4 is a feature of the present invention. Yes, different from the conventional camera system. The hardware configuration of the viewer terminal 3 will be described later with reference to FIG. 2, and an example of the software configuration is shown here. The viewer terminal 3 includes a main image display unit 301, an omnidirectional image display unit 302, an image decompression unit 303, a preview frame calculation unit 304, an operation input unit 305, an operation command generation / interpretation unit 306, and a communication unit 307. Is done.

  The operation input unit 305 detects a user operation input via a mouse, a touch panel, or the like. When the operation input unit 305 detects an operation related to camera control from the user, the operation command generation / interpretation unit 306 and the preview frame calculation unit 304 determine the position and size of the preview frame. In addition, a camera control command corresponding to the position and size of the preview frame is generated, and the generated camera control command is transmitted to the camera device 1 or the camera device 2 via the communication unit 307. The camera control command generated here is generated according to the specifications of the camera devices 1 and 2. The communication unit 307 receives main image data and omnidirectional image data from each camera device together with the camera control result. These image data are expanded into a main image and an omnidirectional image that can be displayed by the image expansion unit 303, and passed to the main image display unit 301 and the omnidirectional image display unit 302.

[Hardware configuration]
FIG. 2 is a diagram illustrating a hardware configuration example of the viewer terminal 3, which is an information processing apparatus. The viewer terminal 3 can be configured by hardware of a general information processing apparatus. The CPU 311 executes programs stored in a program ROM in the ROM 313, programs such as an OS (Operating System) and applications loaded from the hard disk 320 to the RAM 312.

  That is, the CPU 311 executes a program stored in a readable storage medium, thereby functioning as each part of the viewer terminal 3 in FIG. A RAM 312 is a main memory of the CPU 311 and functions as a work area. The keyboard controller 314 controls operation inputs from a keyboard 318 and a pointing device (not shown) (mouse, touch pad, touch panel, trackball, etc.). The display controller 315 controls display on the display 319. The disk controller 316 controls data access to the hard disk (HD) 320 that stores various data. A network controller (NC) 317 is connected to the network 4 and executes communication control processing with the camera apparatuses 1 and 2 connected to the network 4.

[Example of arrangement and image]
FIG. 3 is a diagram illustrating an example of installation of a camera. FIG. 4 is a diagram showing an outline of a display screen displayed on the viewer terminal 3.

  As shown in FIG. 3, the camera device 1 and the camera device 2 are installed on the ceiling in a certain space, and a person passing underneath is photographed. The camera device 1 and the camera device 2 are connected to the viewer terminal 3 via the network 4.

  In the state of FIG. 3, as shown in FIG. 4, the main image 400 and the circular omnidirectional image 510 are displayed on the viewer screen on the viewer terminal 3. A preview frame 511 and a mouse cursor 520 are displayed on the omnidirectional image 510, and the mouse cursor 520 moves when the user operates the mouse, and the preview frame 511 can be moved or changed in size accordingly. The main image 400 is an image corresponding to the preview frame 511.

  Here, it is assumed that the configuration of the viewer screen is common to both the camera device 1 and the camera device 2. A difference between the viewer screens of the camera device 1 and the camera device 2 is that, in the camera device 1, a circular omnidirectional image 510 is a still image obtained by joining a plurality of past images taken by panning and tilting. On the other hand, in the camera apparatus 2, a circular omnidirectional image 510 is a live moving image that is captured by the fisheye lens 201 and compressed without being dewarped. A circular omnidirectional image is also called a fisheye image.

[Image correspondence]
FIG. 5 is a diagram illustrating a correspondence relationship between the spherical model, the circular omnidirectional image, and the preview frame for calculating the preview frame according to the present embodiment.

  The sphere model 500 for calculating the preview frame is shown as a sphere centered on the viewpoint of the camera device. In the case of the camera device installed on the ceiling as shown in FIG. 3, the lower half of the sphere corresponds to the shootable range. . A frame 501 is inscribed or circumscribed on a curved surface within the visual field range of the sphere, and indicates a portion of the visual field range in the plane 502 as a plane region orthogonal to the line-of-sight direction. Here, the omnidirectional image 510 is obtained by projecting an image of a scene projected on the lower half of the sphere onto a plane 503 orthogonal to the vertical axis of the sphere. The preview frame 511 is a rectangle in which each vertex of the frame 501 is projected onto the plane 503 and connected by a straight line.

  FIG. 6 is a diagram for explaining the difference in preview frame between inscribed and circumscribed to the sphere model 500. FIG. 6A is a view of the sphere model 500 viewed from a point on the plane 502. The frame 501 corresponds to the surface 505 when inscribed in the curved surface 506 within the visual field range, and corresponds to the surface 504 when circumscribed to the curved surface 506 within the visual field range. When the curved surface 506 is projected onto the flat surface 503, the edge portion becomes the preview frame 516 shown in FIGS. 6B and 6C. The reference point 508 is a point obtained by projecting the intersecting point of the curved surface 506 and the straight line representing the line-of-sight direction onto the plane 503. The preview frame 515 in FIG. 6B is a rectangle that projects four vertices of the surface 505 on the plane 503 and connects them with straight lines. The preview frame 514 in FIG. 6C is a rectangle that projects four vertices of the surface 505 on the plane 503 and connects them with straight lines.

  Here, since the four vertices of the curved surface 506 match the four vertices of the surface 505, the four vertices of the preview frame 515 and the preview frame 516 also match. Further, since the center point of the curved surface 506 and the center point of the surface 504 coincide with each other at the intersection with the straight line representing the line-of-sight direction, the center point of the preview frame 514 and the reference point 508 also coincide. If both of the preview frames 515 and 514 are displayed on the screen at the same time, it is difficult for the user to visually recognize them, so either one is selected and displayed as a preview frame 511 on the viewer screen. In the present embodiment, the preview frame 516 is not displayed on the viewer screen.

  When the preview frame 515 is selected to be displayed, it is displayed closer to the center of the circular omnidirectional image than the preview frame 516 corresponding to the actually visible range, and the preview frame does not protrude from the omnidirectional image. There is. However, in this case, a deviation occurs between the reference point 508 and the center point of the preview frame 515. On the other hand, when the preview frame 514 is selected to be displayed, the preview frame 514 is displayed outward from the center of the circular omnidirectional image than the preview frame 516 corresponding to the actually visible range, and the preview frame is displayed on the omnidirectional image. May protrude. However, in this case, there is no deviation between the reference point 508 and the center point of the preview frame 514. Depending on these features, the user can choose which to display.

[Processing flow]
Hereinafter, display processing according to the present embodiment will be described with reference to FIGS. Here, the screen transition when any position with respect to the preview frame is designated will be described together with a display example.

  In the present embodiment, an example in which a mouse is used as a device for performing user input to the operation input unit 305 will be described. In this mouse operation, the left button of the mouse is used for the main operation, and the right button is used for the sub operation. However, the present invention is not limited to this, and in the case of a mouse having the reverse setting, the right button and the left button are reversed in the mouse operation in the following processing flow. Unless otherwise specified, the following mouse up and mouse down operations refer to operations using the left button. Note that dragging refers to moving the mouse cursor while the mouse is down.

  9, 11, 13, and 18 are realized by the CPU of the viewer terminal 3 executing a program stored in a ROM or the like.

(Operation when the outside of the preview frame is specified)
FIG. 7 is a diagram for explaining the transition of display when an area outside the preview frame is clicked by a user operation. FIG. 8 is a diagram for explaining the transition of display when the region outside the preview frame is dragged. FIG. 9 is a flowchart showing processing when the operation according to FIG. 7 is performed.

  Assume that the preview frame 511 is displayed on the omnidirectional image 510 at the time when the processing flow shown in FIG. 9 is started, as shown in FIG.

  In S101, the viewer terminal 3 detects whether or not the mouse is down. If the mouse is not down (NO in S101), the process flow ends. When the mouse is down (YES in S101), the viewer terminal 3 determines that the position where the mouse is down (hereinafter referred to as the mouse down position) is outside the currently displayed preview frame 511, and It is determined whether the image is on the omnidirectional image 510. If the above condition is not satisfied (NO in S102), the process flow ends. If the above condition is satisfied (YES in S102), in S103, the viewer terminal 3 displays another preview frame 512 with the mouse-down position as the reference point 508. As shown in FIG. 7B, the preview frame 512 is displayed so that, for example, the line type and the line color are different from each other so as to be distinguishable from the preview frame 511. At this time, the preview frame calculation unit 304 calculates the size of the preview frame 512 so that the size of the curved surface 506 (that is, the zoom position or the angle of view) corresponding to the preview frames 511 and 512 is maintained. To do. For example, in the circular omnidirectional image 510, the preview frame 512 is displayed larger even if it is the same size as the mouse is moved down to a position closer to the center of the circle. This is due to the relationship when projected onto the plane 503 as shown in FIG. Also, the preview frame 512 is rotated from the preview frame 511 in accordance with the positional relationship between the center position of the omnidirectional image 510 and the reference point 508.

  In S104, the viewer terminal 3 detects whether or not dragging has been performed. If dragging has not been performed (NO in S104), the process proceeds to S106. When dragging has been performed (YES in S104), in S105, the viewer terminal 3 determines the preview frame 512 according to the mouse down position and the current position of the mouse cursor 520 moved by the dragging. Redraw. At this time, the preview frame calculation unit 304 does not change the position of the reference point 508 as shown in FIG. 8B, and the position of the mouse cursor 520 is always on the preview frame 512 in accordance with the movement of the mouse cursor 520 by the user operation. Change the size so that Thereafter, the process proceeds to S106.

  In S106, the viewer terminal 3 detects whether or not the mouse is up. If mouse up has not been performed (NO in S106), the process proceeds to S107, and if mouse up has been performed (YES in S106), the process proceeds to S109.

  In S107, the viewer terminal 3 determines whether or not a right click has been performed. If the right click has not been performed (NO in S107), the process returns to S104. If the right click has been performed (YES in S107), the process proceeds to S108. In S108, the viewer terminal 3 cancels the operation on the preview frame 512, and returns to the state where only the preview frame 511 is displayed (the state shown in FIG. 7A). In this case, no camera control command is generated and no transmission is performed. Then, this processing flow ends.

  In S109, the viewer terminal 3 deletes the preview frame 511 and changes to a state in which only the preview frame 512 is displayed. Along with this, the viewer terminal 3 handles the changed preview frame as the preview frame 511. As a specific operation, after the mouse is down, if the mouse is raised without dragging the mouse down (NO in S104 and YES in S106), the preview displayed in FIG. The frame 511 disappears. Then, a new preview frame is displayed as a preview frame 511 as shown in FIG.

  Thereafter, in S110, the viewer terminal 3 transmits a camera control command to the camera device 1 or the camera device 2 in order to perform camera control corresponding to the new preview frame 511. That is, the preview frame calculation unit 304 calculates camera control parameters corresponding to the new preview frame 511, and the operation command generation / interpretation unit 306 generates camera control commands corresponding to the calculated parameters. Then, the communication unit 307 transmits the generated camera control command to the camera device 1 or the camera device 2.

(Operation when the inside of the preview frame is specified)
FIG. 10 is a diagram for explaining the transition of the screen when the preview frame is dragged by a user operation. FIG. 11 is a flowchart illustrating processing when the operation according to FIG. 10 is performed.

  In S201, the viewer terminal 3 detects whether or not the mouse is down. If the mouse is not down (NO in S201), the process flow ends. If the mouse is down (YES in S201), the viewer terminal 3 determines whether or not the mouse-down position is within the preview frame 511 in S202. If it is not within the frame (NO in S202), the process flow ends. If within the frame (YES in S202), the viewer terminal 3 displays the preview frame 512 at the same position and the same size as the preview frame 511 in S203. At this time, the shape of the mouse cursor 520 changes as shown in FIG.

  Thereafter, in S204, the viewer terminal 3 detects whether or not dragging has been performed. If dragging has not been performed (NO in S204), the process proceeds to S206. When dragging has been performed (YES in S204), in S205, the viewer terminal 3 translates the preview frame 511 in accordance with the movement of the mouse cursor 520 as shown in FIG. The preview frame 512 is rotated according to the position and redrawn. At this time, the preview frame calculation unit 304 calculates the size of the preview frame 512 so that the size of the curved surface 506 (that is, the zoom position) that is the visual field range corresponding to the preview frames 511 and 512 is maintained. Thereafter, the process proceeds to S206.

  In S206, the viewer terminal 3 detects whether or not a mouse up has been performed. If mouse up has not been performed (NO in S206), the process proceeds to S207. If mouse up has been performed (YES in S206), the process proceeds to S209.

  In S207, the viewer terminal 3 determines whether or not a right click has been performed. If the right click has not been performed (NO in S207), the process returns to S204. If the right click has been performed (YES in S207), the process proceeds to S208. In S208, the viewer terminal 3 cancels the operation of the preview frame 512, and returns to the state where only the preview frame 511 is displayed (the state of FIG. 10A). In this case, no camera control command is generated and no transmission is performed. Then, this processing flow ends.

  In S209, the viewer terminal 3 deletes the preview frame 511 and changes to a state in which only the preview frame 512 is displayed. Along with this, the viewer terminal 3 handles the changed preview frame as the preview frame 511. As a specific example, it is displayed as shown in FIG.

  Thereafter, in S210, the viewer terminal 3 transmits a camera control command to the camera device 1 or the camera device 2 in order to perform camera control corresponding to the new preview frame 511. That is, the preview frame calculation unit 304 calculates camera control parameters corresponding to the new preview frame 511, and the operation command generation / interpretation unit 306 generates camera control commands corresponding to the calculated parameters. Then, the communication unit 307 transmits the generated camera control command to the camera device 1 or the camera device 2.

(Operation when dragging the preview frame)
FIG. 12 is a diagram for explaining the transition of the screen when the user drags on the preview frame. FIG. 13 is a flowchart illustrating processing when the operation according to FIG. 12 is performed.

  In S301, the viewer terminal 3 detects whether or not the mouse is down. If the mouse is not down (NO in S301), the process flow ends. If the mouse is down (YES in S301), the viewer terminal 3 determines whether or not the mouse-down position is on the preview frame 511 in S302. If it is not on the frame (NO in S302), the process flow ends. If it is on the frame (YES in S302), the viewer terminal 3 displays the preview frame 512 at the same position and the same size as the preview frame 511 in S303. For example, when the mouse is moved down on the frame, the preview frame 511 and the preview frame 512 are superimposed and displayed as shown in FIG. 12A, and the shape of the mouse cursor 520 changes at this time.

  Thereafter, in S304, the viewer terminal 3 detects whether or not dragging has been performed. If dragging has not been performed (NO in S304), the process proceeds to S306. When dragging has been performed (YES in S304), in S305, the viewer terminal 3 enlarges or reduces the preview frame 512 in accordance with the movement of the mouse cursor 520. At this time, the preview frame calculation unit 304 moves the mouse cursor 520 by the user operation so that the mouse cursor 520 is always on the preview frame 512 without changing the position of the reference point 508 as shown in FIG. In addition, the size of the preview frame 512 is adjusted. Thereafter, the process proceeds to S306. Since the processing after S306 is the same as the processing after S206 of FIG. 10, the description thereof is omitted here.

  In the example using FIGS. 7 to 13, the operation based on the mouse operation is described. However, in the configuration using the touch panel operation instead of the mouse operation, for example, tapping the mouse down, untapping the mouse up, swiping the drag, and two-point tap with a finger different from the finger tapping the right click This embodiment can be applied by replacing.

<Second Embodiment>
Next, while switching between rectangular and circular omnidirectional images, pan / tilt / zoom control and image cut-out control can be performed using the preview frame in both, and a preview frame that accurately represents the field of view range when not in operation. A second embodiment to be displayed will be described.

  The processing at the time of the preview frame operation when the system configuration, the installation image, and the omnidirectional image are fixed in a circle or rectangle is the same as the contents shown in the first embodiment, so the explanation is omitted, and only the difference is explained. To do.

  FIG. 14 is a diagram illustrating a correspondence relationship between a sphere model for calculating a preview frame, a rectangular omnidirectional image, and a preview frame.

  The spherical model 500 and the frame 501 for calculating the preview frame are the same as those in FIG. 3 shown in the first embodiment, but the present embodiment is different in that the projection surface is a cylinder 530 surrounding the spherical model. . A frame 532 on the cylinder 530 is a rectangle having the same width as the curved surface within the spherical visual field range. The rectangular omnidirectional image 540 and the preview frame 542 correspond to a cylinder 530 and a frame 532 that are developed in a plane and compressed in the height direction. This corresponds to a projection method called equirectangular projection, and the line 531 with equal tilt angles is wider on the cylinder 530 as it goes down, whereas the line 541 is on the rectangular omnidirectional image 540. It becomes equal intervals like.

  FIG. 15 is a diagram for explaining the transition of the screen when dragging outside the preview frame on the circular omnidirectional image of the present embodiment.

  The processing flow is the same as that in the flowchart of FIG. 9 described in the first embodiment, except that the preview frame 511 is not rectangular but has a shape corresponding to the preview frame 516 of FIG. Although the preview frame 512 that is the target of the mouse operation is rectangular, the preview frame 511 when not operated and after completion of the operation is a frame surrounded by a curve so as to accurately represent the visual field range.

  FIG. 16 is a diagram for explaining the transition of the screen when dragging within the preview frame on the rectangular omnidirectional image of the present embodiment.

  The process flow is the same as the flowchart of FIG. 11 described in the first embodiment. However, like the preview frame on the circular omnidirectional image shown in FIG. 15, the preview frame 542 that is the object of the mouse operation is rectangular, but the preview frame 543 when the operation is not performed and after the operation is completed has a field of view range. It becomes a frame surrounded by a curve so that it can be expressed accurately. By moving the preview frame 543 shown in FIG. 16A to the lower portion of the screen, the preview frame 543 becomes wide as shown in FIG. 16D due to the characteristics of the configuration shown in FIG.

  FIG. 17 shows a screen example in the case of automatically switching between a rectangular omnidirectional image and a circular omnidirectional image. FIG. 18 is a flowchart showing the processing according to FIG.

  The omnidirectional image display unit 302 displays a viewer screen 600. As shown in FIG. 17A, the rectangular omnidirectional image is displayed when the rectangular omnidirectional tab 610 is selected. As shown in FIG. 17B, the circular omnidirectional image is displayed when the circular omnidirectional tab 620 is selected. A pull-down list 630 can be used to select whether to automatically switch tabs. In each omnidirectional image, a dividing line 550 indicating the same tilt angle is displayed. When “automatic switching” is selected from the pull-down list 630, the tab is automatically switched depending on whether the preview frame crosses the dividing line 550 or not. The dividing line 550 indicates a range in which the distance between the line-of-sight direction or the visual field range for the image corresponding to the preview frame and the center of the omnidirectional image is within a predetermined range. The predetermined range here is defined in advance in consideration of image distortion.

  When “automatic selection” is not selected, even if an instruction such as movement to the preview frame is given, automatic display switching is not performed. In addition, it is assumed that the user can switch to any display by selecting a tab at an arbitrary timing.

  The flowchart in FIG. 18 shows processing when the preview frame is dragged. S401 and S402 are added from the flowchart of FIG. Here, with respect to overlapping processes, the same reference numerals are assigned and description thereof is omitted.

  When the mouse up is detected in S206 (YES in S206), in S401, the viewer terminal 3 determines whether or not the position of the reference point 508 of the moved preview frame exceeds the dividing line 550. For example, in the case of a rectangular omnidirectional image, it is determined that the position of the reference point 508 has been exceeded when it moves below the dividing line 550. In the case of a circular omnidirectional image, it is determined that the position of the reference point 508 has been exceeded when it has moved outside the dividing line 550. If exceeded (YES in S401), the process proceeds to S402. If not exceeded (NO in S401), the process proceeds to S209.

  In S402, the viewer terminal 3 switches to the other tab, and the subsequent processing is performed in the selected display form. Specifically, when a rectangular omnidirectional image is displayed, the display is switched to a circular omnidirectional image. The same applies to the reverse case. Then, it progresses to S210. Note that whether or not the dividing line 550 is crossed may be determined by determining whether or not any vertex of the preview frame or the entire preview frame exceeds the dividing line 550.

  By switching the display, it is possible to prevent the preview frame from being displayed in a narrow shape as shown by the preview frame 511 in FIG. 15A on a circular omnidirectional image. Similarly, on a rectangular omnidirectional image, it is possible to avoid displaying a wide shape as shown by the preview frame 543 in FIG. As a result, the distortion of the image within the field of view does not become too large.

  As mentioned above, although one Embodiment of this invention was described, this invention is not limited to these Embodiment, A various deformation | transformation and change are possible within the range of the summary.

<Other embodiments>
The present invention is also realized by executing the following processing. That is, software (program) for realizing the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, etc.) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 1, 2 ... Camera apparatus, 3 ... Viewer apparatus, 301 ... Main image display part, 302 ... Omni-directional image display part, 303 ... Image expansion | extension part, 304 ... Preview frame calculation 305 ... Operation input unit 306 ... Operation command generation / interpretation unit 307 ... Communication unit

Claims (17)

An information processing device connected to a camera device via a network,
Acquisition means for acquiring an omnidirectional image captured by the camera device;
Display processing means for displaying the omnidirectional image acquired by the acquisition means on the display screen of the display means;
Control means for switching a preview frame in the omnidirectional image and an image corresponding to the preview frame in response to detection of a user operation on the preview frame in the omnidirectional image displayed on the display screen;
The information processing apparatus according to claim 1, wherein the preview frame displayed in the omnidirectional image is obtained by projecting a visual field range represented by a spherical model surrounding the camera apparatus onto the omnidirectional image. Generating means for generating a control command to the camera device in response to a user operation on the preview frame in the omnidirectional image displayed on the display screen;
A transmission unit that transmits the control command generated by the generation unit to the camera device;
The information processing apparatus according to claim 1, wherein the acquisition unit acquires a new image from the camera device controlled according to the control command transmitted by the transmission unit.   The information processing apparatus according to claim 2, wherein the control command is a command for control related to any one of pan, tilt, and zoom. The point projected on the omnidirectional image at the point where the line of sight intersects the spherical model is the reference point of the preview frame,
When the control unit is instructed by a user operation to a first position outside the preview frame, the control unit sets the first position as a new reference point of the preview frame, and the preview frame is the center of the omnidirectional image. Switching the display of the preview frame and the image corresponding to the preview frame so that the orientation is in accordance with the direction of the first position with respect to the position and is changed to a size that maintains the original angle of view. The information processing apparatus according to claim 1, wherein the information processing apparatus is characterized. The point projected on the omnidirectional image at the point where the line of sight intersects the spherical model is the reference point of the preview frame,
When the first position in the frame of the preview frame is instructed by a user operation, and the second position in the omnidirectional image different from the first position is instructed, The reference point of the preview frame is translated to the second position, and the preview frame is set to the center position of the omnidirectional image according to the distance and direction between the first position and the second position. The display of the preview frame and the image corresponding to the preview frame are switched so as to be rotated to a direction corresponding to the direction of the reference point and changed to a size that maintains the original angle of view. Item 4. The information processing device according to any one of Items 1 to 3. The point projected on the omnidirectional image at the point where the line of sight intersects the spherical model is the reference point of the preview frame,
When the first position outside the frame of the preview frame is instructed by a user operation and the second position in the omnidirectional image different from the first position is instructed by the user operation, the control means One position is set as a reference point of the new preview frame, the preview frame is oriented according to the direction of the first position with respect to the center position of the omnidirectional image, and the size of the preview frame is the first frame 4. The display of the preview frame and the image corresponding to the preview frame are switched so as to be changed according to the distance between the first position and the second position. The information processing apparatus according to item. The point projected on the omnidirectional image at the point where the line of sight intersects the spherical model is the reference point of the preview frame,
The control means, when a first position on the frame of the preview frame is instructed by a user operation, and when a second position in the omnidirectional image different from the first position is instructed, The size of the preview frame is changed in accordance with the distance between the first position and the second position while maintaining the reference point of the preview frame and the orientation of the preview frame with respect to the center position of the omnidirectional image. As described above, the information processing apparatus according to claim 1, wherein the display of the preview frame and an image corresponding to the preview frame are switched.   The display means displays the preview frame in a non-rectangular shape as a curved region of the visual field range represented by the spherical model while accepting a user operation on the preview frame. The information processing apparatus according to any one of claims 1 to 7.   While receiving the user operation on the preview frame, the display unit displays both the preview frame at the original position and the preview frame while receiving the user operation so that the user operation is completed. The information processing apparatus according to claim 8.   The information processing apparatus according to claim 1, wherein the user operation is performed by a mouse operation or a touch panel operation on the display screen. The display means can display an omnidirectional image by switching between a rectangular shape and a circular shape,
The display means displays a circular omnidirectional image when the distance between the line-of-sight direction or the visual field range for the image corresponding to the preview frame and the center of the circular omnidirectional image falls within a predetermined range, and the predetermined range 11. The information processing apparatus according to claim 1, wherein the information processing apparatus is switched to display a rectangular omnidirectional image when the image does not fit in the image. Means for accepting a setting as to whether or not to automatically switch and display any one of rectangular and circular omnidirectional images by the display means;
The information processing apparatus according to claim 11, wherein the display unit performs switching according to the setting. The preview frame is
While accepting a user operation, the field of view represented by the spherical model surrounding the camera device is displayed as a rectangle that is projected onto the omnidirectional image and connected by a straight line,
The information processing apparatus according to claim 1, wherein the information processing apparatus is displayed in a non-rectangular shape as a curved surface area of a visual field range indicated by the spherical model in response to completion of processing by a user operation. A method for controlling an information processing apparatus connected to a camera device via a network,
An acquisition step of acquiring an omnidirectional image captured by the camera device;
A display processing step of displaying the omnidirectional image acquired in the acquisition step on a display screen of a display means;
A control step of switching between a preview frame in the omnidirectional image and an image corresponding to the preview frame in response to a user operation on the preview frame in the omnidirectional image displayed on the display screen,
A control method for an information processing apparatus, wherein the preview frame displayed in the omnidirectional image is obtained by projecting a visual field range represented by a spherical model surrounding the camera device onto the omnidirectional image. . The preview frame is
While accepting a user operation, the field of view represented by the spherical model surrounding the camera device is displayed as a rectangle that is projected onto the omnidirectional image and connected by a straight line,
The information processing apparatus control method according to claim 14, wherein the information is displayed in a non-rectangular shape as a curved surface area of the visual field range indicated by the spherical model in response to completion of processing by a user operation. Computer
Acquisition means for acquiring an omnidirectional image captured by the camera device;
Display processing means for displaying the omnidirectional image acquired by the acquisition means on the display screen of the display means;
In response to a user operation on the preview frame in the omnidirectional image displayed on the display screen, function as a control unit that switches between the preview frame in the omnidirectional image and the image corresponding to the preview frame,
The program according to claim 1, wherein the preview frame displayed in the omnidirectional image is obtained by projecting a visual field range represented by a spherical model surrounding the camera device onto the omnidirectional image. The preview frame is
While accepting a user operation, the field of view represented by the spherical model surrounding the camera device is displayed as a rectangle that is projected onto the omnidirectional image and connected by a straight line,
17. The program according to claim 16, wherein the program is displayed in a non-rectangular shape as a curved surface area of the visual field range indicated by the spherical model in response to completion of processing by a user operation.