JP6543108B2 - INFORMATION PROCESSING APPARATUS, CONTROL METHOD THEREOF, AND PROGRAM - Google Patents

Description

  The present invention relates to an information processing apparatus, a control method thereof, and a program, and more particularly to a preview frame display method using an omnidirectional image captured by an imaging apparatus.

  2. Description of the Related Art Among network cameras that are becoming widespread in recent years, there are cameras that can remotely control a camera such as pan, tilt, and zoom while capturing a remote image via a network and displaying the image on a monitor. There is also a network camera that can carry a fisheye lens, transform the coordinate system, and cut out and display a part.

  Among GUIs (Graphical User Interfaces) for performing these operations, there are GUIs for specifying pan, tilt, zoom and cutout positions using rectangular or circular omnidirectional images. In a GUI using a rectangular omnidirectional image, a rectangular preview frame is displayed, the center of the preview frame is moved by clicking the mouse, the size of the preview frame is changed by dragging, and the preview frame is redrawn. There are some that can perform pan, tilt, and zoom control by doing this. Further, in a GUI using a circular omnidirectional image, there are some which can change pan / tilt control and a cutout position by clicking an arbitrary position on the omnidirectional image with a mouse.

  Patent Document 1 discloses that pan, tilt, and zoom control are performed by operating a preview frame on a rectangular captured image indicating a range capable of shooting by changing pan, tilt, and zoom with a mouse. There is. Further, Patent Document 2 discloses a GUI for camera control by clicking on a circular panoramic image.

Patent No. 3315554 gazette Patent No. 5464290 gazette

  In the technique of Patent Document 1, a captured image is displayed on a rectangle that indicates a range that can be captured. Therefore, when it is desired to look directly below the camera installed on the ceiling, distortion of the subject may be large. In the technique of Patent Document 2, there is a demand for camera control by operation of an intuitive preview frame as disclosed in Patent Document 1 which uses a circular panoramic image.

  Also, in either the rectangular or circular omnidirectional image, if the corresponding part is accurately surrounded by the preview frame on the omnidirectional image of the currently viewed image, it becomes a complicated shape surrounded by a curve. Therefore, it is difficult to change the smooth position and size of the preview frame by mouse dragging.

  An object of the present invention is to display a preview frame not only on a rectangular omnidirectional image but also on a circular panoramic image, and to enable camera control using it.

In order to solve the above-mentioned subject, the present invention has the following composition. That is, an information processing apparatus connected to a camera device via a network, the acquiring unit acquiring the omnidirectional image photographed by the camera device, and the display of the omnidirectional image acquired by the acquiring unit Display processing means for displaying on a screen, and a frame superimposed on an omnidirectional image displayed on the display screen, for determining the position of a partial image of a predetermined visual field range cut out from the omnidirectional image A frame superimposed on the omnidirectional image and control means for switching the partial image corresponding to the frame in response to detection of a user operation on the frame, the display processing means receiving the user operation In the meantime, the rectangular frame is displayed on the display screen, and while the user's operation on the frame is not received, the frame is based on the visual field range corresponding to the frame. At least partially Ru is displayed on the display screen in a shape composed of curved lines.

  According to the present invention, it is possible to display a preview frame on a circular omnidirectional image and to improve the user's convenience in camera control using it.

A figure showing an example of system configuration. The figure which shows the hardware structural example of a viewer terminal. The figure which shows the example of installation of a camera apparatus. The figure which shows the structural example of a viewer screen. The figure showing the correspondence of a spherical model, circular omnidirectional image, and a preview frame. The figure showing the difference of the preview frame by inscribed in a spherical model and circumscribed. The figure for demonstrating the change of the screen at the time of preview frame outside clicking. The figure for demonstrating the change of the screen at the time of dragging outside a preview frame. Flowchart of processing when clicking and dragging outside the preview frame. The figure for demonstrating the change of the screen at the time of the dragging in a preview frame. The flowchart of the processing at the time of dragging in the preview frame. The figure for demonstrating the change of the screen at the time of a drag on a preview frame. The flowchart of the processing at the time of dragging on the preview frame. The figure showing the correspondence of a spherical model, a rectangular omnidirectional image, and a preview frame. The figure for demonstrating the change of the screen at the time of the preview frame operation on a circular omnidirectional image. The figure for demonstrating the change of the screen at the time of the preview frame operation on a rectangular omnidirectional image. The figure for demonstrating the time of the automatic switching of a rectangular omnidirectional image and a circular omnidirectional image. The flowchart at the time of the process at the time of the automatic switching of a rectangular omnidirectional image and a circular omnidirectional image.

  Hereinafter, an embodiment of the present invention will be described in detail based on the drawings.

First Embodiment
In the first embodiment, an embodiment in which pan, tilt, zoom control, and image cutout control can be performed using a preview frame on a circular omnidirectional image will be described.

[System configuration]
FIG. 1 is a diagram showing an example of a system configuration according to the present embodiment. The system according to the present embodiment includes a camera device 1, a camera device 2, and a viewer terminal 3, and the devices are connected via a network 4.

  The camera device 1 includes a zoom lens 101, an imaging unit 102, an image compression unit 103, a movable camera platform 104, a camera control unit 105, a communication unit 106, and an omnidirectional image storage unit 107. Further, it is assumed that the camera device 1 can perform pan, tilt, and zoom control according to various camera control commands.

  When a pan, tilt, or zoom camera control command is transmitted from the viewer terminal 3 via the network 4, the camera device 1 receives the camera control command from the communication unit 106. Furthermore, the communication unit 106 passes the received camera control command to the camera control unit 105. The camera control unit 105 executes camera control by sending pan and tilt control signals to the movable camera platform 104 and a zoom control signal to the zoom lens 101 in response to a camera control command. At the same time, the image compression unit 103 compresses the image received from the imaging unit 102 into image data that can be distributed via network. Finally, the camera control unit 105 passes the camera control result to the communication unit 106, and the image compression unit 103 passes the compressed image data to the communication unit 106 as main image data. The communication unit 106 transmits the acquired information to an external device.

  Although not shown in FIG. 1, an omnidirectional image obtained by combining and compressing images taken while pan and tilt controlling the camera device 1 by an omnidirectional image generation terminal (not shown) The data of is stored in the omnidirectional image storage unit 107.

  The camera device 2 includes a fisheye lens 201, an imaging unit 202, an image compression unit 203, a dewarping unit 204, a camera control unit 205, and a communication unit 206. In addition, it is assumed that the camera device 2 can perform virtual pan, tilt, and zoom control by image cutout processing.

  When a pan, tilt, or zoom camera control command is transmitted from the viewer terminal 3 via the network 4, the camera device 2 receives the camera control command from the communication unit 206. Furthermore, the communication unit 206 passes the received camera control command to the camera control unit 205. The camera control unit 105 generates an image clipping parameter in response to the camera control command, and sends the image clipping parameter to the dewarping processing unit 204. Next, the dewarping unit 204 performs a dewarping process (coordinate system conversion and cutout process) for reducing distortion of the omnidirectional image acquired from the imaging unit 202 according to the acquired image cutout parameter.

  Finally, the image compression unit 203 compresses the image after the dewarping processing into image data that can be distributed over a network, and passes it to the communication unit 206 as main image data. In addition, in parallel with that, the imaging unit 202 passes the omnidirectional image directly to the image compression unit 203, and the image compression unit 203 compresses the omnidirectional image into image data that can be distributed over a network, and becomes omnidirectional image data. It passes to the communication unit 206. The communication unit 206 transmits each acquired image data to an external device.

  Although these camera devices themselves may use the same configuration as a conventional camera system, the configuration of the viewer device 3 connected to the camera device 1 and the camera device 2 via the network 4 is a feature of the present invention. Yes, different from conventional camera systems. The hardware configuration of the viewer terminal 3 will be described later with reference to FIG. 2, and an example of the software configuration is shown here. The viewer terminal 3 includes a main image display unit 301, an omnidirectional image display unit 302, an image decompression unit 303, a preview frame calculation unit 304, an operation input unit 305, an operation command generation / interpretation unit 306, and a communication unit 307. Be done.

  The operation input unit 305 detects a user operation input via a mouse, a touch panel, or the like. When the operation input unit 305 detects an operation related to camera control from the user, the operation command generation / interpretation unit 306 and the preview frame calculation unit 304 determine the position and size of the preview frame. In addition, a camera control command corresponding to the position and size of the preview frame is generated, and the generated camera control command is transmitted to the camera device 1 or the camera device 2 through the communication unit 307. The camera control command generated here is generated in accordance with the specifications of each of the camera devices 1 and 2. The communication unit 307 receives main image data and omnidirectional image data from each camera device together with the camera control result. These image data are expanded into a main image that can be displayed by the image expansion unit 303 and an omnidirectional image, and passed to the main image display unit 301 and the omnidirectional image display unit 302.

[Hardware configuration]
FIG. 2 is a diagram showing an example of a hardware configuration of the viewer terminal 3 which is an information processing apparatus. The viewer terminal 3 can be configured by hardware of a general information processing apparatus. The CPU 311 executes a program stored in the program ROM in the ROM 313 or a program such as an operating system (OS) or an application loaded from the hard disk 320 into the RAM 312.

  That is, the CPU 311 functions as each part of the viewer terminal 3 of FIG. 1 by executing a program stored in a readable storage medium, and executes the processing of each flowchart to be described later. A RAM 312 is a main memory of the CPU 311 and functions as a work area or the like. The keyboard controller 314 controls operation input from the keyboard 318 and a pointing device (not shown) (mouse, touch pad, touch panel, track ball, etc.). The display controller 315 controls the display of the display 319. A disk controller 316 controls data access to a hard disk (HD) 320 which stores various data. A network controller (NC) 317 is connected to the network 4 and executes communication control processing with the camera devices 1 and 2 connected to the network 4.

[Example of arrangement and image]
FIG. 3 is a diagram showing an example of installation of a camera. FIG. 4 is a diagram showing an outline of a display screen displayed on the viewer terminal 3.

  As shown in FIG. 3, the camera device 1 and the camera device 2 are installed on a ceiling in a certain space, and a person passing below is photographed. The camera device 1 and the camera device 2 are connected to the viewer terminal 3 via the network 4.

  In the state of FIG. 3, as shown in FIG. 4, the main image 400 and the circular omnidirectional image 510 are displayed on the viewer screen on the viewer terminal 3. A preview frame 511 and a mouse cursor 520 are displayed on the omnidirectional image 510, and when the user operates the mouse, the mouse cursor 520 moves, and the preview frame 511 can be moved or changed in size accordingly. The main image 400 is an image corresponding to the preview frame 511.

  Here, it is assumed that the configuration of the viewer screen is common to both the camera device 1 and the camera device 2. A different point in the viewer screens of the camera device 1 and the camera device 2 is that, in the camera device 1, the circular omnidirectional image 510 is a still image obtained by joining a plurality of past images captured by panning and tilting. On the other hand, in the camera device 2, the circular omnidirectional image 510 is a live moving image captured by the fisheye lens 201 and compressed without being dewarped. The circular omnidirectional image is also referred to as a fisheye image.

[Correspondence of image]
FIG. 5 is a view showing a correspondence relationship between a spherical model, a circular omnidirectional image, and a preview frame for calculating a preview frame according to the present embodiment.

  The sphere model 500 for calculating the preview frame is shown as a sphere centered on the viewpoint of the camera device, and in the case of the camera device installed on the ceiling as shown in FIG. 3, the lower half of the sphere corresponds to the imageable range . A frame 501 is inscribed or circumscribed to a curved surface within the visual field range of the sphere, and shows a portion of the visual field range in the plane 502 as a plane area orthogonal to the viewing direction. Here, the omnidirectional image 510 is a projection of the image of the scene projected on the lower half of the sphere on a plane 503 orthogonal to the vertical axis of the sphere. Further, the preview frame 511 is a rectangle in which each vertex of the frame 501 is projected on the plane 503 and connected by a straight line.

  FIG. 6 is a diagram for explaining the difference between the introspection to the sphere model 500 and the preview frame due to the circumscription. FIG. 6A is a view of the sphere model 500 viewed from a point on the plane 502. FIG. The frame 501 corresponds to the surface 505 when inscribed in the curved surface 506 within the field of view, and corresponds to the surface 504 when circumscribed to the curved surface 506 within the field of view. When the curved surface 506 is projected onto the plane 503, the edge portion thereof becomes the preview frame 516 of FIGS. 6 (b) and 6 (c). The reference point 508 is a point obtained by projecting the intersection point of the curved surface 506 and the straight line representing the viewing direction onto the plane 503. The preview frame 515 in FIG. 6B is a rectangle obtained by projecting the four vertices of the surface 505 on the plane 503 and connecting them by straight lines. The preview frame 514 in FIG. 6C is a rectangle obtained by projecting the four vertices of the surface 505 on the plane 503 and connecting the four vertices with a straight line.

  Here, since the four vertices of the surface 506 coincide with the four vertices of the surface 505, the four vertices of the preview frame 515 and the preview frame 516 also match. Further, since the central point of the curved surface 506 and the central point of the surface 504 coincide at an intersection point with the straight line representing the gaze direction, the central point of the preview frame 514 and the reference point 508 also coincide. If both of the preview frames 515 and 514 are displayed on the screen at the same time, it becomes difficult for the user to visually recognize them, and one of them is selected and displayed as the preview frame 511 on the viewer screen. In the present embodiment, the preview frame 516 is not displayed on the viewer screen.

  When the preview frame 515 is selected to be displayed, the preview frame is displayed closer to the center of the circular omnidirectional image than the preview frame 516 corresponding to the actually visible range, and the preview frame does not protrude from the omnidirectional image. There is. However, in this case, a shift occurs between the reference point 508 and the center point of the preview frame 515. On the other hand, when the preview frame 514 is selected to be displayed, the preview frame is displayed on the outer side with respect to the center of the circular omnidirectional image than the preview frame 516 corresponding to the actually visible range. May spill over. However, in this case, no shift occurs between the reference point 508 and the center point of the preview frame 514. Depending on these features, the user can choose which to display.

Processing flow
Hereinafter, the display processing according to the present embodiment will be described using FIGS. 7 to 13. Here, the screen transition when one of the positions with respect to the preview frame is designated will be described together with the display example.

  In this embodiment, an example using a mouse as a device for performing user input to the operation input unit 305 will be described. The mouse operation here uses the button on the left side of the mouse as the main operation, and the button on the right side uses it as the secondary operation. However, the present invention is not limited to this, and in the case of a mouse having the reverse setting, the right button and the left button are reversed in the mouse operation in the following processing flow. In addition, unless otherwise specified, the mouse-up and mouse-down operations described below refer to operations using the left button. The term "drag" refers to moving the mouse cursor while the mouse is down.

  The process flowcharts shown in FIG. 9, FIG. 11, FIG. 13 and FIG. 18 are realized by the CPU of the viewer terminal 3 executing a program stored in the ROM or the like.

(Operation when the outside of the preview frame is specified)
FIG. 7 is a diagram for explaining transition of display when an area outside the preview frame is clicked by a user operation. FIG. 8 is a diagram for explaining transition of display when an area outside the preview frame is dragged. Further, FIG. 9 is a flowchart showing processing when the operation according to FIG. 7 is performed.

  When the processing flow shown in FIG. 9 is started, it is assumed that the preview frame 511 is displayed on the omnidirectional image 510 as shown in FIG. 7A.

  At S101, the viewer terminal 3 detects whether or not mouse-down has been performed. If mouse-down has not been performed (NO at S101), this processing flow ends. When mouse down is performed (YES in S101), viewer terminal 3 has a position where mouse down is performed (hereinafter, mouse down position) outside the frame of preview frame 511 currently displayed, and It is determined whether or not the omnidirectional image 510 is on. When the above condition is not satisfied (NO in S102), this processing flow ends. If the above condition is satisfied (YES in S102), the viewer terminal 3 displays another preview frame 512 having the mouse-down position as the reference point 508 in S103. As shown in FIG. 7B, the preview frame 512 is displayed so that, for example, the line type and the line color are different so as to be distinguishable from the preview frame 511. At this time, the preview frame calculation unit 304 calculates the size of the preview frame 512 so that the size (that is, the zoom position or the angle of view) of the curved surface 506 corresponding to the preview frames 511 and 512 is maintained. Do. For example, in the circular omnidirectional image 510, the preview frame 512 is displayed larger as the mouse is down at a position closer to the center of the circle, even with the same size. This originates in the relationship at the time of projecting on the plane 503, as shown in FIG. Further, the preview frame 512 is rotated from the preview frame 511 according to the positional relationship between the center position of the omnidirectional image 510 and the reference point 508.

  At S104, the viewer terminal 3 detects whether a drag has been performed. If the drag is not performed (NO in S104), the process proceeds to S106. If a drag is performed (YES in S104), in S105, viewer terminal 3 selects preview frame 512 according to the mouse down position and the current position of mouse cursor 520 moved by dragging. Redraw At this time, as shown in FIG. 8B, the position of the reference point 508 does not change, and the position of the mouse cursor 520 is always on the preview frame 512 according to the movement of the mouse cursor 520 by the user operation. Change the size to be Then, it progresses to S106.

  At S106, the viewer terminal 3 detects whether or not mouse-up has been performed. If mouse-up has not been performed (NO in S106), the process proceeds to S107, and if mouse-up has been performed (YES in S106), the process proceeds to S109.

  In S107, the viewer terminal 3 determines whether or not the right click has been made. If the right click is not performed (NO at S107), the process returns to S104, and if the right click is performed (YES at S107), the process proceeds to S108. In S108, the viewer terminal 3 cancels the operation on the preview frame 512, and returns to the state where only the preview frame 511 is displayed (the state of FIG. 7A). In this case, no camera control command is generated and transmission is not performed. Then, the process flow ends.

  In S109, the viewer terminal 3 deletes the preview frame 511, and changes it to a state in which only the preview frame 512 is displayed. Along with this, the viewer terminal 3 treats the changed preview frame as the preview frame 511. As a specific operation, after mouse-down is performed, if mouse-up is performed while dragging the mouse down without dragging (NO in S104 and YES in S106), the preview displayed in FIG. The frame 511 disappears. Then, as shown in FIG. 7C, a new preview frame is displayed as the preview frame 511.

  Thereafter, in S110, the viewer terminal 3 transmits a camera control command to the camera device 1 or the camera device 2 in order to perform camera control corresponding to the new preview frame 511. That is, the preview frame calculation unit 304 calculates a camera control parameter corresponding to the new preview frame 511, and the operation command generation / interpretation unit 306 generates a camera control command corresponding to the calculated parameter. Then, the communication unit 307 transmits the generated camera control command to the camera device 1 or the camera device 2.

(Operation when the inside of the preview frame is specified)
FIG. 10 is a diagram for explaining the transition of the screen when the user drags the inside of the preview frame by the user operation. FIG. 11 is a flowchart showing processing when the operation according to FIG. 10 is performed.

  In S201, the viewer terminal 3 detects whether or not mouse-down has been performed. If mouse-down has not been performed (NO in S201), this processing flow ends. If mouse down is performed (YES in S201), the viewer terminal 3 determines in S202 whether the mouse down position is within the frame of the preview frame 511 or not. If not within the frame (NO in S202), this processing flow ends. If it is within the frame (YES in S202), in S203, the viewer terminal 3 displays the preview frame 512 at the same position and the same size as the preview frame 511. At this time, as shown in FIG. 10A, the shape of the mouse cursor 520 changes.

  Thereafter, in S204, the viewer terminal 3 detects whether a drag has been performed. If the drug is not being performed (NO in S204), the process proceeds to S206. When the drag is performed (YES in S204), the viewer terminal 3 moves the preview frame 511 in parallel according to the movement of the mouse cursor 520 as shown in FIG. The preview frame 512 is rotated and redrawn according to the position. At this time, the preview frame calculation unit 304 calculates the size of the preview frame 512 so that the size (that is, the zoom position) of the curved surface 506 which is the visual field range corresponding to the preview frames 511 and 512 is maintained. Then, it progresses to S206.

  At S206, the viewer terminal 3 detects whether or not mouse-up has been performed. If mouse-up has not been performed (NO at S206), the process proceeds to S207, and if mouse-up has been performed (YES at S206), the process proceeds to S209.

  In S207, the viewer terminal 3 determines whether or not the right click has been made. If the right click is not performed (NO at S207), the process returns to S204, and if the right click is performed (YES at S207), the process proceeds to S208. In S208, the viewer terminal 3 cancels the operation of the preview frame 512, and returns to the state where only the preview frame 511 is displayed (the state of FIG. 10A). In this case, no camera control command is generated and transmission is not performed. Then, the process flow ends.

  In S209, the viewer terminal 3 deletes the preview frame 511 and changes it to a state in which only the preview frame 512 is displayed. Along with this, the viewer terminal 3 treats the changed preview frame as the preview frame 511. As a specific example, it is displayed as shown in FIG.

  Thereafter, in S210, the viewer terminal 3 transmits a camera control command to the camera device 1 or the camera device 2 in order to perform camera control corresponding to the new preview frame 511. That is, the preview frame calculation unit 304 calculates a camera control parameter corresponding to the new preview frame 511, and the operation command generation / interpretation unit 306 generates a camera control command corresponding to the calculated parameter. Then, the communication unit 307 transmits the generated camera control command to the camera device 1 or the camera device 2.

(Operation when dragging on the preview frame)
FIG. 12 is a diagram for explaining the transition of the screen when the user drags on the preview frame by the user operation. FIG. 13 is a flowchart showing processing when the operation according to FIG. 12 is performed.

  In S301, the viewer terminal 3 detects whether or not mouse-down has been performed. If mouse-down has not been performed (NO in S301), this processing flow ends. If mouse down is performed (YES in S301), the viewer terminal 3 determines in S302 whether the mouse down position is on the frame of the preview frame 511 or not. If it is not on the frame (NO in S302), this processing flow ends. If the frame is on the frame (YES in S302), in S303, the viewer terminal 3 displays the preview frame 512 at the same position and the same size as the preview frame 511. For example, when the mouse is down on the frame, the preview frame 511 and the preview frame 512 are superimposed and displayed as shown in FIG. 12A, and at this time, the shape of the mouse cursor 520 changes.

  Thereafter, in S304, the viewer terminal 3 detects whether a drag has been performed. If the drag is not performed (NO in S304), the process proceeds to S306. If a drag is performed (YES in S304), the viewer terminal 3 enlarges or reduces the preview frame 512 in accordance with the movement of the mouse cursor 520 in S305. At this time, as shown in FIG. 12B, the preview frame calculation unit 304 moves the mouse cursor 520 by user operation so that the mouse cursor 520 is always on the preview frame 512 without changing the position of the reference point 508. At the same time, the size of the preview frame 512 is adjusted. Then, it progresses to S306. Since the process after S306 is the same as the process after S206 of FIG. 10, the description here is omitted.

  In the example using FIGS. 7 to 13, the operation assuming the mouse operation is described. However, in the configuration based on touch panel operation instead of mouse operation, for example, tap mouse down, tap mouse up untap, drag swipe, double-tap with a finger different from the finger tapping right click The present embodiment can be applied by replacing.

Second Embodiment
Next, while switching between rectangular and circular omnidirectional images, pan / tilt / zoom control and image clipping control can be performed using the preview frame on both sides, and a preview frame that accurately represents the visual field range except during operation A second embodiment for displaying will be described.

  The system configuration, the installation image, and the process at the time of the preview frame operation when the omnidirectional image is fixed as a circle or a rectangle are the same as the contents shown in the first embodiment, so the description is omitted and only the difference is described. Do.

  FIG. 14 is a diagram showing a correspondence between a sphere model for calculating a preview frame, a rectangular omnidirectional image, and a preview frame.

  The sphere model 500 and the frame 501 for calculating the preview frame are the same as in FIG. 3 shown in the first embodiment, but in this embodiment, the projection plane is a cylinder 530 surrounding the sphere model. . The frame 532 on the cylinder 530 is a rectangle having the same width as the curved surface within the field of view of the sphere. The rectangular omnidirectional image 540 and the preview frame 542 correspond to those in which the cylinder 530 and the frame 532 are expanded in a plane and compressed in the height direction. This corresponds to a projection method called equidistant cylindrical projection, and while the lines 531 at equal intervals of tilt angle become wider on the cylinder 530 as they go down, the lines 541 on the rectangular omnidirectional image 540 become wider. It is equally spaced like.

  FIG. 15 is a diagram for explaining screen transition when dragging outside the preview frame on the circular omnidirectional image of the present embodiment.

  The flow of the process is the same as that of the flowchart of FIG. 9 described in the first embodiment, except that the preview frame 511 is not rectangular and has a shape corresponding to the preview frame 516 of FIG. Although the preview frame 512, which is the target of the mouse operation, is a rectangle, the preview frame 511 when not operated and after the operation is a frame surrounded by a curve so as to accurately represent the visual field range.

  FIG. 16 is a diagram for explaining screen transition when dragging is performed in the preview frame on the rectangular omnidirectional image according to the present embodiment.

  The flow of processing is the same as the flow chart of FIG. 11 described in the first embodiment. However, as in the preview frame on the circular omnidirectional image shown in FIG. 15, the preview frame 542 which is the target of the mouse operation is rectangular, but the preview frame 543 at the time of non-operation and after completion of the operation It will be a box surrounded by a curve to accurately represent it. By moving the preview frame 543 shown in FIG. 16 (a) to the lower part of the screen, it becomes wide as shown in FIG. 16 (d) due to the characteristics of the configuration shown in FIG.

  FIG. 17 shows an example of a screen in the case of automatically switching between a rectangular omnidirectional image and a circular omnidirectional image. FIG. 18 is a flowchart showing the process according to FIG.

  The omnidirectional image display unit 302 displays the viewer screen 600. As shown in FIG. 17A, the rectangular omnidirectional image is displayed when the rectangular omnidirectional tab 610 is selected. As shown in FIG. 17B, a circular omnidirectional image is displayed when the circular omnidirectional tab 620 is selected. In pull-down list 630, it is possible to select whether to automatically switch tabs. In each omnidirectional image, a dividing line 550 representing the same tilt angle is displayed. When “automatic switching” is selected in the pull-down list 630, the tab is automatically switched depending on whether the preview frame straddles the dividing line 550 or not. The dividing line 550 indicates a range in which the distance between the viewing direction or the viewing range with respect to the image corresponding to the preview frame and the center of the omnidirectional image falls within a predetermined range. The predetermined range here is assumed to be previously defined in consideration of the distortion of the image.

  If "automatic selection" is not selected, automatic display switching is not performed even if an instruction such as movement to the preview frame is issued. In addition, it is assumed that the user can switch to any display by selecting a tab at any time.

  The flowchart in FIG. 18 shows the processing in the case where the inside of the preview frame is dragged. S401 and S402 are added from the flowchart of FIG. Here, the same reference numerals are assigned to duplicate processes, and the description is omitted.

  If mouse-up is detected in S206 (YES in S206), the viewer terminal 3 determines in S401 whether or not the position of the reference point 508 of the moved preview frame has exceeded the dividing line 550. For example, in the case of a rectangular omnidirectional image, it is determined that the position of the reference point 508 is exceeded when it moves below the dividing line 550. In addition, in the case of a circular omnidirectional image, it is determined that the position of the reference point 508 has exceeded when moved to the outside of the dividing line 550. If it exceeds (YES in S401), the process proceeds to S402, and if it does not exceed (NO in S401), the process proceeds to S209.

  At S402, the viewer terminal 3 switches to the other tab, and the subsequent processing is performed in the selected display form. Specifically, when a rectangular omnidirectional image is displayed, the display is switched to a circular omnidirectional image. The same is true for the opposite case. Then, it progresses to S210. The determination as to whether or not the dividing line 550 is crossed may be made based on whether any vertex of the preview frame or the entire preview frame exceeds the dividing line 550.

  By switching the display, the shape of the preview frame can be prevented from being displayed in a narrow shape as shown by the preview frame 511 in FIG. 15A on a circular omnidirectional image. Similarly, on a rectangular omnidirectional image, it can be avoided that the image is displayed in a wide shape as shown by the preview frame 543 in FIG. As a result, the distortion of the image in the field of view does not become too large.

  As mentioned above, although one embodiment of the present invention was described, the present invention is not limited to these embodiments, and various modification and change are possible within the limits of the gist.

<Other Embodiments>
The present invention is also realized by performing the following processing. That is, software (program) for realizing the functions of the above-described embodiment is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU or the like) of the system or apparatus reads the program. It is a process to execute.

1, 2 ... camera device, 3 ... viewer device, 301 ... main image display unit, 302 ... omnidirectional image display unit, 303 ... image decompression unit, 304 ... preview frame calculation Unit 305: Operation input unit 306: Operation command generation / interpretation unit 307: Communication unit

Claims (16)

An information processing apparatus connected to a camera apparatus via a network,
Acquisition means for acquiring an omnidirectional image captured by the camera device;
Display processing means for displaying the omnidirectional image acquired by the acquisition means on the display screen of the display means;
In response to detection of a user operation on a frame superimposed on an omnidirectional image displayed on the display screen, the frame for determining the position of a partial image of a predetermined visual field range cut out from the omnidirectional image A frame to be superimposed on the omnidirectional image , and control means for switching the partial image corresponding to the frame ;
The display processing means displays the rectangular frame on the display screen while receiving the user operation, and while the user operation with respect to the frame is not received, the frame corresponds to the visual field range corresponding to the frame. based on the information processing apparatus according to claim Rukoto is displayed on the display screen in the form at least part of which is constituted by a curve. Generation means for generating a control command to the camera device according to a user operation on a frame superimposed on an omnidirectional image displayed on the display screen;
And transmission means for transmitting the control command generated by the generation means to the camera device.
The information processing apparatus according to claim 1, wherein the acquisition unit acquires a new image from the camera device controlled according to the control command transmitted by the transmission unit.   The information processing apparatus according to claim 2, wherein the control command is a command for control regarding any of pan, tilt, and zoom. The display processing means displays both the original position frame and the frame while receiving the user operation so as to be distinguishable until the user operation is completed while the user operation for the frame is received. The information processing apparatus according to any one of claims 1 to 3, wherein The information processing apparatus according to any one of claims 1 to 4, wherein the user operation is performed by a mouse operation or a touch panel operation on the display screen. The said frame superimposed on the said omnidirectional image is obtained by projecting the visual field range represented by the spherical body model surrounding the said camera apparatus on the said omnidirectional image. The information processing apparatus according to any one of the items. A point obtained by projecting the point at which the spherical model intersects the viewing direction onto the omnidirectional image is set as a reference point of the frame ,
When the control unit indicates a first position outside the frame by the user operation, the control unit determines the first position as a new reference point of the frame , and the frame corresponds to the center position of the omnidirectional image. It becomes the direction corresponding to the direction of the first position, and claims to be changed to a size to maintain the original angle, and switches the image corresponding to the display and the frame of the frame The information processing apparatus according to 6 . A point obtained by projecting the point at which the spherical model intersects the viewing direction onto the omnidirectional image is set as a reference point of the frame ,
The control means is configured such that, when a user's operation indicates a first position in the frame of the frame and further indicates a second position in the omnidirectional image different from the first position, the frame of the reference point is moved parallel to the second position, the reference point of the frame with respect to the center position of the omnidirectional image the frame in accordance with the distance and direction of the second position and the first position rotate the direction corresponding to the direction, as changed to a size to maintain the original angle, according to claim 6, characterized in that switching the partial image corresponding to the display and the frame of the frame Information processing equipment. A point obtained by projecting the point at which the spherical model intersects the viewing direction onto the omnidirectional image is set as a reference point of the frame ,
The control means is configured such that a first position outside the frame is indicated by a user operation, and a second position in the omnidirectional image different from the first position is indicated. Position of the frame as a new reference point of the frame , the frame is oriented according to the direction of the first position with respect to the center position of the omnidirectional image, and the size of the frame is the first position and as it is changed according to the distance between the second position, the information processing apparatus according to claim 6, characterized in that switching the partial image corresponding to the display and the frame of the frame. A point obtained by projecting the point at which the spherical model intersects the viewing direction onto the omnidirectional image is set as a reference point of the frame ,
Wherein if the first position on the frame of the frame by the user operation is instructed, further, the second position in different the omnidirectional image and the first position is instructed, the frame said reference point and said frame while maintaining the orientation with respect to the center position of the omnidirectional image, so that the size of the frame is changed according to the distance of the second position and the first position, the 7. The information processing apparatus according to claim 6 , wherein a frame is displayed and the partial image corresponding to the frame is switched. The rectangular frame is a rectangular frame in which the visual field range represented by the spherical model is projected onto the omnidirectional image and connected by a straight line,
The shape of which at least a portion is composed of a curve is a non-rectangular shape as a region of the curved surface of the field of view shown in the sphere model
7. An information processing apparatus according to claim 6, wherein: An information processing apparatus connected to a camera apparatus via a network,
Acquisition means for acquiring an omnidirectional image captured by the camera device;
Display processing means for displaying the omnidirectional image acquired by the acquisition means on the display screen of the display means;
A frame to be superimposed on an omnidirectional image displayed on the display screen, the frame being superimposed on the omnidirectional image according to detection of a user operation on a frame corresponding to an image cut out from the omnidirectional image And control means for switching the image corresponding to the frame
Equipped with
The frame superimposed on the omnidirectional image is obtained by projecting the visual field range represented by a spherical model surrounding the camera device onto the omnidirectional image,
It said display processing means can be displayed on the display screen by switching the omnidirectional image in either rectangular or round,
The display processing means displays a circular omnidirectional image when the distance between the visual line direction or the visual field range and the center of the circular omnidirectional image with respect to the image corresponding to the frame falls within the predetermined range, and the predetermined range If it does not fit on information processing device and switches to display the omnidirectional image of the rectangle. The apparatus further comprises means for accepting setting of whether to automatically switch and display any rectangular or circular omnidirectional image by the display means.
The information processing apparatus according to claim 12 , wherein the display processing unit performs switching in accordance with the setting. A control method of an information processing apparatus connected to a camera apparatus via a network,
An acquisition step of acquiring an omnidirectional image captured by the camera device;
A display processing step of displaying the omnidirectional image acquired in the acquisition step on the display screen of the display means;
A frame to be superimposed on an omnidirectional image displayed on the display screen, the frame being responsive to a user operation on the frame for determining the position of a partial image of a predetermined visual field range cut out from the omnidirectional image A frame to be superimposed on the omnidirectional image , and a control step of switching the partial image corresponding to the frame ;
In the display processing step, while the user operation is received, the rectangular frame is displayed on the display screen, and while the user operation with respect to the frame is not received, the frame is a visual field range corresponding to the frame. based on the control method for the information processing apparatus according to claim Rukoto is displayed on the display screen in the form at least part of which is constituted by a curve. A control method of an information processing apparatus connected to a camera apparatus via a network,
An acquisition step of acquiring an omnidirectional image captured by the camera device;
A display processing step of displaying the omnidirectional image acquired in the acquisition step on the display screen of the display means;
A frame to be superimposed on an omnidirectional image displayed on the display screen, the frame being superimposed on the omnidirectional image according to detection of a user operation on a frame corresponding to an image cut out from the omnidirectional image And a control step of switching the image corresponding to the frame
Have
The frame superimposed on the omnidirectional image is obtained by projecting the visual field range represented by a spherical model surrounding the camera device onto the omnidirectional image,
In the display processing step, it is possible to switch an omnidirectional image with either a rectangle or a circle and to display the image on the display screen.
In the display processing step, a circular omnidirectional image is displayed when the distance between the viewing direction or the visual field range to the image corresponding to the frame and the center of the circular omnidirectional image falls within the predetermined range, and the predetermined range is displayed. The control method of an information processing apparatus characterized by switching so as to display a rectangular omnidirectional image when the image does not fit into. A program for controlling a computer as each means of the information processing apparatus according to any one of claims 1 to 13.