JP6614516B2 - Display device and computer program - Google Patents

Description

  The present invention relates to a display device that displays an image, and more particularly to a display device that displays an image taken by an omnidirectional camera.

  Conventionally, a display device that displays an image of an omnidirectional camera on a monitor is known. Since the omnidirectional camera uses a fish-eye lens with a horizontal viewing angle of 360 degrees to generate an image, the image (hereinafter referred to as “omnidirectional image”) is circular or donut-shaped, and the distortion is greater as the position is closer to the outer periphery. There is a characteristic to become. Therefore, in the conventional display device, in addition to the display of the omnidirectional image for confirming the entire photographing range, an image obtained by performing distortion correction on the entire omnidirectional image or a part of the cutout range (hereinafter referred to as “distortion corrected image” ") Is also possible. This display switching is performed based on a user instruction.

  On the other hand, a display device that displays video from an omnidirectional camera is used in the following scene, for example. In a business scene, a mobile terminal such as a mobile phone is used as a display device to display an image of an omnidirectional camera installed at a construction site, facility, store, or the like. Also, when an omnidirectional camera is used as a surveillance camera, the omnidirectional camera is installed at a monitoring target location such as a railway platform or parking lot, and an image of the omnidirectional camera is displayed at the headquarters or the central command room. The staff monitors this. In addition, when an omnidirectional camera is installed at a high place such as on a utility pole, an operator who installs an omnidirectional camera installs an omnidirectional camera that is used as a surveillance camera. Using the display device, the angle of view is adjusted while checking the image of the omnidirectional camera.

  When the display device is realized by a personal computer having a monitor, there are operation input means such as a mouse and a keyboard, and the user can instruct switching of the display by operating the operation input means (for example, , See Patent Document 1 and Patent Document 2).

Japanese Patent Laid-Open No. 11-136584 JP 2008-301034 A

  However, as described above, when a small portable terminal such as a mobile phone is used as a display device, the operation input means is physically restricted, and the conventional operation input means such as a mouse and a keyboard cannot be employed. There is. As described above, in the case of a display device worn by an operator at a high place, the display device becomes smaller and restrictions on the operation input means become more severe.

  In addition, even when there are relatively few physical restrictions, such as the above-mentioned headquarters or central command room display device, an omnidirectional image and a distortion-corrected image can be operated more intuitively without performing complicated operations. It is desirable to be able to switch the display between.

  The present invention has been made in view of the above problems, and provides a display device capable of switching display between an omnidirectional image and a distortion-corrected image with a simple operation by an intuitive user interface. For the purpose.

  In order to solve the above-described conventional problems, a display device of the present invention includes an image acquisition unit that acquires an omnidirectional image, a touch panel display unit having a panel interface, and distortion of a part or all of the omnidirectional image. A distortion-corrected image generation unit that corrects and generates a distortion-corrected image, and an image that is displayed using the panel interface when the omnidirectional image or the distortion-corrected image is displayed on the display unit An operation detection unit that detects an operation for switching between the display of the omnidirectional image and the display of the distortion-corrected image, and the omnidirectional image or the distortion correction performed on the display unit. A display control unit that displays an image and switches between the display of the omnidirectional image and the display of the distortion-corrected image according to the operation detected by the operation detection unit. We have the example was constructed.

  With this configuration, when an omnidirectional image or a distortion-corrected image is displayed on the touch panel display unit, the user can directly input an operation for switching the display on the displayed image. Therefore, it is possible to intuitively switch the display between the omnidirectional image and the distortion corrected image with a simple operation.

  In the display device, when the omnidirectional image is displayed on the display unit, the operation detection unit pinches the omnidirectional image displayed on the display unit. When it is detected that an out operation has been performed, the display of the omnidirectional image may be switched to the display of the distortion corrected image.

  With this configuration, the pinch-out operation for the omnidirectional image by the user and the display switching in which the omnidirectional image is expanded and the distortion corrected image is displayed intuitively match, so the user can display more intuitively. Can be switched.

  In the display device, when the distortion correction image is displayed on the display unit, the operation detection unit pinches in the distortion correction image displayed on the display unit. When it is detected that an operation has been performed, the display of the distortion correction image may be switched to the display of the omnidirectional image.

  With this configuration, the pinch-in operation on the distortion-corrected image by the user and the switching of the display for returning the distortion-corrected image to the omnidirectional image are intuitively matched, so that the user can switch the display more intuitively. it can.

  In the above display device, the distortion-corrected image may be a panoramic image obtained by panoramic development of the omnidirectional image with the location where the pinch-out operation is performed as a base line.

  With this configuration, since the pinch-out operation by the user and the switching of the display from the omnidirectional image to the panoramic image are intuitively matched, the user can change the display from the omnidirectional image to the panoramic image. Can be performed intuitively with simple operations.

  In the display device, the distortion correction image may be a cut-out image in which a portion where the pinch-out operation has been performed is enlarged.

  With this configuration, since the pinch-out operation and the display of the enlarged cut-out image are intuitively matched, the user can intuitively perform the operation for displaying the cut-out image with a simple operation. Can do.

In the display device, when the omnidirectional image is displayed on the display unit, the operation detection unit taps the omnidirectional image displayed on the display unit. When it is detected that an operation has been performed, the distortion correction image may be displayed on the display unit at a location where the tap operation has been performed on the omnidirectional image.

  With this configuration, the tap operation on the omnidirectional image by the user and the display of the distortion corrected image at the tap location intuitively match, so the user can easily perform the operation for displaying the distortion corrected image, It can be done intuitively.

  In the above display device, the display control unit may determine an image correction parameter based on an operation detected by the operation detection unit, and the display device further includes the display control unit. According to the determined correction parameter, an intermediate image generation unit that generates an intermediate image to be displayed in the middle of switching between the display of the omnidirectional image and the display of the distortion correction image may be provided, During the switching between the display of the omnidirectional image and the display of the distortion corrected image, the intermediate image generated by the intermediate image generation unit may be displayed on the display unit.

  With this configuration, the user can see the display switching process with the intermediate image, so that the user can intuitively recognize the operation performed by the user, and between the display before switching and the display after switching. Correlation can be recognized.

  Another aspect of the present invention is a computer program, which is executed by a display device having a touch panel type display unit, thereby causing the display device to function as the display device.

  In the present invention, when an omnidirectional image or a distortion correction image is displayed on a touch panel type display unit, an operation for display switching can be directly input to the displayed image. It is possible to intuitively switch the display between the omnidirectional image and the distortion corrected image by a simple operation.

The block diagram which shows the structure of the viewer terminal in embodiment of this invention The block diagram which shows the detailed structure of the image correction sequencer in embodiment of this invention (A) The figure which shows the omnidirectional image in embodiment of this invention (b) The figure which shows the drag operation for designating the base line in the case of switching from the omnidirectional image to the panoramic image in embodiment of this invention. c) A diagram showing a pinch-out operation when switching from an omnidirectional image to a panoramic image in the embodiment of the present invention. (d) When switching from an omnidirectional image to a panoramic image in the embodiment of the present invention. The figure which shows an intermediate image (e) The figure which shows the intermediate image in the case of the switch from the omnidirectional image in embodiment of this invention to a panoramic image (f) The figure which shows the panoramic image in embodiment of this invention (A) The figure which shows the panoramic image in embodiment of this invention (b) The figure which shows the intermediate | middle image in the case of switching from the panoramic image to the omnidirectional image in embodiment of this invention (c) Implementation of this invention Diagram showing omnidirectional image in form Flow chart of switching from omnidirectional image to panoramic image and switching from panoramic image to omnidirectional image in the embodiment of the present invention (A) The figure which shows the omnidirectional image in embodiment of this invention (b) The figure which shows tap operation for designation | designated of the cutout image in embodiment of this invention (c) The cutout image in embodiment of this invention (D) The figure which shows the drag operation for the movement of the cut-out image in embodiment of this invention (e) For the switch to the multiscreen in embodiment of this invention (F) The figure which shows the multi-screen in embodiment of this invention The figure explaining switching from the single panorama image to the double panorama image in the embodiment of the present invention (A) A diagram showing a pinch-in operation for switching from a double panorama image to a single panorama image in the embodiment of the present invention. (B) At the time of switching from a double panorama image to a single panorama image in the embodiment of the present invention. (C) The figure which shows the single panorama image in embodiment of this invention (A) The figure which shows the flick operation for the scroll of the single panorama image in embodiment of this invention (b) The figure which shows the scroll of the single panorama image in embodiment of this invention (A) The figure which shows the flick operation for the scroll of the double panorama image in the embodiment of the present invention (b) The figure which shows the scroll of the double panorama image in the embodiment of the present invention (A) The figure which shows the modification of operation for the scroll of the panorama image in embodiment of this invention (b) The figure which shows the scroll of the panorama image in embodiment of this invention (A) The figure which shows the pinch out operation for the high quality image display in embodiment of this invention (b) The figure which shows the high quality image display in embodiment of this invention (c) In embodiment of this invention The figure which shows the drag operation for movement of the cutout area of the high quality picture (A) The figure which shows the omnidirectional image in embodiment of this invention (b) The figure which shows rotation operation for rotation of the omnidirectional image in embodiment of this invention (c) All in the embodiment of this invention Diagram showing rotation of orientation image The figure which shows the switching from the omnidirectional image to a panoramic image of the 1st modification in embodiment of this invention. The figure which shows the switch from the omnidirectional image to a panoramic image of the 2nd modification in embodiment of this invention. (A) Drawing which shows drag operation for panorama development of the 3rd modification in an embodiment of the invention (b) Pinch out operation for panorama development of the 3rd modification in an embodiment of the invention Figure showing (A) Drawing which shows drag operation for double panorama development of the 4th modification in an embodiment of the invention (b) Pinch for double panorama development of a 4th modification in an embodiment of the invention Diagram showing out operation (A) The figure which shows the pinch out operation (radial direction) for the switch of the display from the omnidirectional image of the 5th modification in embodiment of this invention (b) The 5th in embodiment of this invention The figure which shows the multi image containing the cut-out image of a modification (A) The figure which shows the pinch out operation (circumferential direction) for the switch of the display from the omnidirectional image of the 5th modification in embodiment of this invention (b) The 5th in embodiment of this invention Showing a panoramic image of a modified example of (A) The figure which shows the pinch out operation (vertical direction) for the switch from the single panorama image of the 6th modification in embodiment of this invention to a double panorama image (b) The figure in embodiment of this invention The figure which shows the intermediate image of the switching from the single panorama image of 6 modification to a double panorama image (c) The figure which shows the double panorama image of the 6th modification in embodiment of this invention (A) The figure which shows the pinch out operation (horizontal direction) for the switch from the single panorama image of the 6th modification in embodiment of this invention to a double panorama image (b) The figure in embodiment of this invention FIG. 6C is a diagram illustrating an intermediate image for switching from a single panoramic image to a double panoramic image according to the sixth modified example. (C) An intermediate image for switching from a single panoramic image to a double panoramic image according to the sixth modified example of the embodiment of the present invention. (D) The figure which shows the double panorama image of the 6th modification in embodiment of this invention

  Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a viewer terminal as a display device according to an embodiment of the present invention. The viewer terminal 1 includes a panel interface 11, an operation detection unit 12, an image correction sequencer 13, an intermediate image generation unit 14, a distortion correction image generation unit 15, an omnidirectional image acquisition unit 16, an omnidirectional image decoding unit 17, an image queue 18, And an image display unit 19. The image display unit 19 and the panel interface 11 are provided so as to overlap each other, thereby forming a touch panel type display unit.

  The panel interface 11 accepts an operation with a user's finger and outputs operation position information. The operation detection unit 12 receives the information of the operation position from the panel interface 11 and detects the content of the panel operation performed by the user based on the content of the change when the panel operation is changed. The operation on the panel interface 11 by the user's finger includes tap, drag, flick, pinch (pinch out and pinch in), and rotation.

  A tap is an operation of tapping a portion of the panel interface 11 with a finger, that is, an operation of touching the panel interface 11 with one fingertip and then moving the finger away from the panel interface 11 without moving the finger on the panel interface 11. Say. Dragging refers to an operation of dragging the panel interface 11 with a finger, that is, an operation of moving one fingertip while being attached to the panel interface 11, stopping the movement of the finger and releasing the finger from the panel interface 11. Flicking refers to an operation of flicking the panel interface 11 with a finger, that is, an operation of moving one fingertip attached to the panel interface 11 and releasing the finger from the panel interface 11 without stopping the movement of the finger.

  Pinch out refers to an operation of moving one or both of the two fingertips on the panel interface 11 so that the distance between them is increased while the two fingertips are attached to the panel interface 11. In contrast, the pinch-in refers to an operation of moving one or both of the two fingertips on the panel interface 11 so that the distance between the two fingertips is reduced while the two fingertips are attached to the panel interface 11. Rotation means moving one or both of the two fingertips on the panel interface 11 so that the line segment connecting the two fingertips is attached to the panel interface 11 and rotates in the plane of the panel interface 11 This is the operation to be performed.

  In addition to the above, the operation with the user's finger may include a double tap, a long tap, and the like. When the viewer terminal 1 has an electronic compass and can detect the acceleration (posture) of the viewer terminal 1, the user inputs an operation by shaking (shaking) or tilting the viewer terminal 1 itself. May be performed. In this case, when there is a change in the acceleration of the viewer terminal 1, the operation detection unit 12 detects an operation by the user based on the change content.

  The image correction sequencer 13 displays the image on the image display unit 19 by determining correction parameters (center coordinates, range, rotation, distortion, etc.) of the image to be generated and displayed next according to the contents of the panel operation. The display of the omnidirectional image, the intermediate image, and the distortion corrected image (the distortion corrected image includes a panoramic image and a cutout image) is switched. The correction parameter is a parameter used when correcting the omnidirectional image. By applying the correction parameter to the omnidirectional image, a distortion corrected image or an intermediate image is generated. The image correction sequencer 13 corresponds to a display control unit of the present invention.

The intermediate image generation unit 14 generates an intermediate image when switching from an omnidirectional image to a distortion corrected image, or from a distortion corrected image to an omnidirectional image. An intermediate image is an image displayed during switching. The intermediate image generation unit 14 generates an intermediate image corresponding to the content of the panel operation based on the correction parameter input from the image correction sequencer 13. The intermediate image generation unit 14 generates a plurality of intermediate images as necessary in order to make it appear that the omnidirectional image is switched to the distortion corrected image or the distortion corrected image to the omnidirectional image in stages.

  The distortion correction image generation unit 15 performs distortion correction on the omnidirectional image based on the correction parameter input from the image correction sequencer 13 to generate a distortion correction image. The omnidirectional image acquisition unit 16 acquires an omnidirectional image captured by an imaging device (camera) using a fisheye lens. This imaging apparatus has a viewing angle of 360 degrees in the horizontal field of view and 180 degrees in the vertical field of view. The omnidirectional image is a circular or donut-shaped image, and the distortion increases as the distance from the center increases. The imaging device is provided at a high location of the shooting target location and performs shooting in a downward direction. The omnidirectional image acquisition unit 16 acquires an omnidirectional image captured by the imaging device in the form of a compressed image file. The omnidirectional image acquisition unit 16 receives an image file of an omnidirectional image by wireless communication, for example.

  The omnidirectional image decoding unit 17 decodes the image file of the omnidirectional image acquired by the omnidirectional image acquisition unit 16 and develops the omnidirectional image. The image queue 18 temporarily stores the omnidirectional image developed by the omnidirectional image decoding unit 17 and outputs it to the image correction sequencer 13 or the image display unit 19 as necessary. The image display unit 19 displays the intermediate image generated by the intermediate image generation unit 14, the distortion correction image generated by the distortion correction image generation unit 15, and the omnidirectional image developed by the code unit 17 as an omnidirectional image. Display using a monitor. The image display unit 19 corresponds to the display unit of the present invention.

  FIG. 2 is a block diagram showing a detailed configuration of the image correction sequencer. As shown in FIG. 2, the image correction sequencer 13 includes a panel operation interface unit 131, a timer control unit 132, a target image determination unit 133, a display image determination unit 134, a correction parameter calculation unit 135, and an image correction interface unit 136. ing.

  When there is a change in the panel operation, the panel operation interface unit 131 acquires the content of the change from the operation detection unit 12. The timer control unit 132 performs timer control for a display switching period until the final target image is displayed. When the display is switched from the omnidirectional image to the distortion corrected image, the distortion corrected image is the target image, and when the display is switched from the distortion corrected image to the omnidirectional image, the omnidirectional image is the target image. When there is a panel operation, the timer control unit 132 sets a timeout value according to the specific contents (operation position and strength) of the operation (tap, drag, flick, pinch out, pinch in, rotation). The intermediate image is displayed until the timeout occurs, and the target image is displayed after the timeout.

  The target image determination unit 133 determines the target image based on the content of the current panel operation. That is, the target image determination unit 133 sets the distortion corrected image as the target image when there is an operation input for switching the display to the distortion corrected image when the omnidirectional image is displayed. In this case, since the distortion-corrected image includes a panoramic image and a cutout image, the target image determination unit 133 determines whether the target image is a panoramic image or a cutout image. In addition, when the distortion correction image is displayed, the target image determination unit 133 sets the omnidirectional image as the target image when there is an operation input for switching the display to the omnidirectional image.

The display image determination unit 134 determines an image (an omnidirectional image, a distortion correction image, or an intermediate image) to be currently displayed according to timer control by the timer control unit 132. When the image to be displayed is an intermediate image, the correction parameter calculation unit 135 determines a correction parameter for the next intermediate image to be displayed according to the history of the operation input so far, the contents of the current operation input, and the timer value. The correction parameters include parameters related to center coordinates, rotation, and distortion. The image correction interface unit 136 calls an image generation process for each display image determined by the display image determination unit 134.

  Display switching performed using the viewer terminal 1 configured as described above will be described. Display switching includes (1) switching from an omnidirectional image to a panoramic image, (2) switching from a panoramic image to an omnidirectional image, (3) switching from an omnidirectional image to a multi-image including a cut-out image, ( 4) Switch from single panorama image to double panorama image, (5) Switch from double panorama image to single panorama image, (6) Scroll panorama image, (7) Change cutout range of cutout image, (8) All There is a rotation of the orientation image. Hereinafter, it demonstrates in order.

(1) Switching from Omnidirectional Image to Panoramic Image FIG. 3 is a diagram for explaining switching of display from an omnidirectional image to a panoramic image. In the present embodiment, the display portion of the viewer terminal 1 is rectangular, and an omnidirectional image that is originally circular is displayed with a part of the top and bottom cut off. Is also called an omnidirectional image. When an omnidirectional image is displayed as shown in FIG. 3A, the user first selects a base line (development line) when he / she wants to develop it into a panoramic image. In the selection of the base line, the user performs a drag operation in the circumferential direction from the center of the omnidirectional image, as shown in FIG. Next, as shown in FIG. 3C, a pinch-out operation is performed across the dragged line.

  The operation detection unit 12 detects the type of operation (drag and pinch out), speed, and position, and outputs them to the image correction sequencer 13. The operation panel interface unit 131 of the image correction sequencer 13 inputs the contents and position of this operation, and outputs them to the timer control unit 132 and the target image determination unit 133.

  The target image determination unit 133 determines that the operation is a change from the omnidirectional image to the panoramic image based on the currently displayed image (omnidirectional image) and the content and position of the operation. It is determined that the image is a panoramic image, and the result is output to the display image determination unit 134 and the timer control unit 132. The timer control unit 132 determines the time spent for the transition from the omnidirectional image to the panoramic image based on the speed at which the two fingertips are separated in the pinch-out operation, and starts the timer (that is, the faster the fingertip is separated, Shorten the transition time).

  When the timer value from the timer control unit 132 has not reached the timeout value, the display image determination unit 134 indicates that the image to be displayed is an intermediate image in order to display the intermediate image according to the timer value. The image correction interface 136 is notified, and the timeout value and the current timer value or the ratio of the current timer value to the timeout value are output to the correction parameter calculation unit 135.

  The correction parameter calculation unit 135 generates a correction parameter for generating an intermediate image according to the ratio of the current timer value to the timeout value, and outputs the correction parameter to the image correction interface unit 136. When the timer value reaches the timeout value, the display image determination unit 134 notifies the image correction interface unit 136 that the image to be displayed is the target image (that is, the panoramic image).

  When the image correction interface 136 receives a notification from the display image determination unit 134 that the image to be displayed is an intermediate image, the image correction interface 136 calls an intermediate image generation process by the intermediate image generation unit 14 and causes the intermediate image generation unit 14 to perform correction. The correction parameter calculated by the parameter calculation unit 135 is output. The intermediate image generation unit 14 corrects the omnidirectional image using the correction parameter input from the correction parameter calculation unit 135, generates an intermediate image, and outputs the intermediate image to the image display unit 19.

  The timer control unit 132 outputs a timer value to the display image determination unit 134 at a predetermined intermediate image frame rate, and the display image determination unit 134 determines whether or not the timer value has reached the timeout value each time. Until the value reaches the timeout value, the image correction interface 136 is notified that the image to be displayed is an intermediate image, and the correction parameter calculation unit calculates the timeout value and the current timer value or the ratio of the current timer value to the timeout value. It outputs to 135. Then, the correction parameter calculation unit 135 calculates a correction parameter for generating an intermediate image, and the intermediate image generation unit 14 generates an intermediate image using the correction parameter. Thus, an intermediate image is generated at the above-described intermediate image frame rate until timeout.

  FIGS. 3D and 3E show intermediate images generated as described above. The intermediate image is an image in the process of transition from a circular omnidirectional image to a rectangular panoramic image. The upper height is crushed downward, the horizontal direction is extended outward, the base line is cut, and the lower image is gradually lowered. Transition to erase half.

  As a result of the timer value output from the timer control unit 132 at the intermediate image frame rate, when the display image determination unit 134 determines that the timer value has timed out, the image correction interface 136 displays the display value from the display image determination unit 134. The distortion correction image generation unit 15 is notified that the image to be processed is the target image (that is, the distortion correction image). Upon receiving this notification, the distortion corrected image generation unit 15 corrects the omnidirectional image and generates a panoramic image without distortion as shown in FIG.

  The panoramic image may be displayed on the image display unit 19 in all directions with a horizontal angle of view of 360 degrees, or only a portion corresponding to a partial range of the horizontal angle of view of 360 degrees may be displayed. In the latter case, the panoramic image may be scrolled left and right by a scroll operation.

  In the above example, panorama development is performed by drawing the base line vertically downward from the center of the omnidirectional image. However, the present invention is not limited to this, and the base line may be drawn from the center of the omnidirectional image in any circumferential direction. . Also in this case, the intermediate image generation unit 14 and the distortion correction image generation unit 15 develop an omnidirectional image so as to be separated from the base line, and generate an intermediate image and a panoramic image.

(2) Switching from panoramic image to omnidirectional image FIG. 4 is a diagram for explaining switching of display from a panoramic image to an omnidirectional image. When a panoramic image is displayed as shown in FIG. 4A, when the user wants to change this to an omnidirectional image, the user performs a pinch-in operation in the left-right direction on the panoramic image. The operation detection unit 12 detects the type (pinch-in), speed, and position of this operation, and outputs them to the image correction sequencer 13. The operation panel interface unit 131 of the image correction sequencer 13 inputs the contents and position of this operation, and outputs them to the timer control unit 132 and the target image determination unit 133.

  The target image determination unit 133 determines that this operation is a change from a panoramic image to an omnidirectional image based on the currently displayed image (panoramic image) and the content and position of the operation. It is determined that the image is an orientation image, and the result is output to the display image determination unit 134 and the timer control unit 132. The timer control unit 132 determines the time spent for the transition from the panoramic image to the omnidirectional image based on the speed at which the two fingertips approach in the pinch-in operation, and starts the timer (that is, the faster the fingertip approaches, the faster the transition is) Shorten time).

The correction parameter calculation unit 135 generates a correction parameter for generating an intermediate image according to the ratio of the current timer value to the timeout value, and the image correction interface unit 1
To 36. When the timer value reaches the timeout value, the display image determination unit 134 notifies the image correction interface unit 136 to that effect.

  When the image correction interface 136 receives a notification from the display image determination unit 134 that the image to be displayed is an intermediate image, the image correction interface 136 calls an intermediate image generation process by the intermediate image generation unit 14 and causes the intermediate image generation unit 14 to perform correction. The correction parameter calculated by the parameter calculation unit 135 is output. The intermediate image generation unit 14 corrects the omnidirectional image using the correction parameter input from the correction parameter calculation unit 135, generates an intermediate image, and outputs the intermediate image to the image display unit 19.

  The timer control unit 132 outputs a timer value to the display image determination unit 134 at a predetermined intermediate image frame rate, and the display image determination unit 134 determines whether or not the timer value has reached the timeout value each time. Until the value reaches the timeout value, the image correction interface 136 is notified that the image to be displayed is an intermediate image, and the correction parameter calculation unit calculates the timeout value and the current timer value or the ratio of the current timer value to the timeout value. It outputs to 135. Then, the correction parameter calculation unit 135 calculates a correction parameter for generating an intermediate image, and the intermediate image generation unit 14 generates an intermediate image using the correction parameter. Thus, an intermediate image is generated at the above-described intermediate image frame rate until timeout.

  FIG. 4B shows an intermediate image generated as described above. The intermediate image transitions so that the panoramic image is stretched upward and the left-right direction is crushed inward.

  As a result of the timer value output from the timer control unit 132 at the intermediate image frame rate, when the display image determination unit 134 determines that the timer value has timed out, the image correction interface 136 displays the display value from the display image determination unit 134. The image queue 18 is notified that the image to be processed is the target image (that is, the omnidirectional image). Upon receiving this notification, the image queue 18 outputs an omnidirectional image to the image display unit 19, and the image display unit 19 displays the omnidirectional image as shown in FIG. At that time, the position pinched in on the panoramic image may correspond to the upper center of the omnidirectional image as the target image.

  FIG. 5 is a flow chart for switching between (1) and (2). The flow shown in FIG. 5 is repeatedly executed for each frame. The operation detection unit 12 monitors whether or not there is an operation on the panel interface 11 (step S51). If the operation is performed on the panel interface 11 (YES in step S51), the target image determination unit 133 of the image correction sequencer 13 is performed. Determines whether the target image is a panoramic image or an omnidirectional image (step S52).

  Then, the timer control unit 132 updates the timeout value (step S53), and the correction parameter calculation unit 135 generates a correction parameter (step S54). Then, the intermediate image generation unit 14 generates an intermediate image using the generated correction parameter.

  Next, when returning to step S51, since the operation for switching the display has already been completed, it is determined that there is no panel interface 11 operation (NO in step S51), and the timer value in the timer control unit times out. It is determined whether or not (step S56). If the timer value has not timed out (NO in step S56), the correction parameter calculation unit 135 calculates the correction parameter again using the timer value (step S54), and the intermediate image generation unit 14 performs intermediate processing. An image is generated (step S55).

When the above processing is repeated and it is determined in step S56 that the timer value has timed out (YES in step S56), display image determination unit 134 determines that the target image is a panorama according to the determination of target image determination unit 133. It is determined whether or not it is an image (step S57).
. If the target image is a panoramic image (YES in step S57), the distortion corrected image generation unit 15 performs distortion correction on the omnidirectional image to generate a panoramic image (step S58). If the target image is an omnidirectional image, the omnidirectional image is output from the image queue 18 without correction to the image display unit 19 and displayed on the image display unit 19 (step S59).

(3) Switching from an omnidirectional image to a multi-image including a cut-out image FIG. 6 is a diagram illustrating switching from an omnidirectional image to a multi-image including a cut-out image. When an omnidirectional image is displayed as shown in FIG. 6A, in the omnidirectional image, an area in which the difference from the previous frame is equal to or greater than a threshold is detected as a moving object in advance by a moving object detection unit (not shown). When the user wants to cut out one of the moving object areas and display the cut image, the user taps the area to be cut out as shown in FIG.

  The operation detection unit 12 detects the type (tap), speed, and position of this operation and outputs the detected type to the image correction sequencer 13. The image correction sequencer 13 cuts out the moving object region from the omnidirectional image, and the distortion correction image generation unit 15 performs distortion correction on the cut out region to generate a cutout image. As shown in FIG. 6C, the image display unit 19 displays the generated cut image superimposed on the omnidirectional image. At this time, the clipped image is superimposed on the omnidirectional image with the position overlapping the clipped region in the omnidirectional image as an initial position.

  As shown in FIG. 6C, the user can enlarge the clipped image by performing a pinch-out operation on the clipped image superimposed on the omnidirectional image. In addition, the user can reduce the cutout image by performing a pinch-in operation on the cutout image superimposed on the omnidirectional image.

  Furthermore, as shown in FIG. 6D, the user can also display a plurality of cut images superimposed on the omnidirectional image by generating cut images for a plurality of regions. When the cutout image is superimposed on the omnidirectional image and the user performs an operation of shaking (shaking) the viewer terminal 1, the superimposed cutout image is displayed side by side with the omnidirectional image. As shown in (f), a multi-screen is generated.

(4) Switching from Single Panorama Image to Double Panorama Image FIG. 7 is a diagram for explaining switching from a single panorama image to a double panorama image. When one panorama image is displayed (single panorama image), as shown in FIG. 7, by performing a pinch-out operation on this panorama image, two panoramic images on the upper and lower sides are displayed (double Panoramic image). In the double panorama image, the upper and lower panorama images may display panorama images having a horizontal angle of view of 180 degrees that do not overlap each other.

(5) Switching from Double Panorama Image to Single Panorama Image FIG. 8 is a diagram illustrating switching from a double panorama image to a single panorama image. As shown in FIG. 8 (a), when a double panorama image is displayed, when the user wants to change this to a single panorama image, the user touches the upper and lower panorama images one by one, The pinch-in operation is performed up and down so that these distances are close.

  As shown in FIG. 8 (b), the intermediate image correction unit 15 makes the two panoramic images overlap (the upper panoramic image is on the upper surface), and the upper panoramic image is on the lower side. An intermediate image is generated so that the panorama image moves upward, and finally, as shown in FIG. 8C, only the upper panorama image is displayed.

(6) Scrolling of panoramic image FIG. 9 is a diagram illustrating scrolling of a single panoramic image. When a panoramic image is displayed as shown in FIG. 9A, when the user wants to scroll the panoramic image in the horizontal direction, the user flicks in the direction in which the panoramic image is to be scrolled.

  In the example of FIG. 9A, since the panorama image is flicked to the right, the panorama image moves to the right as shown in FIG. 9B. After the flick operation, scrolling may be stopped by moving the panoramic image by an amount corresponding to the strength of the flick operation (the moving speed of the finger when the finger leaves the panel interface 11). If there is, scrolling may be continued at a constant speed. Furthermore, the moving speed of the panoramic image may be determined according to the strength of the flick operation.

  FIG. 10 is a diagram illustrating scrolling of the double panoramic image. When a double panorama image is displayed as shown in FIG. 9A, when the user wants to scroll the double panorama image in the horizontal direction, the user wants to scroll either the top or bottom of the double panorama image. Flick.

  In the example of FIG. 9A, since the upper panorama image is flicked in the right direction, as shown in FIG. 9B, the upper panorama image moves in the right direction. At this time, the lower panorama image also moves to the right according to the upper scroll. As in the case of the single panorama, after the flick operation, the upper and lower panorama images may be moved by an amount corresponding to the strength of the flick operation, or when there is a flick operation, at a constant speed. You may continue scrolling. Furthermore, the moving speed of the upper and lower panoramic images may be determined according to the strength of the flick operation.

  FIG. 11 is a diagram for explaining a modified example of scrolling a panoramic image. The scrolling of the panoramic image can be performed not only by the flick operation as described above but also by an operation of tilting the viewer terminal 1. As shown in FIG. 10A, when a panoramic image is displayed, the viewer terminal 1 is tilted so that the right side of the viewer terminal 1 is downward and the left side is upward. As shown, the panoramic image moves to the right. In addition, the panoramic image moves to the left side by tilting the viewer terminal 1 in the reverse direction, that is, the right side is upward and the left side is downward. In this modified example, the panoramic image moving speed may be varied according to the degree of inclination.

  The panorama image can also be scrolled by a drag operation. In this case, the panoramic image moves in the direction dragged by the drag operation by the dragged amount.

(7) Change of Cutout Range of Cutout Image FIG. 12 is a diagram for explaining change of the cutout range of the cutout image. As shown in FIG. 12A, when an omnidirectional image is displayed, the user wants to display a partial area of the omnidirectional image with high image quality (no distortion and high resolution). Perform a pinch out operation on the region. At this time, only the area detected as the moving object by the moving object detecting means may be the target of the pinch-out operation.

When a pinch-out operation is received, the distortion correction image generation unit 15 generates a cut-out image of the corresponding area, and the image display unit 19 displays the cut-out image on a full screen as shown in FIG. . When the moving object moves in the cut-out image and the user wants to move the area where the cut-out image is generated together with the moving object, as shown in FIG. Drag to. The distortion correction image generation unit 15 generates a new cut image by moving the cut region in the direction and amount according to the drag, and the image display unit 19 is continuously generated while being dragged in this way. To display the cropped image.

(8) Rotation of Omnidirectional Image FIG. 13 is a diagram for explaining rotation of the omnidirectional image. As shown in FIG. 13 (a), when the omnidirectional image is displayed, to rotate this omnidirectional image, as shown in FIG. 13 (b), two arbitrary points in the image are displayed. Touching with two fingers simultaneously, both or either one of the two fingers is moved on the panel interface 11 so that a line connecting the two points rotates in the screen.

  In accordance with such a rotation operation, the image display unit 19 changes the direction of the omnidirectional image to be displayed. Note that, by simultaneously flicking two fingers, the omnidirectional image may be continuously rotated even after the finger speaks from the panel interface 11.

  Various changes can be made to the switching of the display. Hereinafter, modified examples will be described.

(First modification)
FIG. 14 is a diagram illustrating a first modification. The first modified example is a modified example of switching from an omnidirectional image to a panoramic image. When the omnidirectional image is displayed as shown in FIG. 14, when the user wants to expand the panoramic image, the user touches the left and right end portions of the omnidirectional image respectively and spreads it outward. Perform a pinch out operation. When the monitor is small, this pinch-out operation can be performed with one hand (for example, thumb and index finger). When the monitor is large, the pinch-out operation can be performed using both hands as shown in FIG. Just do it.

  By the above pinch-out operation, the omnidirectional image is continuously developed into a panoramic image so as to be pulled outward. At this time, for example, an omnidirectional image can be developed into a panoramic image so that locations touched by two fingers are connected by a straight line and the center coordinates are positioned on a straight line extending in the vertical direction from the midpoint.

  Also in this first modification, the operation of pinching out the left and right ends of the omnidirectional image and the switching of the display in which the omnidirectional image is pulled outward to transition to a panoramic image are sensuously coincident. The user operability and visibility are improved.

(Second modification)
FIG. 15 is a diagram illustrating a second modification. The second modified example is a modified example of switching from an omnidirectional image to a panoramic image. When the omnidirectional image is displayed as shown in FIG. 15, when the user wants to expand the panoramic image, the user touches the upper and lower end portions of the omnidirectional image respectively and narrows it inward. Perform a pinch-in operation. When the monitor is small, this pinch-in operation can be performed with one hand (for example, thumb and index finger). When the monitor is large, as shown in FIG. 15, the pinch-in operation can be performed using both hands. Good.

  By the above pinch-in operation, the omnidirectional image is continuously developed into a panoramic image so as to be crushed inward. At this time, for example, an omnidirectional image can be developed into a panoramic image so that locations touched by two fingers are connected by a straight line and the center coordinates are positioned on the straight line.

  Also in this second modification, the operation of pinching in the upper and lower ends of the omnidirectional image and the switching of the display in which the omnidirectional image is crushed inward and transitioned to a panoramic image match sensuously, User operability and visibility are improved.

(Third Modification)
FIG. 16 is a diagram illustrating a third modification. The third modified example is a modified example of switching from an omnidirectional image to a panoramic image. When an omnidirectional image is displayed as shown in FIG. 16A, when the user wants to expand the omnidirectional image into a panoramic image, the center portion of the omnidirectional image is set to the left and right as shown in FIG. A drag operation is performed so as to make a cut in the direction, and then a pinch-out operation is performed with two fingers up and down so as to straddle the cut line as shown in FIG.

  Through the series of operations described above, the omnidirectional image is cut from the dragged line, and continuously developed into a panoramic image so as to form a donut shape. At this time, for example, the omnidirectional image can be developed into a panoramic image so that the center coordinates are positioned on a straight line extending in the vertical direction from the midpoint of the dragged line.

  According to this third modified example, the operation of cutting and pinching out the central portion of the omnidirectional image and the display switching of expanding the cut so that the omnidirectional image becomes a donut shape and changing to a panoramic image are performed. Are sensuously matched to improve user operability and visibility.

  In the above example, the center of the omnidirectional image is dragged in the horizontal direction to make a cut, and the pinch-out operation is performed up and down to open the cut. Drag the center part vertically to make a cut, and even if you perform a pinch-out operation left and right to open the cut, the omnidirectional image will be expanded to a panoramic image so that it becomes a donut shape You can do it.

(Fourth modification)
FIG. 17 is a diagram illustrating a fourth modification. The fourth modification is a modification of switching from an omnidirectional image to a panoramic image. In this modification, the display is switched from an omnidirectional image to a double panoramic image. When the omnidirectional image is displayed as shown in FIG. 17A, when the user wants to expand it into a double panoramic image, as shown in FIG. 17A, the horizontal direction of the omnidirectional image is displayed. Drag to make a notch in the diameter. And as shown in FIG.17 (b), pinch out operation is performed up and down with two fingers so that the cut line may be straddled.

  Through the series of operations described above, the omnidirectional image is panorama-developed in the upper and lower directions with the dragged line as a boundary, and a double panoramic image is displayed. At this time, for example, the omnidirectional image can be developed into upper and lower panoramic images so that the center coordinates are positioned on a straight line extending in the vertical direction from the midpoint of the dragged line.

  Also in this fourth modification, an operation of cutting and pinching out the central part of the omnidirectional image and an upper half and a lower half at the boundary of the omnidirectional image cut are expanded into a panoramic image respectively. The switching of the display of being arranged in a line is sensuously coincident, and the user operability and visibility are improved.

(Fifth modification)
18 and 19 are diagrams showing a fifth modification. The fifth modified example is a modified example of switching from an omnidirectional image. In this modification, the target image is switched according to the direction in which the pinch out is performed on the omnidirectional image. As shown in FIG. 18A, when a pinch-out operation is performed in the radial direction of the omnidirectional image, as shown in FIG. 18B, a multi-image including a cut-out image is set as a target image. The same operation as described above in “(3) Switching from omnidirectional image to multi-image including cut-out image” is performed. At this time, the person at the pinched out position is enlarged and displayed as a cutout image.

  On the other hand, as shown in FIG. 19A, when the pinch-out operation is performed in the circumferential direction of the omnidirectional image, the panoramic image is set as the target image as shown in FIG. The same operation as the switching to the panoramic image described in “(1) Switching from the omnidirectional image to the panoramic image” is performed. At this time, panorama development is performed using the line segment connecting the center of the pinched-out finger position and the center of the omnidirectional image as the base line.

(Sixth Modification)
20 and 21 are diagrams showing a sixth modification. The sixth modified example is a modified example of switching from a single panoramic image to a double panoramic image. In this modification, the intermediate image is switched according to the direction in which the pinch out is performed on the single panoramic image. As shown in FIG. 20A, when the single panorama image is vertically pinched out, as shown in FIG. 20B, “(4) from the single panorama image to the double panorama image. An intermediate image is generated and displayed in the same manner as described in “Switching”, and a double panoramic image shown in FIG.

  On the other hand, as shown in FIG. 21A, when a single panorama image is horizontally pinched out, as shown in FIG. 21B, a line connecting the positions of the fingers pinched out. The panoramic image is cut by a vertical line segment passing through the center of the minute, and as shown in FIG. 21C, an intermediate image in which one of the two cut panoramic images moves to the top and the other moves to the bottom. Generated and displayed, a double panoramic image shown in FIG. In addition, the part shown with the dotted line in FIG.21 (d) and FIG.21 (d) is a hidden part which is not displayed in an intermediate image. This hidden portion is gradually displayed as the intermediate image progresses.

  As described above, according to the viewer terminal 1 of the present embodiment, when an image is displayed, various display switching can be performed with a simple operation using an intuitive user interface using the panel interface 11. Can be performed. Further, according to the viewer terminal 1 of the present embodiment, an intermediate image is generated and displayed in the process of switching display, so that the user can easily recognize the correlation before and after the display is switched.

  Note that the operation of the viewer terminal 1 may be realized by executing a computer program by an arithmetic processing device provided in the viewer terminal 1.

  As described above, the present invention is capable of intuitively switching display between an omnidirectional image and a distortion-corrected image with a simple operation, and a display device that displays an image captured by an omnidirectional camera. As useful.

DESCRIPTION OF SYMBOLS 1 Viewer terminal 11 Panel interface 12 Operation detection part 13 Image correction sequencer 14 Intermediate image generation part 15 Distortion correction image generation part 16 Omnidirectional image acquisition part 17 Omnidirectional image decoding part 18 Image queue 19 Image display part 131 Panel operation interface part 132 Timer control unit 133 Target image determination unit 134 Display image determination unit 135 Correction parameter calculation unit 136 Image correction interface unit

Claims (14)

An image acquisition unit for acquiring an omnidirectional image;
A panoramic image generation unit that corrects distortion of the omnidirectional image to generate at least two panoramic images;
A display control unit for displaying the acquired omnidirectional image or the generated two panoramic images on a display unit;
An operation detection unit that detects an operation on the image displayed on the display unit,
When the operation detection unit detects a scroll operation on one of the two panoramic images, the display control unit scrolls both the two panoramic images. The image display device according to claim 1,
The image display device, wherein the two panoramic images are panoramic images having a horizontal angle of view of 180 degrees that do not overlap each other. The image display device according to claim 1,
When the operation detection unit detects the scroll operation with respect to one of the two panoramic images, the display control unit scrolls both the two panoramic images in the direction specified by the scroll operation. The image display device according to claim 1,
When the operation detection unit detects the scroll operation with respect to one of the two panoramic images, the display control unit scrolls both the two panoramic images at a speed corresponding to the scroll operation. The image display device according to claim 4,
The image display device, wherein a scroll speed of the two panoramic images changes according to the scroll operation. The image display device according to claim 4,
The image display device, wherein a scroll speed of the two panoramic images does not change according to the scroll operation. The image display device according to claim 1,
The display control unit is an image display device that hides one of the two panoramic images based on an operation detected by the operation detection unit when the two panoramic images are displayed. The image display device according to claim 7,
The display control unit, when one of the two panoramic images is not displayed, moves the other of the two panoramic images toward the center of the display unit. The image display device according to claim 7,
The display control unit redisplays both of the two panoramic images based on an operation detected by the operation detection unit when one of the two panoramic images is not displayed. The image display device according to claim 9,
The display control unit, when redisplaying the two panoramic images, moves the other of the two panoramic images upward or downward of the display unit. The image display device according to claim 1,
When the operation detection unit detects a pinch-in operation when displaying the two panoramic images, the display control unit hides one of the two panoramic images. The image display device according to claim 1,
The panoramic image generation unit corrects distortion from the omnidirectional image to generate one panoramic image,
The display control unit displays the two panoramic images when the operation detection unit detects a pinch-out operation when displaying the one panoramic image. The image display device according to claim 1,
The panoramic image generation unit corrects distortion from the omnidirectional image to generate one panoramic image,
The display control unit, when the operation detection unit detects a pinch-out operation when displaying the one panoramic image, divides the panoramic image into two panoramic images from a position based on the pinch-out operation. Acquiring an omnidirectional image,
Correcting the distortion from the acquired omnidirectional image to generate at least two panoramic images;
Displaying the generated two panoramic images on a display unit in response to a user operation;
An image display method comprising: scrolling both of the two panoramic images when a scroll operation is performed on one of the two panoramic images.

Images