JP2020039052A - Image processing system and HMD - Google Patents

Abstract

To provide an image processing system enabling an entire celestial sphere image data to be easily viewed over a wide range by mounting an HMD.SOLUTION: An image processing system 1 comprises : an omnidirectional camera 11; and a head mounted display (HMD) 13. The omnidirectional camera 11 comprises a transmission unit 111 transmitting image data of a photographed image. The HMD 13 comprises: a photographed image acquisition unit 130 acquiring the image data of the photographed image; an entire celestial sphere image data generation unit 135 generating entire celestial sphere image data on the basis of the acquired image data; an inclination sensor 133 detecting an amount of change in a position of the HMD 13; an image area selection unit 136 selecting an image area from the entire celestial sphere image data on the basis of the amount of change in position when the detected position change amount is within a predetermined range; and a display unit 134 displaying the selected image area. When the amount of change in the position detected by the inclination sensor 133 is larger than a predetermined value, the entire celestial sphere image data generation unit 135 generates the entire celestial sphere image data by rotating it a predetermined angle.SELECTED DRAWING: Figure 3

Description

  The present disclosure relates to an image processing system and an HMD that display an image captured by a spherical camera.

  2. Description of the Related Art In recent years, a celestial sphere camera capable of simultaneously capturing an image with a field angle of 360 ° using a plurality of image sensors has been used. The spherical camera captures an image with an angle of view of 180 ° or more with one image sensor by using a fisheye lens for the optical system, and combines the images captured with the two image sensors with a 360 ° angle of view. Record the image.

  An image obtained by the omnidirectional camera is converted into omnidirectional image data by synthesizing images obtained simultaneously by a plurality of image sensors, and a part of the omnidirectional image data is displayed on the display. When displaying an image with an angle of view of 360 ° based on the spherical image data, for example, a head-mounted display (HMD) is used as a display. An acceleration sensor is attached to the HMD, and an image in the direction in which the user is facing out of the spherical image data is partially displayed on the display.

  If the vertical direction of the omnidirectional camera is reversed or the camera shakes in the vertical direction during imaging, the user viewing the image captured by the omnidirectional camera may feel sick. In order to cope with this problem, Patent Document 1 discloses a technique for generating moving image data whose inclination has been corrected in accordance with the inclination of the imaging unit with respect to the detected reference direction.

JP 2018-67964 A

  By the way, conventionally, it has been practiced to obtain an image in a narrow space where humans cannot enter by using an endoscope apparatus, and observe the inside. The endoscope apparatus is frequently used by doctors to observe the inside of a patient's body. However, a wide field of view cannot be obtained during an operation with an imaging device provided at the end of the current endoscope.

  Then, in order to obtain a wider field of view, it is conceivable to attach the above spherical camera to the distal end of the endoscope and insert it into the patient's body. When using an endoscope device, another person different from the surgeon operates the endoscope device and displays an image on a monitor. However, if the surgeon wears the HMD, the surgeon himself / herself determines the display location. Since it can be changed, the operation can be performed intuitively, and the operability of the endoscope apparatus is improved.

  Also, unlike conventional endoscope devices, if a spherical camera is attached to the end of the endoscope device, not only the insertion direction of the endoscope but also a field of view opposite to the insertion direction can be obtained, In order to confirm the image in the opposite direction, the operator wearing the HMD needs to face in the opposite direction to the front. However, it is difficult for the surgeon to face the front (the direction in which the patient is viewed) while wearing the HMD, and it is difficult to say that 360 ° spherical image data can be easily visually recognized. .

  The present disclosure has been made in view of the above points, and provides a technique in which the HMD is attached to allow easy viewing of omnidirectional image data over a wide range.

  In order to solve the above problem, a display system that includes a spherical camera and a head-mounted display (HMD) and displays spherical image data, wherein the spherical camera converts image data of a captured image. A transmitting unit for transmitting, the HMD includes: a captured image acquiring unit that acquires the image data transmitted by the spherical camera; and the spherical image based on the image data acquired by the captured image acquiring unit. A celestial sphere image data generating unit for generating data, a tilt sensor for detecting a change amount of the position of the HMD, and the position sensor when the change amount of the position detected by the tilt sensor is within a predetermined range. An image area selecting unit that selects an image area from the celestial sphere image data based on the amount of change, and a display unit that displays the image area selected by the image area selecting unit. A spherical image data generating unit that provides an image processing system that generates the spherical image data by rotating the spherical image data by a predetermined angle when a change amount of the position detected by the tilt sensor is larger than a predetermined value. I do.

  A head-mounted display (HMD) for displaying spherical image data, wherein the captured image acquiring unit acquires image data transmitted by the spherical camera, and a head-mounted display (HMD) based on the image data acquired by the captured image acquiring unit. A celestial sphere image data generating unit for generating the celestial sphere image data, a tilt sensor for detecting a change amount of the position of the HMD, and a change amount of the position detected by the tilt sensor within a predetermined range. In some cases, an image area selecting unit that selects an image area from the spherical image data based on the amount of change in the position, and a display unit that displays the image area selected by the image area selecting unit. The celestial sphere image data generation unit generates the celestial sphere image data by rotating the celestial sphere image data by a predetermined angle when the amount of change in the position detected by the tilt sensor is larger than a predetermined value. To provide a HMD.

  A display system that includes a spherical camera, an image processing device, and a head-mounted display (HMD) and displays spherical image data, wherein the spherical camera transmits image data of a captured image. A transmitting unit, wherein the image processing device is a photographed image acquiring unit for acquiring the image data transmitted by the spherical camera, and the spherical image based on the image data acquired by the photographed image acquiring unit A celestial sphere image data generating unit that generates data, a position change amount obtaining unit that obtains a change amount of the position of the HMD, and a position change amount obtained by the position change amount obtaining unit is within a predetermined range. In some cases, an image area selection unit that selects an image area from the spherical image data based on the amount of change in the position, and an image data of the image area selected by the image area selection unit is stored in the HM. An image data transmission unit for transmitting to the celestial sphere image data generation unit, wherein the celestial sphere image data generation unit, when the amount of change in the position acquired by the position change amount acquisition unit is greater than a predetermined value, the celestial sphere The image data is generated by rotating the image data by a predetermined angle, and the HMD transmits a tilt sensor for detecting a change amount of the position of the HMD and the change amount of the position detected by the tilt sensor to the image processing device. An image processing system comprising: a position change amount transmission unit, a display image acquisition unit that acquires the image data transmitted by the image processing device, and a display unit that displays the image data acquired by the display image acquisition unit. provide.

  According to the present disclosure, the omnidirectional image data can be easily visually recognized over a wide range by wearing the HMD.

FIG. 4 is a diagram for explaining a relationship between spherical image data generated by shooting with a spherical camera and a video visually recognized on a head mounted display (HMD). FIG. 1 is a diagram conceptually illustrating a configuration of an image processing system according to a first embodiment. FIG. 1 is a block diagram schematically illustrating a configuration of an image processing system. FIG. 4 is a diagram for explaining a change amount of a position of an HMD. FIG. 9 is a diagram illustrating an example in which spherical image data is rotated. 5 is a flowchart illustrating a flow of processing of the image processing system. It is a block diagram showing roughly the composition of the image processing system of a 2nd embodiment. 5 is a flowchart illustrating a flow of processing of the image processing system.

[Overview]
FIG. 1 is a diagram for explaining the relationship between spherical image data generated by shooting with a spherical camera and a video visually recognized on a head-mounted display (hereinafter, HMD). FIGS. 1A and 1B show the relationship between the image displayed on the HMD and the viewing direction of the user wearing the HMD.

  In FIGS. 1A and 1B, the hatched image area of the spherical image data is the image displayed on the monitor of the HMD when the user wearing the HMD is facing the direction of the arrow. is there.

  The user can freely select (set an image area) which image area is to be displayed on the HMD when the user is facing, for example, by pressing a button provided on the HMD. it can. In the initial setting, for example, an image area in the front direction of one of the two image sensors provided in the spherical camera is displayed on the HMD. In the example shown in FIG. 1A, the setting is made so as to display the shaded image area when the user is facing the direction of the arrow.

  In FIG. 1B, an image area displayed on the HMD when the user is facing leftward by an angle θ from the reference direction is indicated by oblique lines. As shown in FIG. 1B, when the user changes the line-of-sight direction while wearing the HMD, an amount of change in the position of the HMD is detected by a tilt sensor provided in the HMD, and an image displayed on the HMD is displayed. The area is changed. That is, the user can visually recognize an image in an arbitrary direction in the spherical image data by changing the direction of the face.

  For example, if you shoot the inside of a thin tube with a spherical camera, you can check the inside of the tube in the opposite direction to the insertion direction even if you can not reverse the insertion direction of the spherical camera in the tube It is useful that the user can easily check all the spherical image data. However, when the HMD is used as a display for performing a specific operation such as an operation, the user turns only to the front and rarely looks back.

  Therefore, when the user changes the viewing direction by more than a predetermined angle (when the amount of change in the position of the HMD is larger than a predetermined value), the image processing system of the present disclosure converts the spherical image data to a predetermined angle. Rotate just to generate. That is, when the user changes the viewing direction by more than a predetermined angle, the image area displayed on the HMD by the initial setting is changed. If the amount of change in the position of the HMD is within a predetermined range, the image area displayed on the HMD is changed based on the changed image area.

  In this way, the user can visually recognize most of the celestial sphere image data without largely changing the body direction. Further, even when the HMD acquires the image data by wire, the user does not need to rotate the body direction, so that the communication cable can be prevented from being entangled. Hereinafter, some embodiments of the present disclosure will be described with reference to the drawings.

<First embodiment>
FIG. 2 is a conceptual diagram illustrating a configuration example of an external appearance of the image processing system 1 according to the first embodiment. The image processing system 1 includes an endoscope insertion unit 12 provided with a spherical camera 11 at the tip and an HMD 13. The image processing system 1 may be considered as an endoscope system itself. In FIG. 2, the endoscope insertion section 12 and the HMD 13 are connected by wire, but the endoscope insertion section 12 and the HMD 13 may be configured to be able to communicate wirelessly.

  The omnidirectional camera 11 can simultaneously capture an image in a 360 ° direction by an imaging system (11A and 11B) in which a fisheye lens and an imaging element are combined. Further, the omnidirectional camera 11 has a tilt detection sensor (for example, a gyro sensor) not shown, and performs tilt correction of an image displayed on the HMD by, for example, a technique described in JP-A-2018-67964. Is also good.

  FIG. 3 is a block diagram schematically illustrating a configuration of the image processing system 1. As described above, the image processing system 1 includes the omnidirectional camera 11 and the HMD 13.

  The spherical camera 11 includes an imaging system 110 and a transmission unit 111. The transmission unit 111 transmits image data of an image captured by the imaging system 110 (an imaging system combining the two imaging systems 11A and 11B) to the HMD 13 by wire or Transmit by radio.

  The HMD 13 includes a captured image acquisition unit 130, a recording unit 131, a control unit 132, an inclination sensor 133 for detecting a change in the position of the HMD 13, and a display unit 134 for displaying image data transmitted by the control unit 132. The tilt sensor 133 is, for example, a gyro sensor, a magnetic sensor, or an acceleration sensor. The display unit 134 is, for example, an organic EL display.

  Here, the amount of change in the position of the HMD 13 means, for example, the amount of change in two angles (θ, φ) in three-dimensional polar coordinates. The amount may be considered to be zero.

  FIG. 4 is a diagram exemplarily showing the amount of change in the position of the HMD 13. FIG. 4A shows that the position coordinates of the HMD 13 when the user faces the x-axis direction (horizontal direction of z = 0) are (θ = 0 °, φ = 90 °). FIG. 4B shows that the position coordinates of the HMD 13 when the user is pointing left by 60 ° in the horizontal direction and downward by 30 ° in the vertical direction are (θ = 60 °, φ = 120 °). It is shown that.

  The captured image acquisition unit 130 is, for example, a communication interface for exchanging data with the omnidirectional camera 11, and acquires the image data transmitted by the omnidirectional camera 11 by wire or wirelessly. The captured image acquisition unit 130 transmits the acquired image data to the control unit 132.

  The recording unit 131 includes, for example, a ROM (Read Only Memory) in which various programs are recorded, a RAM (Random Access Memory) in which data is temporarily recorded, and an SSD (Solid State) in which an image captured by the spherical camera 11 is recorded. Drive). In the ROM, for example, a program for generating omnidirectional image data based on image data of an image captured by the omnidirectional camera 11 is recorded. Further, the ROM stores a program for selecting an image area from the spherical image data when the amount of change in the position of the HMD 13 is detected.

  The control unit 132 is, for example, a CPU (Central Processing Unit), and functions as an omnidirectional image data generation unit 135 and an image area selection unit 136 by executing a program recorded in the recording unit 131.

  The omnidirectional image data generating section 135 generates omnidirectional image data based on the image data acquired by the captured image acquiring section 130, for example, by a known method.

  The image region selection unit 136 selects an image region to be displayed on the display unit 134 from the omnidirectional image data generated by the omnidirectional image data generation unit 135. In the initial setting, the image area selection unit 136 selects, for example, an image area in the zenith direction of the omnidirectional image data or an image area in the front direction (the direction of the center axis of the fisheye lens) of the image sensor included in the imaging system 11A. The zenith direction of the spherical image data is also the insertion direction of the endoscope insertion section 12.

  When the amount of change in the position of the HMD 13 detected by the tilt sensor 133 is within a predetermined range, the image region selection unit 136 selects an image region from the spherical image data based on the amount of change in the position. As described with reference to FIG. 1, for example, when the user wearing the HMD 13 shifts the face direction in the horizontal direction by an angle θ (−80 ° ≦ θ ≦ 80 °), the spherical image data The image area selection unit 136 selects an image area at a position shifted from the initially set image area by an angle θ (−80 ° to 80 °) in the horizontal direction. Then, the display unit 134 displays the image area selected by the image area selection unit 136. Here, the definition of the angle means that a positive rotation angle means a counterclockwise rotation and a negative rotation angle means a clockwise rotation, but which direction is positive is arbitrary.

  Furthermore, when the amount of change in the position of the HMD 13 detected by the tilt sensor 133 is greater than a predetermined value, the image processing system 1 of the present disclosure causes the spherical image data generation unit 135 to convert the spherical image data to a predetermined position. It is generated by rotating by an angle of.

  Specifically, for example, when the user wearing the HMD 13 rotates the face direction (leftward or rightward) by more than 80 ° from directly in front, the omnidirectional image data generating unit 135 The celestial sphere image data is generated by rotating the celestial sphere image data by 30 °, 90 °, or 180 ° in the horizontal direction (that is, the celestial sphere image data is rotated by 30 °, 90 °, or 180 ° about the vertical direction).

  At this time, since the initially set image area is shifted by an amount corresponding to the rotation of the omnidirectional image data, the image area selected by the image area selecting section 136 is selected from the initially generated omnidirectional image data. As compared with the case, an image area shifted by the rotation angle is selected. In other words, an image area shifted by the rotation angle is displayed on the display unit 134 of the HMD 13.

  FIG. 5 is a diagram illustrating an example in which the spherical image data is rotated. In FIG. 5, when the user wearing the HMD 13 rotates the face direction to the left by more than 90 ° from directly in front, the omnidirectional image data generation unit 135 converts the omnidirectional image data by 180 ° horizontally. An example is shown in which the image is generated by rotating the celestial sphere image data by rotating the celestial sphere image data 180 degrees around the vertical direction. In FIG. 5, the image area displayed on the display unit 134 of the display unit of the HMD 13 is converted from a dotted area to a hatched area.

  According to the above, the user can visually recognize a wide range of the spherical image data without keeping the face facing backward. In particular, since the image area displayed on the display unit 134 can be changed only by changing the direction of the face, it is possible to easily change the image to be visually recognized even in a situation in which both hands are not free to hold the appliance.

  Further, the omnidirectional image data generation unit 135 determines that the amount of change in the position of the HMD 13 detected by the tilt sensor 133 is greater than a predetermined value (first shake width), and that the change in the position of the HMD 13 If the amount of change in the position of the HMD 13 is smaller than a predetermined value (second shake width) for more than the time period, the spherical image data may be generated by rotating the spherical image data by a predetermined angle.

  Specifically, for example, the omnidirectional image data generation unit 135 determines that the user wearing the HMD 13 rotates the face direction by at least 80 ° from directly in front, and the HMD 13 maintains the state for 2 seconds or more. When the time has elapsed, the spherical image data may be generated by rotating the spherical image data horizontally by 30 °, 90 °, or 180 °.

  In this way, if the user accidentally changes the position of the HMD 13, the omnidirectional image data is prevented from rotating, and an image in a desired range can be displayed on the HMD 13.

[Processing flow of image processing system]
FIG. 6 is a flowchart illustrating a flow of processing of the image processing system 1. The operation of the image processing system 1 in each step of FIG. 6 will be described below.
(S501)
An imaging system 110 of the spherical camera 11 captures an image.
(S502)
The transmission unit 111 of the omnidirectional camera 11 transmits image data captured by the imaging system 110 to the HMD 13.
(S503)
The captured image acquisition unit 130 of the HMD 13 acquires image data captured by the spherical camera 11.
(S504)
The spherical image data generator 135 of the HMD 13 generates spherical image data.
(S505)
The image area selection unit 136 of the HMD 13 selects the image area of the initial setting and displays the image data on the display unit 134. For example, the image area selection unit 136 selects an image area corresponding to the center axis direction of the fisheye lens provided in the imaging system 11A of the spherical camera 11.
(S506)
The control unit 132 of the HMD 13 acquires the amount of change in the position of the HMD 13 from the tilt sensor 133.
(S507)
If the amount of change in the position of the HMD 13 is larger than the predetermined value, the process proceeds to S508. Otherwise, the process proceeds to S509.
(S508)
The celestial sphere image data generating unit 135 generates the celestial sphere image data by rotating it by a predetermined angle. (At this time, the default image area selected by the image area selection unit 136 is an image area shifted by the predetermined angle.)
(S509)
The image region selection unit 136 selects an image region from the spherical image data based on the amount of change in the position of the HMD 13. For example, when the polar coordinates of the HMD 13 change from (θ = 0 °, φ = 0 °) to (θ = 60 °, φ = 30 °), the image area selecting unit 136 determines the coordinate position from the spherical image data. Is selected.
(S510)
The image of the image area selected by the image area selection unit 136 is displayed on the display unit.

[Advantages of Image Processing System 1]
The image processing system 1 having the above configuration allows the user to visually recognize most of the celestial sphere image data without largely changing the body orientation. Further, even when the HMD 13 acquires image data by wire, the user does not need to rotate the body direction, so that it is possible to prevent the communication cable from being entangled.

<Second embodiment>
Next, an image processing system according to a second embodiment will be described. High computational power is required to generate celestial sphere image data in real time from images captured by the celestial sphere camera and to select an image area in accordance with changes in the position of the HMD.

  Therefore, in the image processing system of the second embodiment, unlike the image processing system 1 of the first embodiment, generation of omnidirectional image data and selection of an image area are not performed by the HMD, and high calculation performance is achieved. This is performed by an image processing apparatus having the function. Then, the HMD transmits the amount of change in the position to the image processing device, and displays the image transmitted by the image processing device on the display unit.

[Configuration of Image Processing System]
FIG. 7 is a block diagram schematically illustrating a configuration of the image processing system 2 according to the second embodiment. The image processing system 2 includes a spherical camera 11, an image processing device 14, and an HMD 15. The image processing device 14 is, for example, a desktop computer including a high-performance GPU (Graphic Processing Unit). The omnidirectional camera 11 has the same configuration as that of the first embodiment, except that the transmitting unit 111 transmits image data of an image captured by the imaging system 110 to the image processing device 14 instead of the HMD 13.

  The image processing device 14 includes a captured image acquisition unit 140, a recording unit 141, a control unit 142, a position change amount acquisition unit 143, and an image data transmission unit 144. The captured image acquisition unit 140 is, for example, a communication interface that exchanges data with the omnidirectional camera 11, and acquires image data transmitted by the omnidirectional camera 11 by wire or wirelessly. The captured image acquisition unit 140 transmits the acquired image data to the control unit 142.

  The recording unit 141 includes, for example, a ROM (Read Only Memory) in which various programs are recorded, a RAM (Random Access Memory) in which data is temporarily recorded, and an SSD (Solid State) in which images captured by the spherical camera 11 are recorded. Drive). In the ROM, for example, a program for generating omnidirectional image data based on image data of an image captured by the omnidirectional camera 11 is recorded. Further, the ROM stores a program for selecting an image area from the spherical image data when the amount of change in the position of the HMD 15 is detected.

  The control unit 142 is, for example, a CPU (Central Processing Unit), and functions as an omnidirectional image data generation unit 145 and an image area selection unit 146 by executing a program recorded in the recording unit 141. The functions of the celestial sphere image data generation unit 145 and the image area selection unit 146 are the same as those of the image processing system 1 according to the first embodiment, and a description thereof will be omitted.

  The position change amount acquisition unit 143 and the image data transmission unit 144 are each a communication interface for transmitting and receiving data to and from the HMD 15. The position change amount acquisition unit 143 acquires the change amount of the position of the HMD 15 transmitted by the HMD 15. Further, the image data transmission unit 144 transmits image data to be displayed on the display unit 154 of the HMD 15.

  The HMD 15 includes an inclination sensor 151, a position change amount transmission unit 152, a display image acquisition unit 153, and a display unit 154. The HMD 15 according to the second embodiment includes a position change amount transmission unit 152 that transmits a change amount of the position of the HMD 15 to the image processing device 14, and a display image acquisition unit 153 that receives image data transmitted by the image processing device 14. This is different from the first embodiment.

[Processing flow of image processing system]
FIG. 8 is a flowchart illustrating a flow of processing of the image processing system 2. The operation of the image processing system 2 in each step of FIG. 8 will be described below.
(S701)
The imaging system 110 of the spherical camera 11 captures an image.
(S702)
The image data captured by the transmission unit 111 of the spherical camera 11 is transmitted to the image processing device 14.
(S703)
The captured image acquisition unit 140 of the image processing device 14 acquires an image captured by the spherical camera 11.
(S704)
The omnidirectional image data generator 145 of the image processing device 14 generates omnidirectional image data.
(S705)
The image area selection unit 146 of the image processing device 14 selects an image area of the initial setting and transmits image data to the HMD 15. For example, the image area selection unit 136 selects an image area corresponding to the center axis direction of the fisheye lens provided in the imaging system 11A of the spherical camera 11.
(S706)
The display image acquisition unit 153 of the HMD 15 acquires image data and displays an image on the display unit 154.
(S707)
The position change transmitting unit 15 of the HMD 15 transmits the amount of change in the position of the HMD 15 detected by the tilt sensor 151 to the image processing device 14.
(S708)
The position change amount acquisition unit 143 of the image processing device 14 acquires the position change amount of the HMD 15.
(S709)
If the amount of change in the position of the HMD 15 is larger than a predetermined value, the process proceeds to S710. Otherwise, the process proceeds to S711.
(S710)
The spherical image data generation unit 145 of the image processing device 14 rotates the spherical image data by a predetermined angle to generate the spherical image data. (At this time, the initially set image area selected by the image area selection unit 146 is an image area shifted by the predetermined angle.)
(S711)
The image area selection unit 146 of the image processing device 14 selects an image area from the spherical image data based on the amount of change in the position of the HMD 15.
(S712)
The image data transmission unit 144 of the image processing device 14 transmits an image to the HMD 15 (S713).
The display image acquisition unit 153 of the HMD 15 acquires the image data, and displays the image on the display unit 154 of the HMD 15.

[Advantages of Image Processing System]
The image processing system 2 having the above configuration has the same advantages as the image processing system 1 of the first embodiment, and also performs image processing with the image processing device 14 having high calculation capability. The delay of a moving image can be prevented. Further, since the calculation processing in the HMD 15 is reduced, it is possible to prevent the case of the HMD 15 from becoming hot.

  Note that the present disclosure is not limited to the above-described embodiment, and includes various modifications. For example, in the above description, when the amount of change in the position of the HMD 13 detected by the tilt sensor 133 is larger than a predetermined value, the spherical image data generation unit 135 rotates the spherical image data by a predetermined angle. Generated. Alternatively, the position of the image area selected by the image area selection unit 136 may be changed by the same angle while the spherical image data is kept fixed.

  In addition, instead of rotating the spherical image data by increasing the amount of change in the position of the HMD 13 beyond a predetermined value, a speech recognition device is attached to the HMD 13 or the image processing device 14 and, for example, “30 ° rotation by the user” The celestial sphere image data may be rotated by causing the voice recognition device to recognize the voice of "".

  Further, in the above description, the omnidirectional camera 11 has the tilt detection sensor, and corrects the tilt of the image displayed on the HMD 13 by a known technique, for example. Here, when attaching the omnidirectional camera 11 to the end of the endoscope insertion portion, it is preferable that the omnidirectional camera 11 be as small as possible. Therefore, for example, the omnidirectional camera 11 may be downsized by providing the tilt sensor outside the endoscope insertion section.

  The tilt sensor may be attached to, for example, a trocar that fixes the endoscope insertion section. Since the trocar moves integrally with the endoscope insertion section, the movement of the omnidirectional camera 11 and the movement of the trocar match. For this reason, a tilt detection sensor, a communication unit, and a power supply are attached to the trocar, and a change in the attitude of the trocar detected by the trocar tilt detection sensor (change in the attitude of the spherical camera 11) is transmitted to the HMD 13 or the image processing apparatus 14. Alternatively, the inclination of the spherical image data may be corrected.

  Further, the image area selection units (136 and 146) may superimpose information indicating the position of the selected image area in the spherical image data on the image data of the selected image area. For example, the image region selection units (136 and 146) may superimpose an image in which the selected image region is highlighted in the spherical image data or polar coordinates indicating the position of the selected image region. Alternatively, the image area selection unit (136 and 146) superimposes information indicating how many times the celestial sphere image data has been rotated compared to the initial state on the image data of the selected image area. Is also good.

1, 2 image processing system 11 spherical camera 110 imaging system 111 transmitting unit 12 endoscope insertion unit 13, 15 HMD
130, 140 ... photographed image acquisition units 131, 141 ... recording units 132, 142 ... control units 133, 151 ... inclination sensors 134, 154 ... display units 136, 146 ... image area selection units 152 ... position change amount transmission units 133 ... display Image acquisition unit 14: Image processing device 143: Position change amount acquisition unit 144: Image data transmission unit

Claims (9)

A display system that includes a spherical camera and a head-mounted display (HMD) and displays spherical image data,
The spherical camera includes a transmission unit that transmits image data of a captured image,
The HMD is:
A captured image acquisition unit that acquires the image data transmitted by the spherical camera,
An omnidirectional image data generating unit that generates the omnidirectional image data based on the image data obtained by the captured image obtaining unit;
An inclination sensor for detecting an amount of change in the position of the HMD;
When the amount of change in the position detected by the tilt sensor is within a predetermined range, an image area selection unit that selects an image area from the spherical image data based on the amount of change in the position,
A display unit that displays the image area selected by the image area selection unit,
With
The celestial sphere image data generating unit generates the celestial sphere image data by rotating the celestial sphere image data by a predetermined angle when the amount of change in the position detected by the tilt sensor is larger than a predetermined value.
Image processing system. The celestial sphere image data generating unit generates the celestial sphere image data by rotating the celestial sphere image data by 180 ° about the vertical direction when the amount of change in the position detected by the tilt sensor is greater than a predetermined value. ,
The image processing system according to claim 1. The celestial sphere image data generation unit is configured to determine that the change amount of the position detected by the tilt sensor is larger than a predetermined value, and that the change of the position of the HMD is performed for a predetermined time or more after the position of the HMD changes. When the amount is smaller than a predetermined value, the spherical image data is generated by rotating the spherical image data by a predetermined angle,
The image processing system according to claim 1. A head mounted display (HMD) for displaying spherical image data,
A captured image acquisition unit for acquiring image data transmitted by the spherical camera,
A celestial sphere image data generation unit that generates the celestial sphere image data based on the image data acquired by the captured image acquisition unit,
An inclination sensor for detecting an amount of change in the position of the HMD;
When the amount of change in the position detected by the tilt sensor is within a predetermined range, an image area selection unit that selects an image area from the spherical image data based on the amount of change in the position,
A display unit that displays the image area selected by the image area selection unit,
With
The celestial sphere image data generating unit generates the celestial sphere image data by rotating the celestial sphere image data by a predetermined angle when the amount of change in the position detected by the tilt sensor is larger than a predetermined value.
HMD. The celestial sphere image data generating unit generates the celestial sphere image data by rotating the celestial sphere image data by 180 ° about the vertical direction when the amount of change in the position detected by the tilt sensor is greater than a predetermined value. ,
The HMD according to claim 4. The celestial sphere image data generation unit is configured to determine that the change amount of the position detected by the tilt sensor is larger than a predetermined value, and that the change of the position of the HMD is performed for a predetermined time or more after the position of the HMD changes. When the amount is smaller than a predetermined value, the spherical image data is generated by rotating the spherical image data by a predetermined angle,
The HMD according to claim 4. A display system including a spherical camera, an image processing device, and a head-mounted display (HMD), and displaying spherical image data,
The spherical camera includes a transmission unit that transmits image data of a captured image,
The image processing device,
A captured image acquisition unit that acquires the image data transmitted by the spherical camera,
An omnidirectional image data generating unit that generates the omnidirectional image data based on the image data obtained by the captured image obtaining unit;
A position change amount obtaining unit that obtains a change amount of the position of the HMD;
When the position change amount acquired by the position change amount acquisition unit is within a predetermined range, an image region selection unit that selects an image region from the spherical image data based on the position change amount,
An image data transmitting unit that transmits image data of the image region selected by the image region selecting unit to the HMD;
With
The celestial sphere image data generation unit generates the celestial sphere image data by rotating the celestial sphere image data by a predetermined angle when the change amount of the position acquired by the position change amount acquisition unit is larger than a predetermined value. ,
The HMD is:
An inclination sensor for detecting an amount of change in the position of the HMD;
A position change amount transmission unit that transmits the change amount of the position detected by the tilt sensor to the image processing device,
A display image acquisition unit that acquires the image data transmitted by the image processing device,
A display unit that displays the image data acquired by the display image acquisition unit,
An image processing system comprising: The celestial sphere image data generating unit rotates the celestial sphere image data by 180 ° around the vertical direction when the amount of change in the position obtained by the position change amount obtaining unit is larger than a predetermined value. Generate
The image processing system according to claim 7. The celestial sphere image data generation unit is configured such that the change amount of the position acquired by the position change amount acquisition unit is greater than a predetermined value, and that the HMD changes over a predetermined time after the change of the position of the HMD. When the amount of change in the position is smaller than a predetermined value, the spherical image data is generated by rotating the spherical image data by a predetermined angle.
The image processing system according to claim 7.

Images