JP2018157578A - Video display system, video display device and video display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enhance visibility of video by dynamically adjusting brightness of the video in accordance with a line-of-sight direction of a user, to increase a dynamic range for imaging, in a pseudo manner, in viewing a video captured by a panoramic camera on a head mounted display.SOLUTION: In a video display system, a video processing device for holding multiple pieces of video data captured by plural cameras and a video display device to be worn by a user are connected to each other. The video processing device generates panoramic video data having a range of 360 degrees around an installation area of the cameras by using the video data, and transmits the generated panoramic video data to the video display device. The video display device detects a line-of-sight direction of the user, receives the transmitted panoramic video data, extracts video data in a predetermined range containing a detection result of a sensor from the panoramic video data, adjusts brightness of the extracted predetermined range of the video data to brightness in a certain range, and displays the adjusted predetermined range of video data.SELECTED DRAWING: Figure 4

Description

  The present invention relates to a video display system, a video display apparatus, and a video display method for displaying captured video data in a omnidirectional direction.

  In recent years, cameras capable of capturing images in the omnidirectional direction (that is, a range of 360 degrees) (hereinafter referred to as omnidirectional cameras) have begun to appear. Since the omnidirectional camera can capture a wide range of images, it is often captured outdoors, for example, on the assumption of use. The image is mixed with the image in the backlight direction of sunlight, and the brightness of the imaged image is not uniform regardless of the direction. For this reason, the approach of raising the dynamic range of the camera which images the image | video (namely, image | video with low brightness) of the backlight direction of sunlight can be considered.

  Here, as a prior art for displaying an image captured by a camera on a separate display device (for example, a head-mounted display) in order to effectively use the dynamic range of the camera at the time of image capturing, for example, Patent Document 1 It has been known.

  In the display device disclosed in Patent Document 1, a plurality of display units of LCD (Liquid Crystal Display) are normally illuminated by a main backlight. A photometric information signal together with an image signal at the time of shooting is input to the control unit of the display device, and each drive circuit is controlled by the control unit based on the photometric information, and a plurality of auxiliary backlights, a plurality of auxiliary flashes, a plurality of auxiliary The lamp is driven to emit light, and a plurality of LCDs are supplementarily illuminated. As a result, the sense of reality caused by the change in the brightness of the subject at the time of shooting is not impaired.

Japanese Patent Laid-Open No. 7-294876

  However, when the user wants to view the image captured by the omnidirectional camera on a head-mounted display (for example, HMD: Head Mount Display), in the configuration of Patent Document 1, the LCD is supplementarily supported by the backlight. It is only illuminated. For this reason, if the original image is dark, such as an image in the backlight direction of sunlight, the image is not clearly displayed even if the screen on which the image is displayed is brightened with the backlight. There has been a problem that the deteriorated state is maintained.

  For example, when a user wearing a head-mounted display wants to view an image captured by an omnidirectional camera, the head-mounted display displays various directions in the omnidirectional direction in a short time according to the user's movement. To do. For this reason, if the brightness of the video differs depending on the viewing direction of the user, the visibility of the video is not improved even if the video is uniformly illuminated with the same brightness.

  In order to solve the above-described conventional problems, the present invention dynamically adjusts the brightness of an image according to a user's line-of-sight direction when viewing an image captured by an omnidirectional camera on a head-mounted display. An object of the present invention is to provide a video display system, a video display device, and a video display method that artificially increase the dynamic range during imaging and improve the visibility of the video.

  The present invention is a video display system in which a video processing device that holds a plurality of video data captured by a plurality of cameras and a video display device that can be worn by a user are connected, and the video processing device includes the plurality of video processing devices. Using the video data, a video generation unit for generating omnidirectional video data having a range of 360 degrees around an installation area of the plurality of cameras, and the omnidirectional video data generated by the video generation unit A video transmission unit for transmitting the image to the video display device, the video display device receiving the omnidirectional video data transmitted from the video transmission unit, and a sensor for detecting a gaze direction of the user. A video receiving unit, a video extraction unit that extracts a predetermined range of video data including the detection result of the sensor from the omnidirectional video data, and the predetermined range extracted by the video extraction unit A brightness adjustment unit that adjusts the brightness of the video data to a certain range of brightness according to the brightness of the omnidirectional video data, and the video data of the predetermined range adjusted by the brightness adjustment unit is displayed. An image display system is provided.

  According to another aspect of the present invention, there is provided a video display device connected to a video processing device that can be worn by a user and holds a plurality of video data picked up by a plurality of cameras, the sensor detecting a gaze direction of the user An image receiving unit that receives 360-degree image data having a range of 360 degrees around an installation area of the plurality of cameras generated by the image processing device, and the sensor from the omnidirectional image data. A video extraction unit that cuts out a predetermined range of video data including the detection result, and the brightness of the video data of the predetermined range extracted by the video extraction unit is a predetermined range according to the brightness of the omnidirectional video data There is provided a video display device comprising: a brightness adjustment unit that adjusts the brightness of the image; and a video display unit that displays the predetermined range of video data adjusted by the brightness adjustment unit.

  The present invention is also a video display method in a video display device that can be worn by a user and connected to a video processing device that holds a plurality of video data captured by a plurality of cameras, wherein the user's line-of-sight direction And omnidirectional video data having a 360-degree range around the installation area of the plurality of cameras generated by the video processing device is received, and the detected celestial sphere video data Cut out a predetermined range of video data including the user's line-of-sight direction, adjust the brightness of the extracted video data of the predetermined range to a certain range of brightness according to the brightness of the omnidirectional video data, and adjust A video display method for displaying the predetermined range of video data is provided.

  According to the present invention, when the image captured by the omnidirectional camera is viewed on the head mounted display, the brightness of the image is dynamically adjusted according to the user's line-of-sight direction, so that the dynamic range at the time of imaging is simulated. Therefore, the visibility of the video can be improved.

The figure which shows an example of the system configuration | structure of the omnidirectional camera system of this embodiment The figure which shows an example of the omnidirectional video data obtained by the stitching process of the captured image of all the cameras shown in FIG. The block diagram which shows the 1st example of the internal structure of the video processing apparatus of this embodiment. The block diagram which shows the 1st example of the internal structure of the head mounted display of this embodiment. The block diagram which shows the 2nd example of the internal structure of the video processing apparatus of this embodiment. The block diagram which shows the 2nd example of the internal structure of the head mounted display of this embodiment. (A) Explanatory drawing which shows an example of the positional relationship between the omnidirectional camera of this embodiment and a subject, (B) The line of sight that the user faces among the image data captured by the omnidirectional camera shown in (A) Explanatory drawing which shows the example where the image of the direction was cut out and displayed on the head mounted display (A) An explanatory diagram of the origin of the omnidirectional camera, a horizontal reference line, and a horizontal angle θ indicating the user's line-of-sight direction. Graph showing an example relationship The flowchart explaining an example of the operation | movement procedure of the video processing apparatus of this embodiment. The flowchart explaining an example of the operation | movement procedure of the head mounted display of this embodiment.

  Hereinafter, an embodiment (hereinafter referred to as this embodiment) that specifically discloses a video display system, a video display device, and a video display method according to the present invention will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art. The accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

  The omnidirectional camera system 1 as an example of the video display system of the present embodiment includes, for example, a plurality of cameras CAM1... Using video data captured by a plurality of cameras CAM1. Video data having a 360-degree range around the installation area of the CAM 7 is generated, and the video data in the user's line-of-sight direction is cut out and displayed on the head mounted display 4 worn by the user.

  FIG. 1 is a diagram illustrating an example of a system configuration of an omnidirectional camera system 1 according to the present embodiment. FIG. 2 is a diagram showing an example of the omnidirectional video data ASP obtained by the stitching processing of the captured images of all the cameras CAM1... CAM7 shown in FIG. As shown in FIG. 1, an omnidirectional camera system 1 includes an omnidirectional camera 2 that is a camera unit having cameras CAM1... CAM7 that generate captured image data (for example, moving images) for stitching processing, A video processing device 3 that sequentially generates omnidirectional video data ASP by performing stitching processing of captured image data, and video data in a predetermined range among the omnidirectional video data ASP transmitted from the video processing device 3 And a head mounted display 4 that cuts out and displays the image.

  The cameras CAM1 to CAM7 have an imaging unit that can be configured by a lens having a wide angle of view (for example, 120 degrees) and an image sensor, and using data obtained from the imaging unit (that is, an electrical signal of an image), Image data in RGB (Red Green Blue) format or image data in YUV (luminance, color difference) format that can be recognized by a person is generated and stored. The cameras CAM1... CAM7 can be configured by a known technique, and a detailed description of the internal configuration is omitted.

  In FIG. 1, the omnidirectional camera is composed of seven cameras CAM1... CAM7, but is not limited to seven. The cameras CAM1... CAM6 are arranged at equal intervals in the circumferential direction so that the optical axes are substantially radial in the horizontal direction. The camera CAM 7 is disposed so that the optical axis is substantially upward in the vertical direction in order to image the zenith direction. The cameras CAM1... CAM7 are arranged such that two adjacent cameras CAM1. In the omnidirectional camera system 1, the arrangement of cameras that generate captured image data for stitching processing is not limited to the arrangement shown in FIG. 1, and various changes can be made. For example, the camera CAM7 may be omitted.

  The video processing device 3 is configured using a PC (Personal Computer) in which the cameras CAM1... CAM7 are connected via a connector CN via a cable. However, the video processing device 3 and the cameras CAM1... CAM7 are not limited to wired connection, and may be wirelessly connected. The video processing apparatus 3 will be described in detail later with reference to FIGS. 3 and 5, but the CPU (Central Processing Unit) that centrally executes various control processes based on a predetermined control program, and the work area of the CPU Hardware configuration including RAM (Random Access Memory), ROM (Read Only Memory) for storing control programs and data executed by the CPU, speakers, input devices, HDD (Hard Disk Drive), etc. ing.

  Further, at least a part of various functions (stitching processing, which will be described in detail later, average brightness information superimposing processing for each direction, etc.) executed in the video processing device 3 is performed by the CPU (control program for stitching processing). This can be realized by executing an orientation-specific average brightness information superimposing processing program or the like. The video processing device 3 is not limited to the PC, and other information processing devices (for example, a server, a smartphone, a tablet terminal, etc.) that perform the same function can also be used. Further, at least a part of the functions of the video processing device 3 may be replaced by processing using other known hardware.

  In the video processing device 3, as shown in FIG. 2, the omnidirectional video data ASP, which is one panoramic composite image, is synthesized by synthesizing the captured images (here, seven videos) captured by the cameras CAM1. The generation process (that is, stitching process) is performed for each frame. As shown in FIG. 2, video data P1... P7, which are captured image data, are data captured by cameras CAM1. The omnidirectional video data ASP shown in FIG. 2 is data obtained by imaging the omnidirectional camera 2 installed outdoors (for example, the ground). Since the camera CAM 7 is imaging the zenith direction, the omnidirectional video data ASP shows an empty state. Since the other cameras CAM1... CAM6 take images in the horizontal direction, the horizontal state of the ground is shown. With the omnidirectional video data ASP, it is possible to clearly determine the state of a 360 degree range around the place where the omnidirectional camera 2 is installed (ground in the example shown in FIG. 2).

  The video processing device 3 transmits the omnidirectional video data ASP for each frame to the head mounted display 4 via the network NW. The network NW may be a network using wired communication or a network using wireless communication (for example, a wireless LAN (Local Area Network)).

  The head mounted display 4 is mounted and used in a state in which a part of a housing frame extending in a pair of left and right, such as glasses, is placed near the user's ear. In other words, the head mounted display 4 is used while being worn by the user. The head mounted display 4 is connected to the video processing device 3 via the network NW. Although the detailed configuration of the head mounted display 4 will be described later with reference to FIGS. 4 and 6, the head mounted display 4 receives the omnidirectional video data ASP transmitted from the video processing device 3 and receives the user's line-of-sight direction (that is, the head mounted display). A predetermined range of video data including the center direction of the video data display surface in the housing frame of the display 4 as a center direction is cut out and displayed in real time. Note that the video processing device 3 does not display the video data on the head-mounted display 4, or at the same time, the omnidirectional video data ASP after the stitching processing is stored in a known storage device (not shown) such as an HDD. You may memorize | store sequentially.

  FIG. 3 is a block diagram illustrating a first example of the internal configuration of the video processing apparatus according to the present embodiment. The video processing apparatus 3 shown in FIG. 3 includes at least a video input unit 31, a video signal processing unit 33, and a video output unit 35. 3, the connector CN (see FIG. 1) is not shown between the cameras CAM1... CAM7 and the video input unit 31.

  The video input unit 31 has a function as an input / output interface with the cameras CAM1... CAM7, and inputs and acquires video data output from the respective cameras CAM1. The video input unit 31 outputs video data from each of the cameras CAM1 to CAM7 to the video signal processing unit 33.

  The video signal processing unit 33 as an example of the video generation unit is configured by using a processor such as a CPU, an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor), for example, and the camera CAM1 output from the video input unit 31. The omnidirectional video data ASP, which is panoramic video data shown in FIG. 2, is generated by performing stitching processing for each frame of the video data using video data that is captured image data of the CAM 7. The omnidirectional video data ASP includes brightness data for each direction of the frame of the video data for each pixel constituting the omnidirectional video data ASP, and so on. The brightness data is, for example, a luminance value for each pixel when the video data is YUV format data, and is a pixel value when the video data is RGB format data.

  The video signal processing unit 33 stores the omnidirectional video data ASP in a memory (not shown) (for example, a flash memory or an HDD). The video signal processing unit 33 outputs the omnidirectional video data ASP to the video output unit 35.

  The video output unit 35 as an example of the video transmission unit has a function as a communication interface with the head mounted display 4. The omnidirectional video data ASP output from the video signal processing unit 33 is transmitted to the network NW. Via the head mounted display 4. The video output unit 35 may display the omnidirectional video data on a display device (not shown) different from the head mounted display 4.

  FIG. 4 is a block diagram showing a first example of the internal configuration of the head mounted display of the present embodiment. The head mounted display 4 shown in FIG. 4 includes a gyro sensor JS, a communication unit 41, a video input unit 42, a display range determination unit 43, a video cutout deformation unit 44, a display video brightness adjustment unit 45, and a video. The display unit 46 is included at least.

  The gyro sensor JS as an example of the sensor detects an angle in the line-of-sight direction of the user wearing the head mounted display 4 (that is, the direction in which the display surface of the video data in the housing frame of the head mounted display 4 faces the user). . The gyro sensor JS outputs a detection result (that is, a detection value corresponding to the three axes of roll, pitch, and yaw) to the display range determination unit 43.

  The communication unit 41 as an example of the video receiving unit has a function as a communication interface with the video processing device 3, receives the omnidirectional video data ASP transmitted from the video processing device 3, and receives a video input unit. Output to 42.

  The video input unit 42 has a function as an input / output interface with the communication unit 41, and inputs and acquires the omnidirectional video data ASP received by the communication unit 41. The video input unit 42 outputs the omnidirectional video data ASP to the video cutout deformation unit 44.

  The display range determination unit 43 is configured using a processor such as a CPU, MPU, or DSP, for example, and based on the output of the gyro sensor JS (that is, detection values corresponding to the three axes of roll, pitch, and yaw), A predetermined range including the line-of-sight direction as a center (that is, a video data cut-out range) is determined. The display range determination unit 43 outputs the cutout range information that is the determination result to the video cutout deformation unit 44.

  The video cutout deformation unit 44 as an example of the video extraction unit is configured by using a processor such as a CPU, MPU, or DSP, for example, and from the omnidirectional video data ASP output from the video input unit 42, the display range determination unit 43. Display cut-out video data obtained by cutting out the video data corresponding to the cut-out range information output from is generated. The video cutout deformation unit 44 outputs the cutout video data for display to the display video brightness adjustment unit 45.

  In addition, the video cutout deforming unit 44 uses the brightness data for each pixel of the omnidirectional video data ASP included in the omnidirectional video data ASP, so that the video range corresponding to the cutout video data for display (that is, The average value of the brightness of the display video data in the cutout range corresponding to the cutout range information output from the display range determining unit 43 is calculated. The video cutout deformation unit 44 outputs cutout range average brightness information, which is a calculated value, to the display video brightness adjustment unit 45.

  The display video brightness adjustment unit 45 as an example of the brightness adjustment unit is configured by using a processor such as a CPU, MPU, or DSP, for example, and the display cut-out video data output from the video cut-out deformation unit 44 and the cut-out range average Using the brightness information, the cutout range average brightness information is adjusted such that the cutout range average brightness information of the display cutout video data has a certain range of brightness. The adjustment processing of the cutout range average brightness information in the display video brightness adjustment unit 45 will be described later with reference to FIGS. 7B and 8B. The display video brightness adjustment unit 45 outputs the corrected cut-out video data for display after being adjusted to have a certain range of brightness to the video display unit 46.

  The video display unit 46 is configured using, for example, an LCD (Liquid Crystal Display) or an organic EL (Electroluminescence), and displays the cutout video data after the brightness is adjusted by the display video brightness adjustment unit 45.

  FIG. 5 is a block diagram illustrating a second example of the internal configuration of the video processing apparatus according to the present embodiment. A video processing device 3A shown in FIG. 5 includes a video input unit 31, a video signal processing unit 33, a video output unit 35A, and an orientation-specific average brightness information superimposing unit 37. 5, the connector CN (see FIG. 1) is omitted between the cameras CAM1... CAM7 and the video input unit 31. In the description of each part of the video processing apparatus 3A shown in FIG. 5, the same components and operations as those of the video processing apparatus 3 shown in FIG.

  The video signal processing unit 33 outputs the omnidirectional video data ASP to the orientation-specific average brightness information superimposing unit 37.

  The orientation-specific average brightness information superimposing unit 37 as an example of the brightness providing unit is a brightness for each pixel of the omnidirectional video data ASP included in the omnidirectional video data ASP output from the video signal processing unit 33. The average brightness of the video data is calculated for each direction using the data. The azimuth average brightness information superimposing unit 37 superimposes the calculated value for each azimuth of the average brightness of the video data on the omnidirectional video data ASP. As an example of superimposition, the azimuth-specific average brightness information superimposing unit 37 calculates the average value of the brightness of the video data for each azimuth as the header area (not shown) of the omnidirectional video data ASP or the optional area of the payload. It is superimposed (given) by storing it in. The azimuth average brightness information superimposing unit 37 outputs the omnidirectional video data ASP on which the calculated value for each azimuth of the average brightness of the video data is superimposed to the video output unit 35A.

  The video output unit 35A outputs the output of the video signal processing unit 33 (that is, the omnidirectional video data ASP on which the calculated value of the average value of the brightness of the video data is superimposed) via the network NW. Send to 4A.

  Here, the positional relationship between the omnidirectional camera 2 and the subject imaged by the omnidirectional camera 2 will be described with reference to FIG. FIG. 7A is an explanatory diagram illustrating an example of a positional relationship between the omnidirectional camera 2 and the subjects TG1 and TG2 according to the present embodiment. In the example shown in FIG. 7A, the omnidirectional camera 2 is installed so as to be supported on a support base (for example, a stand) placed on the ground FLD in the outdoors where sunlight falls. Around the omnidirectional camera 2, a female subject TG1 and a male subject TG2 stand. When viewed from the center of the omnidirectional camera 2 (the origin O shown in FIG. 8A), the subject TG1 is in the direction of sunlight (that is, the direction in which the average brightness of the video data is relatively high). On the other hand, the subject TG2 stands in the backlight direction of sunlight (that is, the direction in which the average value of the brightness of the video data is relatively low).

  In this case, since the brightness of the video data captured by the camera in which the subject TG1 is included in the angle of view is relatively high, the brightness of the video data becomes too high and the video data is white or difficult to see. The visibility may not be good. In addition, since the brightness of the video data captured by the camera in which the subject TG2 is included in the angle of view is relatively low, the visibility of the video data may be poor, such as the video data becoming too dark.

  FIG. 8A is an explanatory diagram of the origin O of the omnidirectional camera, the horizontal reference line BD, and the horizontal angle θ indicating the user's line-of-sight direction. FIG. 8B is a graph illustrating an example of the relationship between the horizontal angle θ indicating the user's line-of-sight direction and brightness (for example, luminance value). In FIG. 8A, the horizontal angle θ indicating the user's line-of-sight direction indicates the horizontal rotation angle from the horizontal direction reference line BD defined in advance between the video processing device 3 and the head mounted display 4.

  The azimuth-specific average brightness information superimposing unit 37 holds data relating to the azimuth of each camera CAM1... CAM7 constituting the omnidirectional video data ASP. The brightness for each direction is calculated for each frame of the data ASP (see FIG. 8B). The horizontal axis in FIG. 8B indicates the horizontal angle θ, and the vertical axis in FIG. 8B indicates brightness (for example, luminance value). In FIG. 8B, the threshold value Th1 is a threshold value that determines whether the brightness for each direction is too low (in other words, too dark). If the brightness for each direction is less than the threshold value Th1, the video data is dark. It will be too much. On the other hand, the threshold value Th2 is a threshold value that determines whether the brightness for each direction is too high (in other words, too bright). If the brightness for each direction is greater than the threshold value Th2, the video data is too bright.

  As shown in FIG. 8B, for example, when the horizontal angle θ is about 90 degrees (about π / 2), the video data is too bright and the brightness exceeds the threshold Th2. This is based on the fact that the brightness of the video data obtained by imaging is high when the female subject TG1 is standing in the forward light direction of sunlight as shown in FIG. On the other hand, for example, when the horizontal angle θ is about 270 degrees (about 3π / 2), the video data is too dark and the brightness is less than the threshold Th1. This is based on the fact that the brightness of the video data obtained by imaging is low when the male subject TG2 stands in the backlight direction of sunlight as shown in FIG.

  The display image brightness adjustment unit 45 and the display image brightness adjustment unit 45A of the head mounted displays 4 and 4A adjust the brightness of the display cut-out image data to a certain range (for example, from the threshold Th1 to the threshold Th2 shown in FIG. 8B). The brightness of the cut-out video data for display is adjusted so as to be within a predetermined range. FIG. 7B shows an example in which the video in the line-of-sight direction facing the user is cut out from the video data captured by the omnidirectional camera 2 shown in FIG. It is explanatory drawing shown. With the user wearing the head mounted displays 4 and 4A, the head mounted displays 4 and 4A set the brightness of the video data in the direction in which the female subject TG1 is standing (the cutout video data for display) within a certain range. The corrected cutout video data PC1 that has been adjusted so as to be dark is displayed. Further, the head mounted displays 4 and 4A are corrected display cut-out images in which the brightness of the video data (display cut-out video data) in the direction in which the male subject TG2 stands is adjusted to be bright within a certain range. Data PC2 is displayed.

  Further, the display image brightness adjustment unit 45 and the display image brightness adjustment unit 45A of the head-mounted displays 4 and 4A, in addition to adjusting the brightness of the display cut-out image data so as to be within a certain range, The brightness of the clipped video data may be adjusted by increasing / decreasing by a predetermined ratio, and so on. The predetermined ratio is, for example, + 20% and -20%, but these values are not limited.

  FIG. 6 is a block diagram showing a second example of the internal configuration of the head mounted display of the present embodiment. The head mounted display 4A shown in FIG. 6 includes a gyro sensor JS, a communication unit 41, a video input unit 42, a display range determination unit 43, a video cutout deformation unit 44A, a display video brightness adjustment unit 45A, This is a configuration including a display unit 46 and a video superimposition information separation unit 47. In the description of each part of the head mounted display 4A shown in FIG. 6, those having the same configuration and operation as each part of the head mounted display 4 shown in FIG. The head mounted display 4A receives the omnidirectional video data ASP on which the calculated value for each direction of the average brightness of the video data is superimposed.

  The video input unit 42 outputs the omnidirectional video data ASP on which the calculated value for each direction of the average value of the brightness of the video data is superimposed to the video superimposition information separation unit 47.

  The video superimposition information separation unit 47 calculates the average brightness value of the video data according to the direction (in other words, the calculated value of the average brightness value of the video data from the omnidirectional video data ASP on which the calculated value of the average brightness value of the video data is superimposed. For example, the orientation-dependent average brightness information) is separated and output to the display image brightness adjustment unit 45A. The video superimposition information separation unit 47 outputs the omnidirectional video data ASP from which the calculated values of the average value of the brightness of the video data for each direction are separated to the video cutout deformation unit 44A.

  The video cutout deformation unit 44A as an example of the video extraction unit is configured by using a processor such as a CPU, MPU, or DSP, for example, and determines the display range from the omnidirectional video data ASP output from the video superimposition information separation unit 47. Display cutout video data obtained by cutting out video data corresponding to the cutout range information output from the unit 43 is generated. The video cutout deformation unit 44A outputs the cutout video data for display to the display video brightness adjustment unit 45A.

  The display video brightness adjustment unit 45A as an example of the brightness adjustment unit is configured using a processor such as a CPU, MPU, or DSP, for example, and the display cutout video data and the video superimposition information output from the video cutout deformation unit 44A. Using the average brightness information for each direction output from the separation unit 47, the average brightness information of the cutout range is adjusted so that the average brightness information of the display cutout video data has a certain range of brightness. The display video brightness adjustment unit 45A outputs the corrected display cut-out video data after the adjustment so that the brightness is within a certain range to the video display unit 46.

  Next, the operation procedure of the video processing apparatus 3A of the present embodiment will be described with reference to FIG. FIG. 9 is a flowchart for explaining an example of the operation procedure of the video processing apparatus 3A of the present embodiment.

  In FIG. 9, a video input unit 31 inputs video data output from N (for example, N = 7 in this embodiment) cameras CAM1. (S1). The video input unit 31 outputs video data from each of the cameras CAM1 to CAM7 to the video signal processing unit 33.

  The video signal processing unit 33 performs the stitching process for each frame of the video data by using the video data which is the imaged image data of the cameras CAM1... CAM7 output from the video input unit 31, and is shown in FIG. The omnidirectional video data ASP is generated (S2). The video signal processing unit 33 outputs the omnidirectional video data ASP to the orientation-specific average brightness information superimposing unit 37. The azimuth average brightness information superimposing unit 37 divides the area of the omnidirectional video data ASP into X pieces in the horizontal direction (S3). The value of X is, for example, 32 or 64, but is not limited to these values.

  The orientation-specific average brightness information superimposing unit 37 selects an area for which the average value of brightness has not yet been calculated from among the individual areas divided into X pieces, and determines the brightness of the video data in the selected area. An average value is calculated, and the calculated value (that is, the average value) is temporarily stored in a buffer (not shown) (S4).

  The orientation-specific average brightness information superimposing unit 37 determines whether or not the average brightness value of the video data has been calculated for all X regions and temporarily stored in the buffer (S5). When the calculated value of the average value of the brightness of the video data is not stored in the buffer for all X areas (S5, NO), the processing of the video processing device 3A returns to step S4, and X The processing in step S4 is repeated until the calculated value of the average brightness of the video data is stored in the buffer for all the areas.

  On the other hand, when the calculated average brightness value of the video data is stored in the buffer for all X regions, the orientation-specific average brightness information superimposing unit 37 stores the calculated value in the buffer (S5, YES). The calculated value of the average value of the brightness of the video data is superimposed on the omnidirectional video data ASP for all the X areas thus obtained (S6). The azimuth average brightness information superimposing unit 37 outputs a frame of the omnidirectional video data ASP on which the calculated value for each azimuth of the average brightness of the video data is superimposed to the video output unit 35A (S7). The video output unit 35A outputs the output of the video signal processing unit 33 (that is, the omnidirectional video data ASP on which the calculated value of the average value of the brightness of the video data is superimposed) via the network NW. Send to 4A. If the operation of the video processing device 3A has not ended after step S7 (S8, NO), the processing of the video processing device 3A returns to step S1. On the other hand, when the operation of the video processing device 3A is finished (S8, YES), the processing of the video processing device 3A shown in FIG. 9 is finished.

  Next, the operation procedure of the head mounted display 4A of the present embodiment will be described with reference to FIG. FIG. 10 is a flowchart for explaining an example of the operation procedure of the head mounted display 4A of the present embodiment. As a premise of the description of FIG. 10, it is assumed that the omnidirectional video data ASP on which the calculated value for each direction of the average brightness of the video data is superimposed is transmitted from the video processing device 3A.

  In FIG. 10, the communication unit 41 transmits the omnidirectional video data ASP transmitted from the video processing device 3 </ b> A (that is, the omnidirectional video data ASP on which the calculated value for each orientation of the average brightness of the video data is superimposed). Is output to the video input unit 42. The video input unit 42 receives and acquires the omnidirectional video data ASP received by the communication unit 41. The video input unit 42 outputs the omnidirectional video data ASP on which the calculated value for each direction of the average brightness of the video data is superimposed to the video superimposition information separation unit 47 (S11).

  The video superimposition information separation unit 47 calculates the average brightness value of the video data according to the direction (in other words, the calculated value of the average brightness value of the video data from the omnidirectional video data ASP on which the calculated value of the average brightness value of the video data is superimposed. For example, the average brightness information for each direction is separated and output to the display image brightness adjustment unit 45A (S12). The video superimposition information separation unit 47 outputs the omnidirectional video data ASP from which the calculated values of the average value of the brightness of the video data for each direction are separated to the video cutout deformation unit 44A.

  Based on the output of the gyro sensor JS (that is, the detection values corresponding to the three axes of roll, pitch, and yaw), the display range determination unit 43 extracts a predetermined range (that is, cuts out video data) centered on the user's line-of-sight direction. (Range) is determined (S13). The display range determination unit 43 outputs the cutout range information that is the determination result to the video cutout deformation unit 44A.

  The video cutout deforming unit 44A cuts out video data corresponding to the cutout range information output from the display range determining unit 43 from the omnidirectional video data ASP output from the video superimposition information separating unit 47. Is generated (S14). The video cutout deformation unit 44A outputs the cutout video data for display to the display video brightness adjustment unit 45A.

  The display video brightness adjustment unit 45A determines whether or not the brightness of the video data in the cutout range of the video data determined in step S13 is too bright (S15). When the display video brightness adjustment unit 45A determines that the brightness of the video data in the video data cutout range is too bright (for example, when the brightness of the video data in the video data cutout range exceeds the threshold Th2). (S15, YES), the average brightness information of the cutout range is adjusted to be low so that the average brightness information of the display cutout video data has a certain range of brightness (S16). The display video brightness adjustment unit 45A outputs the corrected display cut-out video data after the adjustment so that the brightness is within a certain range to the video display unit 46. After step S16, the process of the head mounted display 4A proceeds to step S19.

  On the other hand, if it is not determined that the brightness of the video data in the cutout range of the video data determined in step S13 is too bright (S15, NO), the display video brightness adjustment unit 45A is determined in step S13. It is determined whether or not the brightness of the video data in the cutout range of the video data is too dark (S17). When the display video brightness adjustment unit 45A determines that the brightness of the video data in the video data cutout range is too dark (for example, when the brightness of the video data in the video data cutout range is less than the threshold Th1). (S17, YES), the average brightness information of the cutout range is adjusted to be bright so that the average brightness information of the display cutout video data has a certain range of brightness (S18). The display video brightness adjustment unit 45A outputs the corrected display cut-out video data after the adjustment so that the brightness is within a certain range to the video display unit 46. After step S18, the process of the head mounted display 4A proceeds to step S19.

  After the processing of steps S16 and S18, or when it is not determined that the brightness of the video data in the video data cut-out range is too dark (S17, NO), the video display unit 46 displays the display video brightness adjustment unit. The cut-out video data after the brightness is adjusted by 45A is displayed (S19). If the operation of the head mounted display 4A has not ended after step S19 (S20, NO), the processing of the head mounted display 4A returns to step S11. On the other hand, if the operation of the head mounted display 4A is finished (S20, YES), the processing of the head mounted display 4A shown in FIG. 10 is finished.

  As described above, in the omnidirectional camera system 1 of the present embodiment, the video processing device 3 uses the plurality of video data captured by the plurality of cameras CAM1... CAM7 to surround the installation area of the plurality of cameras CAM1. 360 degrees of omnidirectional video data is generated, and the omnidirectional video data is transmitted to the head mounted display 4 worn by the user. The head mounted display 4 detects the direction of the user's line of sight with the gyro sensor JS, cuts out a predetermined range of video data including the detection result of the gyro sensor JS from the omnidirectional video data obtained by reception, and cut out the predetermined data. Adjust the brightness of the video data in the range to a certain range of brightness. Further, the head mounted display 4 displays video data in a predetermined range after adjustment.

  As a result, when the user tries to view the omnidirectional video data with the head mounted display 4 attached, the head mounted display 4 displays the direction of the user's line of sight (that is, the direction of the video display surface on the head mounted display 4). ), The video data matching the user's line-of-sight direction is cut out from the omnidirectional video data, and the brightness of the cut-out video data is dynamically adjusted. Therefore, the head mounted display 4 does not need to illuminate the video data with an auxiliary backlight as in Patent Document 1, and can increase the dynamic range at the time of imaging of at least one of the cameras CAM1. Since it can do, the visibility of the image | video of the gaze direction with respect to a user can be improved.

  Further, the head mounted display 4 calculates an average value of brightness (for example, luminance value) of video data in a predetermined range after being cut out so as to match the user's line of sight, and the calculated average value of brightness is constant. Adjust to the brightness of the range. Thereby, the head mounted display 4 does not need to calculate the average value of the brightness of the video data over the entire area of the omnidirectional video data transmitted from the video processing device 3 (that is, the range of 360 degrees), and the user Since the average brightness value only needs to be calculated for video data in a predetermined range that matches the line-of-sight direction, the load of the average value calculation process can be reduced.

  Also, the video processing device 3 calculates the average brightness data for each azimuth of the omnidirectional video data and assigns it to the omnidirectional video data, and the average brightness data for each azimuth is given. The omnidirectional video data is transmitted to the head mounted display 4. The head mounted display 4 obtains an average brightness value of a predetermined range of video data matching the user's line-of-sight direction from the received omnidirectional video data, and obtains the average brightness value after the acquisition within a certain range of brightness. Adjust it. Thereby, since the head mounted display 4 has obtained the average brightness data for each direction over a range of 360 degrees, for example, the predetermined range that matches the user's line-of-sight direction is changed to an arbitrary range and the video data Is displayed, the brightness of the video data to be displayed can be appropriately adjusted in accordance with the average value of the brightness of the video data in an arbitrary range after the change.

  Further, the head mounted display 4 changes the brightness of a predetermined range of video data cut out according to the detection result of the gyro sensor JS every time the user's line-of-sight direction is switched, for example, when the user rotates the neck. The brightness is adjusted, and the adjusted video data in a predetermined range is displayed. As a result, the head mounted display 4 allows the brightness of the video data in a predetermined range corresponding to the direction of the head mounted display 4 (that is, the user's line-of-sight direction) to be within a certain range even when the user changes the viewing direction. It is possible to suppress the deterioration of the visibility of the video data displayed following the display.

  While various embodiments have been described above with reference to the drawings, it goes without saying that the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes and modifications can be made within the scope of the claims, and these are naturally within the technical scope of the present invention. Understood.

  The present invention dynamically adjusts the brightness of an image in accordance with the user's line of sight when viewing an image captured by an omnidirectional camera on a head-mounted display, and artificially increases the dynamic range during imaging. The present invention is useful as a video display system, a video display device, and a video display method for improving video visibility.

DESCRIPTION OF SYMBOLS 1 Spherical camera system 2 Spherical camera 3, 3A Video processing device 4, 4A Head mounted display 31, 42 Video input part 33 Video signal processing part 35, 35A Video output part 37 Average brightness information superimposition part 41 by direction Communication unit 43 Display range determination unit 44, 44A Image cutout deformation unit 45, 45A Display image brightness adjustment unit 46 Image display unit 47 Image superimposition information separation unit ASP omnidirectional image data BD Horizontal reference lines CAM1, CAM2, CAM3, CAM4, CAM5, CAM6, CAM7 Camera CN Connector JS Gyro Sensor NW Network PC1, PC2 Cutout video data TG1, TG2 Subject

Claims (10)

A video display system in which a video processing device that holds a plurality of video data captured by a plurality of cameras and a video display device that can be worn by a user are connected,
The video processing device includes:
A video generation unit that generates omnidirectional video data having a range of 360 degrees around an installation area of the plurality of cameras using the plurality of video data;
A video transmission unit that transmits the omnidirectional video data generated by the video generation unit to the video display device;
The video display device
A sensor for detecting the user's gaze direction;
A video receiver for receiving the omnidirectional video data transmitted from the video transmitter;
A video extraction unit that cuts out a predetermined range of video data including the detection result of the sensor from the omnidirectional video data;
A brightness adjustment unit that adjusts the brightness of the predetermined range of video data extracted by the video extraction unit to a certain range of brightness according to the brightness of the omnidirectional video data;
A video display unit for displaying the predetermined range of video data adjusted by the brightness adjustment unit,
Video display system. The video display system according to claim 1,
The brightness adjustment unit, when the brightness of the predetermined range of video data is higher or lower than the predetermined range of brightness according to the brightness of the omnidirectional video data, Adjusting the brightness of the to be within the certain range,
Video display system. The video display system according to claim 1 or 2,
The brightness adjustment unit calculates an average brightness value of the video data in the predetermined range extracted by the video extraction unit, and adjusts the calculated average brightness value to the certain range of brightness. ,
Video display system. The video display system according to any one of claims 1 to 3,
The video display device adjusts the brightness of the predetermined range of video data cut out according to the detection result of the sensor to the brightness of the predetermined range every time the user's line-of-sight direction is switched. Displaying the predetermined range of video data;
Video display system. A video display device connected to a video processing device that can be worn by a user and holds a plurality of video data captured by a plurality of cameras,
A sensor for detecting the user's gaze direction;
A video receiver that receives the omnidirectional video data having a range of 360 degrees around the installation area of the plurality of cameras generated by the video processing device;
A video extraction unit that cuts out a predetermined range of video data including the detection result of the sensor from the omnidirectional video data;
A brightness adjustment unit that adjusts the brightness of the predetermined range of video data extracted by the video extraction unit to a certain range of brightness according to the brightness of the omnidirectional video data;
A video display unit for displaying the predetermined range of video data adjusted by the brightness adjustment unit,
Video display device. The video display device according to claim 5,
The brightness adjustment unit, when the brightness of the predetermined range of video data is higher or lower than the predetermined range of brightness according to the brightness of the omnidirectional video data, Adjusting the brightness of the to be within the certain range,
Video display device. A video display method in a video display device that can be worn by a user and connected to a video processing device that holds a plurality of video data captured by a plurality of cameras,
Detecting the user's gaze direction,
Receiving omnidirectional video data having a range of 360 degrees around an installation area of the plurality of cameras generated by the video processing device;
From the omnidirectional video data, cut out a predetermined range of video data including the detected user's gaze direction,
Adjust the brightness of the extracted video data in the predetermined range to a certain range of brightness according to the brightness of the omnidirectional video data,
Displaying the adjusted video data of the predetermined range;
Video display method. The video display method according to claim 7,
When the brightness of the predetermined range of video data is higher or lower than the predetermined range of brightness according to the brightness of the omnidirectional video data, the brightness of the predetermined range of video data is set to the predetermined range. Adjust so that
Video display method. The video display method according to claim 7 or 8,
Calculating an average value of brightness of the predetermined range of video data, and adjusting the calculated average value of brightness to a certain range of brightness;
Video display method. A video display method according to any one of claims 7 to 9,
Each time the user's line-of-sight direction is switched, the brightness of the predetermined range of video data cut out according to the detection result of the line-of-sight direction is adjusted to the brightness of the predetermined range, and the adjusted video of the predetermined range Display data,
Video display method.