JP2014176053A - Image signal processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To solve the problem that screen installation and the adjustment of a video position is time-consuming in order to achieve the matching of the position of the screen and the position of the video in the case of configuring a video display device by using a plurality of screens.SOLUTION: Video signal generation means includes: display arrangement detection means for determining positional relations among a plurality of video display devices from an image obtained by photographing a situation that a plurality of video display devices each having a video display panel are arranged; photographic position determination means for determining the photographic position of the image corresponding to each of the video display devices from the image; video generation means for generating a video signal to be displayed at each of the plurality of video display devices (the video signal of a video subjected to the matching of the positional relations among the plurality of video display devices at a view point from a photographing device for each video display device) on the basis of the positional relations among the video display devices and the photographic position; and multi-display control means for outputting each generated video signal to each corresponding video display device.

Description

  The present invention relates to an image signal processing apparatus that effectively generates a multi-viewpoint image for multiple screens.

  A multi-screen image generally includes a case where a plurality of screens are combined to display a single viewpoint video, and a case where a video from a different viewpoint is displayed on each screen. This case corresponds to a case where a multi-view video is displayed on a multi-screen.

  In Patent Literature 1, when shooting images to be displayed on a plurality of screens arranged at predetermined positions, the camera is shot while adjusting the positional relationship of the cameras in accordance with the positional relationship of the screens. In addition, the position of the photographed camera is recorded in the photographed video, and this is read and displayed during playback.

Japanese Patent Laid-Open No. 7-284089

  Even if the positional relationship between the captured image group and the screen to be displayed is taken into consideration, there is no guarantee that the screen group is correctly arranged at a position suitable for the imaging conditions. In addition, there is no guarantee that the viewpoint positions of the multi-viewpoint images captured by the camera are correctly installed. Furthermore, it is difficult to arrange the screen group / camera group correctly, and fine adjustment is required.

  In view of the above, an object of the present invention is to detect the arrangement of screen groups, change the shooting conditions of the camera according to the arrangement, correct the camera image, and eliminate the displacement of the arrangement on the video signal side. .

  In particular, in computer graphics, it is possible to regenerate a captured image in real time, so that an error in setting a screen group can be allowed.

  A video signal generation device according to the present disclosure includes a display arrangement detection unit that determines a positional relationship between a plurality of video display devices from an image obtained by photographing a situation in which a plurality of video display devices having video display panels are arranged, A shooting position determination unit that determines a shooting position of an image with respect to the video display device, a video correction unit that corrects a video signal displayed on each of the plurality of video display devices based on the positional relationship and the shooting position of the video display device, and A multi-display control unit that outputs each corrected video signal to a corresponding video display device.

  The invention described in this application simplifies the installation of multifaceted counties / cameras.

FIG. 3 illustrates an example of a photographing system described in Embodiment 1 FIG. 5 is a diagram illustrating an example of a photographing system in the case of a CG image described in the first embodiment The figure which shows arrangement | positioning of the display demonstrated in Embodiment 1 FIG. 6 is a diagram illustrating an arrangement of cameras described in Embodiment 1. The figure which shows the correction | amendment of the camera image demonstrated in Embodiment 1. The figure which shows the positional relationship of the some display demonstrated to Embodiment 1, and a camera. The figure of the image which image | photographed the some display demonstrated in Embodiment 1 The figure which converted the image | photographed image demonstrated in Embodiment 1 into the image shown by the binary value The figure which performed the outline emphasis process of the image | photographed image demonstrated in Embodiment 1 FIG. 5 is a diagram illustrating an example of a calibration image described in the first embodiment. The figure which shows the structural example of the multi-screen multi-view video generation system demonstrated in Embodiment 1. FIG. Flowchart illustrating image processing described in the first embodiment The flowchart which shows the CG process demonstrated in Embodiment 1.

  Hereinafter, embodiments will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of already well-known matters and repeated descriptions for substantially the same configuration may be omitted. This is to avoid the following description from becoming unnecessarily redundant and to facilitate understanding by those skilled in the art.

  The inventor (s) provides the accompanying drawings and the following description in order for those skilled in the art to fully understand the present disclosure, and is intended to limit the subject matter described in the claims. Not what you want.

(Embodiment 1)
FIG. 1 is a diagram illustrating an example of an imaging system described in this embodiment. FIG. 1 shows a configuration using three cameras and three displays. The camera 1, the camera 2, and the camera 3 that shoot the viewpoint video on each display constitute a video generation unit. Screen 1, screen 2, and screen 3 constitute a video display unit. Furthermore, it has a video correction unit and a screen position detection unit for correcting the video signal of the video generation unit.

  The camera 1, the camera 2, and the camera 3 are cameras intended to capture multi-viewpoint images. Here, a case where the optical axes of the respective cameras have a predetermined angle and are arranged at equal intervals on the same plane is shown. Screen 1, screen 2, and screen 3 are displays intended to display a multi-view video. As in the case of the camera, a case where the angles formed by the normal lines of the screen 1, the screen 2, and the screen 3 are arranged in accordance with the predetermined angle is shown.

  The screen position detection unit is configured by a camera, captures a screen, analyzes the video, and acquires the arrangement of the display.

  An image captured by the video generation unit is recorded or transmitted and input to the video correction unit. The video correction unit corrects a shift in the position of the camera, etc., processes the image to a state suitable for the video display unit based on the arrangement of the display, and transmits the corrected video signal to the video display unit.

  FIG. 2 has a computer graphics generation unit constituted by three virtual cameras 1, a virtual camera 2, and a virtual camera 3 instead of the video generation unit of FIG. 1. The video in the case of FIG. 2 is computer graphics, and video correction can be adjusted with parameters at the time of CG generation.

  The virtual camera 1, virtual camera 2, and virtual camera 3 are virtual viewpoint positions in the CG space. Here, a case where the viewpoint directions (corresponding to the optical axes) of the respective virtual cameras have a predetermined angle and are arranged at equal intervals is shown. Screen 1, screen 2, and screen 3 are displays intended to display a multi-view video. As in the virtual camera, the case where the angles formed by the normal lines of the screen 1, the screen 2, and the screen 3 are arranged at the predetermined angle is shown.

  The screen position detection unit is configured by a camera, captures a screen group, analyzes the video, and acquires the arrangement of the display.

  The image generated by the computer graphics generation unit is recorded or transmitted and input to the video display unit. The computer graphics generation unit renders a state suitable for the video display unit based on the arrangement of the display, and then transmits the rendered image to the video display unit.

  Next, the processing of the video correction unit will be described. In the case of FIG. 1, a video is taken by three cameras. The video correction unit sets the position of the optical axis of the camera in accordance with the arrangement of the display. Here, a case will be described in which displays of equal size are arranged along continuous sides of a regular hexagon. FIG. 3 is a view of the display as viewed from above. As shown in FIG. 3, the display is arranged on a regular hexagon.

  At this time, the viewer can view at the optimum position when the regular hexagon is at the center of the circumscribed circle. The distance from the viewer's display is w / 2 / tan 30 °, where w is the horizontal width of the display.

  Similarly, camera 1, camera 2, and camera 3 are arranged on a regular hexagon as shown in FIG. At this time, the viewer can view at the optimum position when the regular hexagon is at the center of the circumscribed circle.

  The display that the viewer looks at from the above optimal position has a viewing angle of 60 degrees. Therefore, it is preferable that the camera image displayed on the display also has a field of view of 60 degrees. Therefore, the angle of view of the camera is input to the video correction unit from the lens characteristics at the time of shooting. When the angle is larger than 60 degrees (N degrees), the video is cropped for an angle of view of 60 degrees, enlarged and displayed on the display.

  FIG. 5 shows a trimming location. 1/2 / tan N °: Cut out from the center of the screen so as to maintain a ratio of 1/2 / tan 30 °. At this time, the height conforms to the aspect of the display. After cropping, the image is enlarged according to the number of pixels of the display. Similarly, segmentation and enlargement processing are performed for three camera images.

  An error may occur depending on the installation state of the camera. At that time, the images between adjacent cameras are compared, and an inter-pixel difference sum calculation is performed near the screen boundary to identify the position where the value is the smallest. It is estimated that there is a subject common to the cameras.

  As described above, the cutout position is adjusted so that the connection between the screens becomes gentle. Since these are general techniques used in panoramic photography, details are omitted.

  Next, the processing of the computer graphics generation unit will be described. In CG, since there is no error as shown in the live-action camera, the virtual camera is set according to the arrangement of the display.

In the CG virtual camera, the angle of view, start point, and gazing point can be set. Therefore, in the display arrangement of FIG. 3, the angle of view is set to 60 degrees, and the start points of the virtual camera 1, virtual camera 2, and virtual camera 3 are set to the same coordinates. The angle between each gazing point and the starting point is set to 60 degrees. As described above, the real camera arrangement of FIG. 4 can be formed in the virtual space.
Next, the image position detection unit will be described. Normally, the display is rectangular. As shown in FIG. 6, a camera is installed at a position where the three displays of FIG. After these preparations, camera images are acquired. The acquired image is binarized. At this time, an all-white image may be input to each display so that the display surface can be easily detected by the binarization process.

  FIG. 7 shows the screen position photographed image thus obtained. Usually, since a captured image is an RGB 8-bit signal, it is converted into an 8-bit luminance color difference signal, and the luminance signal is compared with a threshold value. Binarization is performed by setting white (255, 255, 255) and black (0, 0, 0), etc. as the luminance signal value equal to or higher than a predetermined luminance signal value. The binarized image is shown in FIG.

  Next, edge detection processing is performed on the binarized image. A pixel in which the difference between adjacent pixels is 255 is converted to white, and the others are black. The detected image is shown in FIG.

  As described above, three trapezoidal figures can be acquired. When the distortion of each side of the trapezoid is examined, the relative position in space between the camera and the display is determined first. For example, in a simple case where the display and the camera are facing each other, the screen size is input in advance, and the distance from the camera is obtained by using a similar relationship from the ratio of the opposite sides of the trapezoid. If the distance between each side is obtained, the coordinates in the space can be acquired, and the relative position between the displays can be calculated. Although details are omitted here, the position of the display can be specified from the inverse matrix of the trapezoidal projective transformation in order to calculate an arbitrary position.

Furthermore, the camera image in the screen position detection unit can be used for calibration.
As calibration, the position of the bezel can be specified by the bezel (display frame) position detection unit, and the reproduced video can be corrected. Calibration here means that there is an image beyond the bezel and the display size and position are adjusted as if the image is viewed from a window frame.

  FIG. 10 shows a calibration image. The calibration image will be described later. Conventionally, a graphic is displayed so as to span adjacent displays. Here, a circle is displayed. Adjust the origin coordinates of each display so that the arcs are connected continuously. However, in the present application, the position of the bezel is specified from the previous display position. In the binarized image of FIG. 8, a black region is sandwiched between white regions.

  For the sake of simplicity, here, it is assumed that the binarized image of the front display is a rectangle, and that the bezel width of the screen 1 and the bezel width of the screen 2 are substantially equal when viewed from the camera image in the screen position detection unit. That is, the screen 2 is in a state of facing the camera and the angle between the screen 1 and the screen 2 is shallow. Further, it is assumed that the angle formed by the screen 2 and the screen 3 is equal to that.

  The width of the rectangle is dw and the width of the bezel is bw. If not corrected, the image becomes discontinuous at the bezel position. Therefore, the video for the bezel is expanded. An image having a width dw-bw trimmed to the left and right by a length of (dw-bw) / 2 from the center of the screen is generated.

  Next, trimming in the vertical direction according to the screen aspect. If the full HD image is 16: 9, trimming is performed so that dw-bw / 16 × 9 has a vertical length. Thereafter, the image is multiplied by dw / (dw−bw). Do this for each screen.

  As a result, the viewers have the illusion that the images are connected across the bezel, and the bezel feels like a window frame, creating a sense of presence. In addition, manual calibration is not required as in the prior art, and an optimal video can always be reproduced.

  Next, a configuration example of the entire multi-screen multi-view video generation system will be described with reference to FIG. The screen position detection unit calculates the spatial position of the displays arranged on substantially the same plane. The camera is arranged so that the normal vector at the center of the display obtained from the calculated result and the optical axis vector of the camera substantially coincide.

  In computer graphics, the vector created by the gazing point and the starting point of the virtual camera is set as the optical axis vector. Next, the video generation unit generates a video by shooting a multi-view video with a camera having a field angle larger than a viewing angle when the display is viewed from a substantially intersection of the normal vectors. The generated video is corrected by the video correction unit. The multi-screen display control unit transmits the corrected video to each display.

  The viewer position detection unit detects the position and orientation of the viewer's head, and when the viewer position deviates from the optimal position, the image correction unit corrects the trapezoidal distortion on the screen viewed by the viewer.

  FIG. 12 is a flowchart showing an image processing flow in the case of using a live-action described in the present embodiment.

  (Step S101) Screen position is detected.

  (Step S102) An image is acquired with a multi-viewpoint camera.

  (Step S103) The angle of view information of the camera is input. For still images, the angle of view is usually calculated from the focal length managed by EXIF as camera information. Even for a moving image, it may be automated by adding a process such as shooting a still image and recording shooting conditions.

  (Step S104) The viewer's head position is detected.

  (Step S105) The deviation from the intersection of the normal vectors extending from the center of each display is calculated from the viewer's head position.

  (Step S106) The image acquired in step S302 is corrected using the camera information and the viewer position correction information.

  (Step S107) The corrected image is transmitted to the corresponding display and displayed.

  FIG. 13 is a flowchart showing an image processing flow when computer graphics described in the present embodiment is used.

  (Step S201) The screen position is detected.

  (Step S202) A virtual camera position that matches the screen position is set.

  (Step S203) The head position of the viewer is detected.

  (Step S204) The deviation from the intersection of the normal vectors of each display is calculated from the viewer's head position.

  (Step S205) The virtual camera position is corrected with the position correction information and CG rendering is performed.

  (Step S206) The rendered image is transmitted and displayed on the corresponding display.

  As described above, the embodiments have been described as examples of the technology in the present disclosure. For this purpose, the accompanying drawings and detailed description are provided.

  Accordingly, among the components described in the accompanying drawings and the detailed description, not only the components essential for solving the problem, but also the components not essential for solving the problem in order to illustrate the above technique. May also be included. Therefore, it should not be immediately recognized that these non-essential components are essential as those non-essential components are described in the accompanying drawings and detailed description.

  Moreover, since the above-mentioned embodiment is for demonstrating the technique in this indication, a various change, replacement, addition, abbreviation, etc. can be performed in a claim or its equivalent range.

  The invention of the present application is a technique that can be used when a plurality of viewpoint videos are displayed on a plurality of screens.

S101 Screen position detection step S102 Video acquisition step S103 Camera information input step S104 Viewer position detection step S105 Correction information generation step S106 Video correction processing step S107 Display transmission step S201 Screen position detection step S202 Virtual camera position setting step S203 Viewer position detection Step S204 Correction information generation step S205 CG rendering step S206 Display transmission step

Claims (5)

A display arrangement detecting unit for determining a positional relationship between the plurality of video display devices from an image obtained by photographing a situation in which a plurality of video display devices having a video display panel are arranged;
A shooting position determining unit for determining a shooting position of the image with respect to the video display device from the image;
A video correction unit that corrects a video signal displayed on each of the plurality of video display devices based on the positional relationship of the video display device and the imaging position;
A multi-display control unit that outputs the corrected video signals to the corresponding video display devices;
A video signal generation apparatus comprising: A viewing position detector for detecting the viewing position of the viewer,
The video correction unit corrects video based on the viewing position, the positional relationship of the video display device, and the shooting position.
The video signal generation device according to claim 1. A screen outer frame detection unit for detecting outer frame positions of the plurality of video display devices;
The video correcting unit corrects the video behind the outer frame on the display screens of the video display devices adjacent to each other,
The video signal generation device according to claim 1. The video correction unit corrects the video to be behind the outer frame of the screen by trimming and enlarging the video according to the ratio of the outer frame of the display screen to the screen.
The video signal generation device according to claim 3. A display arrangement detecting unit for determining a positional relationship between the plurality of video display devices from an image obtained by photographing a situation in which a plurality of video display devices having a video display panel are arranged;
A virtual shooting position determination unit that determines a virtual shooting position of the image with respect to the video display device from the image;
A video generation unit configured to generate, by computer graphics, a video signal to be displayed on each of the plurality of video display devices based on the positional relationship of the video display device and the imaging position;
A multi-display controller that outputs the generated video signal to a corresponding video display device;
A video signal generation apparatus comprising: