JP4019785B2 - Image display system, image processing apparatus, and image display method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image display system, an image processing apparatus, and an image display method. Specifically, motion information corresponding to the image signal of the captured image and the motion in the capturing direction is generated. Further, assuming that an image display area having a wider angle of view than the photographed image is provided, image display is performed based on the display signal. Here, the motion of the image is generated by generating the display signal based on the image signal and the motion information so that the image based on the display signal becomes an image that moves the display position of the captured image in the image display area based on the motion information. Accordingly, the image display position is moved in accordance with the above, and an image display with high presence is performed.
[0002]
[Prior art]
In a conventional image display system, for example, an image display system that receives a broadcast signal by a television device and displays an image of a television program on a screen, or an image display system that projects an image on the center of the screen using an image projection device The production of programs and movies is performed on the assumption that various images are displayed in a fixed image frame. For example, when shooting a moving subject, the shooting direction of the video camera is moved in accordance with the movement of the subject so that the subject is positioned within the fixed image frame. In this case, even if the movement of the subject is large, the image of the subject is positioned within the fixed image frame, and the viewer can recognize how the subject is moving according to the amount and direction of movement of the background.
[0003]
[Problems to be solved by the invention]
By the way, in an image display system that projects a captured large space on a two-dimensional plane of a fixed image frame, the spread of the space and the movement in the space are expressed using human illusions. There will be unnaturalness due to matching. For example, when the subject is moving, the subject cannot be recognized unless the viewpoint is moved in accordance with the movement of the subject in the real space. However, in the above-described image display system, the viewer can continue to recognize the subject only by fixing the position of the viewpoint within the fixed image frame, and the viewpoint is moved by an illusion. Unnaturalness is attached.
[0004]
In addition, as described above, production of programs and movies is performed on the assumption that various images are displayed in a fixed image frame, so that the expressive power of images is narrowed and more realistic. The image cannot be displayed.
[0005]
Furthermore, the same is true not only when the captured natural image is displayed on the image display system, but also when an edited image or an image such as a game, CG, or animation is created and displayed.
[0006]
In view of this, the present invention provides an image display system capable of displaying an image with high presence and no unnaturalness.
[0007]
[Means for Solving the Problems]
An image display system according to the present invention is an image display system that displays a captured image that is acquired by moving a shooting direction, and has an image display area that has a wider angle of view than the captured image. Display means for displaying an image; information generating means for generating an image signal of a photographed image based on the photographing; and motion information corresponding to a motion in the photographing direction; and the motion information generated by the information generating means Display position calculating means for determining the display position of the photographed image in the image display area based on the image signal, and a display signal for displaying the photographed image at the display position determined by the display position calculating means The display means generates the image in the image display area based on the display signal generated by the signal generation means. Display image, the display position calculating means, a display image displayed on the display position determined based on the motion information, when it is determined that more than the image display area, the display image is the The display position is set so as not to exceed the image display area .
[0008]
The image processing apparatus is an image processing apparatus that processes an image using an image signal of a captured image acquired by moving in a shooting direction and motion information corresponding to the movement in the shooting direction. Based on the movement information corresponding to the movement in the shooting direction, the display position calculation means for determining the display position of the shot image in the image display area having a wider angle of view than the shot image, and the display position calculation means possess an image moving means for moving the photographed image on the display position, and display signal generating means for generating a display signal for displaying the captured image on the moved display position by the image moving unit from the image signal, the If the display position calculation means determines that the display image displayed at the display position determined based on the motion information exceeds the image display area, the display image is displayed on the image display area. Setting said display position so as not to exceed the is characterized in that.
[0009]
Further, the image display method is an image display method for displaying a captured image acquired by moving the shooting direction, and generating an image signal of the shot image based on the shooting and responding to the movement in the shooting direction. A first step of generating the motion information, a second step of determining a display position of the captured image in an image display region having a wider angle of view than the captured image based on the generated motion information, and the determined A third step of generating a display signal for displaying the photographed image at a display position from the image signal; a fourth step of displaying the photographed image in the image display region based on the generated display signal ; In the second step, when it is determined that the display image displayed at the determined display position exceeds the image display area, the display image is displayed as the image display. It is characterized in determining the display position so as not to exceed the range.
[0010]
In this invention, the motion information generating means is attached to the imaging means, or the information used for controlling the shooting direction of the imaging means is used as the motion information, and the motion information is generated according to the image signal of the shot image and the movement of the shooting direction The motion information is generated simultaneously with the image signal and motion information. In addition, an image display area having a wider angle of view than the captured image is provided, and image display is performed based on the display signal. Here, the image based on the display signal is moved in the image display area based on the movement information on the display position of the captured image, and for example, the image signal and the movement of the captured image are equal to the movement of the captured image. A display signal is generated based on the motion information. The generated image signal and motion information are transmitted via a transmission path, and a display signal is generated based on the transmitted image signal and motion information. Alternatively, the image signal and motion information are accumulated, and then the accumulated image signal and motion information are read to generate a display signal.
[0011]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows the overall configuration of an image display system according to the present invention. This image display system includes an information acquisition device 10, an image processing device 20, and an image display device 30.
[0012]
The information acquisition device 10 generates an image signal DV of the captured image and motion information MV indicating the motion of the image, and supplies the motion information MV to the image processing device 20. The image processing device 20 performs a display position movement process for moving the display position of the captured image according to the movement of the image based on the image signal DV and the movement information MV. Further, a display signal DS corresponding to the image display device 30 is generated based on the image signal after the display position movement process. The image display device 30 has an image display area with a wide angle of view, and performs image display based on the display signal DS. For example, three screens are arranged on the front surface and both side surfaces of the user VP to form a wide-angle image display area. Further, images are projected from projectors 32L, 32C, 32R provided corresponding to the screens 31L, 31C, 31R. As described above, when the projectors 32L, 32C, and 32R are used to display an image in the image display region having a wide angle of view, the image processing apparatus 20 supplies the display signal DSL supplied to the projector 32L and the projector 32C as the display signal DS. The display signal DSC to be generated and the display signal DSR supplied to the projector 32R are generated.
[0013]
FIG. 2 is a diagram illustrating configurations of the information acquisition apparatus 10 and the image processing apparatus 20. The information acquisition apparatus 10 includes an imaging unit (for example, a video camera) 11 and a motion information generation unit 12 that acquires the movement of the imaging unit 11 in the shooting direction and generates motion information MV.
[0014]
The imaging unit 11 generates an image signal DV of the captured image and supplies it to the image processing device 20. The movement information generation unit 12 is configured using an angular velocity sensor or a three-dimensional acceleration sensor using a magnetic sensor, a three-dimensional gyro, or the like, for example, when the shooting direction in the imaging unit 11 is manually moved. By attaching this sensor to the imaging unit 11 and detecting a change in posture of the imaging unit 11, a movement in the shooting direction is acquired. Further, a sensor signal indicating the acquired movement is supplied to the image processing apparatus 20 as movement information MV.
[0015]
When the shooting direction in the imaging unit 11 is remotely controlled, control data for this remote control is used as the motion information MV. For example, as shown in FIG. 3, when control data is supplied as motion information MV from an external remote control device, the motion information MV is supplied to the control unit 16 and also supplied to the image processing device 20. The control unit 16 generates a drive signal GM based on the supplied motion information MV and supplies it to the direction adjustment unit 17. The direction adjustment unit 17 is configured using, for example, a pan head, and the imaging unit 11 is attached to the direction adjustment unit 17. The direction adjustment unit 17 changes the shooting direction of the imaging unit 11 based on the drive signal GM, and performs a pan operation and a tilt operation based on the motion information MV. Thus, when using control data as the motion information MV, it is not necessary to provide a sensor or the like in the imaging unit 11, so that the information acquisition device 10 has a low cost and simple configuration.
[0016]
The display position calculation unit 21 of the image processing device 20 determines the display position of the image based on the motion information MV supplied from the information acquisition device 10, and displays the display position information PS indicating the determined display position as the image moving unit 22. To supply. The image moving unit 22 sets the display position of the image based on the image signal DV supplied from the information acquisition device 10 to the position indicated by the display position information PS supplied from the display position calculation unit 21. For example, as shown in FIG. 4, it is assumed that an image memory 221 is provided corresponding to the image display area AR formed by the screens 31L, 31C, 31R, and the writing position when the image signal DV is written in the image memory 221 is as follows. By controlling based on the display position information PS, the display position of the image can be set to the position indicated by the display position information PS.
[0017]
The display signal generation unit 23 generates a display signal DS for displaying an image whose display position has been moved in accordance with the image display device 30. For example, when the image display device 30 is configured using the screens 31L, 31C, and 31R, the display signals DSL and DSC are used to display images whose display positions have been moved using the screens 31L, 31C, and 31R. , DSR are generated and supplied to the image display device 30.
[0018]
Here, as described above, the image moving unit 22 uses the image memory 221 provided corresponding to the image display area AR to store the image signal of the image whose display position has been moved. Display signals DSL, DSC, and DSR can be easily generated by reading signals for each area corresponding to each screen. For example, the display signal DSL can be easily generated by reading the signal from the area 221L corresponding to the screen 31L. Similarly, display signals DSC and DSR can be easily generated by reading signals from the areas 221C and 221R corresponding to the screens 31C and 31R. If the black level signal is written in the area where the image signal DV is not written in the image memory 221, only the captured image can be displayed on the screen.
[0019]
The image moving unit 22 includes an image memory that stores the image signal DV for one screen, and an address conversion circuit that converts the read address position based on the display position information PS when reading the stored image signal DV. It may be provided. In this case, the display signal generator 23 reads a signal from the image memory by designating a position on the image display area AR, and generates display signals DSL, DSC, and DSR using the read signals. Here, the image moving unit 22 performs address conversion so that the image signal DV stored in the image memory is read out when the position on the designated image display area AR becomes the image display position. If address conversion is performed in this way, display signals DSL, DSC, and DSR can be generated without using an image memory having a storage capacity for the image display area. When the position on the designated image display area AR is not the image display position, a black level signal is supplied to the display signal generation unit 23 so that only the captured image can be displayed on the screen.
[0020]
Next, the operation of the image presentation system will be described. Here, the imaging unit 11 of the information acquisition apparatus 10 generates an image signal DV of SD size (720 pixels × 480 lines) such as the NTSC system. Further, the size of the image display area AR in the image display device 30 is, for example, a size (2160 pixels × 480 lines) in which three SD size screens are arranged in the horizontal direction, and has a wider angle of view than the captured image in the imaging unit 11. An image display area is provided. In the present embodiment, a case where the vertical size of an image based on the image signal DV is equal to the vertical size of the image display area AR will be described in order to simplify the description.
[0021]
The motion information generator 12 of the information acquisition device 10 generates, for example, an angular velocity Yaω in the horizontal direction as the motion information MV based on the output signal from the sensor and control data. The display position calculation unit 21 of the image processing apparatus 20 calculates the display position of the image based on the motion information MV. In the calculation of the display position, the display position is set so that the shooting direction of the imaging unit 11 matches the image presentation direction when viewed from the viewer. Therefore, as shown in FIG. 5, the reference direction RF of the imaging unit 11 is the front direction when viewed from the viewer, and the shooting direction of the imaging unit 11 is rightward with respect to the reference direction RF as shown in FIG. 5A. 5 has an angle “φ” in the right direction with respect to the front direction, as shown in FIG. 5B. In FIG. 5, the angle “θ” is a horizontal viewing angle when the viewer recognizes one entire screen. That is, using three screens, image display is performed within the range of the viewing angle “3θ”.
[0022]
As described above, when the image display range is limited, when the image display position is moved in accordance with the shooting direction of the imaging unit 11, when the angle “φ” is large, the image display position is changed from the screen. It will come off. Therefore, in order to represent the horizontal movement of the imaging unit 11 as faithfully as possible within a limited image display range, the display position of the image is moved according to the angular velocity Yaω supplied as the movement information MV from the movement information generation unit 12.
[0023]
FIG. 6 shows the display position calculation operation. When the update frequency of the image to be displayed, for example, the frame frequency is 60 Hz, the moving angle of the imaging unit 11 per (1/60) second can be calculated by dividing the angular velocity Yaω by time, and this moving angle is “ω”. And As shown in FIG. 6, since the image display area AR is a polygonal surface, strictly speaking, although the coordinates on the image and the angle in the horizontal direction are different, the image display area AR is approximated to a cylindrical surface, The calculation is performed assuming that the horizontal angle and the coordinate position in the horizontal direction have a linear relationship.
[0024]
When setting the conditions for calculating the display position in this way, as shown in FIG. 6A, the shooting direction is moved by the movement angle ω between the “n−1” frame and the “n” frame. When indicated by the motion information MV, by substituting this movement angle ω into the equation (1), the display position “Xn” indicating the center of the “n” frame image to be displayed can be determined as shown in FIG. 6B. .
Xn = Xn-1 + ω (L / θ) (1)
Note that “L” is the number of horizontal pixels on one screen (for example, L = 720 pixels), and “θ” is the horizontal viewing angle (for example, 48 degrees) as described above. “Xn−1” indicates the coordinates of the image one frame before, and the unit of “Xn” is a pixel.
[0025]
Here, when the front is the coordinate reference “0”, the display image is displayed when the display position “Xn” is smaller than “−L” or when the display position “Xn” is larger than “L”. The area AR is exceeded. Therefore, when the display position “Xn” satisfies the condition “−L ≦ Xn ≦ L”, the display image is moved so that the center of the image becomes the display position “Xn” calculated by the equation (1). Further, when the display position “Xn” satisfies the condition “Xn <−L”, the left end of the image display area AR and the left end of the display image are prevented so that the display image does not exceed the image display area AR. The position of the display image is set so that. Further, when the display position “Xn” satisfies the condition of “Xn> L”, the right edge of the image display area AR and the right edge of the display image are arranged so that the display image does not exceed the image display area AR. The position of the display image is set so as to match.
[0026]
By determining the display position in this way, as shown in FIG. 7A, when the imaging unit 11 performs imaging so that the moving subject OB is positioned at the center position of the imaging screen and generates the image signal DV, As shown in FIG. 7B, the moving object OB is moved and displayed on the screen of the display device 30 in accordance with the actual movement, so that it is possible to display an image with high presence and no unnaturalness. it can. Further, even when the angle “φ” of the shooting direction with respect to the reference direction is increased, it is possible to prevent the display image from being out of the image display area AR.
[0027]
Furthermore, the process of displaying an image by moving the display position in accordance with the actual movement can be performed in real time or non-real time .
When performing image display in real time, as shown in FIG. 8, the information acquisition apparatus 10 is provided with a transmission unit 19 that generates and transmits a transmission signal TD based on the image signal DV and the motion information MV. In addition, the image processing apparatus 20 includes a receiving unit 25 that receives the transmission signal TD, supplies the motion information MV to the display position calculation unit 21, and supplies the image signal DV to the image moving unit 22. The receiving unit 25 is connected via a wired or wireless transmission path. If the transmission unit 19 performs the encoding process and the reception unit 25 performs the decoding process, the signal amount of the transmission signal TD can be reduced. If multiplexing processing is performed, images of different programs can be transmitted simultaneously.
[0028]
When image display is performed in non-real time, the image signal DV and the motion information MV generated by the information acquisition device 10 are supplied to the information storage device 40 as shown in FIG. 9A. The information storage device 40 stores the image signal DV and the motion information MV on a recording medium using optical means or magnetic means. The recording medium may be one which is secured to the information storage device 40, or may be one interchangeable. When performing image display, the image signal DV and motion information MV stored in the information storage device 40 are read out and supplied to the image processing device 20, so that the display position can be adjusted according to the motion of the image as described above. Can be moved. When the image signal DV and the motion information MV are stored in the information storage device 40, if the timing of the image signal DV and the motion information MV do not match, the movement of the image and the movement of the display position are not synchronized, and the motion of the image is not synchronized. There is a risk that the display position may be moved when there is not. For this reason, the image signal DV and the motion information MV are stored in the information storage device 40 so that they can be reproduced in synchronization.
[0029]
Further, the information stored in the information storage device 40 is not limited to the image signal DV and the motion information MV, and as shown in FIG. 9B, the display signals DSL, DSC, DSR generated by the image processing device 20 are stored. You can also. If the display signals DSL, DSC, DSR stored in the information storage device 40 are read and supplied to the image display device 30, the display position can be moved in accordance with the movement of the image.
[0030]
Thus, if the image signal DV and the motion information MV are stored in the information storage device 40, the amount of information can be reduced as compared with the case where the display signals DSL, DSC, DSR are stored. Further, if the display signals DSL, DSC, and DSR are accumulated, it is not necessary to perform arithmetic processing when displaying an image, so that the image can be displayed easily.
[0031]
Furthermore, since the image display can be performed in real time or non-real time, for example, the station side transmits the image signal DV and the motion information MV at the time of transmitting or distributing the program. If the image processing device 20 and the image display device 30 are provided on the viewer side, the program can be viewed with a high sense of reality. If the information storage device 40 in FIG. 9 uses a magnetic tape, an optical disk, or the like, and is made into a package media that can be distributed or sold as shown in FIG. 10A, the package media 41 is reproduced by the player 50. As a result, it is possible to present an image with high presence. Furthermore, it is possible to widely distribute contents that can present highly realistic images. Note that when the image signal DV and the motion information MV are recorded on the package medium 41, the player 50 is assumed to have the function of the image processing device 20 described above.
[0032]
Further, if a storage 42 such as a hard disk device is used as the information storage device 40, as shown in FIG. 10B, for an electronic movie theater or the like that displays an image on a screen without using a film, etc. By distributing the information stored in the storage 42, a more realistic and highly realistic movie appreciation is possible. Note that when the image signal DV and the motion information MV are recorded in the storage 42, the storage playback device 55 has the function of the image processing device 20 described above.
[0033]
By the way, in the above-described embodiment, the case where the image is moved only in the horizontal direction has been described. However, when the size of the image display area AR in the vertical direction is larger than the size of the display image, the same as in the case of the horizontal direction. It is also possible to move the image in the vertical direction. Further, the present invention is not limited to the multi-plane screen composed of the screens 31L, 31C, 31R, and the same processing can be performed even with a curved screen, a cylindrical screen, or a 360 degree screen.
[0034]
【The invention's effect】
According to the present invention, the display position of the captured image in the image display area having a wider angle of view than the captured image is determined based on the motion information, and a display signal for displaying the captured image at the display position is generated from the image signal. Based on this display signal, the captured image is displayed in the image display area. Thus, since the image display position is moved according to the movement of the captured image, it is possible to display an image with high presence.
For example, when it is determined that the display image displayed at the display position exceeds the image display area, the display position is determined so that the display image does not exceed the image display area. Thereby, when the angle of the photographing direction with respect to the reference direction becomes large, it is possible to prevent the display image from being out of the image display area.
[0035]
Further, the motion information generating means is attached to the imaging means, and the image signal and the motion information are generated simultaneously. In addition, since the information used for controlling the shooting direction of the imaging means is the motion information, the display position can be moved in accordance with the motion of the image.
[0036]
Further, since the display signal is generated so that the movement in the shooting direction is equal to the movement of the display position of the shot image, the viewer can acquire the same space or movement as in reality during viewing.
[0037]
Further, since the generated image signal and motion information are transmitted through the transmission path and processed, it is possible to display an image in real time. Further, since the image signal and the motion information are accumulated, and then the accumulated image signal and motion information are read and the display signal is generated, it is possible to display an image with high presence even in non-real time.
[Brief description of the drawings]
FIG. 1 is a diagram illustrating an image display system.
FIG. 2 is a diagram illustrating a configuration of an information acquisition apparatus and an image processing apparatus.
FIG. 3 is a diagram illustrating another configuration of the information acquisition apparatus.
FIG. 4 is a diagram illustrating a configuration of an image memory.
FIG. 5 is a diagram illustrating a relationship between an imaging direction and an imaging presentation direction.
FIG. 6 is a diagram illustrating a display position calculation operation.
FIG. 7 is a diagram illustrating an operation of the image display system.
FIG. 8 is a diagram illustrating a configuration for displaying an image in real time.
FIG. 9 is a diagram showing a configuration for displaying an image in non-real time.
FIG. 10 is a diagram illustrating another configuration when image display is performed in non-real time.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 ... Information acquisition apparatus, 11 ... Imaging part, 12 ... Motion information generation part, 16 ... Control part, 17 ... Direction adjustment part, 19 ... Transmission part, 20 ... Image processing device, 21 ... display position calculation unit, 22 ... image moving unit, 23 ... display signal generation unit, 25 ... reception unit, 30 ... image display device, 40 ... information Storage device 41 ... Package medium 42 ... Storage 50 ... Player 55 ... Storage playback device 221 ... Image memory

Claims (11)

An image display system that displays a captured image acquired by moving a shooting direction,
Display means for displaying the captured image having an image display area having a wider angle of view than the captured image;
Information generating means for generating an image signal of a captured image based on the image capturing and generating motion information according to the motion in the capturing direction;
Display position calculating means for determining a display position of the captured image in the image display area based on the motion information generated by the information generating means;
Signal generating means for generating, from the image signal, a display signal for displaying the captured image at the display position determined by the display position calculating means;
The display means includes
Displaying the captured image in the image display area based on the display signal generated by the signal generating means ;
The display position calculating means includes
When it is determined that the display image displayed at the display position determined based on the movement information exceeds the image display area, the display position is set so that the display image does not exceed the image display area. An image display system characterized by setting . An image moving means for moving the captured image to the display position determined by the display position calculating means;
The image display system according to claim 1, wherein the signal generation unit generates the display signal so as to display the captured image at a position moved by the image moving unit. The information generation means includes an imaging means for generating an image signal of a captured image, and a motion information generation means for generating the motion information,
The movement information generating means is formed using a sensor,
2. The image display system according to claim 1, wherein the image signal and the motion information are generated simultaneously by attaching the motion information generation means to the imaging means. The information generation unit includes an imaging unit that generates an image signal of a captured image, and a direction control unit that controls a shooting direction of the imaging unit,
2. The image display system according to claim 1, wherein the image signal and the motion information are generated simultaneously by using the information used for controlling the photographing direction by the direction control means as the motion information. The image display system according to claim 1, wherein the signal generation unit generates the display signal so that the movement in the photographing direction is equal to the movement of the display position of the photographed image. Supplying the image signal generated by the information generating means to the image moving means via a transmission path;
The image display system according to claim 2, wherein the motion information generated by the information generation unit is supplied to the display position calculation unit via a transmission path. The image signals generated by the information generating means and the motion information, and accumulated in the information storing means,
The image display system according to claim 1, wherein the image processing unit generates the display signal using an image signal and motion information stored in the information storage unit. An image processing apparatus that processes an image using an image signal of a captured image acquired by moving a shooting direction and movement information according to a movement in the shooting direction,
Display position calculating means for determining a display position of the captured image in an image display area having a wider angle of view than the captured image based on movement information according to the movement in the capturing direction;
Image moving means for moving the captured image to the display position determined by the display position calculating means;
A display signal for displaying the captured image on the moved display position by the image moving unit have a display signal generating means for generating from said image signal,
The display position calculating means includes
When it is determined that the display image displayed at the display position determined based on the movement information exceeds the image display area, the display position is set so that the display image does not exceed the image display area. An image processing apparatus characterized by setting .   9. The image processing according to claim 8, wherein the display position calculation unit determines the display position based on the movement information so that the movement in the shooting direction is equal to the movement of the display position of the shot image. apparatus. An image display method for displaying a captured image acquired by moving a shooting direction,
A first step of generating an image signal of a photographed image based on the photographing and generating motion information according to a motion in the photographing direction;
A second step of determining a display position of the captured image in an image display region having a wider angle of view than the captured image based on the generated motion information;
A third step of generating a display signal for displaying the photographed image at the determined display position from the image signal;
A fourth step of displaying the captured image in the image display area based on the generated display signal ,
In the second step,
The display position is determined so that the display image does not exceed the image display area when it is determined that the display image displayed at the determined display position exceeds the image display area. Display method. 11. The image display method according to claim 10, wherein the display signal is generated so that the movement in the photographing direction is equal to the movement of the display position of the photographed image.

Images