JP2000184396A - Video processor, its control method and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a stereo video image without a sense of incongruity to an operator at a remote terminal by unifying lightness of stereo images obtained by 2-system of image pickup means and transmitting them to the remote terminal. SOLUTION: The video processor has two systems of image pickup means 1-1, 1-4 and video images obtained by each of the means are fed to a RAM 1-10 via an I/O 1-13. Then a CPU 1-9 detects luminance of corresponding point with correlation in the respective video images and corrects the luminance to be the same for both the images substantially. Then the corrected video image is transmitted to a remote terminal via a communication control section 1-14 and a network 1-19.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a video processing apparatus, a control method thereof, and a storage medium, and more particularly to a video processing apparatus for transmitting a captured video to a remote terminal via a predetermined network line, a control method thereof, and a storage medium. It is.

[0002]

2. Description of the Related Art Image information captured by a video camera or the like is transmitted to a remote terminal via a so-called network represented by the Internet, and the captured video is live at the remote terminal, ie, a moving image. Displayable technologies have been put to practical use.

[0003] Using this technique, a technique of transmitting two systems of images taken by two video cameras arranged in parallel to a remote place, and displaying a stereoscopic three-dimensional image live in the remote place. Consider.

In general, when attempting to construct this type of photographing system, it would be conceivable to use a camera module or the like built in a home video camera supplied at low cost.

[0005]

However, a camera module or the like built in a home video camera is an AE.
In many cases, the (auto exposure) function cannot be released. Therefore,
What was not a problem when transmitting an image obtained by one video camera may be a problem when transmitting a pair of stereo images obtained by two video cameras. The reason for this is that, because the AE function of the video camera cannot be released, images shot with different AE values are transmitted.

[0006] Even if the photographing can be performed with the same AE value or the AE function can be canceled, it is not always the case that completely the same luminance signal can be obtained due to the individual difference of each camera module. If a pair of stereo images obtained by such a video camera is transmitted as it is, when reproducing the stereo image on a remote terminal, images of different brightness are presented to the left and right eyes of the observer. It is inconvenient because you can't wipe out the discomfort.

[0007]

SUMMARY OF THE INVENTION The present invention has been made in view of the above circumstances, and it is possible to unify the brightness of stereo images obtained by two types of imaging means and transmit the stereo images to a remote terminal, thereby enabling remote control. It is an object of the present invention to provide an image processing apparatus, a control method thereof, and a storage medium that enable a terminal operator to provide a stereo image without a sense of discomfort.

A video processing apparatus for distributing video information obtained from at least two systems of imaging means via a predetermined network, wherein said detection means detects lightness information in each video obtained by said imaging means. And, based on the brightness information in each video detected by the detection means,
A correcting unit for correcting one of the images; and a transmitting unit for combining the image corrected by the correcting unit and transmitting the combined image via the network.

[0009]

Embodiments of the present invention will be described below in detail with reference to the accompanying drawings.

The basic concept of the present embodiment will be briefly described as follows.

[0011] In a video distribution device (camera server) capable of capturing in real time two video signals (stereo video signals) obtained by two video cameras arranged in parallel, the two captured videos are compared. , Correct the brightness so that the corresponding portions of the respective images have the same luminance value, reduce and combine the two corrected images to re-synthesize them as one image, and It transmits stereo images to remote terminals (clients).

Hereinafter, a configuration for realizing this and an operation thereof will be described.

FIG. 1 is a block diagram of the video distribution apparatus of the present embodiment.

The hardware includes photographing devices 1-1 and 1-4 and a video processing device 1-8. The video processing device 1-8 has hardware for capturing video from a video camera, but can be realized by a general-purpose information processing device (for example, a personal computer) and software.

Photographing device 1-1 (similarly for photographing device 1-4)
Denotes an imaging unit 1-2 (1-5) for inputting an image and an exposure control unit 1-3 (1-6) for automatically determining an exposure state at the time of shooting.
Consists of Here, the parameters for determining the exposure state include a shutter speed, an iris, a gain, and a white balance, but are not particularly limited here. In the following description, these are collectively referred to simply as exposure parameters for simplicity.

The video output signal of the imaging unit 1-2 (1-5) has an NTSC format or a YC separation system, and the video is output via an I / O unit 1-13 provided in a video processing device 1-8. The data is sequentially stored in the RAM 1-10 in the video processing device 1-8. At this time, it is assumed that the I / O 1-13 has two suitable interfaces called a capture board or the like capable of capturing an image in real time.

The exposure control section 1-3 (1-6) has a function of automatically adjusting the lightness balance with respect to the entire screen being photographed. The exposure control unit 1-3 (1-6) refers to a luminance signal obtained from the imaging unit 1-2 (1-5) through a control signal line, and sets an optimal exposure parameter calculated based on the information. The imaging unit 1-2 (1-
It is possible to instruct 5).

Image pickup section 1-2 (1-5) and exposure control section 1
The photographing apparatus 1-1 (1-4) including -3 (1-6)
This can be easily realized by using a camera module built in a home video camera.

A video processing device 1-8 is a RAM 1-1 for controlling the CPU 1-9, which controls the entire device, and a RAM 1 for loading and executing an OS, an application functioning as a camera server, and the like, and the like. -1
0 and the following configuration.

1-11 is a boot program and a BIOS.
Is a secondary storage device (such as a hard disk device) that stores an OS, a program functioning as a camera server, and the like. 1-13
Is an I / O for communicating with the photographing apparatus 1-1 (1-14), and 1-14 is a communication control unit for transmitting a stereo image to a remote place via the network 1-19. 1-1
Reference numeral 8 denotes a bus that connects the above processing units. The video processing device 1-8 can be realized by a general-purpose computer as described above. A keyboard 1 is connected to the bus 1-18.
-15, pointing device 1-1 such as mouse
6. The video monitor 1-17 is connected.

The video processing device 1-8 generates a stereo video by performing the processing described later on the two systems of video obtained from the photographing devices 1-1 and 1-4, and generates a stereo video.
A process of transmitting a stereo image to a remote terminal (client) through -19 is performed.

The basic principle of this embodiment will be described with reference to FIG.

FIG. 1A shows a state in which two video cameras arranged in parallel are simultaneously photographing a certain scene. Reference numerals 2-1 and 2-2 denote video cameras, which correspond to the photographing apparatuses 1-1 and 1-4 in FIG. 2-
Reference numeral 3 denotes a scene shot by two video cameras, and reference numerals 2-4 and 2-5 denote video cameras 2 respectively.
1 shows a video at a certain point in time of a video signal corresponding to the scene captured by -1 and 2-2.

Since the two video cameras 2-1 and 2-2 have different photographing positions, the images 2-4 and 2
At -5, the appearance of the scene 2-3 is slightly different. Further, the proper exposure determination results of the exposure control units 1-3 and 1-6 provided in the video cameras 2-1 and 2-2 are different, or individual differences between the video cameras 2-1 and 2-2 are caused. In the images 2-4 and 2-5, the brightness of the entire screen is not the same.

FIG. 2B shows the principle of generating a stereo image by inputting the two-system video signals photographed in FIG. 2A to the image processing device 1-8.

Reference numeral 2-6 in the drawing corresponds to the image processing apparatus 1-8, and schematically shows how two systems of images input into the apparatus are processed and output. .
2-7 and 2-8 are two video cameras 2-1 and 2-
2 shoots and is captured by I / O1-13 and R
It is a video of two systems stored in AM1-10. The image obtained by correcting the correspondence between the two systems of images by using a known image processing technique such as finding a correspondence that maximizes the cross-correlation value and correcting the corresponding locations to the same brightness. Are 2-9 and 2-10.

Next, images 2-11 and 2-12 are obtained by compressing these images to half the size in the vertical direction by the well-known image processing technique. Finally, the two compressed images 2-11 and 2-12 are integrated into one image 2-13, and the finally generated image 2-13 is transmitted to a remote location through a network. Called to the terminal.

The program for realizing the above processing is in the secondary storage 1-12, which is loaded into the RAM 1-10 and executed by the CPU 1-9.
Needless to say, the device may be configured as a dedicated device.

On the other hand, the remote terminal receiving the video 2-13 (which can also be constituted by a general-purpose information processing device) reverses the above-described processing. That is, video 2
-13 is separated in the vertical direction, and the obtained two systems of images are respectively expanded in the vertical direction. The two systems of images obtained in this way are presented as a pair of stereo images. Regarding the method of presentation, various methods such as a head-mounted display, a lenticular screen, a liquid crystal shutter, and polarizing glasses can be considered, but these are not an essential part of the present invention and are not particularly limited.

As described above, before transmitting a stereo image obtained by the two video cameras to the network, means for comparing and correcting the luminance signals thereof is provided. Since the brightness of a pair of stereo signals can be unified, it is possible to provide a stereo image with less discomfort to a remote place at the transmission destination.

FIG. 3 is a flowchart showing a basic processing procedure of the video processing apparatus of the present embodiment. The overall flow is to unify the brightness of two systems of images taken by two video cameras, recompose one image by combining both images, and transmit the combined image.

First, two systems of video images captured using the two video cameras 1-1 and 1-4 are input to the video processing device 1-8 (step S301). this is,
This is performed via the I / O 1-1 provided in the video processing device 1-8. The input two video images are immediately stored in RAM 1-
10 (or the secondary storage 1-12) (step S302). Next, in step S303, RAM1-
The corresponding relationship is obtained by using the cross-correlation value for the two systems of video stored in the secondary storage 10 (or the secondary storage 1-12). Here, the correlation between the two videos is extracted by calculating the cross-correlation value between the two videos and finding the association that maximizes the value. Next, in step S304, brightness correction is performed using the correspondence relationship between the obtained two systems of video so that the two videos have the same brightness. In this method, the brightness correction amount of the video is calculated by comparing the luminance information of the corresponding portions in the video, and the brightness correction of one (or both) video is performed using the correction amount.

Next, the two systems of images having the same brightness are reduced to half the size in the vertical direction, respectively (step S30).
5) The reduced two-system video is combined in the vertical direction and recombined as one video (step S306). These processes can also be realized by a known image processing technique.
Thus, the generated stereo image is transmitted to the network 1-
Is transmitted to a remote place via the remote control 19 (step SS30).
7).

The purpose of performing the processing of steps S305 and S306 is that the two systems are originally a pair of stereo images, and if these are transmitted separately, the two systems are synchronized on the receiving side. Need to be taken,
This is to reduce the occurrence of a situation that is inconvenient for stereoscopic video, such as when one video is lost in the network being transmitted.

In the process of step S304,
There are several methods for correcting the brightness of the two images based on the correspondence between the two systems of images, and a simple example will be described below.

When a correspondence relationship in which the cross-correlation value between two images is maximized is obtained, how much the pixel position of one image (hereinafter, referred to as image A) is different from that of the other image (hereinafter, referred to as image B). Only one piece of information as to whether the image is out of alignment is obtained for the entire image. The amount of this displacement is represented as (δx, δy). Pixels of video A are A (x, y), pixels of video B are B
When expressed as (x, y), A (x, y) becomes B (x + δx, y
+ Δy) (of course, this is not the case if the boundary of the image is exceeded), so that the image A is raster-scanned, and A (x, y) is obtained for each pixel.
And the difference between the luminance signals of B (x + δx, y + δy) as e (x,
y), and calculating the average value of e (x, y) in the entire video, it is possible to calculate the luminance correction amount for all the pixels of the video B.

The above is an example of the method of correcting the brightness of an image when utilizing the correspondence that maximizes the cross-correlation value between the images. However, this is, of course, an example of a very simple process. Improvement of correction accuracy can be expected by an appropriate method.

<Second Embodiment> A second embodiment will be described. In the second embodiment, in addition to the first embodiment, a DSP (Digital Signal Processor) is added to execute processing of a video signal by hardware. As described above, by executing the image processing with relatively high computational cost by hardware, the processing speed of the video signal can be increased and the real-time property can be further improved.

FIG. 4 is a block diagram showing a hardware configuration of the second embodiment. Hereinafter, only portions different from those in FIG. 1 will be described. In FIG. 4, the photographing devices 1-1 and 1
DSP-1-7 is provided between -4 and the video processing device 1-8. The DSP 1-7 is connected to the photographing apparatuses 1-1 and 1-4 via video signal lines, and captures two systems of video images photographed by the photographing apparatuses 1-1 and 1-4, as described in the first embodiment. It has a function to execute such video processing at high speed by hardware.

The basic principle and processing procedure of the second embodiment are based on the case of the first embodiment shown in FIGS. However, since high-speed video processing can be performed using the DSP 1-7, it is possible to implement an image processing method having a higher calculation cost than in the first embodiment. In the first embodiment, a cross-correlation value is used as a method for obtaining correspondence between videos. In the second embodiment, a method for obtaining correspondence using stereo correspondence search is employed.

In the first embodiment, the processing procedure (S303)
Has used a cross-correlation value as a method of obtaining correspondence between videos. This method has a relatively low computational cost and can be executed at high speed, but has a drawback that only one correspondence can be extracted from the entire video. That is, since the two photographing devices 4-1 and 4-4 have different photographing positions, the appearance of the scene is also slightly changed, and it is impossible that the entire image is accurately corresponded without inconsistency. Therefore, in the second embodiment, the processing procedure (S
In 303), a stereo correspondence search that can obtain a correspondence relationship in units of one pixel (or in units of blocks of a certain size), not the entire video, is used. Since this is originally intended to transmit a pair of stereo images obtained from two photographing devices arranged in parallel to a remote place, the accuracy of the photographing device can be improved even if attention is paid only to the arrangement method of the photographing devices. The effectiveness can be expected because some advantageous conditions for realizing a high stereo correspondence search have already been satisfied. The stereo correspondence search is also a well-known image processing technique like the cross-correlation, and a description of a specific algorithm will be omitted.

As described above, the introduction of the DSP and the stereo correspondence search can be expected to realize higher-speed and more accurate brightness correction.

<Third Embodiment> A third embodiment will be described. In order to correct the brightness of images obtained from two video cameras, a cross-correlation value is used in the first embodiment, and a stereo correspondence search is used in the second embodiment. In the present embodiment, in order to reduce the processing cost, the brightness is corrected using only the histogram conversion of the video without finding the correspondence between the videos.

The hardware configuration and the basic principle of the third embodiment are based on the case of the first embodiment shown in FIGS. Since the processing procedure of the third embodiment shown in FIG. 5 is similar to that of the first embodiment shown in FIG. 3, only the different parts will be described below.

Step S of the processing procedure of the third embodiment
Steps 503 and S504 correspond to steps SS303 and S304 in the processing procedure in the first embodiment. In the first and second embodiments, a correspondence between videos is obtained, and the brightness is determined based on the correspondence. Was corrected.
As described above, in the third embodiment, a histogram is obtained (step SS503), and brightness correction is performed by performing histogram conversion of one (or both) images (step S504). Since the histogram conversion is also a known image processing technique, a detailed description is omitted.

As described above, the calculation cost can be reduced by correcting the brightness difference between the images by using a simple method without finding the correspondence between the images, so that expensive equipment such as a DSP is introduced. A certain degree of brightness correction can be realized even without it.

Further, in the embodiment, the image to be corrected has been described as being applied to an image obtained by one of the predetermined photographing devices (one of the video cameras). You may. For example, if there are two images A and B and the overall luminance (total value) of the image A is smaller than the image B, it means that the dynamic range of the image A is narrow. In other words, the video B has a wider dynamic range. When making corrections,
Since the wide range encompasses the characteristics of the narrow range,
In this case, the image B is to be corrected.

Also, in the first, third and second embodiments, a part thereof is realized by software. As described above, since the video processing device described in the embodiment can be realized by a general-purpose information processing device, the present invention records the program code of software that realizes the functions of the above-described embodiment. It is needless to say that the storage medium is supplied to a system or an apparatus, and a computer (or a CPU or an MPU) of the system or the apparatus reads out and executes the program code stored in the storage medium.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiment, and the storage medium storing the program code constitutes the present invention.

As a storage medium for supplying the program code, for example, a floppy disk, hard disk, optical disk, magneto-optical disk, CD-ROM, CD
-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.

When the computer executes the readout program code, not only the functions of the above-described embodiment are realized, but also the OS (Operating System) running on the computer based on the instruction of the program code. ) May perform some or all of the actual processing, and the processing may realize the functions of the above-described embodiments.

Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, based on the instructions of the program code, It goes without saying that the CPU included in the function expansion board or the function expansion unit performs part or all of the actual processing, and the processing realizes the functions of the above-described embodiments.

As described above, according to the present embodiment, a pair of stereo signals is obtained by comparing and correcting the luminance signals of the stereo images obtained by the two video cameras before transmitting them on the network. Since the brightness of the image can be unified, a stereo image with less discomfort can be provided to a remote place at the transmission destination.

[0054]

According to the present invention as described above,
By unifying the brightness of the image obtained by each of the two imaging units and transmitting the unified brightness to the remote terminal, it is possible to provide the remote terminal operator with a stereoscopic image without a sense of incongruity.

[Brief description of the drawings]

FIG. 1 is a schematic configuration block diagram of an embodiment.

FIG. 2 is an explanatory diagram of a basic principle of the embodiment.

FIG. 3 is a flowchart illustrating an operation processing procedure of the apparatus according to the embodiment.

FIG. 4 is a schematic block diagram according to a second embodiment.

FIG. 5 is a flowchart illustrating an operation processing procedure according to the third embodiment.

Continuation of the front page (72) Inventor Hiroaki Sato 3-30-2 Shimomaruko, Ota-ku, Tokyo F-term in Canon Inc. (reference) 5C022 AA11 AB01 AB65 AB68 AC42 AC69 AC75 5C054 AA02 CC02 DA06 EA01 EA05 EA07 ED01 FD01 FD02 FE06 GA01 GA04 GB01 HA18 5C061 AA29 AB01 AB04 AB08 AB11 AB12 AB17 AB21 AB24

Claims (7)

[Claims] 1. A video processing apparatus for distributing video information obtained from at least two systems of image capturing means via a predetermined network, wherein lightness information in each video obtained by said image capturing means is detected. A detecting unit, a correcting unit that corrects one of the images based on the brightness information in each of the images detected by the detecting unit, and a transmission transmitting the combined image corrected by the correcting unit via the network. A video processing apparatus comprising: 2. The image processing apparatus according to claim 1, wherein said detecting means detects a luminance of a correlated corresponding point of each image obtained by said image pickup means. 3. The video processing apparatus according to claim 1, wherein said detection means detects the luminance of the entire video obtained by said imaging means. 4. The video processing apparatus according to claim 1, wherein said correction means reduces a luminance difference. 5. The image according to claim 1, wherein, in the combining process in the transmitting unit, both of the two systems of images obtained by the image capturing unit are reduced to a half size and combined. Processing equipment. 6. A control method for a video processing device for distributing video information obtained from at least two systems of image capturing means via a predetermined network, comprising: brightness information in each video obtained by said image capturing means. A detection step of detecting, based on the brightness information in each of the images detected in the detection step, a correction step of correcting one of the images, and combining the image corrected in the correction step via the network And a transmitting step of transmitting. 7. When a computer reads and executes the program,
A storage medium storing a program code functioning as a video processing device for distributing video information obtained from at least two systems of imaging means via a predetermined network, wherein a storage medium stores a program code for each video obtained by said imaging means. Detecting means for detecting lightness information; correcting means for correcting one of the images based on the lightness information in each of the images detected by the detecting means; and transmitting the combined image corrected by the correcting means. A storage medium storing a program code functioning as a means.