JP2003018561A - Pantoscopic video image recording/reproducing system, conference recording/reproducing system, pantoscopic video image transmitting apparatus, conference video image transmitting apparatus, pantoscopic video image reproducing apparatus, conference video image reproducing apparatus, pantoscopic video image recording/reproducing method, conference video image reproducing method, pantoscopic video image transmitting method, conference video image transmitting method, pantoscopic video image reproducing method, conference video image reproducing method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently reproduce the scene of a conference while maintaining the scene of presence. SOLUTION: A conference recording/reproducing system is provided with a hyperboloid mirror 211 for receiving a pantoscopic video image using a vertical direction as the center or the axis, a plurality of microphones 221 for receiving voice, a voice direction detector (not shown) for detecting a direction of the voice on the basis of the voice inputted into the microphones 221, a recording section (not shown) for recording the pantoscopic image, the voice and the voice source direction, a video image deforming section for deforming an image of a prescribed direction according to the voice direction out of the recorded pantoscopic images into a rectangular output image, and a video image outputting section (not shown) for synchronously outputting the deformed image and the sound according to this image.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a wide-angle image recording / reproducing system that handles wide-angle images, a conference recording / reproducing system, a wide-angle image transmitting device, a conference image transmitting device, a wide-angle image reproducing device, a conference image reproducing device, and a wide-angle image recording. The present invention relates to a reproducing method, a conference recording / reproducing method, a wide-angle image transmitting method, a conference image transmitting method, a wide-angle image reproducing method, a conference image reproducing method, and a program.

[0002]

2. Description of the Related Art In recent years, with the development of telecommunications technology, many companies and groups have come to utilize a video conference system for photographing the state of a conference and transmitting the obtained images to a remote place. In order to further improve the convenience of such a system, there have been conventionally proposed a number of devices for capturing the state of a conference as a video and a system for transmitting a partial video cut out from only a speaker.

As such a conventional technique, there is, for example, Japanese Patent Application Laid-Open No. 5-122689, "TV conference system". This publication discloses a technology relating to a video conference system in which a voice input from a microphone is detected to determine a speaker, and a camera control unit automatically controls a camera based on the determination result to capture the speaker. There is.

Further, there is a "television camera device" disclosed in Japanese Patent Laid-Open No. 11-331827. This publication discloses a technique relating to a television camera device using a fish-eye or ultra wide-angle lens and a variable directivity microphone.
Specifically, an invention is disclosed in which a direction of a sound source position is determined, the sound source position direction is tracked, and an image in the sound source position direction is cut out to generate a video signal.

[0005]

However, the conventional techniques have the following problems. That is, JP-A-5-
In the technique disclosed in Japanese Patent No. 122689, it takes a certain amount of time to change the direction of the camera in the direction of the speaker, and there is a problem that the speaker is projected some time after the speaker starts speaking. there were. In addition, there is a problem that a video image is displayed while the camera is moving, which makes it difficult to view a conference image. That is, there is a problem that the sense of presence is impaired.

Further, in the technique disclosed in Japanese Patent Laid-Open No. 11-331827, when installing the television camera device using a fish-eye lens or an ultra wide-angle lens on a desk or the like, a ceiling or the like is generally not so important. There is a problem that an important subject such as a human face exists in the peripheral portion of the visual field, which occupies most of the visual field, and is affected by peripheral dimming and aberration. That is, there is a problem that the conference cannot be efficiently viewed.

In addition, when such a lens is used, the calculation for distortion correction greatly depends on the position of the image, and the calculation load becomes large. In addition, there is a problem in that the design of such a lens or optical system is very difficult and costly.

Further, in recent years, in addition to the so-called real-time characteristic which is a characteristic of the conventional video conference system, there has been a demand for checking the contents of the conference again.

The present invention has been made in view of the above, and it is an object of the present invention to efficiently reproduce a conference while maintaining a sense of presence.

Another object is to provide a small and inexpensive device.

[0011]

In order to achieve the above object, a wide-angle image recording / reproducing system according to claim 1 is provided.
Wide-angle image input means for inputting a wide-angle image centered on or in the vertical direction, a plurality of voice input means for inputting voice, and a sound source for detecting a sound source direction based on the voice input by the plurality of voice input means Direction detection means, wide-angle image input by the wide-angle image input means, voice input by the voice input means, recording means for recording the sound source direction detected by the sound source direction detection means, and the recording means Image transforming means for transforming an image of a predetermined area in a direction corresponding to the sound source direction into a rectangular output image among the wide-angle images recorded by, an image transformed by the image transforming means, and the image. And audio / video output means for outputting in synchronization with the audio recorded in the recording means.

That is, according to the first aspect of the present invention, a wide-angle image is recorded, and when the image is reproduced, the distortion is corrected and the scene centered in the sound source direction is reproduced.

According to a second aspect of the present invention, there is provided a wide-angle image recording / reproducing system in which a wide-angle image input means for inputting a wide-angle image having a vertical direction as a center or an axis and a wide-angle image input by the wide-angle image input means are predetermined. Image transforming means for transforming into a rectangular output image according to the conversion formula, a plurality of voice input means for inputting voice, and a sound source direction for detecting a sound source direction based on the voice input by the plurality of voice input means. Detection means, an image transformed by the image transforming means, a voice input by the voice input means, a recording means for recording the sound source direction detected by the sound source direction detecting means, and a recording means for recording the sound source direction. Image extracting means for extracting an image of a predetermined area in a direction corresponding to the sound source direction among the rectangular images, and an image extracted by the image extracting means. When, characterized by comprising an image and sound output means for outputting is synchronized with the audio recorded in the recording means corresponding to the image.

That is, according to the second aspect of the present invention, an image developed in a panoramic shape is recorded, and a scene centered on the sound source direction can be reproduced at any time.

A wide-angle image recording / reproducing system according to a third aspect of the present invention is the wide-angle image recording / reproducing system according to the first or second aspect, wherein the wide-angle image input means is a mirror body having a paraboloid of a predetermined shape. It is characterized in that it is composed of a specular body having a hyperboloid of a predetermined shape or a specular body having a predetermined conical shape, and an image pickup device.

That is, the invention according to claim 3 captures a wide-angle image with a simple structure.

The wide-angle image recording / reproducing system according to a fourth aspect is the wide-angle image recording / reproducing system according to the first, second or third aspect, wherein the sound source direction detecting means is input by the plurality of audio input means. It is characterized in that the sound source direction is detected based on the time difference between the generated sounds.

That is, the invention according to claim 4 detects the sound source direction with a simple structure.

A wide-angle image recording / reproducing system according to a fifth aspect is the wide-angle image recording / reproducing system according to any one of the first to fourth aspects, further comprising direction correcting means for correcting a direction corresponding to the sound source direction. Is characterized by.

That is, the invention according to claim 5 reproduces a desired video even when the sound source direction is not correctly detected due to noise or the like.

A wide-area image recording / reproducing system according to a sixth aspect is the wide-angle image recording / reproducing system according to any one of the first to fifth aspects, wherein a predetermined area in a direction corresponding to the sound source direction is fixed. It is characterized in that it is provided with a region fixing means.

That is, the invention according to claim 6 prevents the image blur even when the direction of the sound source slightly moves.

A conference recording / reproducing system according to a seventh aspect is a conference recording / reproducing system in which the wide-angle image recording / reproducing system according to any one of the first to sixth aspects is applied to recording / reproducing of a conference. A speaker position determining means for determining the position of the speaker based on the color distribution of the image or a moving part in the image; and a predetermined area determining means for determining the predetermined area based on the determination result of the speaker position determining means, It is characterized by having.

That is, according to the invention of claim 7, the speaker portion is accurately extracted.

According to the eighth aspect of the invention, the wide-angle image transmitting apparatus has a wide-angle image input means for inputting a wide-angle image centered on or in the vertical direction, a plurality of voice input means for inputting voice, and the plurality of voice input means. A sound source direction detecting means for detecting a sound source direction based on a sound input by a sound input means; data about a wide-angle image input by the wide-angle image input means; and data about a sound input by the sound input means, It is characterized by further comprising: data transmission means for transmitting the data regarding the sound source direction detected by the sound source direction detection means to a predetermined data storage means.

That is, the invention according to claim 8 can capture a wide-angle image and reproduce a scene centered on a sound source.

A wide-angle image transmitting apparatus according to a ninth aspect is the wide-angle image transmitting apparatus according to the eighth aspect, in which a wide-angle image input by the wide-angle image input means is converted into a rectangular output image by a predetermined conversion formula. Image transformation means for transforming so that the data transmission means transmits the data relating to the image transformed by the image transformation means in place of the data relating to the wide-angle image input by the wide-angle image input means. Is characterized by.

That is, the invention according to claim 9 can reproduce an image developed in a panoramic shape.

According to a tenth aspect of the present invention, there is provided the wide-angle image transmitting apparatus according to the ninth aspect, wherein the image is deformed by the image deforming means and detected by the sound source direction detecting means. The data transmission means includes image extraction means for extracting an image of a predetermined region in the sound source direction, and the data transmission means converts the data regarding the wide-angle image input by the wide-angle image input means and the data regarding the sound source direction detected by the sound source direction detection means. Alternatively, or in addition to the data on the wide-angle image input by the wide-angle image input means and the data on the sound source direction detected by the sound source direction detecting means, the data on the image extracted by the image extracting means may be transmitted. Characterize.

That is, the invention according to claim 10 can reproduce a scene centered on the sound source direction.

The wide-angle image transmitting device described in claim 11 is the wide-angle image transmitting device according to claim 8, 9 or 10, wherein the wide-angle image input means is a mirror surface body having a paraboloid of a predetermined shape. It is characterized in that it is composed of a specular body having a hyperboloid of a predetermined shape or a specular body having a predetermined conical shape, and an image pickup device.

That is, the invention according to claim 11 captures a wide-angle image with a simple structure.

A wide-angle image transmitting device according to a twelfth aspect is the wide-angle image transmitting device according to any one of the eighth to eleventh aspects, wherein the sound source direction detecting means is constituted by the plurality of voice input means. It is characterized in that the sound source direction is detected based on the time difference between the inputted voices.

That is, according to the twelfth aspect of the invention, the sound source direction is detected with a simple structure.

A wide-angle image transmitting device according to a thirteenth aspect is the wide-angle image transmitting device according to the twelfth aspect.
The sound source direction detecting means detects the sound source direction based on a time difference between a voice input by a certain voice input means and a voice input by a voice input means farthest from the voice input means. And

That is, according to the thirteenth aspect of the present invention, the sound source direction is detected with high accuracy.

A wide-angle image transmitting apparatus according to a fourteenth aspect is the wide-angle image transmitting apparatus according to any one of the eighth to eleventh aspects, wherein the plurality of audio input means are directional microphones. It is characterized in that the sound source direction detecting means detects the sound source direction based on the strength of the voice input by the directional microphone.

That is, the invention according to claim 14 detects the sound source direction with a simple structure.

A wide-angle image transmitting apparatus according to a fifteenth aspect is the wide-angle image transmitting apparatus according to any one of the eighth to fourteenth aspects, wherein the plurality of voice input means are arranged at the center of gravity of the voice input means. It is characterized in that the respective positions are arranged so as to substantially coincide with the optical center of the wide-angle image input means.

That is, according to the fifteenth aspect of the present invention, various calculations can be simplified by matching the coordinate system of the voice input means with the coordinate system of the wide-angle image input means.

A wide-angle image transmitting device according to a sixteenth aspect is the wide-angle image transmitting device according to any one of the eighth to fifteenth aspects, in which the plurality of audio input means and the image pickup device are provided on the pedestal side. The mirror surface is arranged facing the pedestal via a transparent member, or the plurality of voice input means, the image pickup device and the mirror surface are arranged on the pedestal side, It is characterized in that a second mirror body that reflects the reflected light from the mirror body toward the image pickup device on the pedestal side is arranged to face the pedestal through a transparent member.

That is, according to the sixteenth aspect of the present invention, the electric system is embedded in the pedestal to prevent the division of the image due to the conductive wire or the like.

A wide-angle image transmitting apparatus according to a seventeenth aspect is the wide-angle image transmitting apparatus according to any one of the eighth to sixteenth aspects, in which the speaker elevation angle is based on a plane on which the apparatus is installed. The image extraction means extracts an image based on the sound source direction detected by the sound source direction detection means and the elevation angle set by the elevation angle setting means.

That is, the invention according to claim 17 extracts an image with high accuracy.

A conference image transmitting apparatus according to a eighteenth aspect is a conference image transmitting apparatus to which the wide-angle image transmitting apparatus according to any one of the tenth to seventeenth aspects is applied for recording a conference. A speaker position determining means for determining the position of the speaker based on the color distribution of the image or a moving part in the image; and a predetermined area determining means for determining the predetermined area based on the determination result of the speaker position determining means. It is characterized by having.

That is, according to the eighteenth aspect of the present invention, the speaker portion is accurately extracted.

A wide-angle image reproducing apparatus according to a nineteenth aspect of the present invention is data input means for inputting moving image data in which a wide-angle image is captured, audio data synchronized with the moving image data, and data regarding a sound source direction. Of the moving image data input by the data input unit, an image transforming unit that transforms the moving image data in a predetermined area into a rectangular output image based on the data relating to the sound source direction, and the image transforming unit transforms the moving image data. And a video / audio output unit that outputs the video data and the audio data corresponding to the video data in synchronization with each other.

That is, according to the nineteenth aspect of the present invention, a scene centered in the sound source direction is output while correcting the distorted wide-angle moving image data.

The wide-angle image reproducing device according to the twentieth aspect of the invention is data for inputting moving image data in which a panoramic wide-angle image is captured, audio data synchronized with the moving image data, and data regarding a sound source direction. Input means, image extracting means for extracting moving image data of a predetermined area based on the data regarding the sound source direction from the moving image data input by the data inputting means, and moving image data extracted by the image extracting means, And an image / audio output unit that outputs the audio data corresponding to the moving image data in synchronization with each other.

That is, according to the twentieth aspect of the invention, a scene centered in the sound source direction is output.

A wide-angle image reproducing device according to a twenty-first aspect is the wide-angle image reproducing device according to the nineteenth or twentieth aspect, wherein the moving image data, the audio data, and the sound source direction input by the data input means are included. Recording means for recording data relating to the moving image data, reproduction instruction means for instructing reproduction of the moving image data, and when there is a reproduction instruction by the reproduction instruction means, based on the data recorded by the recording means, An output control means for controlling the image / audio output means to output the moving picture data and the sound data corresponding to the moving picture data in synchronization with each other are provided.

That is, according to the invention of claim 21, a scene centered on the sound source direction can be reproduced at any time.

A wide-angle image reproducing device according to a twenty-second aspect is the wide-angle image reproducing device according to the nineteenth, twenty-third or twenty-first aspect, further comprising direction correcting means for correcting a direction corresponding to the sound source direction. Is characterized by.

That is, according to the twenty-second aspect of the present invention, a desired image is reproduced even when the sound source direction is not the desired direction due to noise or the like.

The wide-angle image reproducing device according to a twenty-third aspect is the wide-angle image reproducing device according to any one of the nineteenth to twenty-second aspects, in which a predetermined region in a direction corresponding to the sound source direction is fixed. It is characterized in that a fixing means is provided.

That is, the invention according to claim 22 prevents the image blur even when the direction of the sound source slightly moves.

A conference image reproducing apparatus according to a twenty-fourth aspect is a conference image reproducing apparatus to which the wide-angle image reproducing apparatus according to any one of the nineteenth to twenty-third aspects is applied for reproducing a conference. A speaker position determining means for determining the position of the speaker based on the color distribution of the moving image data or a moving part in the moving image data; and a predetermined area determining means for determining the predetermined area based on the determination result of the speaker position determining means. , Is provided.

That is, according to the invention of claim 24, the speaker portion is accurately extracted.

According to the wide-angle image recording / reproducing method of the twenty-fifth aspect, an input step of inputting a wide-angle image having a vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound. A recording step of recording a wide-angle image input in the input step, a sound, and a sound source direction; and a predetermined area in a direction corresponding to the sound source direction in the wide-angle image recorded in the recording step. Reproduction step of deforming an image into a rectangular output image, the image deformed by the image deforming step, and the sound of the deformed image recorded in the recording step in synchronization with each other. And a process.

That is, according to the twenty-fifth aspect of the present invention, a wide-angle image is recorded, and when the image is reproduced, the distortion is corrected and the scene centered in the sound source direction is reproduced.

According to the wide-angle image recording / reproducing method of the twenty-sixth aspect, an input step of inputting a wide-angle image having a vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound. And a transformation step of transforming the wide-angle image input by the input step into a rectangular output image by a predetermined conversion formula, an image transformed by the transformation step, a sound, and a sound source direction are recorded. A recording step of extracting the image of a predetermined area in a direction corresponding to the sound source direction from the rectangular image recorded by the recording step; an image extracted by the extracting step; And a reproduction step of reproducing in synchronization with the sound related to the extracted image recorded in the step.

That is, according to the twenty-sixth aspect of the invention, an image developed in a panoramic shape is recorded, and a scene centered on the sound source direction can be reproduced at any time.

A conference recording / playback method according to a twenty-seventh aspect is a conference recording / playback method in which the wide-angle image recording / playback method according to the twenty-fifth or twenty-sixth aspect is applied to the recording / playback of a conference. Alternatively, it includes a speaker position determining step of determining the position of the speaker based on the moving part in the image, and a predetermined area determining step of determining the predetermined area based on the determination result based on the speaker position determining step. Is characterized by.

That is, according to the twenty-seventh aspect of the present invention, the speaker portion is accurately extracted.

The wide-angle image transmitting method according to the twenty-eighth aspect further comprises an input step of inputting a wide-angle image having a vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound. , A transformation step of transforming the wide-angle image input in the input step into a rectangular output image by a predetermined conversion formula, data regarding the image transformed by the transformation step, data regarding voice, and a sound source direction. And a data sending step of sending the data to a predetermined data storage destination.

That is, according to the twenty-eighth aspect of the invention, it is possible to reproduce an image developed in a panoramic shape.

A wide-angle image transmitting method according to a twenty-ninth aspect is the wide-angle image transmitting method according to the twenty-eighth aspect.
Of the images transformed by the transforming step, an extraction step of extracting an image of a predetermined region in the sound source direction is included, and in the data sending step, the data on the wide-angle image and the data on the sound source direction are input in the input step. Alternatively, or together with the data about the wide-angle image and the data about the sound source direction input by the input step,
Data relating to the image extracted in the extraction step is transmitted.

That is, according to the twenty-ninth aspect of the present invention, it is possible to reproduce a scene centered on the sound source direction.

A meeting image sending method according to a thirtieth aspect is a meeting image sending method in which the wide-angle image sending method according to the twenty-ninth aspect is applied for recording a conference, and the color distribution of the image or the image And a predetermined area determining step of determining the predetermined area based on the determination result of the speaker position determining step. .

That is, according to the invention of claim 30, the speaker part is accurately extracted.

According to the wide-angle image reproducing method of the thirty-first aspect, a recording step of recording the moving image data in which the wide-angle image is captured, the audio data synchronized with the moving image data, and the data regarding the sound source direction, Of the moving picture data recorded by the recording step, a transformation step of transforming the moving picture data of a predetermined area into a rectangular output image based on the data regarding the sound source direction, and the moving picture data transformed by the image transforming step. And an output step of outputting the audio data corresponding to the moving image data in synchronization with each other.

That is, according to the thirty-first aspect of the present invention, the scene centered in the sound source direction is output while correcting the distorted wide-angle moving image data.

The wide-angle image reproducing method according to a thirty-second aspect of the invention is a recording for recording moving image data in which a panoramic wide-angle image is captured, audio data synchronized with the moving image data, and data regarding a sound source direction. A step of extracting moving image data of a predetermined area based on the data regarding the sound source direction from the moving image data recorded in the recording step, the moving image data extracted by the extracting step, and the moving image data And an output step of outputting corresponding audio data in synchronization with each other.

That is, according to the thirty-second aspect of the present invention, a scene centered on the sound source direction is output.

The conference image reproducing method described in claim 33 is the conference image reproducing method in which the wide-angle image reproducing method according to claim 31 or 32 is applied for reproducing a conference.
A speaker position determining step of determining the position of the speaker based on the color distribution of the moving image data or a moving portion in the moving image data, and a predetermined area determining step of determining the predetermined area based on the determination result based on the speaker position determining step. It is characterized by including and.

That is, according to the invention of claim 33, the speaker portion is accurately extracted.

The program according to claim 34,
A computer is made to perform each step of the method according to any one of claims 25 to 33. That is, the invention according to claim 34 has the same operational effect as the method according to any one of claims 25 to 33.

[0078]

BEST MODE FOR CARRYING OUT THE INVENTION Embodiments of the present invention will be described in detail below with reference to the drawings. Embodiment 1. In the first embodiment, a conference recording / playback system in which the wide-angle image recording / playback system of the present invention is applied to recording / playback of a conference will be described. Here, first, a brief example of how the conference recording / playback system is used will be briefly described, and then, elements that constitute the conference recording / playback system (conference image transmission corresponding to an image and audio input unit) will be described. The apparatus and the conference image reproducing apparatus corresponding to the recording and reproducing unit of the image and the sound) will be described, and finally the processing flow will be described.

(Example of use of conference recording / playback system) FIG.
[Fig. 4] is an explanatory view outlining a usage example in which the present invention is installed in a conference scene. The conference recording / reproducing system 100 includes a conference image transmitting device 200 for inputting a wide-angle image and audio, and a conference image reproducing device 300 for recording and reproducing the image and audio input by the conference image transmitting device 200.

As shown in the figure, the conference image transmitting apparatus 200
Is installed on the table 1 and collectively captures images of the direction in which the participants (speakers) 2 of the conference are, that is, the entire circumference overlooking the horizontal plane, and also inputs the voice of the conference. The conference image reproducing device 300 is stored in the cabinet 3, records an image from the conference image transmitting device 200, and reproduces the recorded conference contents as needed in response to a user's request (note that in the figure, The monitor for is omitted). Upon reproduction, the conference image reproduction device 300 uses the conference image transmission device 20.
The image of the entire circumference taken from 0 is transformed into a rectangular output image.

Next, each part of the conference recording / playback system 100 will be described. (External Configuration of Conference Image Sending Apparatus 200) FIG. 2 is an external perspective view of the conference image sending apparatus 200 according to the first embodiment.
Further, FIG. 3 shows a conference image transmitting apparatus 200 according to the first embodiment.
FIG. 3A is a front view and FIG. The conference image transmitting device 200 is
It has a camera unit 201 for inputting a wide-angle image with the vertical direction as a center or an axis, and a microphone unit 202 for inputting voice. Here, the wide-angle image means an image including at least the entire circumference (360 °) overlooking the horizontal plane.

As shown in the figure, the conference image transmitting apparatus 200 according to the first embodiment has four microphones 221.
The microphone 221 and the image pickup device (CCD) of the camera unit 201, which will be described later, are arranged on the pedestal 203. The hyperboloidal mirror 211 of the camera unit 201, which will be described later, is arranged so as to face the pedestal 203 with the transparent glass 204. By using the transparent glass 204, it is possible to input an image of the entire circumference without blocking the light incident from the hyperboloidal mirror 211. Note that reference numeral 205 indicates a cable that transmits various data.

(Conference image sending device 200: camera unit 20
1) FIG. 4 shows the conference image transmitting apparatus 2 according to the first embodiment.
It is explanatory drawing which showed the structural example of the camera part 201 of 00.
The camera unit 201 includes a hyperboloidal mirror 211 and a lens 21.
2, a diaphragm 213, and a CCD (Ch
charge coupled device) 214.

Further, the camera unit 201 controls the timing of the CCD 214, and at the same time, the drive processing unit 215 that performs A / D conversion (analog-digital conversion) of the video signal obtained by the CCD 214 and the drive processing unit 215 obtains it. A preprocessing circuit 216 that performs preprocessing such as edge enhancement and γ correction on the digital signal and a motor drive unit 217 that drives the diaphragm 213 to control the iris are provided.

Here, the optical system will be described. The hyperboloidal mirror 211 is a reflecting mirror that enables wide-angle shooting. In Embodiment 1, various explanations are given using a hyperboloidal mirror as an example of a reflecting mirror, but the mode is not limited as long as it is a configuration capable of capturing a wide-angle image. Note that an example of another reflecting mirror will be described in Embodiment 3.

Regarding the technique of capturing an image with the hyperboloidal mirror 211, see, for example, A.I. M. Bruckste
in and T.S. J. Richardson: Omni
view Cameras with Curved
Surface Mirrors, Proc. o
f the IEEE Workshop on Om
nidirectional vision 200
0, pp. 79-84 can be mentioned. According to the same magazine, the use of a hyperboloidal mirror makes it possible to capture an important subject in the horizontal direction, such as a human face, at a relatively high resolution.

FIG. 5 is a hyperboloidal mirror 21 of the first embodiment.
FIG. 6 is a diagram for explaining an optical path when 1 is used, and FIG. 6 is a diagram showing a state of a wide-angle image formed on the surface of the CCD 214 by the hyperboloidal mirror 211 of the first embodiment. As illustrated, the image captured from the hyperboloidal mirror 211 has a donut shape. Note that the central portion in FIG. 6 projects in the direction of the pedestal 203, which is unimportant image information. Therefore, the top portion 218 of the hyperboloidal mirror 211 may be painted black to provide black information. Depending on the mode of use, a reference line may be drawn on the parietal region 218, and the motor drive unit 217 may be driven when the conference image sending device 200 is started up to be used for initial settings such as focus adjustment. Good.

As described above, the camera section 201 includes the general-purpose CCD 214 and the hyperboloid mirror 2 having a simple structure.
It can be constructed by 11. Therefore, a desired subject can be collectively photographed with high resolution, and the inexpensive camera unit 2 can be used.
01 can be provided.

(Conference image sending device 200: Microphone unit 20
2) Next, the contents of the microphone unit 202 will be described. As described with reference to FIG. 2 or FIG. 3, the microphone unit 202 includes a plurality of microphones 221. In the following, this plurality of microphones 22
1 will be appropriately referred to as a microphone array.
As the microphone 221, various types such as a piezoelectric type and a capacitive type (so-called condenser microphone) can be used. As will be described later, by using a plurality of microphones, the sound source direction (speaker direction) can be detected.

(Structure of Conference Image Reproducing Device 300) Next,
The configuration of the conference image reproducing device 300 will be described. Figure 7
FIG. 3 is a diagram showing a configuration example of a conference image reproducing device 300 according to the first embodiment. The conference image reproducing apparatus 300 includes a CPU (Central Pr) that performs various controls and processes.
processing unit) 301 and SDRAM
(Synchronous Dynamic Rand
om Access Memory) 302 and HDD
(Hard Disk Drive) 303, input interface (hereinafter referred to as I / F) 304 for pointing device such as mouse, keyboard, button, etc., power supply 305, and display I / F 306
And DVD (Digital Versatile D
isc) -Large-capacity recording device 307 such as a RAM drive
And an external I / O for connecting to the conference image transmitting apparatus 200.
F308, and are connected via a bus 309. The display I / F 306 is connected to a display such as a CRT.

Next, each component of the conference image reproducing apparatus 300 will be described. The CPU 301 transforms the wide-angle donut-shaped image shown in FIG. 6 into a rectangular output image according to a predetermined program stored in the HDD 303. The CPU 301 also extracts a predetermined area in the sound source direction. The transformation and extraction processing will be described later. The SDRAM 302 is used as a work area of the CPU 301 and also as a storage area for each processing program stored in the HDD 303 and other control programs (for example, OS).

The external I / F 308 is an interface used when inputting the data transmitted from the conference image transmitting apparatus 200, as described above. Here, examples of the data input from the conference image transmitting device 200 include wide-angle images (moving image data), audio data, and sound source direction data. The external I / F 308 can employ various types of I / F, for example, a USB (Universal).
Wired connection such as Serial Bus) or IEEE1394 may be adopted, or IrDA, BlueTo
A wireless connection such as oth may be adopted. External I / F 30
The data input by 8 is stored in the mass storage device 307.

(Details of Wide-angle Conference Recording / Reproducing System 100: Functional Configuration) Next, the wide-angle conference recording / reproducing system 10 will be described.
Image processing for transforming a wide-angle image into a rectangular output image and detection processing for a sound source direction will be described while describing a functional configuration of 0. FIG. 8 shows a conference recording / playback system 10.
It is a block diagram showing an example of the functional composition of 0.

The wide-angle conference recording / playback system 100 has, as its functional configuration, a wide-angle image input unit 801, a voice input unit 802, a sound source direction detection unit 803, and a recording unit 804.
The image transformation unit 805, the direction correction unit 806, the area fixing unit 807, and the image / audio output unit 808 are included.

(Wide-angle conference recording / reproducing system 100: contents of wide-angle image input unit 801) The wide-angle image input unit 801 takes in a wide-angle image centered on or in the vertical direction and sends the image data to the recording unit 804. An example of the wide-angle image is the donut-shaped image shown in FIG. The wide-angle image input unit 801 includes, for example, the hyperboloidal mirror 211, the lens 212, and the diaphragm 213 shown in FIG.
The functions can be realized by the CCD 214, the drive processing unit 215, and the preprocessing circuit 216.

(Conference Recording / Playback System 100: Contents of Audio Input Unit 802 / Source Direction Detection Unit 803) Audio Input Unit 8
Reference numeral 02 designates a voice as an input, converts it into an electric signal (voice data), and converts the voice data into a sound source direction detecting unit 803 and a recording unit 8.
Send to 04. The voice input unit 802 can realize its function by the microphone 221 (see FIG. 2 or FIG. 3). A plurality of microphones 221 are arranged as described above, and each microphone 22
The sound source direction is detected based on the audio data from 1.

The sound source direction detecting section 803 has a voice input section 80.
The voice data is input from 2 and the sound source direction is detected. By detecting the sound source direction, it is possible to extract (cut out) the image of the speaker (speaker) portion from the wide-angle image, and it is possible to efficiently reproduce the conference while maintaining the presence. Next, this sound source direction detection processing will be described.

Here, a method in which the sound source direction detecting section 803 detects the sound source direction based on the arrival time difference of the voices input to the microphone array will be described. FIG. 9 is a diagram for explaining the principle of sound source direction detection by the sound source direction detection unit 803. As shown in the figure, two microphones 221 (which will be referred to as a microphone 1 and a microphone 2 for convenience) are arranged at intervals of a distance l, and a voice
When arriving from a direction, voice data s1 (t) output by the microphone 1 and voice data s2 output by the microphone 2
The relationship with (t) can be expressed by the following equation (1), where t is time v and the sound velocity. s1 (t) = s2 (t− (l · cos θ) / v) (1)

Expression (1) indicates that the voice data of the microphone 1 arrives ahead of the voice data of the microphone 2 by (l · cos θ) / v. The sound source direction detection unit 803 uses this arrival time difference to identify the direction of the speaker's voice.

In specifying the sound source direction, first, the arrival time difference between the voice data of the microphones 1 and 2 is detected. This arrival time difference is, for example, the voice data s1 of the microphone 1.
It is calculated by the cross-correlation value between (t) and the voice data s2 (t + dt) of the microphone 2. Cross-correlation value C (t, dt)
Is calculated by the following equation (2).

[Equation 1]

Equation (2) indicates that the product-sum operation is performed using N samples before time t. N is a positive integer indicating the size of the correlation window.
Although detailed description is omitted, dt that maximizes C (t, dt) is the arrival time difference.

Next, using the microphone interval l, the arrival time difference dt, and the sound velocity v, the angle θ between the voice and the base line of the microphone is calculated by the following equation (3).

[Equation 2] Here, the range of θ is 0 ° or more and 180 ° or less.

By the above procedure alone, the direction is detected only in the front 180 ° range of the microphone 221, and the sound source direction is not specified. That is, the angle θ output by the sound source direction detection unit 803 is actually the angle formed by the arrival direction of the voice and the baseline between the two microphones, and the actual direction of the voice is 2 as shown in FIG. It exists on one of the sides of a cone with the apex angle θ whose apex is the midpoint of two microphones.

In order to solve this problem, correction is performed by using another microphone group which is not parallel to the group consisting of the microphone 1 and the microphone 2. FIG. 11 is an explanatory diagram showing how the four microphones 221 are divided into two groups to detect the sound source direction. As shown in the figure, the grouping is performed by combining a microphone 221 (for example, microphone 1 (microphone 3)) and a microphone 221 (microphone 2 (microphone 4)) that is farthest from the microphone 221.

By using a pair of two microphones that are most distant from each other, the difference in the arrival time of voices is maximized and the accuracy of direction detection is improved. In the first embodiment, the microphone unit 2
The 02 has four microphones 221. However, the direction of the sound source can also be detected with high precision by using three microphones. FIG. 12 is an explanatory diagram for explaining how to adopt a set of microphones when the microphone unit is configured by three microphones. By arranging the microphones in an equilateral triangle, as shown,
Whichever microphone set is used, the sound source direction can be detected with high accuracy. In the example shown,
Although the first set and the second set can be used to detect sound sources in all directions, the third set may be used complementarily.

The sound source direction detecting unit 803 can realize its function by, for example, a control unit (not shown) of the microphone 221. In addition, depending on the mode of use,
CPU 301 on the conference image reproducing device 300 side (see FIG. 7)
The function may be realized by. In this case, it is necessary to separately input the sounds input from the microphone 221 to the conference image reproducing device 300 side.

(Wide-angle conference recording / playback system 100: contents of recording unit 804) The recording unit 804 is a wide-angle image moving image data output from the image input unit 801 and a voice input unit 802.
The audio data output from the sound source and the sound source direction output from the sound source direction detecting unit 803 are recorded. There are various recording methods. For example, moving image data is recorded in a moving image encoding format represented by MPEG. For audio data, MPE
The G audio format may be used or the PCM format may be used.

Regarding the data of the sound source direction, the time when the sound source direction is changed and the azimuth and elevation at that time are recorded at any time to extract (cut out) a screen to be described later.
Can be performed. FIG. 13 is a diagram showing an example of the data structure in the sound source direction. In the figure, the time when the sound source direction changed (Time), the azimuth angle (θ) of the new sound source direction
And the elevation angle (φ) is recorded. This direction data is recorded in the mass storage device 307 in the form of a text file or the like, together with moving image data and audio data.

In the above example, the sound source direction data is not the data combined with the moving image data or the audio data, but if the streaming format such as RealMedia format provided by RealNetworks is used, the sound source direction data is obtained. Can also be embedded in one file. Other MPEG-7
It is also possible to write the sound source direction data in a file using a content description standard of multimedia information such as.

In addition, moving image data and audio data may be recorded in a single file like an MPEG program stream. By using such encoding,
The recording capacity can be reduced. The recording unit 804 is
For example, the function can be realized by the large-capacity recording device 307. Depending on the mode of use, HD
The function may be realized by D303. For example,
Long-term meetings and regular meetings are recorded in the large-capacity recording device 307 composed of a DVD or the like due to the necessity of saving, and HDD 303 is used for short-term meetings such as those with low need for long-term saving. It may be used properly such as recording.

(Wide-angle conference recording / playback system 100: contents of image transforming unit 805 and related units) Next, the image transforming unit 8
05 and functional parts associated therewith will be described. The image transformation unit 805 has a donut shape (or a circular shape).
The wide-angle image is transformed into a rectangular output image. In general, an image obtained by shooting a wide-angle range at a time contains large distortion unlike the shape of the image that can be seen by the human eye. Therefore, in order to reproduce the conference recorded in the recording unit 804 later, the transformation process is required.

The hyperboloidal mirror 2 shown in FIG. 2 or FIG.
The transformation process when 11 is used will be described. The image transformation unit 805 transforms the donut-shaped image shown in FIG. 6 (hereinafter referred to as a donut image) into a normal image having a viewing angle of 360 degrees as shown in FIG. 14 (hereinafter referred to as a panoramic image). ) Is transformed into.

15 and 16 show a hyperboloidal mirror 21.
It is a figure explaining the modification principle when 1 is used. Of these, FIG. 15 is a diagram showing the coordinate system of the donut image and the panoramic image, and FIG. 16 is a diagram showing the relationship between the apex angle φ and the elevation angle φ as viewed from the CCD 214. Note that FIG.
In FIG. 6, the lens 212 and the diaphragm 213 are omitted for simplicity. Here, for convenience of the conversion formula, the lenses 212 to
The optical system of the CCD 214 will be described as a pinhole camera model.

The meaning of each variable in the figure is as follows. (U, v): Coordinates in the donut image (u0, v0): Coordinates of the center of the hyperboloidal mirror in the donut image (θ, φ): Coordinates in the panoramic image r: From (u0, v0) to (u, v) In pixel unit rmax: radius of pixel unit of hyperboloid mirror in donut image θ: azimuth angle φ: elevation angle ψ: vertical angle from optical axis of camera F: focus of hyperboloid mirror F ′: pair with hyperboloid mirror The focal point of the hyperboloid that forms (matches the optical center of the camera)

At this time, the following relationship is established between the apex angle ψ and the elevation angle φ.

[Equation 3] here,

[0116]

[Equation 4] Is. Further, φ _max is the value of the elevation angle corresponding to the position of the radius r _max on the donut image, and this represents the photographing limit value of the camera in the elevation angle direction. The values of r _max and φ _max are generally easily known.

The modification procedure will be described below. (I): polar coordinates (r, θ) corresponding to the point (u, v)
It is obtained by solving the following equation (6). (U, v) = (rcos θ + u0, r sin θ + v0) (6) (ii): The apex angle ψ corresponding to r calculated by the equation (6) is calculated by the following equation (7).

[Equation 5]

Here,

[Equation 6] And ψ _max is the value of the apex angle ψ corresponding to the position of the radius r _max on the donut image and the elevation angle φ _max . The value of ψ _max can be obtained by substituting φ _max in the equation (4). (Iii): The elevation angle φ corresponding to ψ calculated by the equation (7) is calculated by the equation (4).

By the above procedure, an arbitrary point (u, u in the donut image captured by the hyperboloidal mirror 211
v) can be coordinate-converted into a point (θ, φ) in the panoramic image. That is, the donut image is transformed into a panoramic image.

If the processing capacity of the processor is low, it takes time to transform the image data. Therefore, by referring to a predetermined conversion table, (u, v) →
The conversion of (θ, φ) may be performed. FIG. 17 shows (u,
It is explanatory drawing which showed typically the example of the conversion table of v)-> ((theta), (phi)). In the illustrated table, the point (θ, v in the panoramic image corresponding to each point (u, v) in the donut image)
φ) are stored respectively. Therefore, by using this table, it is possible to perform high-speed image transformation while reducing the processing load.

The image transformation unit 805 performs the above conversion processing and outputs a predetermined image area. That is,
In order to efficiently reproduce the meeting while maintaining the realism of the meeting,
The conference recording / playback system 100 extracts and outputs the speaker (speaker) portion of the panoramic image. As shown in FIG. 8, the conference recording / playback system 100 has a speaker position determination unit 809 and a region determination unit 810 as its functional configuration.

The speaker position determination unit 809 determines the speaker position based on the color distribution of the image or the moving portion in the image of the image data input from the wide-angle image input unit 801 or the image data recorded in the recording unit 804. to decide. As a method of determination based on the color distribution of an image, for example, there is a method of detecting a portion where the skin color is locally large. It should be noted that the reason why the determination can be made based on the moving part in the image is that the mouth of the speaker is always moving, and in some cases, the speaker is gesturing and moving his body. Therefore, the speaker position can be determined from the portion of the image with the largest movement amount.

The area determination unit 810 is the speaker position determination unit 80.
It is determined which part of the speaker position judged in 9 is extracted. When the table 1 has an elliptical shape, the camera unit 201
The distance between the speaker and the speaker is different, and the size of the speaker in the wide-angle image or the panoramic image is also different. Therefore, if the area to be output has a uniform size,
In some cases, the speaker may be too large or, on the contrary, too small. The area determination unit 810 determines the area of the speaker portion in an area in which the speaker has an appropriate size.
The image transformation unit 805 appropriately enlarges or reduces this image and displays it.

On the other hand, the direction correcting unit 806 corrects the direction corresponding to the sound source direction. This is the sound source direction detection unit 803.
This is based on the fact that the direction of the sound source detected in 1. may not be the desired direction due to noise such as clap sounds, or sporadic words emitted by people other than the speaker such as a reply. In addition, in practical use, there is a request to project the right side of the area determined by the area determination unit 810, for example. In particular, there is a case where the speaker is giving a presentation and wants to correct the direction when writing on the whiteboard. Therefore, the direction correction unit 8
06 corrects the sound source direction so as to satisfy such a request.

The area fixing unit 807 fixes the image area determined by the area determining unit 810 in the direction corresponding to the sound source direction. That is, the area determining unit 810 determines a relative area of, for example, 160 pixels × 90 pixels, whereas the area fixing unit 807 fixes the area as an absolute position so as not to blur according to the sound source direction. . This is to prevent the image from blurring when the direction of the sound source slightly moves due to the speaker shaking the body.

As described above, the image transformation unit 805
Properly outputs the image of the speaker portion without distortion. Image transforming unit 805, area fixing unit 807, and speaker position determining unit 8
09 is, for example, the conference image reproducing device 30 shown in FIG.
The function can be realized by the CPU 301 of 0 and a predetermined program stored in the HDD 303. In addition, the direction correction unit 806 and the area determination unit 810, for example, the CPU 3 of the conference image reproduction device 300 illustrated in FIG.
01, a predetermined program stored in the HDD 303, a pointing device, a K / B, and a button connected to the input I / F 304, the function can be realized.

(Conference Recording / Reproducing System 100: Contents of Image / Sound Output Unit 808) The image / sound output unit 808 detects the image (moving image data) output from the image transformation unit 805 and the image (moving) of this image. Simultaneously recorded (input) voices are output in association with each other. That is, the image and the sound are output in synchronization with each other. Processor (eg CPU3
Depending on the processing speed of 01), there is a time lag between the sound and the image, and therefore the image and sound output unit 808 reproduces the conference with a natural feeling by synchronizing the image and the sound. The image / audio output unit 808 can realize its function by, for example, the CPU 301 of the conference image reproducing apparatus 300 illustrated in FIG. 3 and a predetermined program stored in the HDD 303.

(Conference Recording / Reproducing System 100: Process Flow) Next, the process flow of the conference recording / reproducing system 100 will be described. FIG. 18 shows a conference recording / playback system 100.
It is an explanatory view showing an example of the processing flow of. The conference recording / playback system 100 first starts recording by pressing a recording start button (not shown) (step S1).
801). After this starting operation, wide-angle images (donut images) centered on or in the vertical direction are sequentially input from the camera unit 201, and voices are sequentially input from the microphone unit 202 (step S1802). The microphone unit 202
With respect to the voice input from, the sound source direction is detected at any time by using the above-mentioned microphone set.

Next, the donut image input from the camera unit 201, the voice input from the microphone unit 202, and the detected sound source direction are recorded (step S1803).
Regarding recording, a recording time, a file name (meeting name), etc. are appropriately added for later reproduction. Since the sound source direction is detected, it is not necessary to record all four channels of each microphone 221 (that is, four microphones 221), and only one of them or the average of four sounds should be recorded. Good. Although the above-mentioned time difference occurs in principle, this time difference does not pose any practical problem in view of the size and the sound speed of the conference image transmitting device 200.

Recording is ended by pressing a recording end button (not shown) at the end of the conference (step S1804). By going through the above steps, it is possible to record the content of the conference composed of the image of the entire circumference, that is, the unprocessed donut image.
By recording the image before processing, it is possible to perform later editing (extraction of image area, correction of sound source direction, etc.).

Next, the reproduction of the recorded conference will be described. When a reproduction start button (not shown) is pressed, reproduction is started (step S1805). In addition,
When a plurality of conferences are recorded on a recording medium (DVD-RAM, for example), the index display is performed to allow the user to select which conference to reproduce, and then the reproduction is started.

The speaker position is determined by using the skin-colored portion in the sound source direction of the recorded donut image as a clue (step S1806), and the area to be displayed is designated (step S).
1807). Here, if it is desired to intentionally adjust the direction of the image, an instruction to correct the direction is issued.

Subsequently, the image transformation is performed so that the designated area of the donut image becomes a rectangular image (step S1808). When transforming an image, it may be transformed using a transformation formula, or a transformation table may be referenced. Finally, the extracted and properly transformed image is output together with the sound (step S1809). By taking such steps, it is possible to efficiently reproduce the conference while maintaining the realism.

In the conference recording / playback system 100,
It is preferable to design the position of the center of gravity of the microphone unit 202 to be on the optical axis of the camera unit 201. The most preferable design is the center of gravity of CCD 214 and a plurality of microphones 2.
The arrangement is such that the center of gravity of 21 matches. By designing or arranging in this way, the coordinate system for calculating the sound source and the coordinate system for converting the image can be matched, and the calculation load can be reduced.

Further, in the present embodiment, the microphone unit 202
Was provided on the pedestal 203, but it is also possible for each participant 2 to detect the direction of the sound source by owning each microphone 221 having a wireless communication means. For example, units that emit radio waves are installed at a plurality of known positions in the conference room, and the positions of the microphones 221 can be detected based on the principle of triangulation from the signal strength and time difference of the radio waves that have reached the microphone 221. At this time, the direction of the microphone 221 having the largest signal amplitude can be detected as the speaker direction. Here, as the wireless communication means, Bluetooth is used.
Communication technologies such as h can be used.

Note that the function of the conference image reproducing apparatus 300 can be realized by a personal computer. In this case, the software that implements each functional unit is stored in the hard disk, and the function can be implemented by appropriately executing the processing program.

As described above, the conference recording / playback system according to the first embodiment can take in all the conference participants at one time with a simple structure using a simple optical system using a hyperboloidal mirror. Also, the conference can be reproduced by recording this content. Regarding reproduction, by deforming and outputting necessary parts, it is possible to reproduce highly realistic meeting contents centered on the speaker. In particular, since the content of the conference is recorded in all directions, it is possible to look back at the conference scene under the conditions preferred by the user.

Embodiment 2. In the second embodiment, a conference image recording / reproducing system that transforms a wide area image into a panoramic image and then records the image will be described. In the second embodiment, the same components as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted. Hereinafter, the external configuration, the hardware configuration, the functional configuration, and the processing flow of the conference recording / playback system 1900 will be sequentially described.

(External Configuration of Conference Recording / Reproducing System 1900) FIG. 19 is a diagram showing an example of the external configuration of the image recording / reproducing system according to the second embodiment. The image recording / playback system 1900 includes a cross button 1901 and an enter button 19
02, a video / audio output terminal 1903, and a medium insertion slot 1904. In the conference recording / playback system 100 according to the first embodiment, the conference image sending device 200 that captures images and sounds and the conference image playing device 300 that records and processes and reproduces the moving image are separate entities. The conference recording / playback system 1900 in the form 2 inputs, stores, processes, and plays back image and sound in one housing.

First, the above-mentioned respective parts appearing in appearance will be described. The cross button 1901 is used to move a menu or pointer displayed on a screen (not shown). For example, enter the meeting name and use it when creating the meeting file. Also, if multiple meetings are recorded,
It is also used when selecting the conference file name to be played. In addition, it is also used to correct the sound source direction such as inputting the speaker's elevation angle.

The enter button 1902 makes various decisions. For example, it is also used when determining the selection target by the cross button 1901. Note that this decision button can also be used as a multi-function button by allocating power on / off and playback stop.

The image / audio output terminal 1903 outputs data processed by the conference recording / playback system 1900, that is, an image signal of an image in which a predetermined speaker is cut out without distortion and an audio signal accompanying the image. Output. The data format is MPEG format or RealAudio as mentioned above.
Although the format may be used, here, a signal format transmitted and received by a VIDEO terminal (VHF / UHF terminal) provided in a normal television is adopted. With such a general-purpose signal format, it becomes possible to reproduce a conference on a normal television without going through a special control circuit.

The medium insertion slot 1904 is a slot for inserting a recording medium for recording a conference. Embodiment 1
Then, a DVD-RAM or the like is assumed, but here, P
A CMCIA socket is adopted, and a card-type HDD with high density and large capacity is inserted. With such a configuration, the device can be downsized. In some cases, it may be a slot into which a DVD-RW or DAT tape can be inserted. If the conference recording / playback system 1900 has a mechanical drive unit, a sound deadening structure is adopted so as not to pick up mechanical noise.

(Conference Recording / Playback System 1900: Hardware Configuration) Next, the hardware configuration of the conference recording / playback system 1900 will be described. FIG. 20 is an explanatory diagram showing an example of the hardware configuration of the conference image recording / playback system according to the second embodiment. Conference recording / playback system 19
00 is a CPU 301, a RAM 2001, a ROM 2
002, an operation unit 2003, an output I / F 2004, a camera unit 2005, a microphone unit 2006, and a removable media unit 2007. The camera unit 200
Reference numeral 5 is a notation for convenience showing the camera unit 201 including the optical system shown in FIG.
Also, this is a notation for convenience showing the microphone unit 202 including the microphone 221 shown in FIG.

The RAM 2001 is used not only as a work area for the CPU 301, but also as a storage area for each processing program stored in the HDD 303 and other control programs (eg OS). ROM2
002 stores universal control information and coefficients. For example, the conversion table (correspondence table) shown in FIG. 17 may be stored.

The operation section 2003 is provided with a cross button 1901.
And a decision button 1902. Output I / F
An image / audio output terminal 1903, a video card, and a video memory 2004 send image signals and audio signals to a video input terminal of a television (not shown). The removable media unit 2007 has a medium insertion slot 1
PCMCIA type large capacity HDD inserted in 904
Drive control of writing / reading.

(Contents of the conference recording / playback system 1900:
Functional Configuration) Next, the functional configuration of the conference recording / playback system 1900 will be described. FIG. 21 is an explanatory diagram showing an example of the functional configuration of the conference recording / playback system 1900. The conference recording / playback system 1900 has, as its functional configuration, a wide-angle image expansion unit 2101 and an image extraction unit 2102 in addition to the functional units described in FIG.

(Conference Recording / Playback System 1900: Contents of Wide-angle Image Expansion Unit 2101)
Transform a donut image into a panoramic image. Embodiment 1
In the conference recording / playback system 100, the image is transformed at the time of playback (see the image transforming unit 805 in FIG. 8), but in the conference recording / playback system 1900 of the second embodiment, the image is transformed at the time of recording. In other words, in the conference recording / playback system 1900, the wide-angle image is expanded into a panoramic image before being recorded in the recording unit 804, and this panoramic image is recorded. The expansion process may be calculated by using Expressions (4) to (8), and the description thereof will be omitted.

If the processing capacity of the CPU 301 (see FIG. 20) is low, it takes a long time to transform the image data, so that a wide-angle image and a panoramic image can be associated with each other by referring to a predetermined conversion table. Good. By using such a table, it is possible to perform high-speed image transformation while reducing the processing load.

The wide-angle image expansion unit 2101 is, for example, C
The function can be realized by the PU 301 and the wide-angle image expansion program stored in the HDD 303. It should be noted that the original information is set to 1 in both the conference recording / playback system 1900 and the conference recording / playback system 100.
Since it has 100%, it is possible to reproduce a scene in a desired sound source direction at any time.

(Conference Recording / Playback System 1900: Contents of Image Extraction Unit 2102) The image extraction unit 2102 includes a recording unit 8
A predetermined image area corresponding to the sound source direction is cut out (extracted) from the panorama image recorded in 04 and output to the image / audio output unit 808. For example, when the conference participant A (see FIG. 6) is speaking, the part corresponding to the participant A is determined based on the sound source direction data and the video data developed and recorded as shown in FIG. Extract. Hereinafter, this cut out image will be referred to as a partial image. FIG. 22 is an explanatory diagram showing an example of image extraction. As illustrated, the image extraction unit 2102 generates partial image data showing only the participant A.

Here, the procedure of image extraction will be described. Figure 2
FIG. 3 is an explanatory diagram illustrating a method of generating partial image data by the image extracting unit 2102 according to the second embodiment. First, the range of angles extracted as partial image data is set in advance. The range of this angle is assumed to be Δθ in the azimuth angle direction and Δφ in the elevation angle direction. Next, the sound source direction detection unit 803
The azimuth angle θ and the elevation angle φ detected by are read. Finally,
In the panoramic image data input from the recording unit 804, the area ((θ-Δθ
/ 2, φ-Δφ / 2), (θ + Δθ / 2, φ-Δφ /
2), (θ-Δθ / 2, φ + Δφ / 2), (θ + Δθ /
2, partial image data is generated by extracting a region surrounded by φ + Δφ / 2).

Depending on the mode of use, the image extraction unit 2102 may directly extract the image from the donut image. At this time, the donut image is as shown in FIG.
Δθ centered on (θ, φ) in the coordinate conversion table
By accessing only the rectangular area of Δφ, partial video data is cut out from the donut image data and transformed. Further, when the image transformation unit 805 is provided as in the first embodiment, the panoramic image generated by the image transformation unit 805 has Δθ × Δφ centered at (θ, φ).
Alternatively, the rectangular area may be directly extracted.

The image extraction unit 2102 is, for example, as shown in FIG.
CPU 301 of the conference recording / playback system 1900 shown in FIG.
Then, the function can be realized by the image extraction program stored in the HDD 303. In the second embodiment, the image / audio output unit 808 is the image extraction unit 2
The image (moving image data) output from 102 and the sound recorded (input) at the same time when this image is captured (input) are output in association with each other. That is, the image and the sound are output in synchronization with each other. Depending on the processing speed of the CPU 301 (see FIG. 20), there is a time lag between the sound and the image, so the image and sound output unit 808 reproduces a natural conference by synchronizing the image and the sound.

(Conference Recording / Playback System 1900: Processing Flow) Next, the processing flow of the conference recording / playback system 1900 will be described. FIG. 24 shows a conference recording / playback system 1
It is explanatory drawing which showed the example of the processing flow of 00. The conference recording / playback system 100 first starts recording by pressing a recording start button (not shown) (step S2401). After this start operation, wide-angle images centered on or in the vertical direction are sequentially input from the camera unit 201, and audio is sequentially input from the microphone unit 202 (step S2402). With respect to the voice input from the microphone unit 202, the sound source direction is detected at any time by using the above-described microphone set.

Next, the wide-angle image (donut image) input from the camera unit 201 is sequentially transformed into a panoramic image (step S2403). The panoramic image, the sound input from the microphone unit 202, and the detected sound source direction are recorded (step S2404). Regarding recording, record time and file name (meeting name) as required for later playback.
And so on.

When the conference is over and the recording end button (not shown) is pressed, the recording is ended (step S2405). By going through the above steps, it is possible to record an image of the entire circumference, that is, the contents including all the states of the conference. Since the saved image includes the image of the entire circumference, it can be edited (extracting the image region, correcting the sound source direction, etc.) as desired by the user later.

Next, the reproduction of the recorded conference will be described. The conference recording / playback system 1900 starts playback by pressing a playback start button (not shown) (step S2406). A recording medium (PCMCI
When a plurality of conferences are recorded on the A type hard disk), an index is displayed to let the user select which conference to reproduce.

The speaker position is determined by using the skin-colored portion in the sound source direction of the recorded panoramic image as a clue (step S2407), and the image of the area to be displayed is extracted (step S2408). Finally, the extracted image is output together with the sound (step S2409). By taking such steps, it is possible to efficiently reproduce the conference while maintaining the realism.

As described above, the conference recording / playback system according to the second embodiment can take in all the conference participants at one time with a simple structure using a simple optical system using a hyperboloidal mirror. Also, the conference can be reproduced by recording this content. In addition, since a donut image expanded into a panoramic image is recorded, it is possible to construct a system with less load during reproduction.

Third Embodiment In the third embodiment, an aspect of the conference recording / playback system in which the camera unit and the microphone unit are different from those in the first and second embodiments will be described.
FIG. 25 is an explanatory diagram showing an example of an external configuration of an apparatus including a camera unit according to the third embodiment. As is clear from the figure, the camera unit 2501 of the conference recording / playback system 2500
Is a specular body 2 having a conical shape instead of a hyperboloid mirror.
502. The description of the conversion formula from the donut image to the panoramic image is omitted, but by appropriately disposing a lens like the lens 212 shown in FIG.
Try to focus on the surface of 214. Depending on the mode of use, it may be a specular body having a paraboloid.

In the above-mentioned examples, the reflecting mirror (hyperbolic mirror 211, conical mirror surface 2502 or parabolic mirror surface) is composed of one piece, but is not limited to this. You may use the reflecting mirror of. 26 is 2
FIG. 6 is an external configuration diagram of a camera unit configured to capture a donut image using a single reflecting mirror. The camera unit 2600 includes a first reflecting mirror 2601 composed of a parabolic mirror or a hyperboloidal mirror, and a second reflecting mirror 2602 for deflecting the reflected light reflected by the first reflecting mirror in the CCD direction. Have. The top of the first reflecting mirror 2601 has a hole for taking in the reflected light from the second reflecting mirror.

Next, the microphone section will be described. FIG. 27
FIG. 9 is an explanatory diagram illustrating a relationship between a microphone unit and a sound source direction according to the third embodiment. The microphone unit 202 according to the first and second embodiments uses the omnidirectional microphone 221 to detect the direction of the main source based on the difference in arrival time of voice. The microphone unit 2701 of the third embodiment has four microphones 2702 having directivity, and determines the sound source direction based on the strength of the sound. For convenience, the four microphones 2702 are microphones 1 to 4.

It is now assumed that the voice strength is 20 for microphone 1, 30 for microphone 2, 20 for microphone 3, and 5 for microphone 4. In this case, it is determined that there is a sound source in the direction of the microphone 2. When the strengths of the microphone 1 and the microphone 3 are compared, both have the same value 20, so that the sound source direction is finally determined to be the microphone 2 direction (the direction indicated by θ = 45 ° in the figure).

Another example will be described. Voice strength is 15 for microphone 1, 30 for microphone 2, 25 for microphone 3, 5 for microphone 4.
It was. In this case, it is initially determined that there is a sound source in the direction of the microphone 2. Comparing the strengths of the microphone 1 and the microphone 3, the strength of the microphone 3 is larger than that of the microphone 1. Therefore, the sound source direction is slightly moved from the microphone 2 direction to the microphone 3 direction (the direction indicated by θ = 30 ° in the figure). ) Is determined. The amount of movement in this direction may be determined in advance according to the characteristics of the directional microphone. As described above, if the directional microphone 221 is used, it is not necessary to perform calculations such as Expressions (1) to (3), and thus the load on the processor can be reduced.

Fourth Embodiment In the fourth embodiment, a general-purpose conference image transmitting device and a conference image reproducing device will be described. Here, “having versatility” means that even if there are a plurality of types of conference image transmitting devices and conference image reproducing devices depending on the configuration of a specular body that captures wide-angle images and the type of microphone,
It means that a meeting can be recorded or played back in any combination. In addition, also in the fourth embodiment, the same components as those in the first to third embodiments are denoted by the same reference numerals, and the description thereof will be omitted.

The conference recording / playback system 28 of the fourth embodiment.
00 has a conference image transmitting device 2801 and a conference image reproducing device 2802. FIG. 28 is a diagram showing functional blocks of the conference image transmitting apparatus 2801 and the conference image reproducing apparatus 2802 according to the fourth embodiment. Conference image transmitting device 280
Reference numeral 1 denotes a wide-angle image input unit 2811 as its functional configuration.
, Voice input unit 2812, and sound source direction detection unit 2813
An elevation angle setting unit 2814, a data transmission unit 2815,
Have.

The wide-angle image input unit 2811 takes in a wide-angle image with the vertical direction as the center or axis, and outputs the image data to the data sending unit 2815. For inputting a wide-angle image, the hyperboloidal mirror 211 shown in Embodiment 1 may be used, and either the conical mirror body 2502 or the parabolic reflector shown in Embodiment 3 may be used. Good.

The voice input unit 2812 inputs voice and converts it into an electric signal (voice data), and sends the voice data to the sound source direction detecting unit 2813 and the data sending unit 2815. For voice input, the omnidirectional microphone 221 described in Embodiment 1 may be adopted, or the directional microphone 2702 described in Embodiment 3 may be used. In the sound source direction detection unit 2813, the voice input unit 2
The sound source direction is detected based on the time difference or strength of the voice input from 812. The principle of detecting the direction of the sound source has already been described and will not be described.

The elevation angle setting unit 2814 sets the elevation angle in the height direction of the speaker. The sound source direction detecting unit is generally shown in FIG.
As described above, the error in the elevation angle direction is large. Therefore, the elevation angle setting unit 2814 sets the elevation angle from the plane where the conference image sending device 2801 is installed. The setting method may be based on, for example, direct detection of the image data (skin color data) of the speaker in addition to the direct setting of the angle ψ using the ten-key pad.

The data sending unit 2815 sends the wide-angle image, the sound, and the data regarding the sound source direction including the elevation angle to the predetermined data storage means. Here, it is sent to the conference image reproducing apparatus 2802. Although data transmission by wire is described in the first to third embodiments, the invention is not limited to this, and data may be transmitted wirelessly. Although various methods can be adopted as a method of transmitting wireless data, for example, a wireless I / F such as IrDA or BlueTooth can be adopted.

Next, the conference image reproducing apparatus 2802 will be described. The conference image reproducing apparatus 2802 has, as its functional configuration, a data input unit 2821 and a recording unit 2822.
An image transforming unit 2823, a region determining unit 2824, and an image / audio output unit 2825 are included. Also, the conference image reproducing apparatus 2802 includes the direction correcting unit 806 and the area fixing unit 8
Has 07. In addition, in the following, each functional unit will be divided.
The function of the conference image reproducing apparatus 2802 can be realized by a personal computer. In this case, the software that implements each functional unit is stored in the hard disk, and the function can be implemented by appropriately executing the processing program.

The data input unit 2821 inputs, from a predetermined data transmission source, moving image data in which a wide-angle image is picked up, audio data synchronized with the moving image data, and data regarding a sound source direction. Here, although the predetermined data transmission source is the conference image transmission device 2801, the moving image data,
The format of the voice data and the data about the sound source direction does not depend on the transmission source device as long as the type of the data can be recognized. The data type can be determined by the file extension or the file header. The wide-angle image is assumed to be a donut image here, but it may be a panoramic image. This type is also determined by the extension and the header. The data input unit 2821 can employ, for example, a wireless I / F such as IrDA or BlueTooth.

The recording unit 2822 includes a data input unit 2821.
The wide-angle image moving image data, audio data, and data regarding the sound source direction including the elevation angle are recorded. There are various recording methods, but as described above, the MPEG format and R
The ealAudio format can be adopted.

The image transforming unit 2823 transforms the wide-angle image into a rectangular output image. For conversion, CC
Since it is designed to be in focus with D214, CC
The image captured by D214 is always a donut image.
Therefore, as described above, the image is transformed by referring to the correspondence table (not shown) between the donut image and the panoramic image. At this time, since the final output image is a portion including the speaker, the image transformation unit 2803 transforms the image of only the image region portion determined by the region determination unit 2824.

The area determining unit 2824 determines the area to be reproduced based on the data regarding the sound source direction including the elevation angle recorded in the recording unit 2822. As described in the first embodiment, the speaker position determination unit 809 may be used in combination to improve the speaker position detection accuracy. Image sound output unit 2
Reference numeral 825 outputs the image (moving image data) output from the image transforming unit 2823 and the sound recorded (input) at the same time when this image was captured (input) in association with each other.

Next, the processing flow of the conference image transmitting apparatus 2801 will be described. FIG. 29 is a flowchart showing an example of the processing flow of the conference image transmitting apparatus 2801 of the fourth embodiment. First, the system of the conference image transmitting device 2801 is activated by the user, and the operation of capturing data (image data and audio data) is started (step S290).
1). Next, it is judged whether or not the capture stop (recording) is instructed (step S2902), and if there is an instruction (step S2902: Yes), the capture is ended.

Unless there is an instruction to stop capturing (step S2902: NO), the image data transmitted from the CCD 214 and the voice data transmitted from the microphone array are continuously input (step S2903). When a certain amount of audio data, for example, the same number of samples as the size N of the correlation window C shown in Expression (2) is input, the sound source direction is detected and the sound source direction data is generated (step S29).
04). The conference image sending device 2801 sequentially outputs the image data, the sound data, and the sound source direction data to a predetermined destination, for example, a PC (step S2905). After that, the operations from step S2902 to step S2904 are sequentially repeated, and the data is transmitted until the user instructs recording stop.

Next, the processing flow of the conference image reproducing apparatus 2802 will be described. FIG. 30 is a flowchart showing an example of the processing flow of the conference image reproducing apparatus 2802 of the fourth embodiment. First, the system of the conference image reproducing device 2802 is activated by the user (step S3001).
Next, an image to be reproduced is selected according to the image displayed on the display (television) not shown (step S3002). FIG. 31 is a diagram showing an example of a screen configuration for selecting an image to be reproduced. As shown in the figure, the files of the conference are named Meeting1 and Meeting2, and each file is image data (MPEG).
-2 Video) and audio data (MPEG Audio)
It can be seen that it is composed of o) and sound source direction data (TEXT).

Next, the conference image reproducing device 2802 reads the wide-angle image data, the audio data and the sound source direction data, and starts the reproducing operation (step S3003). Subsequently, the conference image reproducing apparatus 2802 determines whether or not there is an instruction to stop the reproduction (step S3004), and if so, stops the reproduction. On the other hand, when there is no instruction to stop the reproduction (step S3004: NO), it is determined whether the time to inquire about the sound source direction data has been reached (step S3005). The time at which the sound source direction data is inquired means, for example, the time at which the sound source direction changes, as shown in FIG.

When the inquiry time has been reached (step S3005: Yes), the sound source direction data is accessed and a new sound source direction (values of azimuth θ and elevation φ) is acquired (step S3006). Subsequently, the conference image reproducing apparatus 2802 acquires the azimuth angle θ acquired in step S3006.
And partial image data corresponding to the elevation angle φ is extracted (step S3007), and the extracted partial image data and audio are output (reproduced) in synchronization (step S3008). If the inquiry time has not been reached in step S3005 (step S3005: NO), the currently reproduced partial image data is continued and reproduced (step S3009).

As described above, in the fourth embodiment, even if the conference image transmitting device and the conference image reproducing device are separately configured like the video camera and the VCR, the conference contents are kept realistic. However, it can be reproduced efficiently.

Embodiment 5. FIG. In the fifth embodiment, another configuration example of a general-purpose conference image transmitting device and a conference image reproducing device will be described. In addition, also in the fifth embodiment, the same components as those in the first to fourth embodiments are designated by the same reference numerals, and the description thereof will be omitted.

FIG. 32 is a diagram showing a functional block diagram of the conference image transmitting apparatus and the conference recording / playback apparatus according to the fifth embodiment. The conference recording / playback system 3200 according to the fifth embodiment is
Conference image transmitting device 3201 and conference image reproducing device 320
2 and. The conference image transmission device 3201 has, as its functional configuration, a wide-angle image input unit 3211, a voice input unit 3212, a sound source direction detection unit 3213, a wide-angle image expansion unit 3214, an image extraction unit 3215, and a data transmission unit 3.
216, and.

The wide-angle image input unit 3211 takes in a wide-angle image with the vertical direction as the center or axis, and outputs the image data to the wide-angle image developing unit 3214. For inputting the wide-angle image, any of the hyperboloidal mirror 211, the conical mirror surface body 2502, and the parabolic reflecting mirror may be used as in the fourth embodiment. The voice input unit 3212 inputs a voice, converts the voice into an electric signal (voice data), and outputs the voice data to the sound source direction detection unit 3213 and the data transmission unit 3216. The voice input unit 3212 may use a directional microphone or an omnidirectional microphone. In the sound source direction detecting unit 2813, the voice input unit 3212
The sound source direction is detected based on the time difference or the strength of the voice input from the image extraction unit 3215 and the data transmission unit 32.
Output to 16.

The wide-angle image developing unit 3214 transforms the donut image into a panoramic image and outputs it to the image extracting unit 3215 and the data sending unit 3216. The image extraction unit 3215 extracts an image of a predetermined portion in the speaker direction from the panoramic image output from the wide-angle image expansion unit 3214 based on the sound source direction output from the sound source direction detection unit 3213. The data sending unit 3216 sends the panoramic image (entire region), the extracted image (partial image in the speaker direction), the sound, and the data regarding the sound source direction to a predetermined data storage unit. Here, it is sent to the conference image reproducing apparatus 3202.

Next, the conference image reproducing apparatus 3202 will be described. The conference image reproducing apparatus 3202 has, as its functional configuration, a data input unit 3221 and a recording unit 3222.
And an image / audio output unit 3223 and a direction correction unit 3224. It should be noted that, although the functional units will be explained separately below, the conference image reproducing apparatus 3202 can be realized by a personal computer. In this case, store the software that implements each functional unit in the hard disk,
The function can be realized by appropriately executing the processing program.

The data input unit 3221 inputs moving image data in which a wide-angle image is picked up from a predetermined data transmission source, audio data synchronized with the moving image data, and data regarding the sound source direction. Here, the predetermined image data transmission source is the conference image transmission device 3201, but moving image data (entire image and partial image), audio data, and data related to the sound source direction are in a format that allows the type of the data to be recognized. It does not depend on the source device.

The recording unit 3222 includes a data input unit 3221.
The moving image data of the panorama image and the partial image in the speaker direction, the audio data, and the data related to the sound source direction are recorded. There are various recording methods, but as mentioned above, M
The PEG format and the RealAudio format can be adopted. The image / audio output unit 3223 outputs the partial image (moving image data) in the speaker direction output from the recording unit 3222 and the sound recorded (input) at the same time when this image is captured (input) in association with each other. .

However, there are cases where a partial image in the speaker direction is not properly extracted, or there are cases where it is desired to display an image other than the speaker, for example, a speaker including two adjacent persons or a whiteboard. Therefore, in order to meet such a request, the conference image reproducing device 3202 includes a direction correcting unit 3224. The direction correcting unit 3224 corrects the direction corresponding to the sound source direction so that the user can select a desired voice direction. The selection of the direction by the user will be described later.

Next, the processing flow of the conference image transmitting apparatus 3201 will be described. FIG. 33 is a flowchart showing an example of the processing flow of the conference image transmitting apparatus 3201 according to the fifth embodiment. First, the system of the conference image transmitting device 3201 is activated by the user, and the operation of capturing data (image data and audio data) is started (step S330).
1). Next, it is determined whether or not an instruction to stop capturing (stop recording) is given (step S3302), and if there is an instruction (step S3302: Yes), the capturing is ended.

Unless there is an instruction to stop capturing (step S3302: NO), the image data transmitted from the CCD 214 and the audio data transmitted from the microphone array are input (step S3303). When a certain amount of audio data, for example, the same number of samples as the size N of the correlation window are input, the sound source direction is detected and the sound source direction data is sequentially generated (step S3304). The conference image transmitting apparatus 3201 sequentially expands the wide-angle image (donut image) input in step S3303 into a panoramic image (step S3305), and generates partial image data in the sound source direction in the expanded panoramic image (step S).
3306).

The conference image sending device 2801 sequentially outputs the panorama image data, the partial image data, the sound data and the sound source direction data to a predetermined destination, for example, a PC (step S3307). After that, step S330
The operation from 2 to step S3307 is sequentially repeated, and the data is transmitted until the user instructs the recording stop.

Next, the processing flow of the conference image reproducing apparatus 3302 will be described. FIG. 34 is a flowchart showing an example of the processing flow of the conference image reproducing device 3302 according to the fifth embodiment. First, the system of the conference image reproduction device 3302 is activated by the user (step S3401).
Next, an image to be reproduced is selected according to an image displayed on a display (television) not shown (step S3402). FIG. 35 is a diagram showing an example of a screen configuration for selecting an image to be reproduced. As shown in the figure, the conference files are named Meeting1 and Meeting2, and each file includes panoramic image data (MPEG-2 Video) and audio data (MPEG).
It can be seen that it is composed of Audio), sound source direction data (TEXT), and further partial image data (MPEG-2VIdeo).

Next, the conference image reproducing apparatus 3302 reads the partial image data and audio data and starts the reproducing operation (step S3403). Subsequently, the conference image reproducing device 2802 determines whether or not there is an instruction to stop the reproduction (step S3404), and if so, stops the reproduction. On the other hand, when there is no instruction to stop the reproduction (step S34)
04: NO), and it is determined whether there is an input from the direction correction unit 3224 (step S3405). If the direction has been corrected (step S3405: YES), the specified partial image is extracted from the panoramic image and output (reproduced) together with the sound (step S3406).

On the other hand, if there is no input from the direction correcting unit 3224 (step S3405: NO), the partial image data is output as it is (step S3407). Since the conference image reproducing device 3202 sequentially outputs the partial images extracted in advance, it is sufficient to reproduce Meeting1_pv shown in FIG. 35 unless the direction is corrected.

Next, a configuration example of an image output from the conference image recording / reproducing apparatus 3200 will be described. FIG. 36 shows
FIG. 16 is an explanatory diagram showing a configuration example (screen example) of an image output from the conference image recording / reproducing device 3200. The screen includes not only the image 3601 in the direction of the speaker but also user interfaces such as a mode switching unit 3602, a direction instruction operation unit 3603, and a reproduction operation instruction unit 3604.

Next, each user interface will be described. The mode switching unit 3602 switches whether to reproduce a specific partial image in the wide-angle image data. As shown in FIG. 36, radio buttons can be used to switch the operation mode. That is, "AU
When the radio button labeled "TO" is selected, the partial image processed and extracted based on the sound source direction data and recorded in the recording unit 3222 is automatically reproduced. On the other hand, "MAN
When the radio button labeled "UAL" is selected, the screen shown in FIG.
As shown in FIG. 7, a donut image 3605 is displayed,
The operation shifts to the "manual switching mode" in which the user can manually select the portion to be reproduced.

In the manual switching mode, the direction indicating operation section 360, which is four buttons in which up, down, left, and right arrows are drawn.
The pointer 3607 is moved by 3. Pointer 36
By moving 07, it is possible to move the drawing direction of the partial image data and output the image in which the extracted portion is changed as shown in FIG. By this operation, for example, the drawing content on the whiteboard can be appropriately output. The screen configuration is as shown in FIGS.
The number of screens is not limited to eight, and four-division screens may be simultaneously output, for example, as shown in FIG. Here, reference numeral 3901 is a GUI for switching the output between the 4-split screen and one of the screens.

On the other hand, as shown in the figure, the reproduction operation instructing section 3604 has a GUI to which the functions of reproduction, stop, pause, fast forward, and rewind are assigned, and by pressing each section, The operation corresponding to the function is realized. It should be noted that although the processing is described as software here, buttons may be arranged on the side of the conference image reproducing apparatus 3302 in terms of hardware, and a remote controller may be separately provided to improve convenience.

By providing such a manual switching mode and a 4-split screen, for example, when a scene in which one participant continues to speak for a long time is reproduced later, only the video showing the speaker is reproduced endlessly. However, playing a participant other than the speaker intermittently is less tedious and more realistic. In this way, when it is desired to see an image other than the portion specified by the direction data such as the facial expressions of participants other than the speaker, the mode switching unit 3
602 and the direction instruction operation unit 3603 are particularly useful.

The conference image transmitting apparatus 3 of the fifth embodiment
Although 201 transmits both the panoramic image (entire region) and the extracted image (partial image in the speaker direction), only the partial image may be transmitted depending on the usage mode. At this time, the sound source direction data is the conference image reproduction device 3
Since the image is not extracted and the sound source direction is not determined on the 202 side, it is not necessary to send it to the conference image reproducing apparatus 3202.

As described above, in the fifth embodiment, as in the fourth embodiment, even if the conference image transmitting device and the conference image reproducing device are separately configured, the contents of the conference are kept realistic. However, it can be reproduced efficiently.

In the examples up to this point, a system for recording a conference was mainly described, but the present invention is not limited to this application, and may be used as a security camera by being installed on the sky, for example. it can. It can also be used for the purpose of investigating the ecology of nocturnal animals. In this case, a high sensitivity CCD is used.

[0205]

As described above, in the wide-angle image recording / reproducing system of the present invention (Claim 1), the wide-angle image input means inputs a wide-angle image centered on or in the vertical direction, and a plurality of voice inputs are made. The means inputs the voices respectively, the sound source direction detecting means detects the sound source direction based on the voices input by the plurality of voice input means, and the recording means,
The wide-angle image input by the wide-angle image input means, the voice input by the voice input means, and the sound source direction detected by the sound source direction detecting means are recorded, and the image transforming means is recorded by the recording means. In the wide-angle image, an image in a predetermined area in a direction corresponding to the sound source direction is transformed into a rectangular output image, and the image / audio output unit converts the image transformed by the image transforming unit into the image and the image. Since it outputs in synchronization with the sound recorded in the corresponding recording means, it is possible to record a wide-angle image and correct the distortion at the time of reproduction while reproducing the scene centered on the sound source direction. The recorded scene can be efficiently reproduced while maintaining the realism.

Further, in the wide-angle image recording / reproducing system of the present invention (claim 2), the wide-angle image input means inputs a wide-angle image centered on or in the vertical direction, and the image transforming means,
The wide-angle image input by the wide-angle image input means is transformed into a rectangular output image by a predetermined conversion formula, a plurality of voice input means respectively inputs a voice, and a sound source direction detection means is operated by the plurality of sound source direction detection means. The sound source direction is detected based on the voice input by the voice input unit, and the recording unit detects the image transformed by the image transforming unit, the voice input by the voice input unit, and the sound source direction detecting unit. The recorded sound source direction, the image extracting means extracts an image of a predetermined area in a direction corresponding to the sound source direction from the rectangular image recorded by the recording means, and the image sound output means Since the image extracted by the image extraction unit and the sound recorded in the recording unit corresponding to the image are output in synchronization, the image expanded in the panoramic shape is recorded. The scene around the sound source direction can play at any time, As a result, while maintaining a sense of realism, and the recording scene efficiently reproducible.

The wide-angle image recording / reproducing system of the present invention (claim 3) is the wide-angle image recording / reproducing system according to claim 1 or 2, wherein the wide-angle image input means has a paraboloid of a predetermined shape. Since it is composed of a mirror surface body, a mirror surface body having a hyperboloid of a predetermined shape or a mirror surface body having a predetermined cone shape, and an image pickup device, it is possible to capture a wide-angle image with a simple configuration, and thereby a small size is achieved. It becomes possible to provide an inexpensive device.

A wide-angle image recording / reproducing system according to the present invention (claim 4) is the wide-angle image recording / reproducing system according to claim 1, 2 or 3, wherein the sound source direction detecting means is the plurality of audio input means. Since the sound source direction is detected based on the time difference between the input voices, it is possible to detect the sound source direction with a simple configuration, and thus it is possible to provide a small and inexpensive device.

According to the wide-angle image recording / reproducing system of the present invention (claim 5), in the wide-angle image recording / reproducing system of claims 1 to 4, the direction correcting means corrects the direction corresponding to the sound source direction. Therefore, even when the sound source direction is not correctly detected due to noise or the like, a desired image can be reproduced, and thus, it is possible to efficiently reproduce the recorded scene while maintaining the realism.

The wide-area image recording / reproducing system of the present invention (claim 6) is the wide-angle image recording / reproducing system according to any one of claims 1 to 5, wherein the area fixing means comprises:
Since the predetermined area in the direction corresponding to the direction of the sound source is fixed, image blur can be prevented even when the direction of the sound source slightly moves, which allows the recorded scene to be efficiently performed while maintaining the presence. Can be reproduced.

A conference recording / playback system according to the present invention (claim 7) is a conference recording / playback system in which the wide-angle image recording / playback system according to any one of claims 1 to 6 is applied to recording / playback of a conference. Then, the speaker position determination means determines the position of the speaker based on the color distribution of the image or the moving part in the image, and the predetermined area determination means determines the predetermined area based on the determination result of the speaker position determination means. Since it is determined that the speaker portion can be accurately extracted, the conference can be efficiently reproduced while maintaining the presence.

Further, in the wide-angle image transmitting apparatus of the present invention (claim 8), the wide-angle image input means inputs a wide-angle image centered on or in the vertical direction, and the plurality of voice input means inputs voice. The sound source direction detecting means detects the sound source direction based on the voice input by the plurality of voice input means, and the data transmitting means detects the data regarding the wide-angle image input by the wide-angle image input means and the voice input. Data relating to the voice input by the means, data relating to the sound source direction detected by the sound source direction detecting means,
Is transmitted to a predetermined data storage means, a wide-angle image can be captured and a scene centered on the direction of the sound source can be reproduced. This makes it possible to efficiently reproduce a recorded scene while maintaining a sense of realism.

Further, a wide-angle image transmitting device (claim 9) of the present invention is the wide-angle image transmitting device according to claim 8,
The image transformation means transforms the wide-angle image input by the wide-angle image input means into a rectangular output image by a predetermined conversion formula, and the data sending means transforms the wide-angle image input by the wide-angle image input means. Since the data related to the image transformed by the image transforming means is transmitted in place of the data relating to the image, it is possible to reproduce the image expanded in the panoramic shape, thereby maintaining the realistic sensation and efficiently recording the recorded scene. Reproducible.

A wide-angle image transmitting apparatus according to the present invention (claim 10) is the wide-angle image transmitting apparatus according to claim 9, wherein the image extracting means includes the sound source among the images transformed by the image transforming means. An image of a predetermined region in the sound source direction detected by the direction detection means is extracted, and the data transmission means relates to the data regarding the wide-angle image input by the wide-angle image input means and the sound source direction detected by the sound source direction detection means. In place of the data, or together with the data on the wide-angle image input by the wide-angle image input means and the data on the sound source direction detected by the sound source direction detecting means, the data on the image extracted by the image extracting means is transmitted. Therefore, it is possible to reproduce scenes centered on the direction of the sound source, which allows recording while maintaining a sense of realism. And efficient to be able to reproduce the scene.

A wide-angle image transmitting device (claim 11) of the present invention is the wide-angle image transmitting device according to claim 8, 9 or 10, wherein the wide-angle image input means has a paraboloid of a predetermined shape. Since it is composed of a mirror surface body, a mirror surface body having a hyperboloid of a predetermined shape or a mirror surface body having a predetermined cone shape, and an image pickup device, it is possible to capture a wide-angle image with a simple configuration, and thereby a small size is achieved. It becomes possible to provide an inexpensive device.

Further, a wide-angle image transmitting device (claim 12) of the present invention is the wide-angle image transmitting device according to any one of claims 8 to 11, wherein the sound source direction detecting means has a plurality of voice inputs. Since the sound source direction is detected based on the time difference between the voices input by the means, it is possible to detect the sound source direction with a simple configuration, which makes it possible to provide a small and inexpensive device.

A wide-angle image transmitting apparatus according to the present invention (claim 13) is the wide-angle image transmitting apparatus according to claim 12, wherein the sound source direction detecting means is a voice input by a certain voice input means, Since the sound source direction is detected based on the time difference between the sound input means and the sound input by the sound input means farthest away, it is possible to detect the sound source direction with high accuracy, thereby maintaining the sense of presence while recording. The scene can be reproduced efficiently.

Further, a wide-angle image transmitting device (claim 14) of the present invention is the wide-angle image transmitting device according to any one of claims 8 to 11, wherein the plurality of audio input means are constituted by directional microphones. Since the sound source direction detecting means detects the sound source direction based on the intensity of the voice input by the directional microphone, it is possible to detect the sound source direction with a simple configuration, which is small and inexpensive. It becomes possible to provide a device.

A wide-angle image transmitting apparatus according to the present invention (claim 15) is the wide-angle image transmitting apparatus according to any one of claims 8 to 14, wherein the plurality of voice input means are the voice input means. Since the barycentric position of each is arranged so as to substantially coincide with the optical center of the wide-angle image input means, the coordinate system of the voice input means and the coordinate system of the wide-angle image input means can be made coincident to simplify various calculations. As a result, a small-sized and inexpensive device can be obtained.

Further, a wide-angle image transmitting apparatus (claim 16) of the present invention is the wide-angle image transmitting apparatus according to any one of claims 8 to 15, wherein the plurality of audio input means and the image pickup device are provided. It is arranged on the pedestal side, and the mirror body is arranged so as to face the pedestal through a transparent member, or
The plurality of audio input means, the image sensor and the mirror body are arranged on the pedestal side, and the second mirror body that reflects the reflected light from the mirror body on the pedestal side toward the image sensor on the pedestal side is transparent. Since it is placed facing the pedestal via a member, the electric system can be embedded in the pedestal to prevent the image from being divided by the conductors, etc., which makes it possible to efficiently reproduce the recorded scene while maintaining a sense of realism. To do.

Further, the wide-angle image transmitting device of the present invention (claim 17) is the wide-angle image transmitting device according to any one of claims 8 to 16, wherein the elevation angle setting means is set on a plane on which the device is installed. Since the elevation angle of the reference speaker is set and the image extraction means extracts an image based on the sound source direction detected by the sound source direction detection means and the elevation angle set by the elevation angle setting means, high accuracy is achieved. The image can be extracted, and the recorded scene can be efficiently reproduced while maintaining the realism.

A conference image transmitting apparatus (claim 18) of the present invention is a conference image transmitting apparatus to which the wide-angle image transmitting apparatus according to any one of claims 10 to 17 is applied for recording a conference. , The speaker position determination means determines the position of the speaker based on the color distribution of the image or the moving part in the image,
Since the predetermined area determination means determines the predetermined area based on the determination result of the speaker position determination means, it is possible to accurately extract the speaker portion, and thereby it is possible to efficiently reproduce the conference while maintaining the presence. And

Further, in the wide-angle image reproducing apparatus of the present invention (claim 19), the data input means includes moving image data in which the wide-angle image is captured, and audio data synchronized with the moving image data.
Data relating to the sound source direction is input, and the image transforming means causes the moving image data of the predetermined area to become a rectangular output image based on the data relating to the sound source direction, out of the moving image data input by the data inputting means. Since the image-audio output unit deforms and outputs the moving image data transformed by the image transforming unit and the audio data corresponding to the moving image data in synchronization with each other, the sound source direction is corrected while correcting the distorted wide-angle moving image data. A central scene can be output, which enables the user to efficiently view while maintaining a sense of reality.

Further, in the wide-angle image reproducing apparatus of the present invention (claim 20), the data input means relates to the moving image data in which the panoramic wide-angle image is picked up, the audio data synchronized with the moving image data, and the sound source direction. Data is input, and the image extracting means extracts the moving image data in the predetermined area based on the data regarding the sound source direction from the moving image data input by the data inputting means, and the image / sound outputting means outputs the image. Video data extracted by the extraction means,
Since the audio data corresponding to the video data is output in synchronization with the audio data, a scene centered on the sound source direction can be output,
As a result, it is possible for the user to efficiently view and listen while maintaining a sense of realism.

A wide-angle image reproducing device according to the present invention (claim 21) is the wide-angle image reproducing device according to claim 19 or 20, wherein the recording means stores the moving image data input by the data input means. The audio data and the data about the sound source direction are recorded, the reproduction instructing unit instructs the reproduction of the moving image data, and the output control unit causes the recording unit to reproduce the reproduction instruction. Based on the recorded data, the video / audio output means is controlled to output the moving picture data and the sound data corresponding to the moving picture data in synchronism with each other, so that the scene around the sound source direction is reproduced at any time. As a result, the recorded scene can be efficiently reproduced while maintaining the sense of presence.

According to the wide-angle image reproducing device of the present invention (claim 22), in the wide-angle image reproducing device of claim 19, 20 or 21, the direction correcting means corrects the direction corresponding to the sound source direction. Therefore, even if the sound source direction is not the desired direction due to noise or the like, the desired video can be reproduced, and thus the recorded scene can be efficiently reproduced while maintaining the realism.

A wide-angle image reproducing device according to the present invention (claim 23) is the wide-angle image reproducing device according to any one of claims 19 to 22, wherein the area fixing means corresponds to the direction of the sound source. Since the predetermined area is fixed, the image blur can be prevented even when the direction of the sound source slightly moves, and thus the recorded scene can be efficiently reproduced while maintaining the presence.

Further, the conference image reproducing device of the present invention (claim 24) is a conference image reproducing device to which the wide-angle image reproducing device according to any one of claims 19 to 23 is applied for reproducing a conference. The speaker position determining means determines the position of the speaker based on the color distribution of the moving image data or the moving part in the moving image data, and the predetermined region determining means determines the predetermined position based on the determination result of the speaker position determining means. Since the area is determined, the speaker portion can be accurately extracted, and thus the conference can be efficiently reproduced while maintaining the presence.

Further, in the wide-angle image recording / reproducing method of the present invention (claim 25), in the input step, a wide-angle image having the vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound are set. , In the recording step, the wide-angle image, the sound, and the sound source direction input in the input step are recorded, and in the transforming step, the sound source direction in the wide-angle image recorded in the recording step is recorded. The image of the predetermined area in the direction corresponding to is transformed into a rectangular output image,
In the reproducing step, since the image deformed in the image deforming step and the sound applied to the deformed image recorded in the recording step are reproduced in synchronization with each other, a wide-angle image is recorded and reproduced. It is possible to reproduce the scene centered on the sound source direction while correcting the distortion, and thus it is possible to efficiently reproduce the recorded scene while maintaining the presence.

Further, in the wide-angle image recording / reproducing method of the present invention (claim 26), in the input step, the wide-angle image having the vertical direction as the center or axis, the sound synchronized with the image, and the sound source direction of the sound are set. , In the transformation step, the wide-angle image input in the input step is transformed into a rectangular output image by a predetermined conversion formula, and in the recording step, the image transformed by the transformation step and the audio And a sound source direction, and in the extracting step, an image of a predetermined area in a direction corresponding to the sound source direction is extracted from the rectangular image recorded in the recording step, and in the reproducing step, the extraction is performed. Since the image extracted in the step and the sound related to the extracted image recorded in the recording step are reproduced in synchronization with each other, the image expanded in the panoramic shape is recorded,
Scenes centered on the direction of the sound source can be reproduced at any time, which makes it possible to efficiently reproduce recorded scenes while maintaining a sense of realism.

The conference recording / playback method (claim 27) of the present invention is a conference recording / playback method in which the wide-angle image recording / playback method according to claim 25 or 26 is applied to recording / playback of a conference. In the position determination step, the position of the speaker is determined based on the color distribution of the image or the moving part in the image, and in the predetermined area determination step, the predetermined area is determined based on the determination result based on the speaker position determination step. , The speaker part can be accurately extracted, and thereby the conference can be efficiently reproduced while maintaining the presence.

Further, in the wide-angle image transmitting method of the present invention (claim 28), in the input step, a wide-angle image having the vertical direction as the center or axis, a sound synchronized with the image, and a sound source direction of the sound, In the transforming step, the wide-angle image input in the inputting step is transformed into a rectangular output image by a predetermined conversion formula, and in the data sending step, data regarding the image transformed in the transforming step is input. , The data about the sound and the data about the sound source direction are sent to a predetermined data storage destination, so that the image expanded in the panoramic shape can be reproduced, which allows the recorded scene to be efficiently performed while maintaining the presence. To be reproducible.

The wide-angle image transmitting method (claim 29) of the present invention is the wide-angle image transmitting method according to claim 28, wherein in the extracting step, the image in the sound source direction is selected from the images deformed by the deforming step. An image of a predetermined area is extracted, and in the data sending step, the wide-angle image data and sound source direction input in the input step are replaced with the wide-angle image data and sound source direction data input in the input step. Since the data related to the image extracted in the extraction step is transmitted together with the data related to the above, it is possible to reproduce the scene centered on the direction of the sound source, and thereby it is possible to efficiently reproduce the recorded scene while maintaining the presence. To do.

Further, a conference image transmitting method (claim 30) of the present invention is a conference image transmitting method in which the wide-angle image transmitting method according to claim 29 is applied for recording a conference, and a speaker position determining step Then, the position of the speaker is determined based on the color distribution of the image or the moving part in the image, and the predetermined region is determined in the predetermined region determination step by the determination result of the speaker position determination step. Can be accurately extracted, and this makes it possible to efficiently reproduce the conference while maintaining the presence.

Further, in the wide-angle image reproducing method (claim 31) of the present invention, in the recording step, the moving image data in which the wide-angle image is captured, the audio data synchronized with the moving image data, and the data regarding the sound source direction are recorded. In the recording and transforming step, of the moving image data recorded in the recording step, the moving image data in a predetermined area is transformed into a rectangular output image based on the data regarding the sound source direction, and in the output step, the image Since the video data transformed by the transformation process and the audio data corresponding to the video data are output in synchronization with each other, a scene centered on the sound source direction can be output while correcting the distorted wide-angle video data. , It is possible to efficiently reproduce the recorded scene while maintaining the realism.

According to the wide-angle image reproducing method of the present invention (claim 32), the moving image data in which the panoramic wide-angle image is captured, the recording process, the audio data synchronized with the moving image data, and the sound source direction data are recorded. In the extracting step, the moving image data in the predetermined area is extracted based on the data regarding the sound source direction in the extracting step, and in the extracting step, the extracting step extracts the moving image data in the predetermined area. Since the video data and the audio data corresponding to the video data are output in synchronization with each other, it is possible to output a scene centered on the direction of the sound source, which makes it possible to efficiently reproduce the recorded scene while maintaining the presence. And

A conference image reproducing method (claim 33) of the present invention is a conference image reproducing method in which the wide-angle image reproducing method according to claim 31 or 32 is applied for reproducing a conference. In the determining step, the position of the speaker is determined based on the color distribution of the moving image data or the moving part in the moving image data, and in the predetermined region determining step, the predetermined region is determined based on the determination result based on the speaker position determining process. Therefore, the speaker portion can be accurately extracted, and thus the conference can be efficiently reproduced while maintaining the presence.

A program of the present invention (claim 34)
Causes the computer to execute each step of the method according to any one of claims 25 to 33, so that the conference can be efficiently reproduced while maintaining the presence.

[Brief description of drawings]

FIG. 1 is an explanatory diagram outlining a usage example in which the present invention is installed in a conference scene.

FIG. 2 is an external perspective view of the conference image transmitting apparatus according to the first embodiment.

3A and 3B are a front view and a plan view of the conference image transmitting apparatus according to the first embodiment.

FIG. 4 is an explanatory diagram showing a configuration example of a camera unit of the conference image transmitting apparatus according to the first embodiment.

FIG. 5 is a diagram illustrating an optical path when the hyperboloidal mirror according to the first embodiment is used.

FIG. 6 is a diagram showing a state of a wide-angle image formed on the surface of a CCD by the hyperboloidal mirror of the first embodiment.

FIG. 7 is a diagram showing a configuration example of a conference recording / playback apparatus according to the first embodiment.

FIG. 8 is a block diagram showing an example of a functional configuration of a recorded image reproduction system according to the first embodiment.

FIG. 9 is a diagram illustrating a principle of detecting a sound source direction by a sound source direction detecting unit according to the first embodiment.

FIG. 10 is a diagram illustrating that a direction in which a sound source exists is on a cone.

FIG. 11 is an explanatory diagram showing a manner of division when four sound sources are detected by dividing four microphones into two pairs.

FIG. 12 is an explanatory diagram illustrating how to take a set of microphones when the microphone unit is configured by three microphones.

FIG. 13 is a diagram showing a data configuration example in the sound source direction according to the first embodiment.

FIG. 14 is an explanatory diagram showing a state in which a donut image captured from a hyperboloidal mirror is transformed into a panoramic image.

FIG. 15 is a diagram showing a coordinate system of a donut image and a panoramic image, among the diagrams for explaining the deformation principle when a hyperboloidal mirror is used.

FIG. 16 is a diagram showing the relationship between the apex angle ψ as viewed from the CCD and the elevation angle φ in the diagram illustrating the deformation principle when a hyperboloidal mirror is used.

FIG. 17 is an explanatory diagram schematically showing an example of a conversion table in the case of converting a coordinate system from a donut image coordinate system (u, v) to a panoramic image coordinate system (θ, φ).

FIG. 18 is an explanatory diagram showing an example of a processing flow of the conference recording / playback system according to the first embodiment.

FIG. 19 is a diagram showing an example of an external configuration of an image recording / playback system according to a second embodiment.

FIG. 20 is an explanatory diagram showing an example of the hardware configuration of the conference image recording / reproducing system according to the second embodiment.

FIG. 21 is an explanatory diagram showing an example of a functional configuration of the conference recording / playback system according to the second embodiment.

FIG. 22 is an explanatory diagram showing an example of image extraction according to the second embodiment.

FIG. 23 is an explanatory diagram illustrating a method of generating partial image data by the image extracting unit according to the second embodiment.

FIG. 24 is an explanatory diagram showing an example of the processing flow of the conference recording / playback system according to the second embodiment.

FIG. 25 is an explanatory diagram showing an example of an external configuration of an apparatus including a camera unit according to a third embodiment.

FIG. 26 is an external configuration diagram of a camera unit configured to capture a donut image using two reflecting mirrors.

FIG. 27 is an explanatory diagram illustrating a relationship between a microphone unit and a sound source direction according to the third embodiment.

FIG. 28 is a diagram showing functional blocks of the conference image transmitting apparatus and the conference recording / playback apparatus according to the fourth embodiment.

FIG. 29 is a flowchart showing an example of the processing flow of the conference image transmitting apparatus of the fourth embodiment.

FIG. 30 is a flowchart showing an example of the processing flow of the conference image reproducing apparatus in the fourth embodiment.

FIG. 31 is a diagram showing an example of a screen configuration for selecting an image to be reproduced.

FIG. 32 is a diagram showing a functional block diagram of a conference image transmitting apparatus and a conference recording / playback apparatus according to the fifth embodiment.

FIG. 33 is a flowchart showing an example of the processing flow of the conference image transmitting apparatus 3201 according to the fifth embodiment.

FIG. 34 is a flowchart showing an example of the processing flow of the conference image reproducing apparatus in the fifth embodiment.

FIG. 35 is a diagram showing an example of a screen configuration for selecting an image to be reproduced.

FIG. 36 is an explanatory diagram showing a configuration example (screen example) of an image output from the conference image recording / playback device according to the fifth embodiment.

FIG. 37 is an explanatory diagram showing a state of a change in the configuration of the image shown in FIG. 36 after the “MANUAL” button is selected.

FIG. 38 is a diagram showing an image in which an extracted portion is changed by a direction designating operation unit according to the fifth embodiment.

[Fig. 39] Fig. 39 is an explanatory diagram illustrating another example of the screen configuration, which is an example of a 4-split screen.

[Explanation of symbols]

100, 1900, 2500, 2800, 3200
Conference recording / playback system 200, 2801, 3201 Conference image transmission device 201, 2005, 2501, 2600 Camera unit 202, 2006, 2701 Microphone unit 203, 2032 Pedestal 204 Transparent glass 211 Hyperboloidal mirror 212 Lens 213 Aperture 221,702 Microphone 300, 2802, 3202, 3302 Conference image reproducing device 307 Large-capacity recording device 801, 811, 3211 Wide-angle image input unit 802, 2812, 3212 Audio input unit 803, 3213 Sound source direction detection unit 804, 2813, 2822, 3222
Recording unit 805, 2803, 2823 Image transformation unit 806, 3224 Direction correction unit 807, 2824 Area fixing unit 808, 2825, 3223 Image sound output unit 809 Speaker position judgment unit 810 Area decision unit 1901 Cross button 1902 Decision button 1903 Image sound Output terminal 1904 Medium insertion slot 2003 Operation unit 2007 Removable media unit 2101, 3214 Wide-angle image development unit 2102, 3215 Image extraction unit 2502 Specular body 2601 Second reflecting mirror 2602 Second reflecting mirror 2814 Elevation angle setting unit 2815, 3216 Data transmission 2821, 3221 Data input section 3602 Mode switching section 3603 Direction instruction operation section 3604 Reproduction operation instruction section

Continued front page    F-term (reference) 5C022 AA00 AB68 AC21 AC51                 5C053 FA30 GB11 GB38 HA27 HA29                       HA40 JA01 LA01 LA14                 5C054 AA01 CA04 CC02 CE04 CH01                       DA01 EA01 EA07 EG10 FD02                       GB06 GD01 HA00                 5C064 AA02 AB04 AC03 AC04 AC06                       AC09 AC13 AC18 AC22 AD02                       AD08    (54) [Title of Invention] Wide-angle image recording / reproducing system, conference recording / reproducing system, wide-angle image transmitting device, conference image transmitting device,                     Angle image reproducing device, conference image reproducing device, wide-angle image recording / reproducing method, conference recording / reproducing method, wide-angle image transmission                     Method, conference image transmitting method, wide-angle image reproducing method, conference image reproducing method and program

Claims (34)

[Claims] 1. A wide-angle image input means for inputting a wide-angle image centered on or in the vertical direction, a plurality of voice input means for inputting voice, and a sound source based on voice input by the plurality of voice input means. Sound source direction detecting means for detecting a direction, wide-angle image input by the wide-angle image input means, voice input by the voice input means, and recording means for recording the sound source direction detected by the sound source direction detecting means And an image transforming unit that transforms an image of a predetermined area in a direction corresponding to the sound source direction into a rectangular output image among the wide-angle images recorded by the recording unit, and the image transforming unit transforms the image. Wide-angle image recording, comprising: an image and an image / sound output unit that outputs the image in synchronization with the sound recorded in the recording unit corresponding to the image. Playback system. 2. A wide-angle image input means for inputting a wide-angle image having a vertical direction as a center or an axis, and a wide-angle image input by the wide-angle image input means is transformed into a rectangular output image by a predetermined conversion formula. Image transforming means, a plurality of voice inputting means for inputting voices, a sound source direction detecting means for detecting a sound source direction based on the voices input by the plurality of voice inputting means, and the image transforming means An image, a voice input by the voice input unit, a recording unit that records the sound source direction detected by the sound source direction detection unit, and a rectangular image recorded by the recording unit,
An image extracting unit that extracts an image of a predetermined area in a direction corresponding to the sound source direction, an image extracted by the image extracting unit, and a sound recorded in the recording unit corresponding to the image are output in synchronization with each other. A wide-angle image recording / playback system, comprising: 3. The wide-angle image input means is composed of a mirror body having a paraboloid of a predetermined shape, a mirror body having a hyperboloid of a predetermined shape or a mirror body having a predetermined conical shape, and an image pickup device. The wide-angle image recording / playback system according to claim 1 or 2, wherein: 4. The wide-angle image according to claim 1, 2 or 3, wherein the sound source direction detecting means detects the sound source direction based on a time difference between sounds input by the plurality of sound input means. Recording and playback system. 5. The wide-angle image recording / reproducing system according to claim 1, further comprising direction correcting means for correcting a direction corresponding to the sound source direction. 6. The wide-angle image recording / reproducing system according to claim 1, further comprising area fixing means for fixing a predetermined area in a direction corresponding to the sound source direction. 7. A conference recording / playback system in which the wide-angle image recording / playback system according to any one of claims 1 to 6 is applied to recording / playback of a conference, wherein a color distribution of an image or a moving portion in the image is displayed. A conference recording / playback system comprising: a speaker position determining unit that determines the position of the speaker based on the speaker position; and a predetermined region determining unit that determines the predetermined region based on the determination result of the speaker position determining unit. . 8. A wide-angle image input means for inputting a wide-angle image centered on or in the vertical direction, a plurality of voice input means for inputting voice, and a sound source based on voice input by the plurality of voice input means. Sound source direction detecting means for detecting a direction, data about a wide-angle image input by the wide-angle image input means, data about a voice input by the voice input means, and a sound source direction detected by the sound source direction detecting means A wide-angle image transmission device comprising: a data transmission unit for transmitting data to a predetermined data storage unit. 9. An image transforming unit that transforms the wide-angle image input by the wide-angle image input unit into a rectangular output image by a predetermined conversion formula, wherein the data sending unit is configured by the wide-angle image input unit. 9. The wide-angle image transmitting apparatus according to claim 8, wherein the wide-angle image transmitting apparatus transmits the data regarding the image transformed by the image transforming means, in place of the input data regarding the wide-angle image. 10. An image extracting unit for extracting an image of a predetermined region in the sound source direction detected by the sound source direction detecting unit from among the images deformed by the image deforming unit, wherein the data sending unit includes the wide-angle image. Instead of the data regarding the wide-angle image input by the image input means and the data regarding the sound source direction detected by the sound source direction detecting means, or the data regarding the wide-angle image input by the wide-angle image input means and the sound source direction detecting means 10. The wide-angle image sending device according to claim 9, wherein the wide-angle image sending device sends the data concerning the image extracted by the image extracting means together with the data concerning the sound source direction detected by. 11. The wide-angle image input means is composed of a mirror body having a paraboloid of a predetermined shape, a mirror body having a hyperboloid of a predetermined shape or a mirror body having a predetermined conical shape, and an image pickup device. The wide-angle image transmitting device according to claim 8, 9, or 10. 12. The sound source direction detecting means detects the sound source direction based on a time difference between the voices input by the plurality of voice inputting means. Wide-angle image sending device. 13. The sound source direction detecting means determines a sound source direction based on a time difference between a voice input by a voice input means and a voice input by a voice input means that is farthest from the voice input means. The wide-angle image transmitting apparatus according to claim 12, wherein the wide-angle image transmitting apparatus detects the wide-angle image. 14. The plurality of voice input means is composed of a directional microphone, and the sound source direction detecting means is
The wide-angle image transmitting apparatus according to any one of claims 8 to 11, wherein the sound source direction is detected based on the strength of the sound input by the directional microphone. 15. The plurality of voice input means are arranged so that the barycentric position of the voice input means substantially coincides with the optical center of the wide-angle image input means, respectively. The wide-angle image transmission device according to any one of the above. 16. The plurality of voice input means and the image pickup device are arranged on a pedestal side, and the mirror body is arranged so as to face the pedestal through a transparent member, or the plurality of voice input means. A second mirror-faced body that is disposed on the pedestal side, and that reflects light reflected from the pedestal-side mirror-faced body toward the pedestal-side imager via a transparent member; 16. The wide-angle image transmitting apparatus according to claim 8, wherein the wide-angle image transmitting apparatus is arranged so as to face each other. 17. An elevation angle setting means for setting an elevation angle of a speaker with respect to a plane on which the apparatus is installed, wherein the image extracting means has a sound source direction detected by the sound source direction detecting means and the elevation angle setting means. The wide-angle image transmitting apparatus according to any one of claims 8 to 16, wherein the image is extracted based on the elevation angle set by. 18. A conference image transmission device, in which the wide-angle image transmission device according to any one of claims 10 to 17 is applied for recording a conference, which is based on a color distribution of the image or a moving portion in the image. A conference image transmitting apparatus comprising: a speaker position determining means for determining a position of a speaker by means of a speaker; and a predetermined area determining means for determining the predetermined area based on a determination result of the speaker position determining means. 19. Moving image data in which a wide-angle image is captured,
Data input means for inputting audio data synchronized with the moving picture data and data concerning the sound source direction, and moving picture data of a predetermined area based on the data concerning the sound source direction among the moving picture data inputted by the data input means. Image transforming means for transforming the image data into a rectangular output image, and moving image data transformed by the image transforming means, and image and sound output means for synchronizing and outputting the sound data corresponding to the moving image data. A wide-angle image reproducing device characterized by being provided. 20. Video data obtained by capturing a panoramic wide-angle image, and audio data synchronized with the video data,
Data inputting means for inputting data regarding a sound source direction; image extracting means for extracting moving image data in a predetermined area based on the data regarding the sound source direction from the moving image data input by the data inputting means; A wide-angle image reproducing device comprising: a moving image data extracted by the extracting unit; and an image / audio output unit that outputs the audio data corresponding to the moving image data in synchronization with each other. 21. A recording unit for recording the moving image data input by the data input unit, audio data, and data about a sound source direction, a reproduction instructing unit for instructing reproduction of the moving image data, and the reproduction. When there is a reproduction instruction from the instruction unit, the image / audio output unit is controlled based on the data recorded by the recording unit to synchronize the moving image data with the sound data corresponding to the moving image data. 21. The wide-angle image reproducing device according to claim 19, further comprising: an output control unit that outputs the wide-angle image. 22. A direction correcting means for correcting a direction corresponding to the sound source direction is provided.
20. The wide-angle image reproducing device according to 20 or 21. 23. The wide-angle image reproducing apparatus according to claim 19, further comprising area fixing means for fixing a predetermined area in a direction corresponding to the sound source direction. 24. A conference image reproducing device to which the wide-angle image reproducing device according to any one of claims 19 to 23 is applied for reproducing a conference, the color distribution of the moving image data or a moving part in the moving image data. A conference image reproducing device comprising: a speaker position determining means for determining the position of the speaker based on the above; and a predetermined area determining means for determining the predetermined area based on the determination result of the speaker position determining means. apparatus. 25. An input step of inputting a wide-angle image having a vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound, and a wide-angle image input by the input step, A recording step of recording voice and a sound source direction, and a transformation for deforming an image of a predetermined area in a direction corresponding to the sound source direction in the wide-angle image recorded in the recording step so as to be a rectangular output image A wide-angle image including a step, and a reproducing step of reproducing the image deformed by the image deforming step and the sound of the deformed image recorded in the recording step in synchronization with each other. Recording and playback method. 26. An input step of inputting a wide-angle image having a vertical direction as a center or an axis, a sound synchronized with the image, and a sound source direction of the sound, and a wide-angle image input by the input step is predetermined. A transforming step of transforming into a rectangular output image by the conversion formula of, a recording step of recording the image transformed by the transforming step, a sound, and a sound source direction, and a rectangle recorded by the recording step. Of the shape images,
An extracting step of extracting an image of a predetermined area in a direction corresponding to the sound source direction, an image extracted by the extracting step, and a sound of the extracted image recorded in the recording step are reproduced in synchronization with each other. A wide-angle image recording / reproducing method comprising: 27. A conference recording / playback method in which the wide-angle image recording / playback method according to claim 25 or 26 is applied to a recording / playback of a conference, the method being based on a color distribution of an image or a moving portion in the image. A conference recording / reproducing method comprising: a speaker position determining step of determining a position; and a predetermined area determining step of determining the predetermined area based on a determination result based on the speaker position determining step. 28. An input step of inputting a wide-angle image having a vertical direction as a center or an axis, a voice synchronized with the image, and a sound source direction of the voice, and a wide-angle image input by the input step is predetermined. A transformation step of transforming into a rectangular output image by the transformation formula of, and data relating to the image transformed by the transformation step,
A wide-angle image transmitting method, comprising: a data transmitting step of transmitting data regarding sound and data regarding a sound source direction to a predetermined data storage destination. 29. An extracting step of extracting an image of a predetermined region in the sound source direction from the image deformed by the deforming step, wherein the data transmitting step includes data relating to the wide-angle image input by the input step, and 29. The data relating to the image extracted in the extracting step is transmitted in place of the data relating to the sound source direction, or together with the data relating to the wide-angle image and the data relating to the sound source direction input in the input step. The wide-angle image transmission method described. 30. A conference image transmission method in which the wide-angle image transmission method according to claim 29 is applied for recording a conference, wherein the position of the speaker is determined based on a color distribution of the image or a moving portion in the image. And a predetermined area determining step of determining the predetermined area based on the determination result of the speaker position determining step. 31. Movie data obtained by capturing a wide-angle image,
A recording step of recording audio data synchronized with the moving image data and data relating to the sound source direction, and of the moving image data recorded by the recording step, the moving image data in a predetermined area is rectangular based on the data relating to the sound source direction. And an output step of outputting the moving image data transformed by the image transforming step and the audio data corresponding to the moving image data in synchronization with each other. Wide-angle image reproduction method. 32. Video data obtained by capturing a panoramic wide-angle image, and audio data synchronized with the video data,
A recording step of recording data relating to the sound source direction; an extracting step of extracting moving image data in a predetermined area based on the data relating to the sound source direction from the moving image data recorded by the recording step; and extracting by the extracting step. A wide-angle image reproducing method, comprising: an output step of outputting the synchronized moving image data and audio data corresponding to the moving image data in synchronization with each other. 33. A conference image reproducing method in which the wide-angle image reproducing method according to claim 31 or 32 is applied for reproducing a conference, wherein the speaker is based on a color distribution of the moving image data or a moving portion in the moving image data. And a predetermined area determining step of determining the predetermined area based on a determination result based on the speaker position determining step. 34. A program for causing a computer to execute each step of the method according to any one of claims 25 to 33.