JP4451892B2 - Video playback device, video playback method, and video playback program - Google Patents

Description

  The present invention relates to a system for recording a video of a wide range of scenes acquired using an imaging means having a wide-angle visual field. Specifically, it is used for applications such as a monitoring system, a remote conference system, and a remote education system.

  With the development of telecommunications technology, video conferencing systems that capture the state of meetings and transmit acquired images to remote locations have come to be used by many companies and organizations. In order to further improve the convenience of such a system, a number of systems for capturing a state of a conference as a video and a system for transmitting a partial video obtained by cutting out only a speaker have been proposed. For example, in Japanese Patent Laid-Open No. 5-12289, a video conference system in which a voice is input from a microphone to determine a speaker, and a camera control unit automatically controls the camera based on the determination result to capture the speaker. Has been proposed. However, there is a problem that it takes time to control the camera in order to capture the speaker.

  In order to solve this problem, Japanese Patent Application Laid-Open No. 11-331827 relates to a television camera apparatus using a fish-eye or super wide angle lens and a variable directivity microphone, and determines the direction of the sound source position and tracks the sound source position direction. Then, an invention has been disclosed in which an image in the sound source position direction is cut out to generate a video signal. However, when the TV camera device using a fish-eye or super-wide-angle lens is installed on a desk or the like, the human face is generally located in the peripheral part of the field of view, compared to the case where less important things such as the ceiling occupy most of the field of view. There is a problem that there is an important subject such as. In addition, these lenses are expensive to design and manufacture.

  On the other hand, with the recent development of image coding technology represented by MPEG (Motion Picture Experts Group), the increase in capacity of hard disk drives (hereinafter abbreviated as HDD) and the increase in data transfer speed, It has become technically possible to record an audio signal as electronic information in a personal computer (hereinafter abbreviated as PC). At the same time, there is an increasing need for recording scenes such as conferences on video and looking back on the recorded scenes later. However, all of the above prior arts are oriented toward a video conference system, and by transmitting a video signal obtained by cutting out only a desired part from a video acquired by the apparatus in real time, the same realistic sensation as in a normal conference is achieved. The main focus was on generating and transmitting high-quality video in real time. That is, it was not intended to record the acquired video on a PC or to view the video on the PC later.

The present invention has been made in view of the above-mentioned problems, and a first object of the present invention is to acquire and record a wide range of images with a simple configuration and processing, and at the same time, to capture images of a desired scene with high resolution. It is an object of the present invention to provide a video playback device, a video playback method, and a video playback program that can be acquired and recorded.

A second object of the present invention is to provide a video playback device, a video playback method, and a video playback program that enable a viewer to more easily select a video of a desired scene having a high resolution. To do.

In addition, a third object of the present invention is to provide a video playback device, video playback method, and video that make it possible to acquire and record a video of a desired scene at a high resolution without forcing the user to perform troublesome operations. It is intended to provide a playback program .

The fourth object of the present invention is to provide a video playback device, video playback method, and video playback program that make it possible to display a wide range of acquired videos in a form that is easier for viewers to observe. It is something to try.

The fifth object of the present invention is to provide a video playback device, a video playback method, and a video playback program that make it possible to acquire and record video of a desired scene with high resolution without omission. Is.

The present invention is obtained by a video acquisition unit that acquires video captured by an imaging unit that captures video, a sound source detection unit that acquires audio input from a plurality of microphones that input audio, and the video acquisition unit. A motion detection means for detecting a motion of a subject from the captured video, and an utterance section is extracted based on the voice acquired by the sound source detection means, and the video acquired by the video acquisition means in the extracted utterance section Based on a feature amount of movement of each part of the subject detected by the motion detection means, a video selection means for selecting a partial video corresponding to the subject in which the largest motion is detected in the video, and the video acquisition means And a display unit for displaying the acquired video and for displaying the partial video selected by the video selection unit.

The present invention is also a video playback method executed by a video playback device that plays back an acquired video, a video acquisition step of acquiring video captured by an imaging means for capturing video, and a plurality of inputs of audio. A sound source detecting step for acquiring sound input from the microphone, a motion detecting step for detecting a motion of a subject from the video acquired by the video acquiring means, and an utterance section based on the sound acquired by the sound source detecting step And the largest motion in the video is based on the feature amount of the motion of each part of the subject detected by the motion detection step from the video acquired by the video acquisition step in the extracted speech section. a video selection step for selecting a partial image corresponding to the detected subject, and displays an image acquired by the image acquisition step wherein And having a display step of displaying the partial image selected by the image selection process.

Further, the present invention provides a computer, a video acquisition unit that acquires video captured by an imaging unit that captures video, a sound source detection unit that acquires audio input from a plurality of microphones that input audio, and the video A motion detection means for detecting the movement of the subject from the video acquired by the acquisition means, and a speech section is extracted based on the voice acquired by the sound source detection means, and acquired by the video acquisition means in the extracted speech section. Video selection means for selecting a partial video corresponding to the subject in which the largest motion is detected in the video , based on a feature amount of motion of each part of the subject detected by the motion detection means from the video that has been recorded ; The video acquired by the video acquisition unit is displayed on the display device, and the partial video selected by the video selection unit is displayed. A video reproducing program for functioning as display means for displaying the location.

According to the present invention, in easy disjoint configuration and processing, at the same time to acquire and record a wide range of video, it is possible to acquire and record a high resolution image of the desired scene. That is, the first object is achieved.

Further, according to the present invention, it is possible to more easily select a desired scene image with have high resolution. That is, the second object is achieved.

Further, according to the present invention, without forcing a cumbersome operation to User chromatography THE, it is possible to acquire and record a high resolution image of the desired scene. That is, the third object is achieved.

Further, according to the present invention, a wide range of images that are acquisition, it is possible to display further easily observed form the viewer. That is, the fourth object is achieved.

Further, according to the present invention, by including a common region of one another picture and some even without low, it is possible to further obtain and recorded without omission with high resolution images of a desired scene. That is, the fifth object is achieved.

  First, an example of how the video recording system is used will be briefly outlined, and then an embodiment of the video recording system will be specifically described. In each embodiment, the components and operations thereof are described, and finally the processing flow is described.

  First, a usage example of the video distribution system will be described.

  FIG. 1 is an explanatory diagram outlining an example of use in which the present invention is installed in a conference scene. The video recording system includes a wide-angle camera 200 that acquires a wide-angle video, a camera array 400 that includes a plurality of cameras 401-1 to 401-4 having a normal angle of view, and a microphone 501 that acquires audio during a conference. And a server 300 for taking in and recording video data acquired by the wide-angle camera 200 and the camera array 400 and audio data acquired by the microphone 501.

  As shown in FIG. 1, the wide-angle camera 200 is installed on the table 1 and collects all the surrounding images overlooking the direction where the conference participants (speakers) 2-1 to 2-4 are present, for example, the horizontal plane. Take an image. In addition, each of the cameras 401-1 to 401-4 constituting the camera array 400 is placed, for example, in front of each participant in the conference, and photographs the appearance of each participant. The video acquired by these cameras 401-1 to 401-4 is hereinafter referred to as “partial video”. The server 300 is stored in the cabinet 3 and acquires video data from the wide-angle camera 200 and camera array 400 and audio data acquired by the microphone 501, and records the acquired video data and audio data in the HDD.

  In each of the following embodiments, a case will be described in which the video recording system of the present invention is applied to conference shooting and video recording.

Embodiment 1
First, Embodiment 1 of the present invention will be described.

Configuration FIG. 2 is a diagram showing a configuration of a video recording system according to Embodiment 1 of the present invention. The server 300 is connected to the wide-angle camera 200 and the camera array 400 via the USB hub 320 and the bus 310, and wide-angle video data and at least one partial video data acquired by the camera array 400 are acquired and recorded. The The video data recorded by the server 300 is displayed on the server 300. The video data is distributed via the Internet as necessary, and is displayed on a client PC connected to the Internet.

  Next, the configuration of each part will be described.

1.1.1 Wide-angle camera FIG. 3 is a diagram showing a configuration of a wide-angle camera 200 as the first imaging unit according to the first embodiment. The wide-angle camera 200 as the first image pickup means includes a mirror 211 having a curved surface with a predetermined shape, a lens 212, an aperture 213, an image pickup device 214 such as a CCD (Charge Coupled Device), and the timing of the image pickup device 214. A drive unit 215 that performs control and digitization processing such as analog-digital conversion on the video signal obtained by the imaging device 214, and edge enhancement and γ correction for the digital signal obtained by the drive unit 215 And the like, and a motor drive unit 217 that drives the diaphragm 213 to control the iris.

  The mirror 211 is for enabling wide-angle shooting by reflecting light incident on the optical system. Here, a hyperboloid mirror is used as a mirror having a curved surface with a predetermined shape. FIG. 4 is a diagram for explaining an optical path when the hyperboloid mirror 211 of the present embodiment is used. FIG. 5 is a diagram showing a state of a wide-angle image formed on the surface of the image sensor 214 by the hyperboloid mirror 211 of the present embodiment. As shown in FIG. 5, the image reflected from the hyperboloid mirror 211 and taken into the image sensor 214 has a donut shape (this donut-shaped image is hereinafter referred to as “doughnut image”). The donut image is imaged by the image sensor 214, digitized by the drive unit 215, and sent to the server 300 described later via the preprocessing circuit 216. Note that the central portion in FIG. 4 reflects the direction of the image sensor 214, which is unimportant image information. Therefore, the top portion 218 of the hyperboloid mirror 211 may be painted black to obtain black information. Depending on the mode of use, a reference line may be drawn on the top of the head 218, and when the wide-angle camera 200 is started up, the motor driving unit 217 may be driven to be used for initial settings such as focus adjustment.

  As described above, wide-angle images can be taken with an inexpensive and simple configuration by combining a normal camera and a mirror.

Camera Array The camera array 400 is composed of at least one camera, and each camera captures a part of the scene of the imaging range of the wide-angle camera 200 with higher resolution. Even if the cameras 401 constituting the camera array 400 are arranged separately as shown in FIG. 1, the cameras 401-1 to 401-3 are fixed to the housing 402 as shown in FIG. 6. It doesn't matter. As the imaging device used for the camera 401, various types such as a CCD and a CMOS (Complementary Metal-Oxide Semiconductor) type can be used. The video signal imaged by the image sensor is digitized inside the camera and then sent to the server 300 described later.

  By preparing at least one camera 401 having the above-described configuration, it is possible to acquire a partial video with high resolution with an inexpensive and simple configuration.

Server FIG. 7 is a diagram showing a configuration example of the server 300 in the present embodiment. That is, a CPU (Central Processing Unit) 301 that performs various controls and processes in the video recording system 100, an SDRAM (Synchronous Dynamic Access Memory) 302, an HDD (Hard Disk Drive) 303, an inting device such as a mouse 311, and the like. Various input interfaces (hereinafter abbreviated as I / F) 304 such as a keyboard 312, a power source 305, a display I / F 306 for connecting a display such as a CRT (Cathode Ray Tube), the wide-angle camera 200 and the camera External I / F 307 for connecting an external device such as the array 400, and a large size such as a DVD (Digital Versatile Disc) + RW drive And the amount recording unit 308, and by connecting via the bus 313.

  Next, each component of the server 300 will be described. The CPU 301 performs various processes and controls such as video acquisition / recording from the wide-angle camera 200 and the camera array 400 according to a predetermined program stored in the HDD 303. The SDRAM 302 is used as a work area of the CPU 301, and also stores each processing program stored in the HDD 303 and an OS (Operating System) such as Windows (registered trademark) NT Server (registered trademark of Microsoft Corporation, USA). Used as an area. The HDD 303 is also used as an area for recording the acquired video.

  Examples of the external I / F 307 include various I / F boards, USB (Universal Serial Bus), IEEE 1394, or wireless I / Fs such as IrDA and Bluetooth. The wide-angle video data and the plurality of partial video data acquired by the camera array 400 are synchronized by connecting the wide-angle camera 200 and the camera array 400 to the server 300 via a high-speed serial interface such as USB 2.0. Can be obtained automatically. Data acquired via the external I / F 307 is recorded in the HDD 303 or the mass storage device 308.

Operation FIG. 8 is a diagram in which the video recording system according to the present embodiment shown in FIG. 2 is rewritten into a functional block diagram. Hereinafter, the operation of each unit shown in FIG. 8 will be described in detail.

First Imaging Unit The first imaging unit 31 includes the wide-angle camera 200 described in 1.1.1 above, and performs an operation of outputting acquired and digitized wide-angle video data.

Second imaging unit The second imaging unit 32 includes the camera array 400 described in 1.1.2 above, and performs an operation of outputting acquired and digitized partial video data.

Deformation Part FIG. 9 is a diagram for explaining the operation of the deformation part 33 in the first embodiment. The transformation unit 33 transforms the wide-angle video data acquired by the first imaging unit 31 into a video (hereinafter referred to as a panoramic video) close to a perspective transformation image captured by a normal camera as shown in FIG. It is. In general, an image obtained by a camera capable of photographing a wide-angle range includes large distortion unlike the shape of an image that can be confirmed by the human eye, as described above, so that it is convenient for viewing later. It is preferable to apply a deformation process to. Hereinafter, the method described in the literature (A.M. Brookstein and T.J. Richardson: “Omniview Cameras with Curved Surface Mirrors”, Proc. Of the IEEE Workshop on Oct. 84) A method of transforming wide-angle video data (the donut video shown in FIG. 6) into a panoramic video will be described.

FIG. 10 is a diagram for explaining the principle of image deformation in a camera using a hyperboloidal mirror. FIG. 12A shows an operation example of the deforming unit 33. The donut image is coordinate-converted into a panoramic image projected on a cylindrical surface with the horizontal axis as the azimuth and the vertical axis as the elevation angle. FIG. 10B is a diagram showing the geometric structure of the wide-angle camera 200, and the optical system of the camera in FIG. 10B is a central projection model. Here, the meaning of each variable in the figure is as follows.
(U, v): coordinates (u ₀ , v ₀ ) in the donut image: coordinates (θ, φ) of the center of the hyperboloid mirror in the donut image: coordinates r: (u ₀ , v ₀ ) to (u , V) pixel unit distance r _max : radius of the hyperboloid mirror in the donut image θ: azimuth angle φ: elevation angle ψ: apex angle F from the optical axis of the camera F: focus of the hyperboloid mirror F ′: It coincides with the focal point of the hyperboloid paired with the hyperboloid mirror and the optical center of the camera.

  At this time, the following relationship is established between the apex angle ψ and the elevation angle φ.

ここで、

here,

である。また、φｍｉｎはドーナッツ映像上の半径ｒｍａｘの位置に対応する仰角φの値であり、これはカメラの仰角方向の撮影限界値を表す。ｒｍａｘとφｍｉｎの値は一般に容易に知ることができる。
以下、変形の手順を説明する。 （ｉ）点（ｕ， ｖ）に対応する極座標 （ｒ，θ）を、次式を解くことにより求める。

It is. Φmin is the value of the elevation angle φ corresponding to the position of the radius rmax on the donut image, and this represents the photographing limit value in the elevation angle direction of the camera. In general, the values of rmax and φmin can be easily known.
Hereinafter, a modification procedure will be described. (I) The polar coordinates (r, θ) corresponding to the point (u, v) are obtained by solving the following equation.

（ｉｉ） （３）式により算出されたｒに対応する頂角ψを次式により求める。

(Ii) The vertex angle ψ corresponding to r calculated by the equation (3) is obtained by the following equation.

ここで、

here,

であり、ψ_ｍａｘはドーナッツ映像上の半径ｒ_ｍａｘの位置及び仰角φ_ｍｉｎに対応する頂角ψの値である。ψ_ｍａｘの値は、（１）式にφ_ｍｉｎを代入することにより求めることができる。

Ψ _max is the value of the apex angle ψ corresponding to the position of the radius r _max and the elevation angle φ _min on the donut image. The value of ψ _max can be obtained by substituting φ _min into equation (1).

(Iii) The elevation angle φ corresponding to ψ calculated by equation (4) is obtained by equation (1).

  Through the above procedure, an arbitrary point (u, v) in the donut image captured by the hyperboloid mirror can be coordinate-converted to a point (θ, φ) in the panoramic image. That is, the donut video is transformed into a panoramic video.

  FIG. 11 is a diagram for explaining a coordinate conversion table used in the deforming unit 33. When panoramic video is recorded from shooting at a time, the calculation time required for the deformation process becomes a problem. Therefore, it is preferable to create a coordinate conversion table based on the above procedure as shown in FIG. is there. In the coordinate conversion table of FIG. 11, the coordinates (u, v) of the donut image corresponding to each point (θ, φ) are stored.

  The above deformation process is executed by the CPU 301 in the server 300. At this time, the HDD 303 stores in advance a predetermined program for performing the deformation process.

Encoding Unit The encoding unit 34 in FIG. 8 is a format suitable for video recording, with at least one of the partial video data acquired by the second imaging unit 32 in FIG. 8 and the panoramic video data output by the deformation unit 33. Encode to Here, there are various formats suitable for video recording, but a format such as a moving image encoding format represented by MPEG is used. The encoding unit 34 always encodes the video as long as the acquisition of the video data continues, and continuously transmits the encoded data to the recording unit 35.

  The above encoding process is executed by the CPU 301 in the server 300. At this time, an MPEG encoding program is installed in the HDD 303 in advance.

Recording Unit The recording unit 35 records at least one partial video data encoded by the encoding unit 34 and wide-angle video data acquired by the first imaging unit 31.

  The recording unit 35 can realize its function by the HDD 303. Note that the function may be realized by the large-capacity recording device 308 depending on the mode of use. For example, a long-time meeting or a regular meeting is recorded on a large-capacity recording device 308 composed of a DVD + RW or the like because of the necessity for storage, and a short-term meeting or the like that has a low necessity for long-term storage. Different usage such as recording in the HDD 303 may be performed.

Embodiment 2
A curved mirror having a curvature in one direction can also be used for the wide-angle camera 200.

Configuration The configuration of the second embodiment is shown in FIG. 4 as in the first embodiment.

  Hereinafter, the configuration of the wide-angle camera 200 in the present embodiment will be described. Note that the configurations of the server 300 and the camera array 400 are also the same as those described in the first embodiment, and a description thereof will be omitted.

Wide Angle Camera FIG. 12 is a diagram showing a configuration of a wide angle camera 200 in the present embodiment. As shown in FIG. 12, the wide-angle camera 200 includes a camera 219 having a normal angle of view and a mirror 211 having a curvature in one direction. Can be taken. FIG. 13 is a diagram showing a state of a wide-angle image projected on the camera 219 when the mirror 211 is used, and a scene behind the camera 219 can be taken. As shown in FIG. 13, the captured image has a shape compressed in the horizontal direction in a state where the horizontal angle of the incident light is proportional to the horizontal coordinate of the position of the image to be captured. . It is also possible to improve the camera 219 so as to reduce the reflection in the image.

  As described above, wide-angle images can be taken with an inexpensive and simple configuration by combining a normal camera and a mirror.

Operation The block diagram for each function in the present embodiment is shown in FIG. 9 as in the first embodiment. Hereinafter, the operation of each unit shown in FIG. 9 will be described in detail. Note that the operations of the first imaging unit 31, the second imaging unit 32, and the recording unit 35 are the same as those in the first embodiment described above, and a description thereof will be omitted.

Deformation Section In Embodiment 2 of the present invention, it is possible to obtain a panoramic image by simply stretching the video data acquired by the wide-angle camera 200 in the horizontal direction. Similar to the case of using a hyperboloidal mirror, a coordinate conversion table as shown in FIG. 11 may be created to store the coordinates (u, v) of the untransformed image corresponding to each point of the panoramic image. .

  When this wide-angle camera 200 is used, it is possible to generate and display a panoramic video on the client side that displays the video without providing the deformation unit 33 in the video recording system 100. Now, it is assumed that an image having a horizontal (horizontal) shooting range of 180 degrees, a vertical (vertical) shooting range of 60 degrees, and a size of 352 × 240 pixels is acquired by the wide-angle camera 200. In this case, a panoramic image can be obtained by extending the horizontal length to three times, that is, 1056 pixels. Also, the machine name of the server 300 is “vidserv”, the wide-angle video data name distributed from the video recording system is “movie.rm” (a data format called RealVideo, which will be described later), and communication between the video display terminal and the server 300. The protocol used is RTSP (Real Time Streaming Protocol). At this time, the process for executing the enlargement process can be described using SMIL (Synchronized Multimedia Integrated Language) recommended by W3C (World Wide Web Consortium) as shown in FIG. As shown in FIG. 14, when the size of the display area specified in the <region> tag is different from the image size of the associated video data “movie.rm”, the display is performed by specifying the “fit” attribute as “fill”. Video data is enlarged and reduced in accordance with the size of the area. That is, a panoramic video can be displayed by designating a display area having a desired enlargement ratio for video data in a state where the fit attribute value is designated as described above. The above deformation process is executed simultaneously with the video display in the video display terminal of the client. Accordingly, since it is not necessary for the server 300 to execute the deformation process at the time of video recording, it is possible to record wide-angle video data at a low processing cost.

Encoding Unit The operation of the encoding unit 34 is the same as that of the first embodiment described above. At least one of the partial video data acquired by the second imaging unit 32 and the panoramic video data output by the deforming unit 33 are processed. , Encode each into a format suitable for video recording.

  If the deforming unit 33 is not present, the encoding unit 34 encodes the wide-angle video data acquired by the first imaging unit 31 instead of the wide-angle video data deformed by the deforming unit 33.

Embodiment 3
The third embodiment of the present invention relates to a video recording system that identifies the correspondence between wide-angle video data acquired by the wide-angle camera 200 and each partial video data acquired by the camera array 400. . Examples of the “correspondence” here include the following.
Positional relationship between the wide-angle camera 200 and each camera 401 constituting the camera array 400 Positional relationship between the wide-angle video data and each partial video data When the above correspondence is unknown, when the video is reproduced, the partial video Even if switching is requested, there is no guarantee that the desired partial video data will be displayed. In order to solve this problem, measures such as installing the camera array 400 may be considered so that the partial video data to be reproduced is switched counterclockwise when the left arrow button of the video selection button 603 is pressed in order. However, there is a problem that the installation work of the video recording system 100 becomes troublesome because the switching order of the partial videos must correspond to the arrangement order of the cameras.

  FIG. 15 is a diagram showing an example of a display screen using the above correspondence in the video display terminal held by the client. In the figure, a bar 604 is installed on the lower side of the wide-angle image 602, and the shooting range corresponding to the currently displayed partial image 601 is indicated by a black bar 605. Further, the shooting ranges of partial videos other than the currently displayed partial video 601 are indicated by gray bars 606, respectively. Here, when the client moves the cursor 607 by operating a mouse (not shown) and clicks on the gray bar 606 showing a predetermined partial video, the client selects a partial video to be requested for distribution to the server 300. Information is transmitted and the partial video 601 from the camera array 400 sent via the server 300 is switched to the corresponding partial video. As described above, the above-described correspondence is specified, so that the installation work of the video recording system 100 is facilitated. In addition, the client can select a desired video more easily, and can further understand the scene to be photographed from the distributed video.

  Embodiment 3 of the present invention relates to a video recording system for realizing such an operation.

Configuration Similar to the first embodiment, the configuration of the third embodiment of the present invention is shown in FIGS.

Operation FIG. 16 is a diagram showing the video recording system according to the third embodiment of the present invention in functional blocks. In addition to the block diagram of the first embodiment shown in FIG. The specific part 36 is added. Hereinafter, the operation of each unit shown in FIG. 18 will be described in detail. Note that the operations of the first imaging unit 31, the second imaging unit 32, the deformation unit 33, and the encoding unit 34 are the same as those in the first embodiment, and a description thereof will be omitted.

Identification Unit The identification unit 36 performs an operation of identifying the correspondence between the wide-angle video data acquired by the wide-angle camera 200 and each partial video data acquired by the camera array 400. This operation will be described below.
Method of Using Identifiers Attached to Each Camera 401 Constructing Camera Array 400 FIG. 17 is a diagram illustrating another operation example of the specifying unit 36. As shown in FIG. 17A, an identifier 403 is assigned to each camera 401 constituting the camera array 400, and the camera 401 is disposed at a position where it can be captured by the wide-angle camera 200. In this state, the video data acquired by the wide-angle camera 200 is as shown in FIG. In this video data, the positional relationship between the wide-angle camera 200 and each camera 401 can be specified by detecting the image coordinates where the identifier 403 is reflected. Here, the identifier 403 includes
Stickers with mathematical numbers,
barcode,
Color code,
2D barcode,
The operation of reading these identifiers from video data is already well known in the field of pattern recognition.
Method of Using Video Data Acquired by Wide Angle Camera 200 and Camera Array 400 FIG. 18 is a diagram illustrating another example of the operation of the specifying unit 36. In this operation example, a portion having a high degree of similarity between wide-angle video data and each partial video data is detected.

  Here, an operation when template matching is used as means for detecting a portion having a high degree of similarity will be described. First, as shown in FIG. 18A, a template 608 having a size of (2DX + 1) × (2DY + 1) is generated from each partial image acquired by the camera array 400. Next, as shown in FIG. 18B, the template 608 is moved on the wide-angle image 602, and the normalized cross-correlation value S between the template 608 and the point (m, n) in the wide-angle image 602 is Calculate based on the formula.

ここで、（６）式における各記号の意味は以下の通りである。
Ｉ_１（ｘ， ｙ）：テンプレート上の点（ｘ， ｙ）における濃度
Ｉ_２（ｘ， ｙ）：広角の映像上の点（ｘ， ｙ）における濃度
以上の計算に基づき、正規化相互相関値Ｓが最大となる広角の映像６０２における点（ｍ，ｎ）を求め、該点の位置に対応するカメラ４０１を特定すればよい。以上の動作を、全ての部分映像に対して実行することにより、広角カメラ２００と各々のカメラ４０１との位置関係を特定することができる。

Here, the meaning of each symbol in the equation (6) is as follows.
I ₁ (x, y): density at point (x, y) on template I ₂ (x, y): density at point (x, y) on wide-angle image Normalized cross-correlation based on the above calculation A point (m, n) in the wide-angle video 602 with the maximum value S may be obtained, and the camera 401 corresponding to the position of the point may be specified. The positional relationship between the wide-angle camera 200 and each camera 401 can be specified by executing the above operation on all partial videos.

Although it has been described that the similarity of images is obtained based on the cross-correlation of density, this is merely an example. You may obtain | require the similarity of an image | video based on another characteristic, such as a color space and an outline of an image.
Manual Identification Method FIG. 19 is a diagram showing a display screen of the server 300 according to the third embodiment. This display screen appears immediately before the video recording system 100 is started and video recording is started. Thereafter, the user first switches the displayed partial video 601 by operating the video selection button 603. Then, a message 609 that prompts the user to manually input the positional relationship between the currently displayed partial video 601 and the wide-angle video 602 is presented on the display screen. At this time, the user manually inputs the positional relationship by operating a mouse (not shown) to move the cursor 607 and clicking a predetermined point on the wide-angle image 602. When manual input is completed, a cross-shaped pointer 610 is attached to a position corresponding to the partial image 601 in the wide-angle image 602. The positional relationship between the wide-angle video 602 and each partial video 301 can be specified by executing the above operation on all the partial videos.

  This method is particularly effective when the arrangement positions of the wide-angle camera 200 and the camera array 400 are unchanged from the start to the end of video recording. On the other hand, the methods (1) and (2) are effective even when the arrangement position of the camera 401 is changed in the middle.

  The above processing is executed by the CPU 301 in the server 300. At this time, the HDD 303 stores in advance a predetermined program for performing the specific processing.

Recording unit The recording unit 35 records wide-angle video data encoded by the encoding unit 34 and at least one partial video data. At this time, it is preferable to record not only the video data but also the correspondence specified by the specifying unit 36 because the display screen shown in FIG. 17 can be presented on the video display terminal. . The video recording operation is the same as that in the first embodiment.

Embodiment 4
The fourth embodiment of the present invention relates to a video recording system that automatically selects each partial video data acquired by the camera array 400.

  In the first to third embodiments, the client selects a partial video for which distribution is requested when the video is played back later. However, it is troublesome to manually select a partial video each time.

  FIG. 20 is a diagram showing a display screen on which a partial video 601 to be displayed is automatically selected on a video display terminal held by a client. As shown in the figure, when a check box 611 written “AUTO” is checked, the mode is switched to a mode in which the partial video 601 is automatically selected and distributed. On the other hand, the server 300 automatically selects a partial video in which an important scene such as a speaker is shown and distributes it to the client together with a wide-angle video. Accordingly, the client can understand the scene to be photographed more deeply from the distributed video without troublesome operations.

  The fourth embodiment relates to a video recording system for realizing such an operation.

Configuration FIG. 21 is a diagram showing a configuration of a video recording system according to Embodiment 4 of the present invention. A wide-angle camera 200, a camera array 400, and a microphone array 500 are connected to the server 300, and wide-angle video data, a plurality of partial video data, and a plurality of audio data are acquired and recorded. The video data and audio data recorded by the server 300 are displayed and reproduced on the server 300. Further, the video data and the audio data are distributed via the Internet as necessary, and are displayed and reproduced on a client PC connected to the Internet.

  Next, the configuration of each part will be described. Note that the configurations of the wide-angle camera 200, the camera array 400, and the server 300 are the same as those in the first embodiment, and a description thereof will be omitted.

Microphone array The microphone array 500 includes at least two microphones 501-1 and 501-2. As the microphones 501-1 and 501-2 to be used, various types such as a piezoelectric type and a capacitive type (so-called condenser microphone) can be used. As with the camera 401, each microphone 501-1, 501-2 is fixed separately to each other even if the microphones 501-1, 501-2 are arranged separately. It may be arranged. FIG. 22 is a diagram illustrating the configuration of the wide-angle camera 200 and the microphone array 500 according to the fourth embodiment. As described above, the wide-angle camera 200 and the microphone array 500 may be integrated into one housing. As shown in FIG. 22, the image sensor 214 of the camera unit 201 constituting the wide-angle camera 200 and the microphones 501-1 and 501-2 constituting the microphone array 500 are arranged on the pedestal 202.

  Audio signals acquired by the microphones 501-1 and 501-2 are digitized inside the microphones and then sent to the server 300. Similar to the camera array 400, the microphone array 500 is connected via the external I / F 307 of the server 300, specifically via a high-speed serial interface such as USB 2.0, so that the partial video and audio are synchronized. It is possible to get to.

Operation FIG. 23 is a diagram illustrating functional blocks of the video recording system according to the fourth embodiment. In addition to the block diagram of the third embodiment shown in FIG. 16, an audio acquisition unit 37, a sound source detection unit 38, and a video selection unit 39 are added. Hereinafter, the operation of each unit shown in FIG. 23 will be described in detail. The operations of the first imaging unit 31, the second imaging unit 32, the deforming unit 33, and the specifying unit 36 are the same as those in the above-described embodiment, and thus the description thereof is omitted.

Audio Acquisition Unit The configuration and operation of the audio acquisition unit 37 is to output digital audio data acquired by the camera array 400 described in 4.1.1 above.

Sound source detection unit The sound source detection unit 38 detects the position or direction of the speaker based on the audio data acquired by the audio acquisition unit 37. An example of the operation will be described below.
Method by Difference in Arrival Time of Sound Input to Microphone Array 500 This method is effective when a plurality of microphones 501 are fixed at known positions in a certain housing. FIG. 24 is a diagram for explaining the operating principle of the sound source detection unit 38 according to the fourth embodiment of the present invention. As shown in FIG. 24, two microphones 501-1 and 501-2 (referred to as microphone 1 and microphone 2 for convenience) are arranged at a distance l, and sound enters from the θ direction. case, the relationship between the speech data _s 1 the microphone 1 outputs (t), the voice data _s 1 the microphone 1 outputs (t), the

ｖ：音速
となり、マイク１の音声データがマイク２の音声データに対して

v: The sound speed is set, and the sound data of the microphone 1 is compared to the sound data of the microphone 2.

だけ時間が進んでいることとなる。この原理を利用して、話者の音声の方向を特定する手順を説明する。

Only time will be advanced. A procedure for specifying the direction of the speaker's voice using this principle will be described.

First, the arrival time difference between the audio data of the microphone 1 and the microphone 2 is detected. This arrival time difference is calculated by, for example, a cross-correlation value between the voice data s ₁ (t) of the microphone ₁ and the voice data s ₂ (t + dt) of the microphone 2. Here, the cross-correlation value C (t, dt) is calculated by the following equation.

ここで、Ｎは相関窓の大きさを示す正の整数であり、（８）式は時刻ｔ以前のＮ個のサンプルを用いて積和演算が行われることを示す。このとき、Ｃ （ｔ， ｄｔ）を最大化するｄｔが到達時間差となる。

Here, N is a positive integer indicating the size of the correlation window, and Equation (8) indicates that the product-sum operation is performed using N samples before time t. At this time, dt that maximizes C (t, dt) is the arrival time difference.

  Next, the angle θ formed by the voice and the base line of the microphone is calculated using the microphone interval l, the arrival time difference dt, and the sound velocity v.

ここで、θの値域は０°以上１８０°以下とする。

Here, the range of θ is 0 ° or more and 180 ° or less.

  Note that the direction is detected only in the range of 180 ° in front of the microphones 501-1 and 501-2 only by the above procedure, and the sound source direction is not specified. That is, the angle θ output from the sound source detection unit 38 is actually an angle formed by the direction of voice arrival and the base line between the two microphones, and the actual direction of the voice is two microphones as shown in FIG. It exists anywhere on the side of the cone of apex angle θ with the midpoint as the apex.

  In order to solve this problem, correction is performed using another microphone pair that is not parallel to the pair composed of the microphone 1 and the microphone 2. FIG. 26 is an explanatory diagram showing how the four microphones 501-1, 501-2, 501-3, and 501-4 are divided into two groups and the sound source direction is detected. As shown in FIG. 26, the grouping is performed according to the microphones 501-1 and 501-3 (for example, the microphone 1 (microphone 3)) and the microphones 501-2 and 501-4 (the microphone 2) that are the farthest from the microphone. (Mic 4)).

By using a pair of two microphones that are the farthest away, the difference in arrival time of the voice is maximized, and the accuracy of direction detection is improved. Here, the microphone array 500 includes four microphones 501-1, 501-2, 501-3, and 501-4. However, the direction of the sound source can be detected with high precision using three microphones. FIG. 27 is an explanatory diagram for explaining how to use a set of microphones when the microphone array 500 is configured by three microphones 501-1, 501-2, and 501-3. As shown in FIG. 27, by arranging the microphones in an equilateral triangle, the direction of the sound source can be detected with high accuracy regardless of which microphone set is employed. In the example shown in FIG. 27, the first set and the second set can be adopted to detect sound sources in all directions, but the third set may be used complementarily.
Method Using Directional Microphone Array It is also possible to detect the direction of the speaker by using a directional microphone that can input only a limited range of speech. FIG. 28 is an explanatory diagram for explaining the relationship between the microphone array 500 and the sound source direction in the fourth embodiment. The microphone array 500 includes four microphones 501 having directivity, and determines the sound source direction based on the sound intensity. For convenience, the four microphones 501-1, 501-2, 501-3, and 501-4 are referred to as microphones 1-4.

  Assume that the voice intensity is 20 for the microphone 1, 30 for the microphone 2, 20 for the microphone 3, and 5 for the microphone 4. In this case, it is determined that there is a sound source in the direction of the microphone 2. Comparing the intensities of the microphone 1 and the microphone 3, since both values are the same 20, the sound source direction is finally determined to be the direction of the microphone 2 (direction indicated by θ = 45 ° in the figure).

  FIG. 29 is a diagram illustrating another example of the operation of the sound source detection unit 38 in the fourth embodiment. Assume that the voice intensity is 15 for microphone 1, 30 for microphone 2, 25 for microphone 3, and 5 for microphone 4. In this case, it is initially determined that there is a sound source in the direction of the microphone 2. Comparing the intensities of the microphone 1 and the microphone 3, since the intensity of the microphone 3 is greater than the microphone 1, the direction of the sound source is slightly moved from the direction of the microphone 2 to the direction of the microphone 3 (the direction indicated as θ = 30 ° in the figure). ). The amount of movement in this direction may be determined in advance according to the characteristics of the directional microphone.

  The function of the sound source detection unit 38 described above is executed by the CPU 301 in the server 300. At this time, the HDD 303 stores a predetermined program for realizing the function in advance.

Video Selection Unit The video selection unit 39 automatically selects a partial video to be recorded using the correspondence specified by the specifying unit 36 and the position or direction of the speaker detected by the sound source detection unit 38. To choose.

  FIG. 30 is a diagram illustrating an example of the operation of the video selection unit 39 according to the fourth embodiment, and it is seen from above that six participants 2 of A to F surround the table 1 and hold a conference. Is. On the table 1, a wide-angle camera 200 and a microphone array 500 are installed, and one camera 401 (not shown) is installed for each participant. Assume that the direction of the sound source detected by the sound source detection unit 38 is as indicated by an arrow 381 in the figure. At this time, the video selection unit 39 is a camera arranged closest to the direction of the sound source based on the direction of the sound source and the correspondence between the wide-angle camera 200 specified by the specifying unit 36 and each camera 401. 401 is selected. That is, in the figure, the camera 401 that is shooting the participant E is selected.

  The function of the video selection unit 39 described above can be realized by the CPU 301 in the server 300. At this time, the HDD 303 stores a predetermined program for realizing the function in advance.

Encoding Unit The operation of the encoding unit 34 encodes the partial video data selected by the video selection unit 39 and the panoramic video data output by the transformation unit 33 into a format suitable for video recording. There are various formats suitable for video recording. For example, video data is encoded in a video encoding format typified by MPEG. Further, not only video data but also audio data may be encoded, and the audio data is encoded in a format such as MPEG audio format.

  In addition, video data and audio data may be stored in one file and recorded as an MPEG program stream. By using this file format, the number of files to be recorded in the recording unit 35 at the subsequent stage is reduced, so that management of the recorded files can be further facilitated.

  The encoding unit 34 always encodes the video as long as the acquisition of the video data continues, and continuously transmits the encoded data to the recording unit 35.

  The above encoding process is executed by the CPU 301 in the server 300. At this time, an MPEG encoding program is installed in the HDD 303 in advance.

Recording Unit The recording unit 35 records at least one partial video data encoded by the encoding unit 34 and wide-angle video data acquired by the first imaging unit 31. At this time, it is preferable to record not only the video data but also the correspondence relationship specified by the specifying unit 36 and the information about the partial video data selected by the video selecting unit 39.

  Here, an example of an operation for recording information on the selected partial video data will be described. FIG. 31 is a diagram showing a recording example of information related to partial video data. In FIG. 31, the time (Time) when the selected partial video is changed and a new data name (File) are recorded. These data are recorded in the HDD 303 together with moving image data and audio data in the form of a text file or the like. In this way, by recording the time when the selected partial video data changed and the data name at that time as needed, it is possible to play back the appropriate partial video data when playing back the video later. It becomes.

  Note that the information on the partial video data may be described using a content description standard for multimedia information such as MPEG-7.

  The recording unit 35 can realize its function by the HDD 303. Note that the function may be realized by the large-capacity recording device 308 depending on the mode of use. For example, a long-time meeting or a regular meeting is recorded on a large-capacity recording device 308 composed of a DVD + RW or the like because of the necessity for storage, and a short-term meeting or the like that has a low necessity for long-term storage. Different usage such as recording in the HDD 303 may be performed.

Embodiment 5
The fifth embodiment of the present invention relates to a video recording system that automatically selects each partial video data acquired by the camera array 400, as in the fourth embodiment described above. Each configured camera 401 and each microphone 501 configuring the microphone array 500 are configured to have a one-to-one correspondence. Here, the “one-to-one correspondence” is defined as “there is one microphone 501 arranged at a position or a direction substantially matching each camera 401”.

Configuration The configuration of the video recording system 100 in the present embodiment is shown in FIG. 21 as in the fourth embodiment.

  Next, the configuration of each part in the above diagram will be described. Note that the configurations of the wide-angle camera 200 and the server 300 are the same as those in the first embodiment, and a description thereof will be omitted.

Camera Array and Microphone Array FIG. 32 is a diagram showing the appearance of the camera 401 and the microphone 501 according to the fifth embodiment of the present invention. As illustrated, the camera 401 and the microphone 501 have a structure integrated with a common housing 502. The microphone 501 has directivity and can input only a limited range of sound. One integrated camera 401 and microphone 501 is installed per participant.

FIG. 33 is a diagram in which the video recording system according to the present embodiment is rewritten into a functional block diagram. The sound source detection unit 38 is deleted from the block diagram of the fourth embodiment shown in FIG. Further, the connection from the specifying unit 36 to the video selection unit 39 is deleted. The operations of the first imaging unit 31, the second imaging unit 32, the deforming unit 33, the encoding unit 34, the recording unit 35, the specifying unit 36, and the sound acquisition unit 37 are the same as those in the above-described fourth embodiment.

Video Selection Unit By using the integrated camera 401 and microphone 501 described above, the correspondence between each camera 401 and microphone 501 is known. Therefore, the video selection unit 39 may select a partial video acquired by the camera 401 corresponding to the microphone 501 from which the largest signal amplitude is obtained.

Embodiment 6
This embodiment relates to a video recording system that automatically selects each partial video data acquired by the camera array 400, as in the above-described fourth to fifth embodiments.

Configuration Similar to the first to third embodiments, the configuration is shown in FIGS.

FIG. 34 is a diagram in which the video recording system according to the present embodiment is rewritten into a functional block diagram. In addition to the block diagram of the third embodiment shown in FIG. A motion detection unit 40 is added. In the following, the operation of each unit shown in FIG. 34 will be specifically described. The operations of the first imaging unit 31, the second imaging unit 32, the deforming unit 33, the encoding unit 34, the recording unit 35, and the specifying unit 36 are the same as those in the above-described embodiment, and thus description thereof is omitted.

Motion Detection Unit The motion detection unit 40 detects the motion of the subject in the wide-angle video data and outputs the feature quantity of the motion in each part in the video. Here, the “motion feature amount” refers to the magnitude of the movement of the subject.

  Motion detection in a moving image can be realized by a known technique such as a method of obtaining a difference between frames of the previous time and the current time, a method using an optical flow, and the like. With these technologies, it is possible to detect the position of the subject and the magnitude of the movement in the wide-angle video data. According to this operation, when the present invention is used as a remote monitoring system, a partial video from a camera that captures a moving subject is recorded, which is preferable.

  Further, when the video recording system according to the sixth embodiment is used as a remote conference system, it is preferable to automatically detect the position or direction of the speaker by detecting the movement of the lips of the participant. . Detection of lip movement can be realized by well-known techniques such as literature (M. Kass, A. Witkin and D. Terzopoulos: “SNAKES: Active Control Models”, ICCV, pp. 259-268 (1987)). Further, when the microphone 501 can be used as in the fourth to fifth embodiments, the detection accuracy of the speaker is improved by detecting the movement of the lips together with the extraction result of the utterance section based on the voice data. You can also For example, Japanese Patent Laid-Open No. 6-43897 output by the present applicant discloses a system for recognizing a conversation by using audio features extracted from audio data and dynamic visual features of a face extracted from video data. Is disclosed. This operation makes it possible to detect the position or direction of the speaker more stably even when a lot of noise other than speech is occupied in the voice data.

  The function of the motion detection unit 40 described above may be implemented inside the wide-angle camera 200 or may be realized by the CPU 301 in the server 300. In the latter case, the HDD 303 stores a predetermined program for realizing the function in advance.

Video Selection Unit The video selection unit 39 according to the sixth embodiment should be recorded using the correspondence specified by the specifying unit 36 and the feature amount of the motion of the subject detected by the motion detection unit 40. The partial video is automatically selected. The video selection unit 39 first specifies the image position where the largest motion is detected in the wide-angle video data based on the feature quantity of the motion of the subject. Next, based on the identified image position and the correspondence relationship between the wide-angle camera 200 identified by the identifying unit 36 and each camera 401, the same procedure as described in the above-described fourth embodiment is performed. The camera 401 arranged closest to the position is selected. As a result, it is possible to automatically select a partial video obtained by photographing the subject in which the largest movement is detected.

  The function of the video selection unit 39 described above is executed by the CPU 301 in the server 300. At this time, the HDD 303 stores a predetermined program for realizing the function in advance.

Embodiment 7
In the above-described sixth embodiment, the motion of the subject is detected in the wide-angle video, but the motion of the subject may be detected in each partial video data acquired by the camera array 400.
7.1 Configuration The configuration of the seventh embodiment of the present invention is shown in FIGS. 2 to 7 as in the first to third embodiments.

Operation FIG. 35 is a diagram in which the video recording system according to the present embodiment is rewritten into a functional block diagram. In the block diagram of the sixth embodiment shown in FIG. 2 Video data output by the imaging unit 32 is input to the motion detection unit 40, and the connection from the specifying unit 36 to the video selection unit 39 is deleted. In the following, the operation of each unit shown in FIG. 35 will be specifically described. The operations of the first imaging unit 31, the second imaging unit 32, the deforming unit 33, the encoding unit 34, and the recording unit 35 are the same as those in the above-described embodiment, and thus description thereof is omitted.

Motion Detection Unit The motion detection unit 40 according to the present embodiment detects the motion of the subject in each partial video data and outputs the feature quantity of the motion in each partial video data. Here, the “motion feature amount” refers to the magnitude of the movement of the subject as in the sixth embodiment. The movement of the subject in each partial video data is also detected by the well-known technique described in the sixth embodiment.

  Further, when the video recording system according to the present embodiment is used as a remote conference system, it is preferable that the position or direction of the speaker is automatically detected by detecting the movement of the participant's lips in the partial video. It is. This operation can also be realized by the well-known technique described in the sixth embodiment. Further, in this embodiment, since the face of each participant is photographed largely by the camera 401, the movement of the lips of the participant can be detected more stably than in the above-described sixth embodiment. .

  The function of the motion detection unit 40 described above may be implemented inside the camera 401 or may be realized by the CPU 301 in the server 300. In the latter case, the HDD 303 stores a predetermined program for realizing the function in advance.

Video Selection Unit In the present embodiment, the video selection unit 39 automatically selects a partial video to be recorded based on the motion of the subject in the partial video detected by the motion detection unit 40. Specifically, the partial video in which the largest movement is detected is identified from the feature amount of the subject motion in each partial video, and this is automatically selected as the partial video to be recorded. Here, since this embodiment does not necessarily require the specifying unit 36, it is possible to select an appropriate partial video with a simpler configuration and processing than in the above-described sixth embodiment.

  The function of the video selection unit 39 described above is executed by the CPU 301 in the server 300. At this time, the HDD 303 stores a predetermined program for realizing the function in advance.

Others In the above-described sixth embodiment or the present embodiment, it is preferable that each camera 401 included in the camera array 400 includes an imaging region that is partially in common with other cameras. FIG. 36A is a diagram illustrating a screen of the video display terminal when each camera does not include a common shooting area. As shown in FIG. 36, when the participant A is standing and moving, the image selection unit 39 displays a partial image including a shooting area closest to the participant A (in the figure, a black bar 605). Automatically select the one shown). However, when the participant A moves to a place where none of the cameras 401 can shoot, a partial video in which no important subject is captured is selected. As described above, there arises a problem that a partial video image obtained by continuously tracking a moving subject cannot be recorded.

  Therefore, as shown in FIG. 36B, this problem can be solved if each camera is arranged so as to include a common photographing region. In the figure, the bar indicated by diagonal lines indicates a range in which two or more cameras 401 are overlapped. As shown in FIG. 6, when the camera array 400 is configured by fixing each camera 401 to the housing 402, each camera 401 may be fixed so that the shooting ranges partially overlap each other.

Embodiment 8
Note that the video recording system 100 according to the present invention can realize its functions by a PC. In this case, the functions can be realized by storing software that realizes the above-described units in a hard disk and appropriately executing a processing program.

Embodiment 9
In addition, the program can be stored in a recording medium such as a CD-ROM. As shown in FIG. 37, the function can be realized by mounting a CD-ROM 308 storing the program on a PC and appropriately executing the program. Needless to say, the recording medium for storing the program is not limited to the CD-ROM 308, and may be another medium such as a DVD-ROM.

  Each of the above embodiments is merely an example of the present invention, and the scope of rights of the present invention should not be limited or reduced as in the above embodiment. For example, in each embodiment, the wide-angle camera 200, the camera array 400, and the microphone array 500 have been described using a configuration example in which they are connected to a USB hub. However, these connection forms are limited to the above description. is not. For example, another interface such as a PCI bus, IEEE 1394, or Bluetooth may be used.

  Further, as the mirror 211 used in the wide-angle camera 200, a hyperboloid mirror and a curved mirror having a curvature in one direction have been described in the embodiment, but other forms such as a parabolic mirror and a conical mirror may be used. It doesn't matter.

  In the description of the first imaging unit 31, it has been described that the video data digitized by the wide-angle camera 200 is output. However, the wide-angle camera 200 may output an analog video signal. In this case, digital video data can be output by combining the wide-angle camera 200 and a video capture board that digitizes analog video signals. That is, the same operation as that of the first imaging unit 31 described in the above embodiment can be realized.

  In the description of the second imaging unit 32, it has been described that the partial video data digitized by each camera 401 constituting the camera array 400 is output. However, each of these cameras 401 outputs an analog video signal. It does not matter. In this case, by combining these cameras 401 with a video capture board that performs digitization processing on multi-channel analog video signals, digital partial video data can be output. That is, the same operation as that of the second imaging unit 32 described in the above embodiment can be realized.

  In the description of the sound acquisition unit 37, it has been described that the sound data digitized by each microphone 501 constituting the microphone array 500 is output. However, each of these microphones 501 outputs an analog sound signal. It doesn't matter. In this case, digital audio data can be output by combining these microphones 501 and an audio capture board that digitizes multi-channel analog audio signals. That is, the same operation as that of the sound acquisition unit 37 described in the above embodiment can be realized.

  Further, although the encoding unit 34 and the recording unit 35 have been described as being mounted on the same server 300, an encoding PC may be installed separately from the server 300. In this case, the encoded data is transferred from the encoding PC to the server 300 via the telecommunication line.

  In the description of the operation of the encoding unit 34, it has been described that the MPEG encoding process is performed by the CPU 301 in the server 300, but the configuration of the encoding unit 34 is not limited to this. For example, the video data may be converted into the MPEG format using a video capture board incorporating an MPEG encoding IC. The same applies to audio data.

  Further, in the description of the operation of the motion detection unit 40, it has been described that the “feature feature amount” indicates the magnitude of the movement of the subject, but may be different, for example, the shape of the movement locus of the subject.

It is a figure which shows the usage example of the video recording system which concerns on this invention. 1 is a diagram illustrating a configuration of a video recording system according to Embodiment 1. FIG. 2 is a diagram illustrating a configuration of a wide-angle camera 200 according to Embodiment 1. FIG. 2 is a diagram illustrating a structure of a wide-angle camera 200 according to Embodiment 1. FIG. It is a figure which shows the image | video image | photographed by the wide angle camera 200 shown by FIG. 3 is a diagram showing an example of a camera array 400 according to Embodiment 1. FIG. It is a figure which shows the structure of the server 300 which concerns on Embodiment 1. FIG. FIG. 3 is a block diagram showing an operation according to the first embodiment. FIG. 6 is a diagram for explaining the operation of a deforming unit 33 in the first embodiment. It is a figure explaining the principle in the deformation | transformation part. It is a figure explaining the coordinate conversion table used by the deformation | transformation part 33. FIG. It is a figure which shows the structure of the wide angle camera 200 which concerns on Embodiment 2. FIG. It is a figure which shows the image | video image | photographed with the wide angle camera 200 shown by FIG. It is a figure which shows the example which implement | achieves the operation | movement of the deformation | transformation part 33 in Embodiment 2 simultaneously with an image display. It is a figure which shows an example of the display screen of a video display terminal at the time of using the video delivery system which concerns on Embodiment 3. FIG. FIG. 10 is a block diagram showing an operation according to the third embodiment. FIG. 10 is a diagram illustrating an example of an operation of a specifying unit 36 according to Embodiment 3. FIG. 10 is a diagram illustrating an example of an operation of a specifying unit 36 according to Embodiment 3. It is a figure which shows an example of the display screen of the server 300 which concerns on Embodiment 3. FIG. It is a figure which shows an example of the display screen of a video display terminal at the time of using the video recording system which concerns on Embodiment 4. FIG. FIG. 10 is a diagram illustrating a configuration of a video recording system according to a fourth embodiment. It is a figure explaining the structure of the wide-angle camera 200 in Embodiment 4, and the microphone array 500. FIG. FIG. 10 is a block diagram showing an operation according to the fourth embodiment. It is a figure explaining the principle of operation of the sound source detection part 38 in Embodiment 4. FIG. It is a figure explaining the problem of the sound source detection part 38 in Embodiment 4. FIG. FIG. 10 is a diagram illustrating an arrangement example of microphones 501 according to Embodiment 4. FIG. 10 is a diagram illustrating another arrangement example of microphones 501 in the fourth embodiment. It is a figure explaining operation | movement of the sound source detection part 38 in Embodiment 4. FIG. It is a figure explaining operation | movement of the sound source detection part 38 in Embodiment 4. FIG. FIG. 10 is a diagram for explaining the operation of a video selection unit 39 in the fourth embodiment. It is a figure explaining the data which the video selection part 39 in Embodiment 4 outputs. It is a figure which shows the structure of the camera 401 in Embodiment 5. FIG. FIG. 10 is a block diagram showing an operation according to the fifth embodiment. FIG. 20 is a block diagram showing an operation according to the sixth embodiment. FIG. 20 is a block diagram showing an operation according to the seventh embodiment. It is a figure which shows the problem at the time of using the video delivery system which concerns on Embodiment 6 and 7. FIG. FIG. 38 is a diagram illustrating a configuration example according to the ninth embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Table 2 Participant 3 Cabinet 31 1st imaging part 32 2nd imaging part 33 Deformation part 34 Encoding part 35 Recording part 36 Identification part 37 Audio | voice acquisition part 38 Sound source detection part 39 Image | video selection part 40 Motion detection part 200 Wide-angle camera 211 Mirror 212 Lens 213 Diaphragm 214 Image sensor 215 Drive unit 216 Pre-processing circuit 217 Motor drive unit 300 Server 310 Bus 320 USB hub 330 Internet 350 Client PC
400 Camera array 401-1 to 401-4 Camera 500 Microphone array 501 Microphone 600 Display window

Claims (4)

Video acquisition means for acquiring video captured by an imaging means for capturing video;
Sound source detection means for acquiring sound input from a plurality of microphones for inputting sound;
Motion detection means for detecting the movement of the subject from the video acquired by the video acquisition means;
An utterance section is extracted based on the voice acquired by the sound source detection means, and the motion of each part of the subject detected by the motion detection means from the video acquired by the video acquisition means in the extracted utterance section. Video selection means for selecting a partial video corresponding to the subject in which the largest movement is detected in the video based on the feature amount ;
A video reproduction apparatus comprising: a display unit that displays the video acquired by the video acquisition unit and displays the partial video selected by the video selection unit. The imaging means includes
A mirror body having a curved surface of a predetermined shape, and an imaging device for imaging the video,
The video imaged by the imaging means is
The video playback apparatus according to claim 1, wherein the video playback apparatus includes at least an entire periphery overlooking a horizontal plane. A video playback method executed by a video playback device for playing back acquired video,
A video acquisition step of acquiring video captured by an imaging means for capturing video;
A sound source detection step for acquiring sounds input from a plurality of microphones for inputting sounds;
A motion detection step of detecting the movement of the subject from the video acquired by the video acquisition means;
An utterance section is extracted based on the voice acquired by the sound source detection step, and the motion of each part of the subject detected by the motion detection step is extracted from the video acquired by the video acquisition step in the extracted utterance section. A video selection step of selecting a partial video corresponding to the subject in which the largest movement is detected in the video based on the feature amount ;
A display step of displaying the video acquired by the video acquisition step and displaying the partial video selected by the video selection step. Computer
Video acquisition means for acquiring video captured by an imaging means for capturing video;
Sound source detection means for acquiring sound input from a plurality of microphones for inputting sound;
Motion detection means for detecting the movement of the subject from the video acquired by the video acquisition means;
An utterance section is extracted based on the voice acquired by the sound source detection means, and the motion of each part of the subject detected by the motion detection means from the video acquired by the video acquisition means in the extracted utterance section. Video selection means for selecting a partial video corresponding to the subject in which the largest movement is detected in the video based on the feature amount ;
A video reproduction program for causing the display device to display the video acquired by the video acquisition unit and to function as a display unit for displaying the partial video selected by the video selection unit on the display device.

Images