JP2002524936A - Automatic audio and image recording system - Google Patents

Abstract

(57) Abstract The present invention particularly relates to an automatic recording and recording system for a video conference, the system comprising: a recording device and control means (20) of the recording device (10); Scene analysis means (40) for automatically framing the scene being played. The present invention is provided by scene analysis (40), which selects (30) one or a group from the people in the scene that is being filmed at the direction of the participant and analyzes the scene with respect to the selected person or group. An intelligent interface is provided for automatic framing (30) based on the information.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

TECHNICAL FIELD OF THE INVENTION

The present invention relates in particular to video conferencing recording and automatic recording systems. The need for such a system is becoming more pronounced, especially in the field of video conferencing, but also relates to other situations, for example, recording images of multiple people. The present invention is a system in which audio and images are controlled by automated devices, rather than by one or more people (photographers and sound engineers in the television field).

[0002]

[Prior art]

In any event, in the case of video conferencing, participants may make use of certain systems with remote controls for controlling the recording by the cameras, but these systems have disadvantages as seen below. In particular, the teleconferencing system comprises audio and video recording means, which have equipment (cameras and microphones) that cannot change direction or cannot control direction using a remote controller.

[0003] Conventional video conferencing systems use various cameras that are remotely controllable and can be remotely adjusted via a serial communication link. The remote controller continuously scans the camera's attitude and orientation,
In addition, the zoom of the camera can be changed continuously. Orienting the camera in the direction occupied by a person or group of people is possible, but not easy. The direction of the space (six directions for two cameras) can be stored by the cameras. By activating buttons on the remote control or operating the serial port, the camera can be pointed in either of these directions. The advantage of this function is that you can directly reach one direction of space without working by combining continuous buttons (posture, orientation). Thus, if the pre-recorded position of the space corresponds to a seat around the table, the user of the remote controller need only move from one person to the next in turn.

[0004] Some systems are based on acoustic location, whereby the orientation of the camera is controlled based on a sound analysis of the scene. The audio analysis of the scene is performed by a plurality of microphones, which make it possible to determine the direction of the sound or sound source. Identify the direction of the audio source,
Audio sources can be selected one by one and tracked dynamically. Picture-tel's Lime Light feature, which manufactures and sells video conferencing systems, is based on sound-based positioning, which allows it to detect and track sound sources and dynamically orient the camera. .

[0005] Other systems are based on the visual location of people. This visual position setting is based on analysis by processing of image output of the camera. If the camera's field of view is wide enough and the system is sophisticated enough, it will be able to detect everyone there. Unlike sound location, visual location benefits the consistency of faces in the image and the absence of reflections (the absence of mirrors in videoconferencing and normal recording rooms). Several techniques can be used separately or together to detect people: motion detection, skin color detection, elliptical shape detection, whole face detection (eg using neural networks), For example, detection of faces with eyes, nose, etc.

[0006] There are many face detection systems, some combining automatic recording device capabilities. Examples include systems described in the following literature: P1-M. Collobert, R. Reraud, G. Le Tourneur, O. Bernier, JE Viall.
et, Y. Mahieux, D. Collobert, LISTEN: a System for Locating and Tracking
^{Individual Speakers, 2 nd Int. Conf} . On Automatic Face and Gesture Recog
nition, Killington, USA, October 1996, pp 283-288, P2-Hunke, Locating and Tracking human faces with neural network, Te
chnical Report CMU-CS-94-155, 1994, P3-Yang, Wu, Waibel, Focus of attention in video conferencing, Tech
nical Report CMU-CS-96-150, 1996.

[0007]

[Problems to be solved by the invention]

However, the majority of all systems described herein have drawbacks, or in any case are not sufficient for the needs identified below. In particular, a system that has the capability to scan attitude, orientation and focal length using a remote controller has the following disadvantages: To obtain the appropriate frame, the user has to use multiple buttons on the remote control (posture, orientation and focal length). ) Must be pressed to get the desired result. This functionality is impractical, slow, prone to operational errors, and therefore rarely used. This is impractical when it is desired to periodically or continuously match the camera frame to a group of people who are changing positions or a participant moving naturally. Similarly, it is impractical to change people regularly.

The ability to access a pre-recorded direction of space using a remote controller has the following disadvantages: The first disadvantage is that the position must be pre-recorded. Therefore, the position cannot be changed quickly and continuously. A second drawback is in assuming that people actually occupy pre-recorded positions and do not move. In fact, even if the chair is fixed to the floor, people move,
If the frame is barely centered or fixed to a person,
It turns out that it goes out of the frame. This drawback applies in the case of videoconferencing systems, where people unknowingly fall outside the frames defined by the pre-recorded directions of the space.

A second disadvantage is that the ability to orient the camera in this spatial direction is not always combined with a specific focal length. The user of the remote controller must operate the adjustment button of the remote controller to adjust the size of the frame to the position actually occupied by a certain person. This is because if the person tends to go out of the frame, the focal length can be reduced and the person returned into the frame without changing the direction of the space pointed to by the camera. However, the frame is no longer fixed to that person, and there may be more than one person in the frame. It is possible to combine and store the focal lengths with each of the stored directions of space, but this, on the other hand, allows the user to pre-determine the focal length as deemed appropriate,
On the other hand, you have to remember this. Furthermore, this stored focal length is fixed, and therefore non-adaptive, and can only be changed slightly in direction.

The ability to access a predetermined direction of space may be suitable for certain stable situations (remote monitoring), but cannot be adapted to special situations. The camera faces a certain direction in the space, but has no idea what is in the space, that is, whether there is a person in the space or the sky. Another second disadvantage is that the number of directions in the space which can be stored by the camera and is easily accessible to the remote controller is limited to six. This disadvantage can generally be overcome by storing these directions in a computer and using a remote controller with a large number of buttons.

[0011] Sound localization has the following disadvantages: due to the presence of fake sound sources (noise in chairs, doors, appliances, etc.) overlapping with the spoken language of the people, and due to multiple reflections of the sound sources from walls. Position setting by sound is often inaccurate. Furthermore, the acoustic activity of words is inherently intermittent (when people stop speaking and listen). Finally, the position setting by sound is easily affected by the amplitude (magnitude) of the sound source.

[0012] Visual positioning has the following disadvantages: The main disadvantages of visual positioning are the complexity and speed and strength of the algorithm (robu).
stness). Nevertheless, a plurality of systems, such as the system developed by the applicant or the documents cited above by the applicant, can be operated on either a workstation or a personal computer (PC). The ability to automatically record images of a group of people has been implemented by the applicant and is complex, but indeed extremely useful. The frame always fits the number and location of participants in the video conference. However, in practice, as described above, even if it is more elaborate and more pleasing than everything else, it requires that some of the participants be automatically framed,
It's not enough because it was obviously necessary to be able to return to the whole frame quite easily.

[0013]

[Means for Solving the Problems]

To this end, the present invention, when instructed by the participant, selects one (or a group of people) from the people in the scene being filmed and determines the selected person (or group of people). We propose an intelligent interface that performs automatic framing based on information provided by scene analysis. The present invention therefore relates to an automatic recording and recording system, in particular for video conferencing, comprising a recording device and control means for the recording device, and a scene for operating the control means to automatically frame the scene being shot. It has analysis means. The system of the present invention provides a means for selecting a person or group of people from the people in the scene being filmed, and information provided by the means for analyzing the scene about the selected person or group of people. A means for automatic framing based on the

The present invention particularly relates to an automatic recording and recording system for video conferencing, the system comprising a recording device and control means for the recording device, scene analysis means for providing a position signal to the control means, and people in the scene being photographed. Has a physical interface with a remote control, said remote control selecting any one or group of people in the scene, Automatically framing around a person or group, or selecting all people so that the entire framing of the scene can be performed, and the framing means has a logical interface, Interface derived from person selected by remote control and scene analysis It is characterized mainly in that communication with the position information is established, and the position information of this person or group regarding the scene being photographed can be given to the control means.

[0015]

BEST MODE FOR CARRYING OUT THE INVENTION

In one embodiment of the invention, the remote control is a general purpose remote controller operating a device capable of transmitting control signals to the logical interface. The signal transmitted by the remote controller may be an infrared signal or an electromagnetic signal. A control signal from the remote controller may be received and retransmitted at the transceiver. The control signal from the remote controller may be received and retransmitted by a voice recognition device or a body motion recognition device. The remote control may consist of a remote controller of the image analysis camera, from which the control signal is received and retransmitted to the logical interface.

In another embodiment, the remote control is a general purpose remote controller,
Control signals from the remote controller are received by the analysis camera and retransmitted. In another embodiment, the remote control has a graphic interface. In this case, the remote control further includes a screen on which scenes and various selectable areas are displayed. The remote control has a computer input / output peripheral that selects the identified area.

In one aspect of the operation, the scene analysis means receives the local analysis signal (A) and selects one person or a group of people in the scene being locally photographed by the selection means. It is possible, and the automatic framing means uses information about a scene that is locally photographed. In another aspect of the operation, the analyzing means receives a signal (A ') for scene analysis or corresponding to scene analysis from a remote system, and a person or one person in the scene being shot from a distance by the selecting means. A group of people can be selected, automatic framing means can control the framing of a scene being shot from a distance, and a control signal is transmitted to the remote system.

[0018]

【Example】

Other advantages of the present invention will become apparent from a reading of the description given in conjunction with the accompanying drawings below, by way of indication and not limitation. FIG. 1 schematically shows an automatic recording and recording system, in which an audiovisual resource 10 is provided for capturing and recording the sound of a scene 50.

In the case of a videoconferencing system, the system is made up of one or more people who want to communicate with people in remote locations, and these people are referred to as participants P1
Called Pn. The audiovisual resource 10 includes a recording device and a recording device. The recording device is, for example, a series of microphones located near the participant. The recording device is composed of one or more cameras that shoot scenes.

The audiovisual resource 10 is controlled by a normal control device 20, and the control device 20 can output a control signal to the recording device / recording device (recorder) 10 according to information received as an input at the interface 30. . This is described in detail below. In the present invention, the information received as input is provided by the interface 30 based on selections made by the scene analyzer 40 and participants. The scene analyzer may be a video recorder or an audio analyzer or a screen analyzer or an audio screen analyzer combined with a sound recorder.

In the embodiment shown, the device is an existing screen analyzer. For this purpose, a fixed analytical camera 60 (the camera may be mobile) is used, so that it is possible to provide the necessary signals used to make an analysis of the observed visual scene. Become. The resulting scene analyzer comprises for this purpose a camera 60 and means 40 for processing the signal A provided by the camera. These means, for example,
It consists of a microcomputer or workstation equipped with an existing special scene analysis program. More precisely, in the illustrated embodiment, the faces of the people present in the field of view are detected by a neural network, and the program implements an algorithm for tracking the detected faces. Other known methods may be used. In another embodiment, the scene analysis device 40 may be used with a movable camera.

In a third embodiment, a scene analysis device may be used with or consist of a plurality of fixed or movable cameras. The various recorders 10 are controlled by a controller 20 that receives control signals from the interface 30 of the present invention. The embodiment illustrated in the schematic diagram of FIG. 2, which has been implemented in practice, has a device 20 for controlling a motorized camera 11 for recording images and an acoustic antenna 12 for recording sounds. A set of people and a single person's video recordings are detailed below. This is consistent with Applicants' actual working example.

The same method can be used to record and record a group of people.
A group is a group that subdivides the whole population. In the actually implemented preferred embodiment, the scene analysis is visual. That is, the location of the people is defined, but whether they are speaking is ignored. In other embodiments that require audiovisual analysis (and thus include sound), the recording device is selected based on the audiovisual information.

When a user of the system intends to record all participants P 1 to Pn via the interface 30, the control device 20 controls the camera 11 according to a technical rule that recording is performed as long as the restrictions of the camera 11 allow. So that everyone in the analysis area is framed. If the user of the system intends to record a specific person via the interface 30, the control device 20 will follow the rules of recording,
For example, the person may be centered laterally, for example, with his or her eyes approximately one-third above the image. In one embodiment, if one or more people are visible on either side of the selected person, the recording will keep this person away from others in the image, as restrictions on the camera and recording rules allow. To do.

If the user wishes to record from all participants via the interface 30, the device 20 controls the recording and records sound fields from multiple participants. This sound field can be obtained in various ways: 1) from a large field formed by the acoustic antenna 12; 2) output by various recording devices (microphones or acoustic antennas) selected based on an analysis of the scene. By accumulating the sound field.

If the user wants to record from one person via interface 30,
Device 20 controls the recording and records the sound field from various people. This sound field can be obtained in various ways: 1) from the narrow, steerable field of the acoustic antenna 12; This is the embodiment described above; 2) From one or more recording devices selected from visual or audiovisual information.

The interface 30 allows the system user to record and record according to their requirements (wide-angle shooting for all participants, narrow-range shooting for a particular person). By sending the commands output by the interface, the recording device and the recording device can be oriented as a function of the audiovisual scene analyzed by the scene analysis device. For this purpose, the interface has a logical interface 31 and a physical interface 32.

The physical interface 32 can be configured according to various embodiments described below with reference to FIGS. Logical interface 31 includes scene analysis signal 40 according to the preferred embodiment.
Consists of programs loaded into the system to process The logical interface 31 retrieves location information about the people in the scene, which is the output of the scene analysis process, and establishes communication between this location information and selection information provided by the participant via the physical interface. The logical interface 31 interprets (ie, decodes) the information received from the unit 40 and provides position control signals, which are interpreted by the controller 20 to provide the desired framing of the selected person or group. . Several embodiments of the physical interface 32 are shown below.

The first embodiment comprises a graphic interface 32 A attached to a microprocessor or workstation P as shown in FIG. Using the mouse 320 and clicking on the window labeled "All", represented by E, the user can select to record and record all the people in the scene. Using the mouse and clicking on the window with the number of the desired person P1-pn or group of people, the user can choose to record and record one person in the scene.

In this graphic interface 32 A, the assignment of numbers to people is replaced by an image 321 of people obtained by the analysis system. This image may be obtained at a moment determined by the user of the system or may be updated automatically during the meeting. Graphic interface 32A with images of people 321 is more ergonomic to the user because the interface can display the image that the user selects. It is also possible to provide a video conference screen with an overlay of images of people attending the video conference in a reduced format.

In another embodiment, mouse 320 may be replaced with a touch screen and / or voice recognition device R. Another embodiment implemented for logical interface 32 is illustrated in FIG. In this embodiment, the remote controller 32 of the camera 60 that analyzes a visual scene
B is used to allow a user of the system to send control signals to the camera 60.
This remote control was adapted and used for ease of use and speed. The infrared remote controller 32B is in communication with the analysis camera 60 (command I
NST). The remote control of the analysis camera has a plurality of buttons, including a button corresponding to a position memory and a “home” button H corresponding to a standby position of the camera.

The location memory is not used to select the direction of space, only the fact that the buttons are activated. The position in the position memory is pre-initialized by the system when the camera is in the standby position. In one embodiment, the position of the analysis camera 60 is not affected at all by the activation of the positions 1 to 6 or the “home” button H, since the analysis camera is fixed. To this end, the user presses the “Home” button H,
, 40, 30 and 20 activate the recording of all the people in the scene. In addition, the user presses one of the buttons 1 to 6 corresponding to the memory containing the location, via the devices 60, 40, 30 and 20 to record the corresponding person (in this version up to 6). Operate.

The remote controller may or may not use an analysis camera as a transmission repeater, and may use another remote controller. For example, a remote controller with general purpose programming can be used instead of the remote control of the analytical camera to allow the user to select recording and recording. This embodiment is not shown because it corresponds to the schematic diagram of FIG. 4, except that the remote controller 32B in this case is a general purpose remote controller.

At the stage of manufacturing hardware to implement the application of the present invention, some codes (location memory, “home”, etc.) of the analysis camera must be pre-recorded on a general-purpose remote controller. When the remote controller 32B is actually used, the infrared signal transmitted by the remote controller is decoded and analyzed by the logical interface 31. Using a universal remote control, on the one hand, the analysis camera 60 which is not available
(The general purpose remote control does not learn the corresponding code), on the other hand, has assigned buttons, "+" and "-" buttons, to better respond to user requests, It is possible to move from one person to the next in sequence, as he or she is accustomed to television channels.

Reference can now be made to FIG. 5, which corresponds to another embodiment of the present invention. A commercially available transceiver 70 may be used because it does not rely on an analytical camera as a repeater between the remote controller and the interface 31 and to provide a wide range of control pallets to the user. The transmitting / receiving device 70 receives the infrared signal I output from the remote controller 32B.
It takes in the NST and returns a code to the logical interface 31 via, for example, an RS232 communication port connected to the interface 30.

FIG. 6 shows an embodiment in which the logical interface 32 has a voice remote control 32 B combined with an existing voice recognition device 80. FIG. 7 shows an embodiment in which the logical interface 32 has a body movement remote control 32 B combined with an existing body movement recognition device 90. Said embodiment of the interface allows the user to select the recordings and recordings obtained on the recorder physically present in the room where the user is.
This is represented by the signal A (local, in-situ) used in the scene analysis of FIG.

In another embodiment, the interfaces 31, 32 allow the recording and recording equipment physically present in a room (without a user), for example, the room in which he or she is participating in a video conference. It becomes possible to control. Thus, a user participating in a video conference will select and obtain the desired recording and recording. For this purpose, a signal A ′ (remote) for scene analysis or a signal corresponding to the analysis (signal already analyzed by a remote system to which no video signal is provided) is applied to the input of the analyzer 40. In this embodiment, the signal C '(and the signals required to configure the graphic interface) transmitted by the far-infrared remote controller or the graphic interface is transmitted together with the image, audio, and other signals of the video conference. In this embodiment, inconsistencies in recorder control between a local (in-situ) room and a remote room may occur, but this must be handled well.

[Brief description of the drawings]

FIG. 1 represents a schematic diagram of the invention.

FIG. 2 represents a more detailed diagram of the invention.

FIG. 3 illustrates a particular embodiment of a physical interface.

FIG. 4 illustrates another embodiment of a physical interface.

FIG. 5 illustrates another embodiment of a physical interface.

FIG. 6 illustrates another embodiment of a physical interface.

FIG. 7 illustrates another embodiment of a physical interface.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 10 Recording device and recording device 20 Control means 30 Selection means 40 Scene analysis means 31 Logical interface 32 Physical interface 32A, 32B Remote controller 60 Image analysis camera 70 Transceiver

──────────────────────────────────────────────────続 き Continuing on the front page (72) Inventor Corobert, Michel, France 22560 Purmour Bodu Gueradre Vra 4 (72) Inventor, Bernier, Olivier, France 22300 Lanyon Lou Querveno 50a F-term (reference) 5C054 AA02 AA05 CA04 CA08 CC02 CF05 CG05 EA01 EA07 FC12 FF02 FF03 FF07 GB01 HA25 5C064 AA02 AC04 AC06 AC08 AC12 AC18

Claims (11)

[Claims] 1. A sound and image automatic recording system, a control unit (20) of a recording device and a recording device (10), a scene analysis unit (40) for supplying a position signal to the control unit (20), Means (30) for selecting one or a group from the people in the scene, said selection means having a physical interface (32) with remote controls (32A, 32B), Selecting any one or a group of people in the scene to automatically frame around this person or group, or selecting all people to frame the entire scene; The framing means has a logical interface (31), which is a remote control (32A or 3A). Establishing communication between the person selected in B) and the position information resulting from the scene analysis and providing the position information of this person or group with respect to the scene being photographed to the control means (20). And the system. 2. The system according to claim 1, wherein the remote control is a general-purpose remote controller operating a device capable of transmitting control signals to a logical interface. 3. The system according to claim 2, wherein the control signal from the remote controller (32B) is received and retransmitted at a transceiver (70). 4. The system according to claim 2, wherein the control signal from the remote controller (32B) is received and retransmitted by a voice recognition device (80) or a body motion recognition device (90). 5. A remote control (32B) comprising an image analysis camera (60).
The system according to claim 1, characterized in that the control signal from the remote controller (32) is received by an analysis camera (60) and retransmitted to a logical interface (31). . 6. A remote control (32B) is a general-purpose remote controller, and a control signal from the remote controller (32B) is transmitted to an analysis camera (32B).
The system of claim 1, wherein the system is received and retransmitted by (60). 7. The remote control is a graphic interface (3).
The system of claim 1, comprising 2A). 8. The system according to claim 7, wherein the remote control further comprises a screen (321) in which scenes and various selectable areas are displayed. 9. The system according to claim 7, wherein the remote control comprises a computer input / output peripheral (320) for selecting the identified area. 10. The scene analysis means receives the local analysis signal (A), and it is possible to select one person or a group of people in a scene which is locally photographed by the selection means. 10. The system according to claim 1, wherein the automatic framing means uses information about the scene being locally photographed. 11. The analysis means receives a signal (A ') for scene analysis or corresponding to scene analysis from a remote means, and a person or a group in a scene which is remotely shot by the selection means. The framing of a scene being remotely shot by the automatic framing means can be controlled, and the control signal is transmitted to a remote system.
The system according to any one of claims 1 to 11.