JP2015177490A - Image/sound processing system, information processing apparatus, image/sound processing method, and image/sound processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately generate a profile by using images and sounds.SOLUTION: An image/sound processing apparatus includes: detection means for detecting a region of a detection target from an image signal imaged on a predetermined position; sound signal separation means for separating a sound signal in each detection target corresponding to the detection target region detected by the detection means from sound signals collected on the predetermined position; sample data selection means for selecting sample data for generating a profile of an image or a sound from the image signal or the sound signal on the basis of the detection target region and an arriving direction of the sound signal in each detection target; and profile generation means for generating a profile of the image or the sound by using the sample data selected by the sample data selection means.

Description

  The present application relates to a video / audio processing system, an information processing apparatus, a video / audio processing method, and a video / audio processing program.

  For example, a technique for generating minutes by converting voice information recorded at a meeting or the like into text information is known. In such a technique, there is a need for a technique for memorizing who speaks during a conference, and for identifying a speaker even when a plurality of conference participants speak at the same time, and recognizing speech with high accuracy. . In order to recognize speech accurately, a speech profile that stores the characteristics of individual users in advance is created, and speech information is calibrated and converted into text information using the created speech profile as an index. It has been broken.

  In addition, a technique is known in which a meeting is photographed with a camera or the like, and a speaker and voice are associated with each other using a photographed video signal. For example, a method is known in which a video object is separated from an input video signal shot by a camera or the like, a sound object is separated from an input audio signal, a correlation between the video object and the sound object is obtained, and the video object and the sound object are associated with each other. (For example, refer to Patent Document 1).

  However, in the method of Patent Document 1, a process of extracting a video object from an input video signal and a process of extracting an acoustic object from an input audio signal are performed independently. Therefore, for example, when noise or occlusion occurs in a video signal or an audio signal, an appropriate correlation between the two cannot be obtained, and the object detection accuracy decreases.

  In one aspect, an object of the present invention is to generate a profile with high accuracy using video and audio.

  In one aspect, detection means for detecting a detection target region from a video signal imaged at a predetermined position, and detection target region detected by the detection means from a sound signal collected at the predetermined position From the video signal or the audio signal, based on the audio signal separation means for separating the audio signal for each detection target corresponding to, the region of the detection target, and the arrival direction of the audio signal for each detection target, Sample data selection means for selecting sample data for generating a video or audio profile, and profile generation means for generating a video or audio profile using the sample data selected by the sample data selection means Have.

  A profile can be generated with high accuracy using video and audio.

It is a figure which shows an example of schematic structure of a video / audio processing system. It is a figure which shows an example of a function structure of a video / audio processor. It is a figure which shows an example of the hardware constitutions of an audio-video processing apparatus. It is a flowchart which shows the flow of a profile production | generation process. It is a flowchart which shows the flow of a matching process with a video object. It is a flowchart which shows the flow of a sample data selection process. It is a figure for demonstrating matching with a video object. It is a figure for demonstrating the correlation of a video object area | region and an estimation area | region.

  Hereinafter, embodiments will be described in detail.

<Video / audio processing system: schematic configuration>
FIG. 1 is a diagram illustrating an example of a schematic configuration of a video / audio processing system. A video / audio processing system 1 illustrated in FIG. 1 includes a conference recording device 10 and a video / audio processing device 20 which is an example of an information processing device. The video / audio processing system 1 shown in FIG. 1 acquires video and audio during a conference, for example, and generates a profile with high accuracy using the acquired information.

  The conference recording device 10 is installed at a predetermined position such as a conference room, for example, and records surrounding video signals and audio signals (sound source signals). It is assumed that the video signal and the audio signal are associated (synchronized) based on time information. The conference recording apparatus 10 includes, for example, a wide-angle camera and a microphone array (hereinafter referred to as “microphone array”). The conference recording device 10 shoots (captures) a conference participant in a conference room or the like using, for example, a wide-angle camera, generates a video signal, and records (collects) voices spoken by the conference participant using, for example, a microphone array. Generate multi-channel audio signals.

  The microphone array of the conference recording apparatus 10 preferably has a plurality of microphones arranged on the radiation at predetermined intervals, for example, in order to be able to specify the sound source position. The audio signals recorded in the microphones are respectively It should be stored independently. Note that the camera of the conference recording apparatus 10 is not limited to a wide-angle camera. Further, the microphone of the conference recording apparatus 10 is not limited to the microphone array.

  The video / audio processing device 20 detects an area of an object (for example, a person such as a conference participant) as a detection target from the video signal obtained from the conference recording apparatus 10, and the object (for example, a person) from the multi-channel audio signal. Each audio signal is separated, and sample data is collected from the recorded video signal or multi-channel audio signal. Further, the video / audio processing device 20 generates a video or audio profile using the collected sample data of the video signal or sample data of the audio signal.

  The video / audio processing device 20 is realized by, for example, a general PC (Personal Computer), a server, and the like, and characteristic functions are mainly provided by software processing. The video / audio processing device 20 includes, for example, an audio processing device 21, a video processing device 22, a CPU (Central Processing Unit) 23, a main storage device 24, and an auxiliary storage device 25.

  The audio processing device 21 inputs an audio signal recorded (collected) by a microphone array or the like of the conference recording device 10 via a signal line. The audio input interface from the microphone array or the like can use a USB (Universal Serial Bus) interface or the like according to the connection specification with the mixer or the like, but is not limited to this. .

  Further, the input audio signal may be a plurality of analog signals. In that case, the audio processing device 21 converts a plurality of input analog signals into digital data by A / D conversion, and stores the converted data in the auxiliary storage device 25.

  The video processing device 22 inputs a video signal photographed (captured) by a wide-angle camera or the like of the conference recording device 10 via a signal line, and stores the input video signal in the auxiliary storage device 25. Note that when the input video signal is an analog signal, the video processing device 22 may convert the analog signal into digital data by A / D conversion.

  The CPU 23 executes various processes in the video / audio processing apparatus 20. The main storage device 24 is a high-speed storage device, and when the video / audio processing device 20 is turned on, all or a part of the software code stored in the auxiliary storage device 25 is copied and used for the control of the CPU 23. It is done.

  The auxiliary storage device 25 is a hard disk, a flash memory, an optical disk, or the like. The auxiliary storage device 25 stores software necessary for this embodiment. The main storage device 24 and the auxiliary storage device 25 may be configured as one storage device.

  The configuration of the video / audio processing system 1 described above is not limited to the example of FIG. 1. For example, the video / audio processing device 20 may be incorporated in the conference recording device 10. The conference recording apparatus 10 and the audio / video processing apparatus 20 may be connected by a communication network represented by a LAN (Local Area Network), the Internet, or the like. The communication network may be wired or wireless, or a combination thereof.

  In the present embodiment, for example, an audio profile and a video profile are automatically generated with high accuracy in order to more accurately acquire the voice and person video of a conference participant. Although the voice of a specific person can be extracted by using the microphone array, it is not known who the individual voice signals are by simply separating the specific voice signals.

  For example, there is no problem if the speaker is always in a specific position during the conference, but for example, a person may stand up and go in front of the whiteboard, or a person may enter and move between the seats. In such a case, the voice signal of the speaker can be separated by the conventional voice separation process, but it cannot be determined which voice signal is who.

  In this embodiment, for example, an audio profile is generated by using information from a video signal in a complementary manner with respect to a separated audio signal. In this embodiment, it is also possible to generate a video profile using information from an audio signal in a complementary manner with respect to a video signal.

<Video / audio processor>
FIG. 2 is a diagram illustrating an example of a functional configuration of the video / audio processing apparatus. As shown in FIG. 2, the video / audio processing apparatus 20 includes an audio signal separation unit 31, an object detection unit 32 as an example of a detection unit, and a profile acquisition unit 33.

  The profile acquisition unit 33 includes an initial image profile generation unit 40, an arrival direction detection unit 41, a video object association unit 42 as an example of an association unit, a sample data selection unit 43, and a profile generation unit 44. .

  The audio signal separation means 31 performs a process of separating the multi-channel audio signals input (sound collection) by the microphone array of the conference recording apparatus 10 into audio signals for each object (for example, a person) as an example of a detection target. Do. As a method for separating the audio signal for each object, for example, a technique such as blind source separation or independent component analysis can be used.

  In the blind signal source separation, for example, a plurality of measurement value sequences obtained by mixing a plurality of unknown signal sequences in an unknown linear mixed system are used to separate the respective sound signals. Independent component analysis is one of the multivariate analysis methods, and analyzes an input signal based on the criterion of independence and separates it into respective audio signals. Note that the separation method is not limited to this.

  The object detection unit 32 extracts an object area as a detection target from, for example, a video signal. The object area is, for example, a face area of a conference participant or the like, but is not limited thereto, and may be a preset object or the like.

  The object detection means 32 can extract an object region by performing image matching using, for example, an initial image profile of one or more image samples. The object region described above can be extracted by extracting features of a face (or an object) by image analysis or the like, for example.

  The profile acquisition means 33 collects sample data from the recorded video or audio based on the area of the object and the arrival direction of the audio signal for each object, and uses the video or audio of the collected sample data, A video profile or an audio profile is generated. Note that the audio profile is, for example, an acoustic model (for example, a collection of sound waveforms or a feature amount of sound), and the video profile is, for example, for detecting each individual (for example, a conference participant). Although it is identification information etc., it is not limited to this.

  The initial image profile generation means 40 acquires an image feature amount (for example, a feature amount of a face image) for each individual from the video signal obtained from the conference recording apparatus 10, and generates the acquired information as a profile of the initial image.

  For example, Haar-Like or the like can be used as the facial image feature acquisition method for the initial image profile generation means 40, but the method is not limited to this. The Haar-like feature amount is a scalar amount obtained as a difference value of the average brightness of the rectangular area, for example, and the value represents the intensity of the brightness gradient.

  The arrival direction detection unit 41 detects, for example, the arrival direction (sound source direction) of the sound at a predetermined position at a predetermined position from the audio signal for each object (for example, a person) separated by the audio signal separation unit 31.

  The arrival direction detection means 41 detects the arrival direction of sound by using the difference in time until the sound reaches each microphone in the microphone array, for example. The arrival direction detection means 41 can detect the arrival direction by, for example, calculating the phase difference of the audio signal, but is not limited to this. As an example of the calculation method, the contents described in JP 2011-71683 A can be used.

  The video object association means 42 estimates which object (detection target) in the video corresponds to the audio signal whose arrival direction (sound source direction) is detected by the arrival direction detection means 41, and based on the estimation result, A video object (for example, an object in the video) and an audio object (for example, an audio signal or a sound source direction) are associated with each other.

  The video object association unit 42 considers the video shot (captured) by the wide-angle camera of the conference recording apparatus 10 as, for example, a video obtained by projecting a three-dimensional object onto the projection plane and projects the arrival direction of the audio signal onto the projection plane. Thus, the estimated position of the object is narrowed down.

  In addition, the video object association unit 42 performs pattern matching on the object (video object) in the video existing at the estimated position using the initial image profile generated by the initial image profile generation unit 40, and the object ( For example, a specific person is detected.

  The sample data selection means 43 selects sample data (sample video signal, sample audio signal, etc.) for generating a profile from the video signal and audio signal obtained from the conference recording apparatus 10.

  The sample data selection means 43 collects sample data having a high correlation with the object region, for example, using at least one of the position and direction of the audio signal and the start or end time of the audio as the sample data of the audio signal. To do. Further, the sample data selection means 43 collects a sample having a high correlation with the audio signal by using at least one of the position and direction of the video object and the start or end time of the operation as the sample data of the video signal, To do.

  For example, if the same speaker is in a meeting, the position of the video object and the utterance timing should be correlated with the position of the audio signal and the utterance timing. Therefore, the sample data selection unit 43 can collect good sample sound and images simultaneously by improving the detection accuracy using the above-described conditions.

  For example, when one or more objects are detected, the sample data selection unit 43 determines whether to extract as sample data according to the matching degree (also referred to as detection reliability or similarity) of the face image.

  For example, suppose that A and B are in a certain sound source direction. In this case, the sample data selection means 43 uses the image feature quantity stored in the initial image profiles of Mr. A and Mr. B, the image feature quantity of Mr. A extracted as a video object, and the image feature quantity of Mr. B. Match and match the higher similarity as sample data.

  In the present embodiment, when the image features of both Mr. A and Mr. B are not similar (for example, when the similarity is equal to or less than a certain value), the sample data may not be extracted.

  The profile generation unit 44 uses the sample data obtained by the sample data selection unit 43 to generate a video or audio profile in units of individuals (for example, for each conference participant) or in units of multiple people.

  In the case of generating a voice profile, it is possible to accumulate speech speech of a target individual (person), generate an acoustic model based on the accumulated voice, and generate a language model. The language model plays a role of eliminating ambiguity caused by speech processing and realizes high recognition accuracy. Usually, learning of a language model requires large-scale learning data that matches a domain to be recognized and an utterance style using digitized text.

  However, for example, “Language Model Learning Information Processing Society Research Report from Bayesian Reasoning, SLP, Spoken Language Information Processing 2010-SLP-82 (16), 1-6, 2010-07-15, http: // /Www.phontron.com/paper/neubig10slp82.pdf "It is also possible to perform language model learning by adopting a method capable of learning a language model and a word dictionary using continuous speech data and an acoustic model. is there.

<Hardware configuration>
Next, a hardware configuration example of the video / audio processing device 20 according to the present embodiment described above will be described with reference to the drawings. FIG. 3 is a diagram illustrating an example of a hardware configuration.

  As shown in FIG. 3, the video / audio processing device 20, which is an example of a computer, includes a CPU 50, a RAM (Random Access Memory) 51, a ROM (Read Only Memory) 52, an HDD 53, an I / F unit 54, An LCD (Liquid Crystal Display) 55 and an operation unit 56 are provided.

  The CPU 50, RAM 51, ROM 52, HDD 53, and I / F unit 54 are connected via a bus 57. In addition, an LCD 55 and an operation unit 56 are connected to the I / F unit 54.

  The CPU 50 is a calculation means and controls the operation of the entire video / audio processing apparatus 20. The RAM 51 is a volatile storage medium capable of reading and writing information at high speed, and is used as a work area when the CPU 50 processes information.

  The ROM 52 is a readable non-volatile storage medium, and stores programs such as firmware. The HDD 53 is a nonvolatile storage medium that can read and write information, and stores an OS (Operating System), various control programs, application programs, and the like.

  The I / F unit 54 is connected to various hardware and networks. The LCD 55 is a display unit for displaying, for example, each screen for executing each process in the present embodiment. The operation unit 56 is, for example, a keyboard, a mouse, various hard buttons, a touch panel, and the like, and is a user interface for receiving information input from the user.

  In the above-described computer hardware configuration, a program stored in a storage medium such as the ROM 52, the HDD 53, or an optical disk is read into the RAM 51, and the CPU 50 performs an operation with the read program, thereby configuring the software control unit. Is done. With the combination of the software control unit configured in this way and hardware, it is possible to realize the functions of the devices constituting the system according to the present embodiment.

<Profile generation process>
FIG. 4 is a flowchart showing the flow of profile generation processing. As shown in FIG. 4, the video / audio processing apparatus 20 acquires at least one face image of a user whose profile is to be created by the initial image profile generation unit 40 and calculates the image feature amount of each individual (person). Thus, an initial image profile is generated (S10). Here, as the image feature amount, for example, the Haar-Like described above can be used, but the image feature amount is not limited thereto.

  In the process of S10, for example, a face image may be taken with the wide-angle camera of the conference recording apparatus 10. As a method of detecting a human face area from an image, a method of scanning the inside of the image sequentially from the end (Window scan) and matching with a template prepared in advance is generally used. Can be used as the initial image profile.

  Note that when a face region is detected using an initial image profile for an input image, since simple matching lacks robustness, determination is performed using a discriminator during normal matching. For example, it is also possible to automatically collect sample images and configure the discriminator using the finally obtained image to obtain an initial image profile.

  Next, the audio / video processing apparatus 20 performs a process of separating the multi-channel audio signal input from the microphone array of the conference recording apparatus 10 into a plurality of audio signals by the audio signal separating unit 31 (S11).

  Next, the audiovisual processing device 20 detects the arrival direction (sound source direction) of the sound at a predetermined position for each of the audio signals separated in the process of S11 by the arrival direction detection unit 41 (S12). Next, the video / audio processing device 20 estimates to which object in the video the audio signal separated in the processing of S11 corresponds to the object (video object) in the video. Are associated (S13).

  Next, when one or more target objects are detected in the process of S13, the video / audio processing apparatus 20 uses the sample data selection unit 43 as sample data according to the matching degree (detection reliability) of the face image. It is determined whether to extract (select sample data) (S14).

  Next, the video / audio processing device 20 determines whether it is the last frame of the video shot by, for example, a wide-angle camera (S15). If it is determined that it is not the last frame (NO in S15), the video / audio processing device 20 proceeds to the processing of S11. Return and continue processing. If the video / audio processing device 20 determines that it is the last frame (YES in S15), the profile generation unit 44 generates, for example, an audio profile (S16), and the process ends.

  In the profile generation process described above, for example, sample data is selected until the captured video signal is completed, and at the time when the video signal is completed, an audio profile is generated based on the sample data selected so far. . Note that the profile generation processing in the present embodiment is not limited to this, and for example, a video profile may be generated.

<Association with video object>
FIG. 5 is a flowchart showing the flow of the association process with the video object. In the process of FIG. 5, the process of S <b> 13 shown in FIG. 4 described above uses the video and audio analysis results in a complementary manner to improve the mutual detection accuracy.

  As illustrated in FIG. 5, the video object association unit 42 acquires the sound source direction (sound arrival direction) for each time from the arrival direction detection unit 41 (S20). The arrival direction detection means 41 analyzes the multi-channel audio input obtained from the conference recording apparatus 10 as described above, and estimates the arrival direction of the sound at each time. Acquisition of sound source localization necessary for estimating the direction of sound arrival (the position of the sound source is localized by the difference in sound pressure, phase, time, etc. of the sound reaching the left and right ears) It is possible to use a technique.

  Next, the video object associating means 42 estimates the presence range of the speaker serving as the sound source (S21). In the processing of S21, for example, based on sound peak value data for each detected time, the peaks of the spatial spectrum of the sound are accumulated for the entire meeting or a necessary section during the meeting to generate histogram data. .

  In the process of S21, clustering such as a k-means method is performed on the generated histogram data. The number of clusters is, for example, the number of participants in the conference, and a space having a margin with an appropriate angle from the obtained cluster center can be estimated as the speaker existence range. In addition, about the process of S21, it is not limited to this.

  Next, based on the sound source direction obtained in the process of S20 and the information on the range of the speaker obtained in the process of S21, the video object association unit 42 determines the utterance state (which Whether the speaker is speaking is estimated (S22). In the process of S22, if the peak of the spatial spectrum is within a certain speaker range at a certain time, it can be estimated that the speaker is speaking, but the present invention is not limited to this. .

  Next, the video object association unit 42 performs utterance separation on the portion where the utterance is superimposed, and estimates a position vector for the target speaker (S23). Next, the video object association unit 42 calculates the degree of correlation between the position vector of the target speaker (audio signal) and the position information of the video object (S24).

  In the process of S24, in order to convert the position vector of the target speaker obtained in the process of S23 into a position on the image, an image obtained from the conference recording apparatus 10, for example, an image obtained by projecting a three-dimensional object onto the projection plane Then, the position vector of the target speaker is projected onto this projection plane, and the estimated area is calculated.

  Further, the video object association means 42 uses the correlation between the video object area obtained by pattern matching and the estimated area as the estimated area and B as the previously obtained video object area. A∩B) / (A∪B) ”.

  Next, the video object association unit 42 calculates the reliability of the video object (S25) and ends the process. In the process of S25, if the object reliability Conf obtained from the correlation obtained in the process of S24 is already obtained, the correlation between A and B is, for example, “correlation between A and B = Conf × (A∩B ) / (A∪B) ”.

  Thus, the reliability of the detected object is calculated by looking at the correlation between the position vector of the audio signal and the position information of the video object, rather than detecting the video object alone by the image included in the video signal, Detection with resistance to occlusion, noise, and the like becomes possible.

  In this embodiment, once a video object and a sound source signal are associated with each other, a face image is tracked on the video in subsequent frames, and the sound source direction is accurately estimated from the position and direction of the object in the video. It is possible.

  For example, the direction in which the audio signal should arrive can be estimated in advance from the video. For example, when three or more microphones are used, if the direction of arrival is calculated from a specific channel pair from the beginning when calculating the direction of arrival of the sound source, the direction of arrival of the sound source can be extracted efficiently and accurately. Is possible. In the present embodiment, it is also possible to evaluate the likelihood of the uttered voice using the generated voice profile.

  Conversely, it is also possible to improve the detection accuracy of the video object using an audio signal. For example, when detecting the face of the speaker A from the image, “abrupt illumination change (cannot cope with a sudden illumination change even when the camera internal parameter is automatically changed”), “speaker face orientation” It is known that detection accuracy deteriorates due to factors such as “shielding object”.

  If the conventional image tracking technique is used, it is possible to cover using the previous frame information. However, for example, if the image is lost at some point in the frame, tracking may be interrupted.

  Therefore, in this embodiment, first, the direction of the sound source for each time is estimated from the speech signal, the existence range of the speaker as the sound source is estimated, and the position vector of the target speaker is estimated by looking at the correlation of the utterance time and the like. To do. The process itself for separating the speech of the target speaker from the audio signal can use a conventional method described in, for example, Japanese Patent Application Laid-Open No. 2007-233239.

<Sample data selection process>
FIG. 6 is a flowchart showing the flow of sample data selection processing. FIG. 6A is a flowchart showing the flow of the voice sample data collection process. In the example of FIG. 6A, when the face image profile obtained from the initial image profile generation means 40 or the like is given as the initial data, the initial data of the audio profile is generated.

  As shown in FIG. 6A, the sample data selection unit 43 detects, for example, a video object indicating a face image of the target person by matching with the face image profile, and acquires an audio signal associated with the video object. (S30).

The sample data selection unit 43 obtains a correlation (for example, the degree of correlation obtained by the processing of S24 in FIG. 5) when associating with the video object, and the correlation is higher than a preset threshold (first threshold). It is judged whether it is high (S31). When the sample data selection unit 43 determines that the correlation is higher than the threshold value (Th ₁ ) (YES in S31), the sample data selection unit 43 adopts it as sample data (S32) and determines that the correlation is higher than the threshold value (Th ₁ ). If not (NO in S31), the process is terminated.

  FIG. 6B is a flowchart showing the flow of video sample data collection processing. As shown in FIG. 6B, the sample data selection means 43 acquires a video object indicating the face image of the person by matching with the face image profile obtained from the initial image profile generation means 40 or the like (S40). .

The sample data selection unit 43 obtains a correlation when associating with the audio signal, and determines whether the correlation is higher than a preset threshold (second threshold) (S41). When the sample data selection unit 43 determines that the correlation is higher than the threshold value (Th ₂ ) (YES in S41), the sample data selection unit 43 employs the sample data selection unit 43 as a positive sample (S42).

If the sample data selection unit 43 does not determine that the correlation is higher than the threshold value (Th ₂ ) (NO in S41), the sample data selection unit 43 adopts it as a negative sample (S43) and ends the process. In the present embodiment, by using a negative sample, for example, the probability of erroneous detection can be reduced. For example, it is possible to reduce the probability of mistaking Mr. A and Mr. B.

  When acquiring the above-described video object indicating the face image, it is possible to accumulate the face image of the target individual, construct an identification function, and perform detection using this identification function. Specifically, the image (screen) is scanned in order from the edge (corner) of the window on the basis of the image included in the video, and the feature amount in each window is given to the discriminator, and the discriminator is TRUE or FALSE. To determine. In the case of TRUE, it is determined that there is a high possibility that the image in the window is the person's face image.

  The classifier described above includes SVM (support vector machine), which is called “supervised learning”, and records image samples separately in advance as negative (false examples) and positive (positive examples). This sample can be used to discriminate between negative and positive.

<Associating with video objects>
FIG. 7 is a diagram for explaining the association with the video object. FIG. 7 illustrates a method used when associating an audio signal with a video object in the process of S13 of FIG.

  In the example of FIG. 7A, a detected object (three-dimensional object) is projected onto a projection plane with respect to a video (camera shot video) shot by a camera (for example, a wide-angle camera) included in the conference recording apparatus 10. An example is shown. The video object associating means 42 estimates the video object from the entire area of the captured video by projecting the sound source direction (the direction of arrival of the audio signal) obtained by the processing of S12 in FIG. 4 onto this projection plane. It is possible to narrow down the position (estimated region).

  The video object association unit 42 associates a sound source with each sound acquired by the microphone array with reference to the video object closest to the estimated sound source direction.

  In FIG. 7B, as shown in FIG. 7A, when the video shot by the camera is not an angled video image of the speaker A and the speaker B, the video signals of the speaker A and the speaker B are displayed. This shows an example of converting the video into an angle video facing the screen.

  As shown in FIG. 7B, by using an image in which the face of a person (speaker) is converted into an image of an angle facing the screen, association is performed after controlling (correcting) the orientation of the face. Therefore, it is possible to appropriately associate the sound source direction with the video object. As shown in FIG. 7, it is possible to appropriately associate the audio signal with the video object by converting the video signal into a position in the real world and comparing it with the position in the absolute direction of the audio signal.

<Correlation between video object area and estimated area>
FIG. 8 is a diagram for explaining the correlation between the video object area and the estimated area. 8A and 8B are diagrams for explaining a method for obtaining the degree of correlation between the position vector of the target speaker (voice signal) and the position information of the video object in the process of S24 of FIG. FIG.

  That is, as shown in FIGS. 8A and 8B, the estimated area A calculated in the process of S24 and the area of the video object obtained by pattern matching (the video object area B obtained in advance). ) To obtain the correlation between the position vector of the audio signal and the position information of the video object.

  8A shows a region A∩B (shaded portion), and FIG. 8B shows a region A 斜 B (shaded portion). At this time, the correlation degree is defined as “correlation degree = (A∩B) / (A∪B)”. If the reliability Conf of the video object is obtained in the process of S25, the correlation between A and B is “Conf × (A∩B) / (A∪B)”.

That is, as shown in FIGS. 8A and 8B, the larger the overlapping area between the estimated area and the video object area obtained in advance, the higher the degree of correlation. Therefore, when the degree of correlation is larger than a preset threshold (for example, Th ₁ and Th ₂ described above), it is possible to select as sample data and generate a video profile and an audio profile using the selected sample data. It becomes.

  According to the above-described embodiment, it is possible to generate a profile with high accuracy using video and audio. For example, when a video profile or audio profile is generated from recorded video or audio such as a meeting, an object in the video (a person's face) and an object in the audio (a person's speaking voice source) are detected in a complementary manner. Thereby, even if noise or occlusion occurs in one of the video and audio signals, the detection is corrected based on the other information. Therefore, it is possible to detect an object with high accuracy.

  In this embodiment, the minimum sample data (sample image signal, sample audio signal, etc.) is selected according to the degree of correlation from each object information in the detected video and audio, and the sample data is used. Efficiently generate profiles. Thus, for example, when generating an image and sound profile from a recorded image such as a meeting, only the image sample and sound sample necessary for creating a personal profile are complementarily collected, so that the profile can be generated automatically and accurately. It becomes possible.

  The preferred embodiments of the disclosed technology have been described in detail above, but the invention is not limited to the specific embodiments according to the disclosed technology, and within the scope of the disclosed technology described in the claims, Various modifications and changes are possible.

1 Video / Audio Processing System 10 Conference Recording Device 20 Video / Audio Processing Device (Example of Information Processing Device)
21 Audio processing device 22 Video processing device 23, 50 CPU
24 main storage device 25 auxiliary storage device 31 audio signal separation means 32 object detection means (an example of detection means)
33 Profile acquisition means 40 Initial image profile generation means 41 Arrival direction detection means 42 Video object association means (an example of association means)
43 Sample data selection means 44 Profile generation means 51 RAM
52 ROM
53 HDD
54 I / F 55 LCD
56 Operation unit 57 Bus

JP 2011-71684 A

Claims (8)

Detection means for detecting a region to be detected from a video signal imaged at a predetermined position;
An audio signal separating unit that separates an audio signal collected at the predetermined position into audio signals for each detection target corresponding to the detection target area detected by the detection unit;
Sample data selection means for selecting sample data for generating a video or audio profile from the video signal or the audio signal based on the detection target area and the arrival direction of the audio signal for each detection target When,
A video / audio processing system comprising: profile generation means for generating a video or audio profile using the sample data selected by the sample data selection means.   Based on the detection target region and the arrival direction of the audio signal for each detection target, the audio signal in which the arrival direction is detected corresponds to the detection target included in the video signal, and the estimation result The video / audio processing system according to claim 1, further comprising: an association unit that associates the detection target included in the video signal with the audio signal based on the video signal. The sample data selection means includes
Collecting at least one of the position, direction and start / end time of the sound as sample data of the sound signal, and collecting sample data having a correlation with the detection target area greater than a threshold value. The video / audio processing system according to claim 1, wherein the system is a video / audio processing system. The sample data selection means includes
As the sample data of the video signal, collecting sample data whose correlation with the audio signal is greater than a threshold at at least one of the position, direction, and start or end time of the detection target included in the video signal. The video / audio processing system according to any one of claims 1 to 3, wherein the system is a video / audio processing system. The detection means includes
5. The video / audio processing system according to claim 1, wherein the detection target region is extracted by performing image matching using an initial profile of at least one or more image samples. Detection means for detecting a region to be detected from a video signal imaged at a predetermined position;
An audio signal separating unit that separates an audio signal collected at the predetermined position into audio signals for each detection target corresponding to the detection target area detected by the detection unit;
Sample data selection means for selecting sample data for generating a video or audio profile from the video signal or the audio signal based on the detection target area and the arrival direction of the audio signal for each detection target When,
An information processing apparatus comprising: profile generation means for generating a video or audio profile using the sample data selected by the sample data selection means. A video / audio processing method executed by a video / audio processing system,
A detection procedure for detecting a detection target region from a video signal imaged at a predetermined position;
An audio signal separation procedure for separating an audio signal collected at the predetermined position into an audio signal for each detection target corresponding to a detection target area detected by the detection procedure;
Sample data selection procedure for selecting sample data for generating a video or audio profile from the video signal or the audio signal based on the detection target area and the arrival direction of the audio signal for each detection target When,
And a profile generation procedure for generating a video or audio profile using the sample data selected by the sample data selection procedure. Computer
Detecting means for detecting a detection target region from a video signal imaged at a predetermined position;
An audio signal separating unit that separates an audio signal collected at the predetermined position into audio signals for each detection target corresponding to the detection target area detected by the detection unit;
Sample data selection means for selecting sample data for generating a video or audio profile from the video signal or the audio signal based on the detection target area and the arrival direction of the audio signal for each detection target ,as well as,
A video / audio processing program for functioning as profile generation means for generating a video or audio profile using the sample data selected by the sample data selection means.

Images