JPH08286680A - Sound extracting device - Google Patents

Abstract

PURPOSE: To detect a position of an object whose position is uncertainty and to extract a sound generated by the object based on its position. CONSTITUTION: When a person A moves, the inside of a room 50 is continuously photographed by plural television cameras 16, a sound extracting device recognizes its position based on the picture information regarding a region provided with specific feature quantity in a human head of which almost surface is covered by hair and which has much black part and almost spherical shape as a region corresponding to a head part P. Next, sound signals of each collected sound are delayed by a delay time in accordance with a distance between positions of each microphone 22 and a position of a head part P so that a voice part of the person A in a voice signal of a collected sound collected by plural microphones 22 is synchronized along time base. And only a voice of the person A is extracted by adding and averaging a delayed voice signal.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a sound extraction device, and more particularly to a sound extraction device for extracting a sound emitted by an object (in the present invention, a human or an object expected to emit a sound).

[0002]

2. Description of the Related Art Conventionally, the problems to be solved by the invention
When inspecting the degree of deterioration of buildings and bridges,
The degree of aging was inspected by calculating the squeaking sound and the like emitted from a predetermined part of the building mainly by simulation. However, since the value calculated by the simulation is a predicted value to the last, in order to perform a more rigorous inspection, the actual squeaking sound emitted from a predetermined part of the building may be discriminated from the surrounding noise and extracted. Was wanted.

On the other hand, in relation to the above, the voice signals of the sounds collected by the plurality of microphones are superimposed on the same time axis, and for each of these sounds, the microphones are responsive to the distance between the microphone and the target sound source. A technique is known in which only the sound emitted from a target sound source is extracted by performing an appropriate delay operation and then performing averaging. It is also known that this technology is applied to a handheld video camera that simultaneously shoots and records, and the sound emitted from the subject during shooting is extracted to match the focus of the image of the subject with the focus of recording. (See Japanese Patent Laid-Open No. 5-305855).

However, the above-mentioned technique relating to the hand-held video camera is actually effective only for a single subject located within a narrow area such as the photographing field of view of the video camera, and is hand-held. Since the number of microphones attached to the video camera is small and the intervals between them are small, it is relatively difficult to collect highly realistic voices because of the large influence of noise.

By the way, conventionally, there is an image recognition technique in which a plurality of television cameras are arranged on the ceiling and the position of an object existing in a room is detected based on image information taken by the plurality of television cameras.

However, when an object moves, the moving object is photographed by moving a plurality of television cameras and adjusting the focus in accordance with the moving object. As described above, since it is necessary to move the television camera to adjust the focus, there is a problem that a delay time occurs until the image data including the object is obtained.

The present invention has been made in consideration of the above facts, and combines the image recognition technology and the sound extraction technology related to the above position detection to detect the position of an object whose position is indeterminate. It is a first object of the present invention to provide a sound extraction device that can detect a sound emitted by an object based on the detected position. A second object of the present invention is to provide a sound extraction device capable of more efficiently detecting the position of an object and extracting the sound emitted by the object based on the detected position.

[0008]

In order to achieve the first object, the invention according to claim 1 is a photographing means for photographing an area including an object as a sound source, and an area photographed by the photographing means. Image recognition means for recognizing the position of the object based on the image information, a plurality of microphones arranged at a predetermined position for collecting sounds emitted by the object, and a sampling sound collected by each of the plurality of microphones Time-series data of a plurality of collected sounds is selected from the series data, and the time-series data of the selected collected sounds is set to the position of the object recognized by the image recognition means and the position of the microphone collecting the selected collected sound. Based on this, the sounds emitted by the objects are shifted in synchronization, and the time-series data of the shifted sampled sounds are averaged. Characterized in that it has extracting means for extracting a sound object emits, the.

According to the first aspect of the present invention, the photographing means photographs the area including the object as the sound source, and the image recognition means recognizes the position of the object from the image information of the area photographed by the photographing means.

For example, as shown in FIG. 2, from the image information in the room 50 taken by each of the plurality of television cameras 16 installed on the ceiling 52, the head of the target person A as an object is obtained as follows. The position of the part P is recognized.
That is, according to the image information, a region having a characteristic amount peculiar to the human head, such as a substantially spherical shape with many black surfaces covered with hair and the like, is defined as the head P of the target person A. Is extracted as a region corresponding to. A region corresponding to the extracted head P is among a number of rectangular parallelepiped regions obtained by virtually equally dividing the room 50 in each of the arrow X direction, the arrow Y direction, and the arrow Z direction. Recognize which area it corresponds to.

On the other hand, the sound produced by the object is sampled by a plurality of microphones (hereinafter, abbreviated as microphones) arranged at predetermined positions. For example, as shown in FIG. 2, the collected sound including the voices of the two target persons A and B and some noise is collected by the microphone arranged on the ceiling of the room where the two persons are present. It is assumed that the time-series data of the collected sound collected by each microphone has the waveform shown in FIG. 1A (the number of microphones is 7 for convenience of explanation, but the present invention is not limited to this). The number can be increased more than just one).

As shown in FIG. 1A, the target person A in the waveform of the time-series data of the collected sound collected by each microphone
Of the target person B and the voice of the target person B are shifted along the time axis (horizontal axis) for each microphone. That is, the time required for the voice of the target person to reach the microphone differs depending on the size of the distance between each target person and each microphone. For example, since the microphone 1 is close to the target person A and far from the target person B, the time-series data of the microphone 1 shows a portion corresponding to the voice of the target person A first along the time axis, and the target person B later. The part corresponding to the voice of will appear.

The extracting means selects the time-series data of the plurality of collected sounds from the time-series data of the collected sounds collected by each of the plurality of microphones. Here, you may select the time-series data of the collected sounds collected by all microphones,
As in the invention described in claim 8 described later, even if the time-series data of the collected sound collected by the microphone separated by a predetermined distance or more from the position of the object is excluded and the time-series data of other collected sounds is selected. good.

Then, the extraction means determines the time series data of the selected sampling sound based on the position of the object recognized by the image recognizing means and the position of the microphone collecting the selected sampling sound. Shift to be in sync.

For example, when the target person A in FIG. 2 is used as an object and the voice of the target person A is extracted as an example, description will be given.
By dividing the distance between the head P of the target person A and each microphone by the speed of sound, the delay time of sound collection by each microphone for the voice uttered by the target person A is obtained. Then, as shown in FIG. 1B, the time-series data obtained by delaying the time-series data of the collected sound collected by the microphone by the delay time along the time axis is obtained for each microphone. As a result, the portions of the microphones corresponding to the voice of the target person A are substantially synchronized (aligned in the same phase) along the time axis. On the other hand, the portions other than the voice of the target person A and corresponding to the voice of the target person B and other noise remain in a state where the phases are not aligned along the time axis.

Further, the extraction means extracts the sound emitted by the object by averaging the time-series data of the shifted collected sounds. For example, the microphone 1 shown in FIG.
Synchronously add (superimpose) all the time series data in
The amplitude of the waveform after addition is divided by the number of microphones “7”.
As a result, as shown in FIG. 1C, the amplitude of the arithmetically averaged time-series data becomes extremely small in a portion corresponding to the voice of the target person B other than the voice of the target person A and other noises, and an error is almost generated. Since the amplitude value is within the range of
Only the part corresponding to the voice of will be extracted.

As described above, according to the first aspect of the present invention, the position of the object can be recognized, and the sound emitted by the object can be extracted based on the position while being discriminated from the ambient noise.

Further, in order to achieve the first object, the invention according to claim 2 is the invention according to claim 1, wherein the image recognition means causes the object to generate a sound based on the image information of the area including the object. It is also characterized in that the direction in which the sound is emitted is recognized, and the position where the sound emitted by the object can be satisfactorily extracted is recognized again based on the position of the object and the direction in which the object emits sound.

According to the second aspect of the invention, the image recognizing means also recognizes the direction in which the object makes a sound from the image information of the area including the object. For example, the direction in which the target person A shown in FIG. 2 makes a sound (voice) is recognized as follows. That is, first, after recognizing the head P in the manner described above, the body S located below the head P is recognized,
Since the chest width L2 is smaller than the shoulder width L1 in the body S, it is estimated that the target person A faces the arrow V direction or the opposite direction. Next, based on the general characteristic that the proportion of hair on the surface of the head P is higher on the side where the face is not located than on the side where the face is located, the back side of the paper in FIG. 2 is the front side of the paper. Since the degree of black is higher than that of the head P, it is estimated that the head P faces the direction of the arrow V, and the direction in which the target person A speaks is recognized as the direction of the arrow V.

Further, the image recognizing means is based on the position of the object and the direction in which the object emits a sound,
A position at which the sound emitted by the object can be satisfactorily extracted, that is, a position at which all the frequency components ranging from the low frequency region to the high frequency region can be extracted at substantially the same level as the original sound (for example, a predetermined distance in the direction in which the sound is emitted from the position of the object ( dozens
Recognize (cm) separated position) as the position of the object.

Since the sound is extracted as described above based on the position of the object recognized in this way, that is, the position at which the sound emitted by the object can be satisfactorily extracted,
In particular, when the directivity of the sound emitted by the object is strong, it is possible to extract the sound with higher accuracy.

In order to achieve the first object, according to the invention of claim 3, in the invention of claim 1, when the object moves, the photographing means follows the movement of the object. It is characterized in that an area including is photographed.

In the invention according to the third aspect, when the object moves, the photographing means follows the movement of the object and photographs the area including the object. Thereby, the image recognition means recognizes the position of the moving object from the image information of the photographed area, and the extraction means detects the position of the moving object from the moving object in the above-described manner based on the position of the object recognized by the image recognition means. Extract the sound of. Therefore, the sound from the moving object can be extracted.

In order to achieve the first object, according to the invention of claim 4, in the invention of claim 1, when there are a plurality of objects, the photographing means photographs a region including a plurality of objects. However, the image recognition means recognizes the position of each of the plurality of objects from the image information of the photographed area, and the extraction means extracts the sound from each of the plurality of objects.

In the invention according to the fourth aspect, when there are a plurality of objects, the photographing means photographs the area including the plurality of objects, and the image recognition means determines each of the plurality of objects from the image information of the photographed area. Recognize position individually. Then, the extraction means extracts the sound from each of the plurality of objects based on the position of each of the plurality of objects recognized by the image recognition means in the manner described above. As a result, the sound from each of the plurality of objects can be extracted for the plurality of objects.

Further, in order to achieve the first object, the invention according to claim 5 is the invention according to claim 1, wherein at least one of a sound velocity and a sound propagation path is provided in a region including the object and the plurality of microphones. Further comprising a detection means for detecting an acoustic environment state that is a factor affecting, the extraction means, when the acoustic environment state detected by the detection means changes, based on the changed acoustic environment state, It is characterized in that the shift of the time series data of the collected sound is corrected.

In the invention according to claim 5, the detecting means has an acoustic environment condition that is a factor affecting at least one of the sound velocity and the sound propagation path in the region including the object and the plurality of microphones, such as temperature and wind force.
Detect the wind direction. Then, when the acoustic environment state detected by the detecting means changes, the extracting means corrects the shift of the time-series data of the collected sound based on the changed state of the acoustic environment, for example, as follows.

That is, the ratio of the sound velocity corresponding to the detected temperature calculated in advance to the standard sound velocity is determined by referring to the sound velocity correction table stored in advance corresponding to the detected temperature to obtain the sound velocity corresponding to the detected temperature. The ratio to the standard sound velocity is obtained, the delay time of sound collection in each microphone is corrected based on the ratio, and the delay operation is performed according to the corrected delay time. Alternatively, the sound collection speed at each microphone is corrected by dividing the distance between the position of the object and the position of each microphone by the speed of sound corresponding to the detected temperature, and the delay operation is performed according to this corrected delay time. To do.

Further, for example, the propagation path of sound from the position of the object under the detected wind force and wind direction to the position of each microphone is obtained by simulating various wind force values and wind direction values in advance. Estimated based on the information about the bend (change) of the propagation path, and by dividing the distance along the estimated propagation path by the sound velocity, the delay time of the sound collection at each microphone was corrected, and this was corrected. Delay operation is performed according to the delay time.

As described above, it is possible to extract sounds with high accuracy according to changes in the acoustic environment condition.

Further, in order to achieve the first object, the invention according to claim 6 is the invention according to claim 1, wherein the extracting means extracts the high frequency range based on the information on the directivity of the high frequency range. It is characterized in that time series data of sounds are weighted and averaged.

FIG. 11 shows a region where a component of each frequency band of sound propagates. It can be seen that the treble range propagates almost only in the sound direction (arrow D), while the bass range propagates in a wider range. That is, the directivity of the sound differs depending on the frequency band, and it is general that the directivity is lower in the low sound range and the directivity is higher in the high sound range. Therefore, the microphone located in the direction in which the object emits sound collects almost all frequency components from the low range to the high range, while other microphones collect the low range but not the high range. No sound is collected.

However, according to the invention of claim 6,
Extraction means, based on the information about the directivity of the high range, the high-range sampled sound by the microphone located in the direction in which the object emits sound, the high-range sampled sound by other microphones,
In order to correct the imbalance of, the weighted average is applied. This makes it possible to prevent the high range from becoming relatively weaker than the low range.

Further, in order to achieve the first object, the invention according to claim 7 is the invention according to claim 1, wherein the image recognition means causes the object to generate a sound based on the image information of the area including the object. Further, the direction of emission, the position and the direction of the reflection surface of the sound located around the object are further recognized, and the extraction means makes the position of the microphone collecting the selected sampled sound, the position of the object, the object emits sound. Based on the direction and the position and orientation of the reflecting surface, the time-series data of the selected collected sound,
One of the direct sound from the object and the reflected sound reflected by the reflecting surface is shifted so as to be synchronized.

By the way, when the sound emitted by the object is reflected by a predetermined reflection surface and collected by the microphone, since the reflected sound is usually much weaker than the direct sound, other noise is added by averaging. Automatically removed with the ingredients. However, in the case where the reflecting surface is located near the object and in the direction in which the object emits sound, and the microphone is located in the traveling direction of the sound reflected by the reflecting surface, the Since the reflected sound becomes louder than the direct sound, the effect of extracting the sound emitted by the object is higher when the reflected sound on the reflecting surface is collected.

In view of the above, according to the present invention, the image recognizing means determines, based on the image information of the area including the object, the direction in which the object emits sound, the position of the sound reflecting surface located around the object, and Also recognize the direction. Extraction means, based on the position of the microphone that collected the selected collected sound, the position of the object, the direction in which the object emits sound, and the position and orientation of the reflective surface, the time-series data of the selected collected sound. ,
Either the direct sound from the object or the reflected sound reflected by the reflecting surface shifts in synchronization.

For example, when the reflecting surface is located near the object and is located in the direction in which the object emits sound and the microphone is located in the traveling direction of the sound reflected by the reflecting surface, The delay operation is executed according to the propagation time of the reflected sound reflected by the surface. In this way, more appropriate sound extraction can be performed according to the arrangement of the reflective surface around the object.

[0038] In order to achieve the first object, the invention according to claim 8 is the invention according to claim 1, wherein the extraction means outputs a sampling sound collected by each of the plurality of microphones. Among the series data, time series data of collected sounds collected by a microphone located at a predetermined distance or more from the position of the object is excluded from the selection target.

In general, sound is attenuated according to its propagation distance. Therefore, when the sound emitted by the object is collected by the microphone through a long propagation distance, the collected sound collected by the microphone is Since it contains only a small amount of the sound component emitted by the object, the degree of contribution to the formation of the time-series data is small when the time-series data of the sound of the object is obtained.

Therefore, in the invention according to the eighth aspect, the extraction means collects the plurality of microphones that are far from the position of the object, that is, the microphones that are located at a predetermined distance or more determined in advance by an experiment. The time series data of sound is excluded from the selection target. This allows
The load of processing related to sound extraction (shifting and averaging processing by the extraction means) can be reduced without lowering the accuracy of sound extraction.

For the same purpose as described above, among a plurality of microphones, a microphone having a small volume of a collected sound taken by the microphone, that is, a microphone having a volume of the collected sound smaller than a predetermined volume level obtained by an experiment in advance is used. The collected sounds that have been taken may be excluded from the selection targets.

In order to achieve the first object, the invention according to claim 9 is the invention according to claim 1, wherein the sound emitted by the object extracted by the extracting means is output to a predetermined voice recognition device. It further comprises an output means for

By the way, according to the first aspect of the present invention, the time series data of the sampled sounds taken by more microphones arranged near the object are shifted and averaged as described above to obtain a signal pair. It is possible to improve the noise ratio and extract the sound of the object. Moreover, it is also possible to extract a sound having a higher signal-to-noise ratio than the sound collected by a normal microphone. Such a good sound can be utilized as an input to the voice recognition device.

Therefore, in the invention according to the ninth aspect, the output means outputs the sound from the object extracted by the extraction means to a predetermined voice recognition device. Accordingly, it is possible to input the voice uttered by a person (one or more people) in the area where the sound can be extracted by the sound extraction device to the voice recognition device. In particular, the present invention can be applied to a case where an elderly person or a handicapped person with physical disabilities controls on / off of switches of home electric appliances and the like by voice using a voice recognition device.

Further, in order to achieve the second object, the invention according to claim 10 is provided with a wide-angle fixed focus lens arranged at a predetermined position, and photographing for photographing an area including an object as a sound source. Means, image recognition means for recognizing the position of the object based on the image information of the area photographed by the photographing means, a plurality of microphones arranged at a predetermined position for collecting sounds emitted by the object, and the plurality of microphones. Select the time series data of a plurality of collected sounds from the time series data of the collected sounds collected by each of the microphones, and select the time series data of the selected collected sounds,
Based on the position of the object recognized by the image recognition means and the position of the microphone collecting the selected collected sound, the sound emitted by the object is shifted in synchronization, and the time-series data of the shifted collected sound is averaged. Accordingly, an extraction unit that extracts the sound emitted by the object is included.

According to the tenth aspect of the invention, the photographing means includes a wide-angle fixed focus lens arranged at a predetermined position. As a result, regardless of whether the object is moving or stationary, the object can be photographed without changing the direction of the photographing means (such as a television camera) following the movement of the object. . In addition, since the height of objects, people, animals, etc. from the floor and the ground is almost fixed, and the wide-angle fixed-focus lens has the characteristic that the depth of focus is large. The object can be photographed without having it, that is, without performing focus adjustment.

In this way, the position of the object can be quickly recognized without changing the orientation of the photographing means or adjusting the focus following the movement of the object. Further, since the mechanical actuation mechanism for changing the orientation of the photographing means and adjusting the focus as described above is not required, the structure of the photographing means and the sound extraction device can be simplified and the mechanical operation can be simplified. Durability can be improved by reducing the number of operating parts.

The wide-angle fixed-focus lens is arranged at, for example, a flat portion such as the ceiling of a room, a corner formed by two surfaces of the ceiling and a wall, and a total of three surfaces including the ceiling and the wall. It can be a corner made by.

According to the tenth aspect of the invention, from the image information of the area photographed by the photographing means as described above,
The position of the object can be recognized by the image recognition means and the sound emitted by the object can be discriminated from the ambient noise and extracted by the extraction means in the same manner as in the first aspect of the invention.

In order to achieve the second object, the invention according to claim 11 is the same as the invention according to claim 10,
A plurality of the photographing means are provided, each of the photographing means further includes an area sensor arranged at an image forming point of the wide-angle fixed focus lens, and the image recognizing means is photographed by the plurality of photographing means. A shape recognizing unit that processes different photographic information to recognize the shape of the object, and a three-dimensional coordinate calculating unit that calculates the three-dimensional coordinates of the object recognized by the shape recognizing unit are included.
It is characterized by the following.

According to the invention described in claim 11, a plurality of photographing means are provided, and each photographing means further comprises an area sensor arranged at the image forming point of the wide-angle fixed focus lens. That is, the image of the object photographed by the photographing means through the wide-angle fixed focus lens is formed on the area sensor. In this way, the regions including the object are photographed from different positions by the plurality of photographing means.

The shape recognition means processes the different photographing information photographed by the plurality of photographing means to recognize the shape of the object. To recognize the shape of an object, for example, as in the invention of claim 12 described later,
Of a large number of cubic microspaces obtained by virtually subdividing the three-dimensional space along each of the X-axis, Y-axis, and Z-axis, the region formed by the microspace occupied by the object You may recognize by asking,
Further, for example, different photographing information is converted into flattened image information by a plurality of photographing means, and at least the front surface, the back surface, the left side surface of the object are converted from the converted image information.
The image information of the right side surface and the plane surface may be obtained, and the obtained image information may be combined and recognized.

The three-dimensional coordinate calculation means calculates the three-dimensional coordinates of the object recognized by the shape recognition means or a predetermined part of the object. In general, an object is not limited to points, but is a set of points. Therefore, all three-dimensional coordinates of points included in an object may be calculated, or all points belonging to a boundary that demarcates the object and the three-dimensional space. The three-dimensional coordinates of may be calculated. Further, for example, the height or length may be obtained by presetting (storing) a specific position and calculating the three-dimensional coordinates of the specific position.

The shape recognizing means and the three-dimensional coordinate calculating means, which constitute the image recognizing means in this way, can promptly obtain the three-dimensional coordinates of the object and promptly recognize the position of the object.

In order to achieve the second object, the invention according to claim 12 is the same as the invention according to claim 11,
The shape recognizing means calculates an X-ray space in a three-dimensional space based on different image information captured by the plurality of capturing means.
Of a large number of cubic microspaces obtained by virtually subdividing along the axes, Y-axis and Z-axis,
The feature of the present invention is that the shape of the object is recognized by obtaining the area formed by the minute space occupied by the object.

According to the twelfth aspect of the present invention, a large number of cubic shapes obtained by virtually subdividing the three-dimensional space along the X-axis, Y-axis, and Z-axis by the shape recognition means are obtained. The shape of the object is recognized by determining the area formed by the minute space occupied by the object in the minute space. Here, the minute area can be subdivided to the limit of the resolution of the area sensor. Therefore, the shape of the object can be recognized in detail.

In order to achieve the second object, the invention of claim 13 is the same as the invention of claim 11,
The shape recognizing means calculates an X-ray space in a three-dimensional space based on different image information captured by the plurality of capturing means.
Of a large number of cubic microspaces obtained by virtually subdividing along the axes, Y-axis and Z-axis,
Characteristic of recognizing the shape of an object by extracting each of the minute spaces included in the viewing angle for projecting the object from each photographing means and determining the area formed by the minute spaces included in all of the extracted minute spaces And

In the image information photographed by the plurality of photographing means according to the eleventh aspect, there is a shadow (blind spot) area as shown in FIG. Therefore, as described in claim 13, the shape recognition means sets the three-dimensional space along each of the X-axis, Y-axis, and Z-axis based on the different image information captured by the plurality of image capturing means. Of a large number of cubic microspaces obtained by virtually subdividing, microspaces included in the viewing angle for projecting an object from each photographing means are extracted, and included in all of the extracted microspaces. The shape of the object is recognized by finding the area formed by the minute space.

By recognizing the shape of the object in this way, the shadow (blind spot) region can be eliminated, and the shape of the object can be accurately recognized.

In order to achieve the second object, the invention of claim 14 is the same as the invention of claim 10,
A plurality of the photographing means are provided, each photographing means further comprises an area sensor arranged at an image forming point by the wide-angle fixed focus lens, and the image recognition means forms an image on the area sensor of each photographing means. It is characterized in that the acquired two-dimensional coordinates are acquired, and the position of the object is recognized based on the plurality of acquired two-dimensional coordinates.

According to the fourteenth aspect of the present invention, the three-dimensional coordinates can be calculated back if the two-dimensional coordinates of the points on the area sensors of the plurality of photographing means are known. The two-dimensional coordinates imaged on the area sensor can be acquired, and the position (three-dimensional coordinates) of the object can be accurately recognized based on the acquired two-dimensional coordinates. Further, in order to achieve the second object, the invention according to claim 15 comprises a wide-angle fixed focus lens arranged at a predetermined position and an area sensor arranged at an image forming point by the lens, A photographing means for photographing an area including an object as a sound source, a reflecting means arranged in the vicinity of the photographing means for reflecting the image of the object so as to form an image on the area sensor, and a reflecting means for reflecting the image of the object. Two-dimensional coordinates of each of the object image formed on the area sensor and the object image formed on the area sensor without being reflected by the reflection unit are acquired, and the acquisition is performed. Image recognition means for recognizing the position of the object by calculating the three-dimensional coordinates of the object based on the plurality of two-dimensional coordinates. A plurality of microphones arranged at different positions for collecting sounds emitted by the object, and time series data of a plurality of collected sounds selected from time series data of the collected sounds collected by each of the plurality of microphones, and selected. Based on the position of the object recognized by the image recognition means and the position of the microphone collecting the selected sampling sound, the time-series data of the sampling sound is shifted so that the sound emitted by the object is synchronized, and the shifted sampling is performed. Extraction means for extracting the sound emitted by the object by averaging the time-series data of the sound.

According to the fifteenth aspect of the present invention, the photographing means includes a wide-angle fixed focus lens arranged at a predetermined position and an area sensor arranged at an image forming point of the lens. Moreover, the area including the object is photographed by the photographing means.

The reflecting means is arranged near the photographing means. The reflecting means may be arranged along a wall, may be L-shaped, or may be curved, as shown in (G) to (L) of FIG. 24, for example. The image of the object is reflected by the reflecting means so as to form an image on the area sensor.

Then, the image recognizing means includes areas of the object image reflected on the area sensor by being reflected by the reflecting means and the object image formed on the area sensor without being reflected by the reflecting means. Obtain two-dimensional coordinates on the sensor. In this way, even if there is only one photographing means, a plurality of object images are formed on the area sensor provided in the single photographing means, and a plurality of two-dimensional coordinates are acquired. Therefore, similarly to the above-described fourteenth aspect of the invention, the image recognition means can accurately recognize the position (three-dimensional coordinate) of the object based on the acquired plurality of two-dimensional coordinates.

As described above, since the reflecting means can form another object image on the area sensor, the three-dimensional coordinates of the object can be calculated and the position of the object can be calculated even if there is only one photographing means. Can be accurately recognized.

In the fifteenth aspect of the invention, the position of the object is recognized by the image recognition means as described above, and the sound emitted by the object is detected by the extraction means in the same manner as in the first aspect of the invention. It can be extracted by discriminating it from noise.

In order to achieve the first and second objects, the invention according to claim 16 is the invention according to any one of claims 10 to 15, wherein the image recognition means is
The position in which the sound emitted by the object can be satisfactorily extracted is also determined based on the position of the object and the direction in which the object emits sound based on the image information of the area including the object. It is characterized by recognizing again.

According to the sixteenth aspect of the invention, similarly to the second aspect of the invention, the sound is extracted based on the position of the object, that is, the position at which the sound emitted by the object can be favorably extracted. Particularly, when the directivity of the sound emitted by the object is strong, or when the part (face) of the object that emits the sound is large, the sound can be extracted with higher accuracy.

Further, in order to achieve the first and second objects, the invention according to claim 17 is the invention according to any one of claims 10 to 15 in which when a plurality of objects are present, The photographing means photographs an area including a plurality of objects, and the image recognition means recognizes each position of the plurality of objects from image information of the photographed area,
The extraction means extracts a sound from each of the plurality of objects.

According to the seventeenth aspect of the invention, when there are a plurality of objects, the photographing means photographs the area including the plurality of objects and the image recognition means photographs the same as in the fourth aspect of the invention. The position of each of the plurality of objects is recognized based on the image information of the region. And
The extracting means extracts the sound from each of the plurality of objects based on the position of each of the plurality of objects recognized by the image recognizing means in the same manner as the invention according to claim 1 described above. As a result, the sound from each of the plurality of objects can be extracted for the plurality of objects.

In order to achieve the first and second objects, the invention according to claim 18 is the invention according to any one of claims 10 to 15, wherein the object and the plurality of microphones are When the acoustic environment state detected by the detecting means is changed, the detecting means further includes a detecting means for detecting an acoustic environment state which is a factor affecting at least one of the sound velocity and the sound propagation path in the area including In addition, the shift of the time-series data of the collected sound is corrected based on the changed acoustic environment state.

According to the eighteenth aspect of the present invention, when the extraction means changes the acoustic environment state detected by the detection means, the extraction means is based on the changed state of the acoustic environment. Similarly, the shift of the time series data of the collected sound is corrected. This makes it possible to extract sounds with high accuracy according to changes in the acoustic environment state.

In order to achieve the first and second objects, the invention according to claim 19 is the invention according to any one of claims 10 to 15, in which the extracting means is in the high range. It is characterized by weighting and averaging time-series data of collected sounds in the high frequency range based on information on directivity.

According to the nineteenth aspect of the present invention, as in the case of the above-described sixth aspect of the invention, the extracting means uses the microphones located in the direction in which the object emits sound based on the information on the directivity in the high frequency range. In order to correct the imbalance between the sampling sound of the range and the sampling sound of the high range by other microphones,
Weight and average. This makes it possible to prevent the high range from becoming relatively weaker than the low range.

In order to achieve the first and second objects, the invention according to claim 20 is the invention according to any one of claims 10 to 15, wherein the image recognition means is
From the image information of the area including the object, the direction in which the object emits sound, and the position and direction of the reflection surface of the sound located around the object are further recognized, and the extraction means collects the selected collected sound. Based on the position of the object, the position of the object, the direction in which the object emits sound, and the position and direction of the reflective surface, the time-series data of the selected sampled sound is reflected by the direct sound from the object or the reflective surface. One of the reflected sounds is shifted so as to be synchronized with each other.

According to the twentieth aspect of the invention, similarly to the above-described seventh aspect of the invention, for example, the reflecting surface is located at a position close to the object, and the object is located in the direction in which sound is emitted. When the microphone is located in the traveling direction of the sound reflected by the reflecting surface, the delay operation is executed according to the propagation time of the sound reflected by the reflecting surface. In this way, more appropriate sound extraction can be performed according to the arrangement of the reflective surface around the object.

In order to achieve the first and second objects, the invention according to claim 21 is the invention according to any one of claims 10 to 15, wherein the extracting means is the plurality of Among the time-series data of the sound collected by each of the microphones, the time-series data of the sound collected by the microphone located at a predetermined distance or more from the position of the object is excluded from the selection target. And

According to the twenty-first aspect of the invention, as in the case of the eighth aspect of the invention described above, the extracting means is a microphone far from the position of the object among the plurality of microphones, that is, a predetermined distance or more previously obtained by an experiment. Collected sounds taken by remotely located microphones are excluded from selection. As a result, it is possible to reduce the load of processing related to sound extraction (shifting and averaging processing by the extraction unit) without degrading the accuracy of sound extraction.

Further, in order to achieve the first and second objects, the invention according to claim 22 is the object according to any one of claims 10 to 15, wherein the object extracted by the extracting means. It is characterized by further comprising output means for outputting the sound emitted by the device to a predetermined voice recognition device.

According to the twenty-second aspect of the invention, the output means outputs the sound of the object extracted by the extraction means to a predetermined voice recognition device, as in the case of the ninth aspect of the invention. Accordingly, it is possible to input the voice uttered by a person (one or more people) in the area where the sound can be extracted by the sound extraction device to the voice recognition device.

[0081]

BEST MODE FOR CARRYING OUT THE INVENTION

[First Embodiment] Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.
Will be described. The first embodiment shows an example in which only the voice of the target person A in the predetermined room 50 shown in FIG. 2 is extracted.

As shown in FIGS. 2 and 3, the sound extraction device 10 according to the first embodiment includes a plurality of television cameras 16 arranged at predetermined positions on the ceiling 52 of the room 50 and the television cameras 16. The extraction position arithmetic processor 14 that is connected and sets the extraction position of the sound based on the image information captured by the television camera 16, and a plurality of (n in FIG. 2) arranged in a matrix at substantially equal intervals on the ceiling 52. Is an example of 8 × 8 microphones), a microphone array unit 18 including microphones 22, a voice extraction board 12 that is connected to each microphone 22 and extracts a voice of a target person from sounds collected by the microphones 22, And an output terminal board 20 for outputting the extracted sound.

Each microphone 22 is connected to the sound collecting section 24, an amplifier filter 26 connected to the sound collecting section 24 for performing noise cutting and amplification of a voice signal, and connected to the amplifier filter 26 to convert an analog signal into a digital signal. And an A / D converter 28 for performing conversion. The extraction position calculation processor 14 includes a CPU 14A and a ROM 1
4B and RAM 14 mainly used as a working storage area
C and input / output controller (hereinafter referred to as I / O) 14
D and D, and these CPU 14
A, ROM 14B, RAM 14C and I / O 14D are connected to each other by a bus 14E.

The sound extraction board 12 is connected to the microphones 22 in a one-to-one correspondence with each other via the digital line 30, and n input buffers for temporarily storing the sound data transmitted from the microphones 22. Memory i (i: 1, 2
... Input buffer memory group 32 composed of n)
A processor 34 connected to each input buffer memory i for controlling the entire voice extraction board 12 and the like, and for temporarily storing voice data corresponding to each microphone 22 connected to each processor 34 and output from the processor 34. n
It corresponds to the output buffer memory group 44 composed of one output buffer memory i (i: 1, 2, ... N), and each microphone 22 connected to each output buffer memory i and output from each output buffer memory i. Adder 46 for adding the audio data to be reproduced, and a D / A converter 48 connected to the adder 46 for converting a digital signal into an analog signal
And are provided. The processor 34 is similar to the extraction position calculation processor 14 in that the CPU 38, R
It is configured to include an OM 40, a RAM 42 and an I / O 36, which are connected to each other by a bus 37. The I / O 36 includes the input buffer memories i, the output buffer memories i, and the extraction position calculation processor 14 described above.
Is connected. In addition, the processor 34 transmits the control signals and the like for synchronizing the operation of each component in the sound extraction device 10 to each component,
That is, each microphone 22, the input buffer memory group 32, the output buffer memory group 44, the adder 46, and the D / A converter 48 are connected to each other via the control signal line 43. The ROM 40 stores in advance a control program for voice extraction processing, which will be described later, position information regarding the respective positions where the microphones 22 are arranged, a delay table which will be described later, and the like.

The output terminal board 20 is also provided with an audio output terminal 21, and the audio output terminal 21 is connected to the D / A converter 48 of the audio extraction board 12.

The ROM 14B built in the extraction position calculation processor 14 stores in advance the position information indicating the respective arrangement positions of the television camera 16 and a control program for the extraction position calculation process described later.

Next, the operation of the first embodiment will be described. When the start button (not shown) of the sound extraction device 10 is turned on by the operator, the control routine of the extraction position calculation processing shown in FIG.
U14A causes the CPU 38 of the voice extraction board 12 to execute the control routine of the voice extraction process shown in FIG. Note that all of these control routines are repeatedly executed at predetermined time intervals.

First, the control routine of the extraction position calculation processing shown in FIG. 4 will be described. In step 102, shooting information from each TV camera 16 is fetched. Next Step 1
In 04, the position of the head P of the target person A (see FIG. 2) is calculated from the captured photographing information. In addition, as a position at this time, as shown in FIG.
Direction, arrow Y direction, arrow Z direction using information indicating in which region of the target person A is located among a large number of rectangular parallelepiped-shaped regions obtained by equally dividing You can In Figure 2, room 50 is 1 in each direction
The case of dividing into 6 equal parts is shown as an example. That is, step 1
In 04, from the photographed image, a region having a characteristic amount peculiar to the human head, such as a substantially spherical shape with many surfaces covered with hair and a black portion, is identified as the head of the target person A. It is extracted as a region corresponding to P, and the position of the head P on the virtual three-dimensional coordinates described above is calculated based on the position of the extracted region on the captured image.

In step 104, the orientation of the head P of the target person A is also estimated. That is, first, the head P shown in FIG.
Recognizing the torso S located below the human body S, and based on the general feature that the chest width L2 is smaller than the shoulder width L1 in the torso S, the target person A is determined from the chest width L2 and the shoulder width L1.
Is in the direction of arrow V or the opposite direction.
Next, based on the general characteristic that the proportion of hair on the surface of the head P is higher on the side where the face is not located than on the side where the face is located, the back side of the paper in FIG. 2 is the front side of the paper. Since the degree of black is higher than that of the target person A, it is estimated that the target person A faces the arrow V direction.

In the next step 106, step 104
The position separated by a predetermined distance (for example, about 30 cm) in the direction of arrow V from the position of the head P obtained in
Set as the extraction position for. Then, in the next step 108, the position information of the set extraction position is transmitted to the voice extraction board 12.

Next, the control routine of the voice extraction processing executed by the CPU 38 of the processor 34 provided in the voice extraction board 12 shown in FIG. 5 will be described. Step 20
At 0, it is judged whether or not the information on the extraction position transmitted from the extraction position calculation processor 14 in the above step 108 is received. If the extraction position information has not been received, the control routine is ended, and if the extraction position information has been received, the routine proceeds to step 202. In step 202, ROM4
Suitable for extracting the sound at the extraction position by excluding the microphones 22 installed at a position more than a predetermined distance from the extraction position based on the installation position information of each microphone 22 extracted from 0 and the received extraction position information Select the microphone 22

On the other hand, the sound emitted from the target person A is first captured by the sound collecting section 24 of the microphone 22, and further, the noise is cut by the amplifier filter 26 and is amplified by a predetermined amplification factor, and then the sound is amplified by a predetermined amplification factor as shown in FIG. The audio signal is as shown in. Then, those audio signals are sent to the A / D converter 28.
Is converted into digitized voice data by.

Then, in step 203 of the voice extraction processing, the voice data collected and converted as described above is taken in from each of the microphones 22 selected in step 202 via the digital line 30, and the voice data is collected. Write to the input buffer memory i corresponding to each microphone 22. That is, the audio data corresponding to the audio signal as shown in FIG. 1A is written in the input buffer memory i. At this time, data is sequentially written from a predetermined reference address of the input buffer memory i. Then, when the voice extraction processing routine is executed next time, a new reference address which is shifted from the reference address by a predetermined address is set, and the new reference address is written in order. When the writing to the input buffer memory i is completed three times, a new reference address is returned to the head address of the input buffer memory i at the next fourth time, and the audio data is written in order from the head address. Thus, the input buffer memory i is used as a so-called ring buffer.

In the next step 212, the delay time corresponding to the distance between the position of one microphone 22 of the selected microphones 22 and the extraction position is fetched from the delay table stored in advance in the ROM 40. It should be noted that the delay table indicates that for each extraction position of the extraction positions that can fluctuate within the room 50, the distance between the extraction position and each microphone 22 is divided by the sound velocity at the standard room temperature to obtain the sound It is a table in which the propagation time (delay time) is recorded, and is prepared in advance by the number of extraction position candidates that can vary within the range of the room 50.

In the next step 214, an address obtained by shifting the audio data from the one microphone 22 by the memory address corresponding to the delay time from the predetermined reference address (that is, the write start address to the input buffer memory i) is obtained. As the beginning of the extraction, it is extracted from the input buffer memory i. As a result, the sound data written in the input buffer memory i before the sound emitted by the target person A reaches the one microphone 22 is cut off, and the sound emitted by the target person A and reaching the one microphone 22 is cut off. Will be taken out.

Then, in the next step 216, the extracted voice data is written in the output buffer memory i corresponding to the one microphone 22. That is, the audio data corresponding to the audio signal as shown in FIG. 1B is written in the output buffer memory i. The output buffer memory i is also used as a so-called ring buffer, like the input buffer memory i.

Then, the above steps 212, 214,
216 is executed for all the selected microphones.
When the processes of steps 212, 214, and 216 have been executed for all the selected microphones, the determination at step 218 is affirmative, and the process proceeds to step 220, where the adder 46 adds the audio data corresponding to each of the selected microphones. Let

In the next step 222, the added voice data is output to the D / A converter 48 by shifting the decimal point position to the upper position by the number of digits of INT (log ₂ M). As a result, it is possible to obtain substantially the same result as when the added voice data is divided by the number M of microphones. Here, in addition to the above, the calculation result of the adder 46 is stored in the processor 34.
You can take in and perform normal division.

Thereafter, the voice data output from the adder 46 is converted into an analog voice signal as shown in FIG. 1C by the D / A converter 48, and the converted voice signal is output from the output terminal board 20. It is sent to the output terminal 21. By connecting an audio reproducing device or the like to the audio output terminal 21, the extracted voice of the target person A can be reproduced and heard.

As is clear from the above description, the delay operation and the addition averaging as described above are performed on the sounds collected by a plurality of (7 in the example of FIG. 1) microphones 22 to achieve the purpose. Since the noise components other than the voice of the target person A are extremely small in amplitude, only the voice of the target person A can be extracted.

The extraction position calculation process (FIG. 4) and the voice extraction process (FIG. 5) described above are repeatedly executed at predetermined time intervals. Accordingly, when the target person A moves, the inside of the room 50 is continuously photographed by the plurality of TV cameras 16, and the position of the head P that changes with the movement of the target person A based on the image information and The orientation is obtained, and the extraction position corresponding to the position and orientation of the head P at that time is set. Then, by performing the above-described voice extraction processing on the voice extraction board 12 according to the extraction position, even if the target person A moves, that voice can be extracted.

In the voice extraction processing of the first embodiment, the microphones close to the set extraction position (for example, 7 microphones) are selected, only the voice data from the selected microphones are fetched, and the input buffer memory is used. Although the example of writing the audio data from all (n) microphones is once taken and written to each input buffer memory, only the audio data from the selected microphones (for example, 7 microphones) is written.
It is also possible to shift the memory address corresponding to the delay time and fetch it from the input buffer memory.

Further, in the voice extraction processing of the present invention, the sound of the target person (or the target object) is collected by a large number of microphones arranged near the extraction position, and the collected voice signal is processed as described above. By performing the delay operation and the averaging, it is possible to extract the sound with the improved signal-to-noise ratio. Moreover, it is also possible to extract a sound having a higher signal-to-noise ratio than the sound collected by a normal microphone. Such a good sound,
It can be used as an input to a voice recognition device. That is, it is possible to input to the voice recognition device the voice spoken by a person (one or more people) in the area where the sound can be extracted by the sound extraction device.

[Second Embodiment] Next, a second embodiment of the present invention will be described. The second embodiment shows an example in which the voice of the target person A and the voice of the target person B in the predetermined room 50 shown in FIG. 2 are separately extracted. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

As shown in FIG. 6, the sound extraction device 10 according to the second embodiment is provided with a plurality (N) of the sound extraction boards 12 described in the first embodiment. A voice data relay board 56 for connecting the microphone 22 and each voice extraction board 12 is installed. In addition, the extraction position calculation processor 14 uses the sound extraction boards 1
2 is connected to the processor 34. Further, the output terminal board 20 is provided with a sound output terminal 21 corresponding to each sound extraction board 12, and each sound output terminal 21 is connected to the D / A converter 48 of the corresponding sound extraction board 12.

Next, the operation of the second embodiment will be described.
When the start button (not shown) of the sound extraction device 10 is turned on by the operator, the control routine of the extraction position calculation processing shown in FIG. 7 for the plurality of extraction positions is shown by the CPU 14A of the extraction position calculation processor 14 in FIG. The same control routine of the voice extraction processing as that of the first embodiment is executed by each CPU 38 of the two voice extraction boards 12.
Executed respectively.

The control routine of the extraction position calculation processing shown in FIG. 7 will be described. In the following description, the target person A,
For the sake of convenience, B is referred to as target persons 1 and 2, respectively. In step 102, the shooting information from each TV camera 16 is fetched, and in the next step 103, the number of target persons is “2”.
Is substituted into the variable K and the variable L is initialized to "1".

In the next step 105, the calculation of the position of the head of the target person L (that is, the target person 1) and the estimation of the direction are performed in the same manner as in the first embodiment, and the next step 107
Then, the extraction position L for extracting the voice of the target person L
(Namely, extraction position 1) is set. Then, in the next step 109, the information of the extraction position L is transmitted to the corresponding voice extraction board L.

In the next step 110, it is judged whether or not the variable L is equal to the variable K indicating the number of target persons.
It is determined whether or not the processes of 7 and 109 are completed. In this case, it is initially denied, and the routine proceeds to step 112, where the variable L is incremented by one. As a result, the value of the variable L becomes "2".

Thereafter, the process returns to step 105 and the target person L
(That is, the target person 2), the above step 105,
The processing of 107 and 109 is performed. When these processes are completed, in step 110, the variable L and the variable K are equal and the determination is affirmative, and the control routine ends.

The voice extraction boards 12 corresponding to the target persons 1 and 2 respectively receive the information of the extraction position 1 or the extraction position 2 transmitted from the extraction position calculation processor 14 in the above step 109, and use the received information as the received information. Based on this, the voice extraction processing shown in FIG. 5, which is the same as in the first embodiment, is executed. Although description is omitted, the voices of the target persons 1 and 2 can be independently extracted by the voice extraction processing in the voice extraction boards 12 corresponding to the target persons 1 and 2, respectively.

In the second embodiment, an example in which a plurality of voice extraction boards 12 are provided and sound is extracted from one extraction position by each voice extraction board 12 is shown, but the immediacy of voice extraction is not so great. When the demand is not high, the voice extraction processing may be sequentially executed on each of the plurality of extraction positions on the single voice extraction board 12.

[Third Embodiment] Next, a third embodiment of the present invention will be described. In the third embodiment, the room 5 is considered in consideration of the influence of the temperature change in the room 50 shown in FIG.
An example of extracting only the voice of the target person A in 0 will be shown.
The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

As shown in FIG. 8, the sound extraction device 10 according to the third embodiment includes a plurality of temperature sensors 58, and one temperature sensor 58 is provided at each of a plurality of temperature measurement points in the room 50. Are installed one by one. Each temperature sensor 58 is connected to the I / O 36 in the processor 34. Also,
The ROM 40 in the processor 34 stores in advance temperature distribution information for estimating the temperature distribution in the room 50 based on the temperatures at the plurality of temperature measurement points in the room 50 measured by the temperature sensor 58.

Next, the operation of the third embodiment will be described.
When the start button (not shown) of the sound extraction device 10 is turned on by the operator, the same control routine of the extraction position calculation processing as that of the first embodiment shown in FIG. 4 is shown in FIG. 9 by the CPU 14A of the extraction position calculation processor 14. The control routine of the voice extraction processing is the CPU of the voice extraction board 12.
38, respectively. Hereinafter, the description of the extraction position calculation process will be omitted, and the voice extraction process in the third embodiment will be described with reference to FIG. 9.

At step 203, the selected microphone 22
For each of the above, the voice data is captured from the microphone 22, and the input buffer memory i of the captured voice data is input.
And write to, and in the next step 204, for one microphone 22 of the selected microphones 22,
The distance between the microphone 22 and the extraction position is calculated.

In the next step 205, a plurality of temperature sensors 5
The temperature at a predetermined temperature measurement point of the room 50 is fetched from each of the eight, and in the next step 206, the temperature distribution information stored in the ROM 40 is referred to based on the fetched temperatures of the plurality of temperature measurement points. The temperature distribution in the room 50 is estimated, and the average temperature on the sound propagation path until the sound emitted from the extraction position reaches the microphone 22 is calculated.

In the next step 207, the sound velocity on the sound propagation path is calculated based on the average temperature on the sound propagation path, and in the next step 208, the distance between the microphone 22 calculated in step 204 and the extraction position is calculated. , The propagation time of the sound that reaches the microphone 22, that is, the delay time for the microphone 22 is calculated by dividing by the sound velocity calculated in step 207. Then, in the next step 209, the calculated delay time is associated with the identification number of the microphone 22 and RA
Store in the delay table secured in M42. The delay table in the third embodiment is used as a temporary storage area for temporarily storing the calculated delay time for each microphone 22.

The above steps 204 to 209 are executed for each of the selected microphones 22. When the execution is completed for all the selected microphones 22, the delay table in which the delay time for each of the selected microphones 22 is recorded is completed. Then, as in the first embodiment,
In step 214, the voice data from one microphone 22 is shifted from the input buffer memory i by shifting the memory address corresponding to the delay time for the microphone 22 obtained from the delay table. In the next step 216, the extracted audio data is written in the output buffer memory i.

When the processing of these steps 214 and 216 has been completed for all the selected microphones 22, the affirmative answer is obtained in step 218 and the operation proceeds to step 220. In steps 220 and 222, the selected microphone 22
The audio data in each of the above are averaged and output to the D / A converter 48. The voice data is converted into an analog voice signal by the D / A converter 48, and the converted voice signal is output to the voice output terminal 21 of the output terminal board 20.

As described above, according to the third embodiment, it is possible to extract the sound with high accuracy according to the change in the temperature in the room 50.

The sound extraction device 10 of the present invention can extract a sound in the same manner as described above in consideration of the bending of the sound propagation path due to the influence of the wind (wind direction, wind force). For example, as shown in FIG. 10, a train 6 traveling in the direction of arrow R
When the 4 crosses the iron bridge 66, a specific measurement portion 66 of the iron bridge 66
The case of extracting the squeaking sound emitted by A will be described.
In this case, because of the outdoor acoustic environment, the influence of wind in addition to temperature affects sound propagation. For example, the sound propagation path emitted from the measurement site 66A of the iron bridge 66 and reaching one microphone 22A is not the straight path indicated by the broken line K1 but the curved path indicated by the solid line K2, and the sound propagation path length L1 (the curved path The length) is longer than the distance L2 (the length of the straight path) between the measurement site 66A and the microphone 22A. Therefore, in the sound extraction device 10, the wind force is measured by the anemometer 60 and the wind direction meter 62 is used.
To detect the wind direction. And what kind of path (curved path) the sound propagation path is due to the influence of wind force and wind direction
Then, it is calculated how much the propagation path length L1 becomes longer than the distance L2 by the extraction position calculation processor 14 or the processor 34 of the voice extraction board 12, and the microphone is calculated based on the calculated propagation path length L1. The delay time at 22A is calculated. Similarly, for the other microphones 22, the sound propagation path length is obtained and the delay time is calculated. And
Based on the calculated delay time, the subsequent delay operation and arithmetic averaging are performed to extract the sound emitted from the measurement site 66A. In this way, the sound can be extracted in consideration of the bending of the sound propagation path due to the influence of the wind (wind direction, wind force).

[Fourth Embodiment] Next, a fourth embodiment of the present invention will be described. The fourth embodiment shows an example in which the voice of the target person C in the room 50 shown in FIG. 11 is extracted in consideration of the difference in directivity depending on the frequency of the voice. The same parts as those in the first embodiment are designated by the same reference numerals and the description thereof will be omitted.

The sound extraction device 10 according to the fourth embodiment.
Since the configuration is the same as the configuration of the sound extraction device 10 in the first embodiment described above, the description thereof will be omitted. However, the ROM 4 in the processor 34 of the voice extraction board 12
In 0, a weighting table in which weighting constants described later are recorded is stored in advance.

Next, the operation of the fourth embodiment will be described.
First, the difference in directivity depending on the frequency of sound will be described. As shown in FIG. 11, the directivity of sound varies depending on the frequency. Generally, the directivity is lower at lower frequencies and stronger at higher frequencies. Therefore, the microphones located in the direction D in which the target person C speaks collects sounds of frequencies in almost all ranges from low frequencies to high frequencies, whereas other microphones collect sounds of low frequencies. However, high frequency sounds are not collected much.

Therefore, in the fourth embodiment, the volume of the high range of the collected sound collected by the microphone 22 located in the direction D and the high range of the collected sound collected by the other microphone 22. In order to correct the imbalance between the volume and the volume, an example in which the above problem is solved by performing a weighting operation on both is shown.

Since the extraction position calculation process is the same as that of the first embodiment, the description thereof will be omitted, and the voice extraction process will be described with reference to FIG.

In steps 200, 202 and 203, the microphone is selected based on the extraction position information received from the extraction position calculation processor 14 as in the first embodiment, and the voice data is taken in from the selected microphone and the voice is extracted. Data is written in the input buffer memory i. In the next step 213, the delay time is fetched from the delay table corresponding to the relative position of the extraction position with respect to one microphone 22, and weighted from the weighting table corresponding to the relative position of the extraction position with respect to the microphone 22 and the sound direction. Take in a constant. The direction D in which the person C speaks
The weighting constant corresponding to the microphone 22 located at is set to a value relatively smaller than the weighting constant corresponding to the microphone 22 located at a position deviated from the direction D.

In the next step 214, the voice data from the microphone 22 is shifted from the input buffer memory i by shifting the memory address corresponding to the delay time, and is fetched in the next step 217, as in the first embodiment. The high frequency components in the audio data are weighted (amplified or reduced in level) according to the weighting constants and written in the output buffer memory i.

The above steps 213, 214 and 217 are
It is executed for each of the selected microphones 22. Thus, the high-frequency component of the collected sound collected by the microphone 22 located in the direction D is reduced in level, while the high-frequency component of the collected sound collected by the microphone 22 located outside the direction D is reduced. Is amplified in level.

In the next steps 220 and 222, the voice data in each of the selected microphones 22 is added and averaged to obtain D
It outputs to the / A converter 48. Audio data is D / A
Converted into an analog audio signal by the converter 48,
The converted audio signal is output to the audio output terminal 21 of the output terminal board 20.

According to the fourth embodiment, the high frequency component of the collected sound collected by the microphone located in the direction D,
The level imbalance between the high frequency component of the collected sound collected by the microphone 22 located at a position deviated from the direction D and the high frequency component of the high frequency sound due to the strong directivity of the high frequency sound is improved. It is possible to prevent a relative decrease in level for low-frequency sounds.

In the above first to fourth embodiments,
An example is shown in which only the direct sound emitted from the target person (or the target object) and directly reaching the microphone is extracted. In general, the reflected sound that reaches the microphone after being reflected from the wall surface or the like as the reflecting surface is much smaller in size than the direct sound, and is therefore removed together with other noise components by averaging.

However, when the wall surface is close to the target person and the target person is located in the direction in which the sound is emitted, the reflected sound on the wall surface becomes louder than the direct sound.
Rather, it can be said that the effect of extracting the sound emitted by the target person is higher when the reflected sound is collected.

Therefore, when it is recognized that the wall surface is close to the target person and the target person is located in the direction of producing sound based on the image information captured by the television camera 16, the processor 34 In the voice extraction processing executed by the CPU 38, the propagation distance of the reflected sound reflected by the wall surface is adopted as the distance between the microphone and the extraction position for calculating the delay time for each microphone, not the direct distance between them. The delay time may be calculated according to the propagation distance of the reflected sound, and the delay operation may be performed according to the delay time according to the propagation distance of the reflected sound.

As a result, the sound that directly reaches each microphone from the target person is removed as a noise component, and the reflected sound that reaches each microphone after being reflected by the wall surface is extracted as the sound of the target person. Thus, when the reflected sound reaching each microphone is more suitable for extracting the sound of the target person (object) than the direct sound, the reflected sound can be extracted.

The sound extraction device of the present invention can be applied as follows in addition to the various embodiments described above. For example, in the case of expanding the voice of a questioner in the audience at a lecture hall, the audience is photographed with a plurality of TV cameras, and a staff member uses a mouse or the like in the vicinity of the questioner's mouth on the screen showing the questioner. When pointed, the extraction position calculation processor sets the vicinity of the questioner's mouth to the extraction position. And
Extract the sound from the extraction position with the voice extraction board,
The sound extracted from a predetermined speaker is output. This eliminates the need to bring the microphone to the position of the questioner in the audience each time, which helps to facilitate a smooth lecture.

Further, in the case of continuously extracting (tracing) the sound emitted from a moving body whose moving route is fixed, such as a train 64 shown in FIG. 10, at substantially equal intervals on the moving route. It suffices to set a plurality of extraction positions (for example, iron bridge parts 66B, 66C, 66D) in advance and sequentially extract sounds at these extraction positions along the passage of time. This eliminates the need for the process of grasping the movement of the moving body from the image captured by the television camera 16 to set the extraction position, and the sound can be traced following the rapid movement of the moving body.

[Fifth Embodiment] Next, a fifth embodiment of the present invention will be described. In the fifth embodiment, in extracting the sound of the object, an example in which an image including the object is captured by a plurality of television cameras having a wide-angle fixed focus lens, and the position of the object is recognized based on the image data Indicates.

As shown in FIG. 13, a plurality of TV cameras 16 (four as an example) are installed on the ceiling 52, and a fish-eye lens 16A as a wide-angle fixed focus lens is installed on each TV camera 16. ing. The viewing angle of each fisheye lens 16A is preset to 90 ° or more. Therefore, regardless of whether the object is moving or stationary, the object can be photographed without moving the television camera 16.

The fish-eye lens may be, for example, equidistant projection.
There are various types such as n) type, stereoscopic projection type, equisolid angle projection type, orthographic projection type, and any fisheye lens can be used in the present embodiment. However, in the following, the equidistant projection type fisheye lens is used. An example will be described. Also, each TV camera 16 is CC
A D- (Charge-Coupled Device) area image sensor 16B (see FIG. 18) is provided. In addition, since the heights of objects, people, animals, and the like from the floor and the ground are almost fixed, and the fish-eye lens 16A as a wide-angle fixed focus lens has a large depth of focus. Even if does not have a focus adjusting mechanism, it is possible to clearly form an object image on the CCD area image sensor 16B. In this way, each of the plurality of television cameras 16 shoots a predetermined area including the object from different positions.

Next, the operation of the fifth embodiment will be described. When the operator designates the target person A as an object and turns on a start button (not shown) of the sound extraction device 10, the control routine of the voice extraction processing shown in FIG. CP
At the same time the execution is started by U38, the control routine of the extraction position calculation processing shown in FIG. 14 is started by the CPU 14A. In the following, the description of the voice extraction process is omitted,
The extraction position calculation process in the fifth embodiment will be described with reference to FIGS.

In step 120 shown in FIG. 14, object classification processing is performed. In this object classification processing, the subroutine shown in FIG. 15 is executed. FIG.
In step 140 of the above, the image data A when the object (target person A) does not exist in the room 50 is stored in the ROM 1
4B, and in the next step 142, the image data B taken by each television camera 16 is fetched and stored in the RAM 14C. In the next step 144, the target person A present in the room 50 is recognized by taking the difference between the image data B and the image data A (see FIG. 17).

Next, in step 146, a timer of a predetermined time T is set, and in the next step 148, the process waits for the predetermined time T, and when a time-out occurs, step 150
Go to.

At step 150, each television camera 16
The image data C captured in (i.e., the image data after a predetermined time T has elapsed from the image data B) is captured. Then, in the next step 152, the image data B stored in the RAM 14C is read, the image data B and the image data C are compared, and in the next step 154, whether or not the target person A is moving based on the comparison result. To judge.

If the target person A is not moving (still), a negative decision is made in step 154 and the process returns to the main routine of FIG. On the other hand, if the target person A is moving, an affirmative decision is made in step 154 and the operation proceeds to step 156, in which the traveling direction of the target person A is obtained from the difference between the image data B and the image data C (see FIG. 17), The front and back of the target person A are determined from the traveling direction. And
In the next step 158, the information regarding the traveling direction and the front and back of the target person A is stored in the RAM 14C, and the process returns to the main routine of FIG.

At the next step 122, the position and height of the target person A are calculated. As shown in FIG. 18, the focal length of the equidistant projection type fisheye lens 16A fixed to the point O is f, and the point Q is the point O lowered from the point O vertically to the floor surface 54 of the room 50.
To H, the distance from the point Q to the point P on the floor 54 of the target person A is R, and the height of the target person A (when the tip of the target person A in the ceiling direction is point P ′, The distance between P'and the point P) is h. Further, the angle formed by the point POQ is θ, and the point P ′ is
The angle formed by OQ is θ ', CCD area image sensor 16
The distance corresponding to the height of the object image on the CCD surface of B is h ′, the point of the object image h ′ formed corresponding to the point P is p, and the point formed of the object image h ′ is the point P ′. If the point formed by the image formation is p ′, the distance from the image center of the CCD surface (center of the CCD surface) o to the point p is r, and the distance from the image center o of the CCD surface to the point p ′ is r ′, the angle is θ, θ ',
The distances r and r'can be calculated by the following equations (1) to (4).

Θ = tan ⁻¹ (R / H) (1) θ ′ = tan ⁻¹ {R / (H−h)} (2) r = fθ .. (3) r '= f?' (4) Therefore, the height h and the distance R can be obtained by the following equations (5) and (6).

H = H {1-tan (r / f) / tan (r ′ / f)} (5) R = Htan (r / f) (6) Note that the distance H And the focal length f are predetermined, and the equations (5) and (6) are stored in the ROM 14B.
Therefore, in this step 122, the equation (5) is stored in the ROM.
CCD of one TV camera 16 read from 14B
The height h is calculated from the information on the surface, the formula (6) is read out, the distance R is calculated from the information on the CCD surfaces of the two TV cameras 16, and the two distances R of the target person A are calculated from the calculated two distances R. Calculate the dimensional position.

In the next step 124, the above step 1
In the X direction in the three-dimensional space with the position calculated in 22 as the center,
A matrix-shaped minute space (hereinafter referred to as a voxel) that is virtually subdivided along the Y direction and the Z direction is set.
As a result, the image data C is converted into a set of voxels. FIG. 19 conceptually shows voxels occupied by the target person A when the target person A is projected from the four television cameras A, B, C, and D.

That is, the voxels located within the viewing angle of the target person A when the target person A is projected from each TV camera include the shadow (blind spot) portions R _A , R _B , R _C , and R _D. Are set as voxels occupied by the target person A.
The voxels are the CCD area image sensor 16B.
It is possible to subdivide to the resolution limit of.

In the next step 126, the primary narrowing is performed in which the voxels occupied by the target person A in the image data are limited based on the height h of the target person A as follows.

The height h of the target person A can be set in advance from the average height of an adult, so that FIG.
As shown in (D), when the target person A is projected from each TV camera, among the voxels positioned within the viewing angle of the target person A, those having a height range of 0 to h are Filter as occupied voxels. The region formed by the voxels narrowed down here is referred to as a primary narrowed region.

Next, at step 128, the secondary narrowing-down is performed from the primary narrowing-down area in each image data to the area overlapping all of them. As a result, FIG.
Region R _A shadow shown in 9, R _B, R _C and R _D are excluded from voxels target person A is occupied, as shown in FIG. 21, it is narrowed to a voxel 70 occupied by the target person A. In the next step 130, the voxel 70 accurately recognizes the position and shape of the object. Since the voxels can be subdivided to the resolution limit of the CCD area image sensor 16B, the shape of the object can be recognized in detail.

In the next step 132, as shown in FIG. 22, dimensions such as the height and thickness of the voxel 70 and RO beforehand are set.
Information about human characteristics such as the color difference of the head, the positions of the eyes, the nose, the mouth, the ears, the length and position of the arms, the direction of the toes, and the degrees of freedom of the joints, which are stored in the M14B, and the target person A moves. If the dummy model 72 is stored in the RAM 14C, the dummy model 72
Convert to.

At the next step 134, the extraction position setting process subroutine shown in FIG. 16 is executed. In step 160 of FIG. 16, a predetermined number (two as an example) of TV cameras whose target is the head of the target person A is selected, and the head of the target person A on the CCD surface of each selected TV camera is selected. Take in the two-dimensional coordinates corresponding to the position.
In selecting the television camera, for example, the object image when the target person A is photographed may be selected in descending order, or the television camera that captures the front of the target person A may be selected. The two selected TV cameras are referred to as camera L and camera R, respectively.

In the next step 162, three-dimensional coordinates are calculated. As shown in FIG. 23, the three-dimensional coordinate C of the camera L is (X, 0, Z) and the three-dimensional coordinate C ′ of the camera R is (X ′,
0, Z). Further, the coordinate P _L on the CCD surface of the camera L corresponding to the position of the head of the target person A is (α ₁ ,
β ₁ ), coordinates P from the image center O _{L on} the CCD surface of the camera L
The distance to _L is r, and the coordinate P _R on the CCD surface of the camera R corresponding to the position of the head of the target person A is (α ₁ ',
beta ₁ '), the distance from the image center O _R of the CCD of the camera R to the coordinates P _R r', that is two light intersect when virtual light emitted from the coordinate P _L and coordinates P _R, i.e. , The three-dimensional coordinate P of the head of the target person A is (x, y, z).

Further, the coordinates of the intersection point S of the perpendicular leg drawn parallel to the Z-axis from the three-dimensional coordinate position of the camera L and the plane including the point P and perpendicular to the Z-axis are (X, 0, z). And the coordinates of the intersection S'of the perpendicular leg dropped from the three-dimensional coordinate position of the camera R parallel to the Z axis and the plane including the point P and perpendicular to the Z axis are (X ', 0, z). To do. Further, the angle formed by the point PCS is θ ₁ , the angle formed by the point PC ′S ′ is θ ₁ ′, the angle formed by the point PSS ′ is φ, and the angle formed by the point PS ′S is φ ′.

The distance r from the image center O _L to the image on the CCD surface is obtained by the above equation (3) as r = fθ ₁ .

[0160] In addition, each α _1, β _{1 is, α 1 = fθ 1 cos (} π-φ) = - fθ 1 cosφ β 1 = fθ 1 sin (π-φ) = fθ 1 sinφ ··· (7) Is. Here, sin φ = y / {(x−X) ² + y ² } ^1/2 (8) cos φ = (x−X) / {(x−X) ² + y ² } ^1/2 , Α ₁ and β ₁ are α ₁ = −fθ ₁ (x−X) / {(x−X) ² + y ² } ^1/2 ... (9) β ₁ = fθ ₁ y / {(x -X) ² + y ² } ^1/2 ... (10) By dividing Expression (10) by Expression (9), y = (β ₁ / α ₁ ) (X−x) (11) Similarly, y = (β ₁ ′ / α ₁ ′) (X '−x) (12) Eliminating y from equation (11) and equation (12), x = (α ₁ β ₁ 'X'-α ₁ 'β ₁ X) / (α ₁ β _{1 '-α 1' β 1)} by (13) can be determined X-coordinate of the three-dimensional coordinates P.

Next, x is eliminated from the equations (11) and (13), and y = β ₁ β ₁ ′ (X−X ′) / (α ₁ β ₁ ′ −α ₁ ′ β ₁ ). The Y coordinate of the three-dimensional coordinate P can be calculated by (14).

By the way, since θ ₁ = tan ⁻¹ [{(x−X) ² + y ² } ^1/2 / (Z−z)], β ₁ / (from the formulas (7) and (8) fsin φ) = tan ⁻¹ [{(x−X) ² + y ² } ^1/2 / (Z−z)] Therefore, z = Z − [{(x−X) ² + y ² } ^1/2 / tan [ (Β ₁ / f) × {(x−X) ² + y ² } ^1/2 / y] (15) Further, from the formula (11), {(x−X) ² + y ² } ^1/2 = (X−X) × {1+ (β ₁ / α ₁ ) ² } ^{1/2 From} formula (11) and formula (14), (x−X) = (X′−X) / {1− (α ₁ Since '/ α ₁ ) × (β ₁ / β ₁ ')}, the equation (15) is expressed by z = Z + [(X′−X) × {1+ (β ₁ / α ₁ ) ² } ^1/2 / {1- (α ₁ '/ α ₁ ) × (β ₁ / β ₁ ')}] / tan {(α ₁ ² + β ₁ ² ) ^1/2 / f} (16) Yes, three-dimensional coordinates The Z coordinate of P can be obtained.

Since the three-dimensional coordinates of each television camera 16 are predetermined, the ROM is determined in step 162.
Equations (13), (14) and (16) are read from 14B, and the coordinates P _L (α ₁ , β ₁ ) on the CCD surface of the camera L and the coordinates P _R on the CCD surface of the camera R read in step 160 are read. The value of (α ₁ ', β ₁ ') is calculated by using equations (13) and (1
By substituting in 4) and (16), the three-dimensional coordinates P (x, y, z) of the head of the target person A can be obtained.

At the next step 164, the orientation of the head of the target person A is estimated as in the first embodiment described above (similar to the processing at step 104 in FIG. 4). In the next step 166, a predetermined distance (eg, about 3) from the position of the head obtained in step 162 in the direction of arrow V (see FIG. 13).
A position separated by 0 cm) is set as an extraction position for the target person A. And the next step 168
Then, the position information of the set extraction position is transmitted to the voice extraction board 12 and the process returns.

As described above, according to the fifth embodiment, since the wide-angle fixed focus lens 16A is used for photographing, it is not necessary to move the television camera 16 or adjust the focus. Therefore, it is possible to shorten the time until the object (target person A) is captured, and it is possible to quickly recognize the position of the object.

Further, since the mechanism for changing the direction of the television camera and adjusting the focus is unnecessary, the work for capturing the object can be automated and the driving part is eliminated, so that the durability of the television camera can be improved. The reliability can be increased.

Also, since one object is photographed by a plurality of television cameras, the three-dimensional coordinates can be calculated even if there are obstacles such as furniture that obstruct the visual field and other objects. it can.

Further, since the television camera is arranged on the ceiling of the room forming the three-dimensional space, the wall surface can be effectively used.

In the fifth embodiment, the plurality of television cameras 16 are arranged on the ceiling 52, but FIGS.
As shown in (F), it may be arranged near the wall or embedded in the wall, and a two-sided corner portion composed of the ceiling and the wall, or three surfaces composed of the ceiling and the two-sided wall. You may arrange in the corner part of. Further, as shown in (M) to (O) of FIG. 24, the equidistant projection type fisheye lens 16A may be directed to the center of the room.

Further, in the fifth embodiment, the image data A when the object does not exist in the room 50 is read in step 140 in the object classification processing shown in FIG. 15, but this step 140 is not performed. ,
Based on the image data B captured by the television camera 16 and the image data C after a predetermined time T has elapsed from the image data B,
You may make it recognize an object.

In addition, in the fifth embodiment, two television cameras are used to shoot an image including an object.
You may use three or more TV cameras.

Further, in the fifth embodiment, the equidistant projection type fisheye lens is used. However, as described above, the equisolid angle projection type fisheye lens, the stereoscopic projection type fisheye lens and the orthographic projection type fisheye lens are also used. Similarly, the three-dimensional coordinates of the head of the target person A can be calculated. The following equations (1) ′ to (6) ′ are equations corresponding to the equations (1) to (6) when the equisolid angle projection type fisheye lens is used.

Θ = tan ⁻¹ (R / H) (1) ′ θ ′ = tan ⁻¹ {R / (H−h)} (2) ′ r = 2fsin ( θ / 2) (3) 'r' = 2 fsin (θ '/ 2) ... (4)' h = H [1-tan {2 sin ^-1 (r / 2f)} / tan {2sin ⁻¹ (r ′ / 2f)}] (5) ′ R = Htan {2sin ⁻¹ (r / 2f)} (6) ′ [Sixth embodiment] Next, a sixth embodiment according to the present invention will be described. In the sixth embodiment, one TV camera and one TV camera are used to extract the sound of the object.
An example in which the three-dimensional coordinates of the object are calculated based on the image data including the object obtained by using the mirror and the position of the object is recognized will be described. Note that the sixth embodiment is substantially the same as the fifth embodiment, so that FIG.
Throughout FIG. 16, the same parts are designated by the same reference numerals,
Description is omitted.

As shown in FIG. 25, each television camera 16
1 side of the CCD area image sensor 16B
A vertically long mirror 74 is fixed to the ceiling 52 in the vertical direction (Z direction) in parallel with the direction of the end surface (X direction).

Next, various quantities such as the position, distance and angle of the equidistant projection type fisheye lens 16A, the CCD area image sensor 16B and the mirror 74 of the sixth embodiment are shown in FIGS.
Will be described with reference to. Note that FIG. 26 shows the details of the above-mentioned various amounts when the distance between the equidistant projection type fisheye lens 16A and the CCD area image sensor 16B is negligible.

As shown in FIG. 25, the center of the upper end of the mirror 74 on the same XY plane as the CCD surface of the CCD area image sensor 16B is located at the origin O (0, 0, 0, 0, 0, 0, 0) of the three-dimensional coordinate.
Take 0). The image center H of the CCD surface is separated from the origin O in the Y direction by a distance h, and the three-dimensional coordinate of the image center H is set to (0, h, 0). In addition, the three-dimensional coordinates of a predetermined part (for example, head) P of the target person A are (x, y, z),
The light emitted from the point P is refracted by the equidistant projection type fisheye lens 16A and forms an image at the point D on the CCD surface. The two-dimensional coordinates of the point D on the CCD surface are (α _D , β _D ). Further, the light emitted from the point P and reflected by the mirror 74 is refracted by the equidistant projection type fisheye lens 16A and imaged at the point R on the CCD surface.
The two-dimensional coordinates of the point R on the CCD surface are (α _R , β _R ). The virtual TV camera 1 without the mirror 74
7 is assumed and the three-dimensional coordinate of the image center H'of the CCD surface is (0, -h, 0), the light emitted from the point P is refracted by the virtual equidistant projection fisheye lens 17A. Virtual C
Point R'on the CCD surface of the CD area image sensor 17B
It is assumed that the image is formed on the image, and the point R and the virtual point R ′ described above are symmetrical with respect to the mirror 74. Also CCD
The distance from the image center H on the surface to the point D is r _D , and the distance from the image center H on the CCD surface to the point R is r _R.

As shown in FIG. 26, an arbitrary point on the perpendicular drawn from the point H in the Z direction is defined as a point V, and an arbitrary point on the perpendicular drawn from the point H ′ in the Z direction to the point V. The angle formed by the point PHV is the angle θ _D , and the angle formed by the point PH'V 'is the angle θ _R' . Also, three-dimensional coordinates (x, y, 0) point S to the point represented by the distance the distance B _R between points S and H, 'distance the distance B _R of _the' points S and H, the point The distance between P and the point H is defined as a distance A _D , and the distance between the point P and a point H ′ is defined as a distance A _{R ′} .

Next, the operation of the sixth embodiment will be described. In step 160 in the extraction position setting process shown in FIG. 16, one TV camera 16 for shooting the target person A is selected (for example, the TV camera with the smallest distance r _D is selected), and the head of the target person A is selected. Point D (α _D , β _D ) and point R (α _R , β on the CCD surface corresponding to the position
Each two-dimensional coordinate value of _R ) is taken in.

In the next step 162, three-dimensional coordinates are calculated. Here, the various quantities described above will be further described with reference to FIGS.

The angles θ _D and θ _{R ′} are respectively θ _D = tan ⁻¹ (B _D / Q) = tan ⁻¹ [{(y−h) ² + x ² } ^1/2 / z] θ _{R ′} = tan ⁻¹ (BR _′ / Q) = tan ⁻¹ [{(y + h) ² + x ² } ^1/2 / z], the distances r _D and r _R can be calculated from the above equation (3) as follows. It is represented by a formula.

R _D = f · tan ^-1 [{(y-h) ² + x ² } ^1/2 / z] r _R = f · tan ^-1 [{(y + h) ² + x ² } ^1/2 / z _{Incidentally, α D = r D cos (} π-φ D) = - r D cosφ D ··· (17) β D = r D sin (π-φ D) = r D sinφ D ··· (18) α _R = r _R cos φ _{R '} (∵ φ _R' = φ _R ) ・ ・ ・ (19) β _R = r _R sin φ _{R '} (∵ φ _{R '} = φ _R ) ・ ・ ・ (20) Also, cos φ _D = (Y−h) / {(y−h) ² + x ² } ^1/2 ... (21) sin φ _D = x / {(y−h) ² + x ² } ^1/2 ... (22) cos φ _{R '} = (y + h) / {(y + h) ² + x ² } ^1/2 ... (23) sin φ _R' = x / {(y + h) ² + x ² } ^1/2 ... (24) Therefore, equation (17) and equation (21) and equation (1
8) and equation (22), α _D = −fθ _D (y−h) / {(y−h) ² + x ² } ^1/2 ... (25) β _D = fθ _D x / {(y -H) ² + x ² } ^1/2 ... (26) Eliminating fθ _D from these two equations, y = h− (α _D / β _D ) x (27) Similarly, α _R = fθ _{R ′} (y + h) / {(y + h) ² + x ² } _{^{1/2 ··· (28) β R =}} fθ R 'x / {(y + h) 2 + x 2} 1/2 ··· (29) y = -h + (α R / β R) x ··· ( 30) (27) and (can be obtained X coordinate of from _{30) x = 2hβ D β R} / (α D β R + α R β D) 3 -dimensional coordinate P by ... (31).

Next, by substituting the equation (31) into the equation (27), y = h (α _R β _D −α _D β _R ) / (α _D β _R + α _R β _D ) (32) ), The Y coordinate of the three-dimensional coordinate P can be obtained.

Β _D = r _D sin φ _D = f θ _D sin φ _D = f · tan ⁻¹ [{(y−h) ² + x ² } ^1/2 / z] · sin φ _D z = {(y-h) 2 + x 2} 1/2 / tan (β D / fsinφ D) = {(y-h) 2 + x 2} 1/2 / tan [(β D / f) × {( y−h) ² + x ² } ^1/2 / x] By the way, from the formula (31) and the formula (32), {(y−h) ² + x ² } ^1/2 = 2hβ _R (α _D ² + β _D ² ) ^1/2 / (α _D β _R + α _R β _D ), so z = 2hβ _R (α _D ² + β _D ² ) ^1/2 / [(α _D β _R + α _R β _D ) × tan {( α _D ² + β _D ² ) ^1/2 / f}] ... (33) The Z coordinate of the three-dimensional coordinate P can be obtained.

Incidentally, the image center H from the mirror 74 on the CCD surface
The distance h up to is predetermined. Therefore, in step 162, the equations (31), (32), and (33) are read from the ROM 14B and the C read in in step 160 is read.
Point D (α _D , β _D ) and point R (α _R , β _R ) on the CD surface
Substituting each two-dimensional coordinate value of
The dimensional coordinate P (x, y, z) is calculated.

As described above, according to the sixth embodiment, 1
Since the three-dimensional coordinates of the head of the target person A can be calculated by one TV camera, the number of TV cameras installed on the ceiling 52 can be reduced.

In the sixth embodiment, one television camera and one mirror installed on the ceiling 52 are used to calculate the three-dimensional coordinates of the object.
As shown in (G) to (L), the mirror may be attached to the wall surface, or one TV camera and a plurality of mirrors may be used. Also, curved mirrors may be used. When a plurality of mirrors are used, more object images are formed on the CCD surface, so even if a blind spot occurs due to another object (furniture, pillar, etc.)
Dimensional coordinates can be calculated.

[Seventh Embodiment] Next, a seventh embodiment according to the present invention will be described. In the seventh embodiment, an example of recognizing the shape of an object without setting voxels when extracting the sound of the object will be shown. Since the seventh embodiment is substantially the same as the fifth embodiment, the same parts in FIGS. 13 and 16 are designated by the same reference numerals and the description thereof will be omitted. In addition, in the seventh embodiment, in order to simplify the description, it is assumed that the target person A is captured by the television cameras A, B, C, and D as illustrated in FIG. 28.

The extraction position arithmetic processor 14 in the seventh embodiment converts the distorted image data including the object image into flattened image data, and based on the converted image data, at least the front of the object image,
It has a function of obtaining image data of the back surface, the left side surface, the right side surface, and the plane, and synthesizing the obtained image data to recognize an object.

Next, the operation of the seventh embodiment will be described. At step 121 in the extraction position calculation process shown in FIG. 27, the image data captured by the television cameras A, B, C and D is captured. The image of the image data captured in step 121 is distorted as shown in FIGS. 29 (A) to (D). In the next step 123, the image data of these distorted images are converted into flattened image data to obtain image data as shown in FIGS.

At the next step 125, the image data of the front face, the back face, the left side face, the right side face and the plane of the target person A is obtained from the flattened image data. 31A to 31C show image data of the front face, the right side face, and the plane of the target person A obtained in step 125, respectively. In the next step 127, the image data of the front surface, the back surface, the left side surface, the right side surface and the plane surface obtained in step 125 are combined.
Thereby, the shape of the object can be recognized. In the next step 134, the synthesized target person A
The extraction position setting process shown in FIG. 16 is executed based on the image data of the same as in the fifth embodiment.

As described above, according to the seventh embodiment, the shape of the object can be recognized without setting voxels.

The TV cameras 16 of the first to seventh embodiments are visible light TV cameras. However, for example, an infrared camera is used to shoot in a wavelength range other than visible light. Good. With this configuration, the object can be photographed even when the illumination lamp is not turned on, and thus the object can be used as a crime prevention device or a monitoring device.

In the fifth to seventh embodiments, the television camera 16 including the fish-eye lens 16A as a wide-angle fixed focus lens and the CCD area image sensor 16B is applied to the sound extraction device 10 of the first embodiment. By doing so, an example of efficiently (quickly) obtaining the position and shape of the object has been shown. However, in the sound extraction device 10 of the second to fourth embodiments, the fisheye lens 16A as a wide-angle fixed focus lens and the CCD are used. The same effect can be obtained by applying the television camera 16 including the area image sensor 16B.

As is clear from the above description, the present invention includes the following technical aspects.

[0195] The photographing means is arranged on a ceiling of a room forming a three-dimensional space.
The sound extraction device according to any one of 4 above.

15. The sound extraction device according to claim 1, wherein the photographing means photographs in a wavelength range other than visible light.

The shape recognizing means converts the distorted image information including the object image into flattened image information,
11. The object is recognized by obtaining image information of at least a front surface, a back surface, a left side surface, a right side surface, and a plane of an object image based on the converted image information, and synthesizing the obtained image information to recognize an object. 15. The sound extraction device according to any one of 14 to 14.

Height, thickness, head, arms, hands, which are characteristics of a person,
The shape recognition unit further includes a storage unit that stores in advance at least one of feature information as information about the foot, face, eyes, nose, mouth, ear, toes, and joints, and the shape recognition unit stores the feature information stored in the storage unit. 11. The object is recognized as a person based on the characteristic information and the image information captured by the image capturing unit.
15. The sound extraction device according to any one of 14 to 14.

[0199]

According to the first aspect of the present invention, it is possible to recognize the position of an object and to extract the sound emitted by the object based on the position by discriminating the noise from the ambient noise.

Further, according to the invention described in claims 2 and 16, it is possible to obtain higher accuracy especially when the directivity of the sound emitted by the object is strong or when the part (face) of the object which emits the sound is large. The effect that sound can be extracted is obtained.

According to the third aspect of the invention, the effect that the sound from the moving object can be extracted can be obtained.

Further, according to the invention described in claims 4 and 17, it is possible to obtain the effect that the sound from each of the plurality of objects can be extracted even for the plurality of objects.

Further, according to the invention described in claims 5 and 18, it is possible to obtain the effect that the sound can be extracted with high accuracy according to the change in the state of the acoustic environment.

Further, according to the invention described in claims 6 and 19, it is possible to prevent the treble range from being relatively weaker than the low frequency component.

Further, according to the invention described in claims 7 and 20, it is possible to obtain an effect that more appropriate sound can be extracted according to the arrangement condition of the reflecting surface and the like around the object.

According to the eighth and twenty-first aspects of the present invention, the load of processing relating to sound extraction (shifting and extraction processing by the extraction means) can be reduced without lowering the accuracy of sound extraction. The effect is obtained.

According to the ninth and twenty-second aspects of the present invention, the voice uttered by a person (one or more people) in the area where the sound extraction device can extract the sound is input to the voice recognition device. The effect that can be obtained is obtained.

According to the invention as set forth in claim 10, there is an effect that the position of the object can be promptly recognized without changing the direction of the photographing means and the focus adjustment following the movement of the object. can get. Also,
Since a mechanical operating mechanism for changing the direction of the photographing means and adjusting the focus is unnecessary, the structure of the photographing means and the sound extraction device can be simplified, and the number of mechanical actuation parts is reduced. The effect that the durability can be improved is also obtained. Furthermore, there is an effect that the sound emitted by the object can be discriminated and extracted from the ambient noise based on the recognized position of the object.

According to the eleventh aspect of the invention, the three-dimensional coordinate of the object is promptly obtained by the shape recognizing means and the three-dimensional coordinate calculating means constituting the image recognizing means, and the position of the object is promptly recognized. The effect that can be obtained is obtained.

According to the twelfth aspect of the invention, since the minute area can be subdivided to the limit of the resolution of the area sensor, the shape of the object can be obtained by obtaining the minute area occupied by the object from the image information. The effect of being able to recognize the details is obtained.

According to the thirteenth aspect of the present invention, it is possible to eliminate shadow regions in different image information captured by a plurality of image capturing means, and to accurately recognize the shape of the object. The effect is obtained.

According to the fourteenth aspect of the present invention, the two-dimensional coordinates formed on the area sensor are acquired, and the three-dimensional coordinates of the object are calculated based on the acquired two-dimensional coordinates. The effect is obtained.

According to the fifteenth aspect of the invention, since the object image can be formed on the area sensor by the reflecting means, the position of the object can be quickly recognized even if there is only one photographing means. In addition, the effect that the sound emitted by the object can be discriminated from the ambient noise and extracted based on the position can be obtained.

[Brief description of drawings]

FIG. 1 is a schematic diagram showing the principle of sound collection according to the present invention.

FIG. 2 is a schematic diagram showing a sound collection environment according to the first to fourth embodiments.

FIG. 3 is a schematic configuration diagram of a sound extraction device according to first and fourth embodiments.

FIG. 4 is a flow chart showing a control routine executed by the sound collection position calculation processor according to the first, third and fourth embodiments.

FIG. 5 is a flowchart showing a control routine executed by a processor of the voice extraction board according to the first and second embodiments.

FIG. 6 is a schematic configuration diagram of a sound extraction device according to a second embodiment.

FIG. 7 is a flow chart showing a control routine executed by the sound collection position calculation processor according to the second embodiment.

FIG. 8 is a schematic configuration diagram of a sound extraction device according to a third embodiment.

FIG. 9 is a flowchart showing a control routine executed by the processor of the voice extraction board according to the third embodiment.

FIG. 10 is a configuration example in which the sound extraction device of the present invention is applied to the extraction of sounds outdoors.

FIG. 11 is a schematic diagram showing a difference in directivity depending on a sound range of a sound according to the fourth embodiment.

FIG. 12 is a flowchart showing a control routine executed by the processor of the voice extraction board according to the fourth embodiment.

FIG. 13 is a schematic diagram showing a sound collection environment according to fifth to seventh embodiments.

FIG. 14 is a flowchart showing a control routine executed by a sound collection position calculation processor according to fifth and sixth embodiments.

FIG. 15 is a flowchart showing a subroutine of object classification processing.

FIG. 16 is a flowchart showing a subroutine of extraction position setting processing.

FIG. 17 is an explanatory diagram illustrating a concept of separating objects.

FIG. 18 is an explanatory diagram illustrating various amounts such as the height of an object.

FIG. 19 is an explanatory diagram illustrating a relationship between a shadow portion of an object and a voxel.

20A is a diagram showing voxels based on image data of the television camera A, FIG. 20B is a diagram showing voxels based on image data of the television camera B, and FIG. 20C is image data of the television camera C. 3D is a diagram showing voxels according to FIG. 3, and FIG. 3D is a diagram showing voxels according to image data of the television camera D.

FIG. 21 is an explanatory diagram illustrating the concept of voxels narrowed down by the second narrowing down.

FIG. 22 is an explanatory diagram illustrating a concept of converting voxels narrowed down by the second narrowing down to a dummy model.

FIG. 23 is a conceptual diagram illustrating various amounts when three-dimensional coordinates are calculated by two television cameras.

FIG. 24 is a diagram showing various arrangements of a television camera or a mirror.

FIG. 25 is a configuration diagram of a three-dimensional position recognition device according to a sixth embodiment.

FIG. 26 is an explanatory diagram for explaining the positions of the CCD area image sensor and the like of the sixth embodiment.

FIG. 27 is a flow chart showing a control routine executed by the sound collection position calculation processor according to the seventh embodiment.

FIG. 28 is a plan view showing the arrangement of an object and a television camera of the seventh embodiment.

29A is a diagram showing an image of image data of the television camera A, FIG. 29B is a diagram showing an image of image data of the television camera B, and FIG. 29C is image data of the television camera C. It is a figure which shows the image of, and (D) is a television camera D.
It is a figure which shows the image of the image data of.

FIG. 30A is a diagram showing an image when the distorted image data of the television camera A is converted into flattened image data, and FIG. 30B is a diagram showing the distorted image data of the television camera B. It is a figure which shows the image when converting into image data, (C) is a figure which shows the image when converting the image data of the distorted television camera C into the planarized image data, (D) is distorted. It is a figure which shows the image when converting the image data of the television camera D into the planarized image data.

FIG. 31A is a diagram showing an image of image data directly in front, FIG. 31B is a diagram showing an image of image data directly beside, and FIG. 31C is a diagram showing an image of image data immediately above. Is.

[Explanation of symbols]

10 sound extraction device 12 audio extraction board 14 extraction position calculation processor 16 TV camera (imaging means) 16A equidistant projection fisheye lens (wide-angle fixed focus lens) 16B CCD area image sensor (area sensor) 21 audio output terminal 22 microphone 32 input buffer Memory 34 Processor 44 Output buffer memory 46 Adder 58 Temperature sensor 60 Anemometer 62 Anemoscope 74 Mirror (reflecting means)

─────────────────────────────────────────────────── ─── Continuation of the front page (51) Int.Cl. ⁶ Identification code Internal reference number FI Technical display location H04N 7/18 G06F 15/62 415 9061-5H 15/70 460C G10K 15/00 MB (72) Inventor Kenji Kageyama 1-5 Otsuka, Inzai-cho, Inba-gun, Chiba Prefecture Takenaka Corporation Technical Research Institute (72) Inventor Tatsumi Nakajima 1-5, Otsuka, Inzai-cho, Inba-gun Chiba Prefecture Takenaka Corporation Technical Research Institute (72) Inventor Yoshitaka Wanaka Naka 1-5, Otsuka, Inzai-cho, Inba-gun, Chiba Prefecture Technical Research Institute of Takenaka Corporation (72) Inventor Yusei Yamada 1-5 Otsuka, Inzai-cho, Inba-gun, Chiba Prefecture Technical Research Institute of Takenaka Corporation (72 ) Inventor Kenichi Unno 1-5 Otsuka, Inzai-cho, Inba-gun, Chiba Prefecture In the Technical Research Institute of Takenaka Corporation (72) Inventor Nobuyoshi Murai Inba, Chiba Prefecture 1-5 Otsuka, Inzai Town, Gunnaka Takenaka Corporation Technical Research Institute

Claims (22)

[Claims] 1. A photographing means for photographing an area including an object as a sound source, an image recognizing means for recognizing the position of the object from image information of the area photographed by the photographing means, and arranged at a predetermined position. A plurality of microphones for collecting sounds emitted by the object, and time series data of a plurality of collected sounds selected from time series data of the collected sounds collected by each of the plurality of microphones, and a time series of the selected collected sounds. The data is shifted based on the position of the object recognized by the image recognition means and the position of the microphone collecting the selected sampling sound so that the sound emitted by the object is synchronized, and the time-series data of the shifted sampling sound. A sound extraction device comprising: an extraction unit that extracts the sound emitted by the object by averaging the. 2. The image recognition means also recognizes a direction in which the object emits sound based on image information of an area including the object, and further, a sound emitted by the object based on the position of the object and the direction in which the object emits sound. The sound extraction device according to claim 1, wherein a position at which the sound can be properly extracted is recognized again as a position of the object. 3. The sound extraction device according to claim 1, wherein when the object moves, the photographing means photographs the region including the object in accordance with the movement of the object. 4. When there are a plurality of objects, the photographing means photographs an area including the plurality of objects, and the image recognition means recognizes each position of the plurality of objects from image information of the photographed area, The sound extraction device according to claim 1, wherein the extraction unit extracts a sound from each of the plurality of objects. 5. A detection means for detecting an acoustic environment state that is a factor affecting at least one of a sound velocity and a sound propagation path in a region including the object and the plurality of microphones, and the extraction means includes: The sound extraction device according to claim 1, wherein, when the acoustic environment state detected by the detection means changes, the shift of the time-series data of the collected sound is corrected based on the changed acoustic environment state. 6. The sound extraction device according to claim 1, wherein the extraction means weights and averages time-series data of collected sounds in the high frequency range based on information on directivity in the high frequency range. 7. The image recognizing means further recognizes a direction in which the object emits sound and a position and direction of a sound reflecting surface located around the object based on image information of an area including the object, and the extracting means. Is the time series data of the selected collected sound based on the position of the microphone collecting the selected collected sound, the position of the object, the direction in which the object emits sound, and the position and the direction of the reflective surface. The sound extraction device according to claim 1, wherein either the direct sound from the sound or the reflected sound reflected by the reflection surface is shifted so as to be synchronized. 8. The extraction means, when the time-series data of the sound collected by each of the plurality of microphones is a sound collected by a microphone located at a predetermined distance or more from the position of the object. The sound extraction device according to claim 1, wherein the sequence data is excluded from the selection target. 9. The sound extraction device according to claim 1, further comprising output means for outputting a sound generated by the object extracted by the extraction means to a predetermined voice recognition device. 10. A photographing means for photographing a region including an object as a sound source, comprising a wide-angle fixed focus lens arranged at a predetermined position, and a position of the object based on image information of the region photographed by said photographing means. Image recognizing means for recognizing, a plurality of microphones arranged at a predetermined position for collecting sounds emitted by the object, and a plurality of collecting time-series data of collected sounds collected by each of the plurality of microphones. The time-series data of the sound is selected, and the time-series data of the selected sampled sound is generated based on the position of the object recognized by the image recognition means and the position of the microphone collecting the selected sampled sound. Are synchronized so that the time-series data of the shifted sampled sounds are averaged. A sound extraction device including: an extraction unit configured to extract a sound emitted by a sound source. 11. A plurality of the photographing means are provided,
Each photographing means further includes an area sensor arranged at an image forming point by the wide-angle fixed focus lens, and the image recognition means processes different photographing information photographed by the plurality of photographing means to shape the object. 11. The sound extraction according to claim 10, further comprising: a shape recognizing means for recognizing the object, and a three-dimensional coordinate calculating means for calculating the three-dimensional coordinate of the object recognized by the shape recognizing means. apparatus. 12. The shape recognizing means, based on different image information captured by the plurality of image capturing means,
Of a large number of cubic microspaces obtained by virtually subdividing the three-dimensional space along each of the X-axis, Y-axis, and Z-axis, the region formed by the microspace occupied by the object The sound extraction device according to claim 11, wherein the shape of the object is recognized by obtaining the shape. 13. The shape recognition means, based on different image information captured by the plurality of imaging means,
Of a large number of cubic microspaces obtained by virtually subdividing a three-dimensional space along each of the X-axis, Y-axis, and Z-axis, within the viewing angle at which an object is projected from each photographing means. The sound extraction device according to claim 11, wherein the shape of the object is recognized by extracting each of the contained microspaces and obtaining a region formed by the microspaces included in all of the extracted microspaces. 14. A plurality of the photographing means are provided,
Each photographing means further comprises an area sensor arranged at an image forming point by the wide-angle fixed focus lens, and the image recognition means acquires two-dimensional coordinates formed on the area sensor of each photographing means, The sound extraction device according to claim 10, wherein the position of the object is recognized based on the acquired plurality of two-dimensional coordinates. 15. A photographing means for photographing a region including an object as a sound source, comprising: a wide-angle fixed focus lens arranged at a predetermined position; and an area sensor arranged at an image forming point by the lens, Reflecting means arranged near the photographing means for reflecting the image of the object so as to form an image on the area sensor, an object image reflected by the reflecting means and formed on the area sensor, and the reflection The two-dimensional coordinates of each of the object images formed on the area sensor without being reflected by the means are acquired on the area sensor, and the three-dimensional coordinates of the object are calculated based on the acquired two-dimensional coordinates. Image recognition means for recognizing the position of the object by calculation, and the object being emitted at the predetermined position A plurality of microphones for collecting sounds, and time series data of a plurality of collected sounds is selected from time series data of the collected sounds collected by each of the plurality of microphones, and the time series data of the selected collected sounds is Shifting the sound emitted by the object so as to be synchronous, based on the position of the object recognized by the image recognition means and the position of the microphone collecting the selected sampling sound, and averaging the time-series data of the shifted sampling sound. A sound extraction device having: an extraction unit that extracts a sound emitted by the object. 16. The image recognition means also recognizes a direction in which the object emits sound based on image information of an area including the object, and further, a sound emitted by the object based on the position of the object and the direction in which the object emits sound. 16. The sound extraction device according to claim 10, further recognizing a position at which can be extracted satisfactorily as a position of the object. 17. When there are a plurality of objects, the photographing means photographs an area including the plurality of objects, and the image recognition means recognizes each position of the plurality of objects from image information of the photographed area, The sound extraction device according to any one of claims 10 to 15, wherein the extraction means extracts a sound from each of the plurality of objects. 18. A detection means for detecting an acoustic environment state that is a factor affecting at least one of a sound velocity and a sound propagation path in a region including the object and the plurality of microphones, the extraction means comprising: 16. When the acoustic environment state detected by the detection means changes, the shift of the time-series data of the collected sound is corrected based on the changed acoustic environment state. The sound extraction device according to item. 19. The method according to claim 10, wherein the extraction means weights and averages time-series data of collected sounds in the high frequency range based on information on directivity in the high frequency range.
The sound extraction device according to any one of 5 above. 20. The image recognizing means further recognizes a direction in which the object emits a sound and a position and a direction of a sound reflecting surface located around the object, based on image information of a region including the object, and the extracting means. Is the time series data of the selected collected sound based on the position of the microphone collecting the selected collected sound, the position of the object, the direction in which the object emits sound, and the position and the direction of the reflective surface. The sound extraction device according to any one of claims 10 to 15, wherein either the direct sound from the sound or the reflected sound reflected by the reflection surface is shifted in synchronization. 21. When the sampling means collects a sound collected by a microphone located at a predetermined distance or more from the position of the object in the time-series data of the sound collected by each of the plurality of microphones. 16. The sound extraction device according to claim 10, wherein the series data is excluded from the selection targets. 22. An output means for outputting the sound generated by the object extracted by the extraction means to a predetermined voice recognition device.
The sound extraction device according to any one of 1.