JP2012058314A - Acoustic processing system and machine employing the same - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an acoustic processing system for instantaneously extracting voices useful for avoiding danger by extracting voices of a person at a position to be extracted for the safety of the person around a machine.SOLUTION: An acoustic processing system has: a sound input section 201 comprised of a plurality of microphones for collecting sounds; a danger level calculation section 206 for calculating a danger level associated with contact with a person or an object around a machine caused by operating the machine; a sound extraction section 203 which inputs a signal outputted from the sound input section 201 and outputs a separation signal corresponding to the danger level calculated by the danger level calculation section 206; and a sound output section 219 for outputting the separation signal outputted from the sound extraction section 203.

Description

  The present invention relates to an acoustic processing technique suitable for an operator or driver operating a relatively large machine such as a construction machine, a vehicle, or a work machine to grasp the situation of a person around the machine, and more particularly, to a person around the machine. The present invention relates to a sound processing system suitable for safety and a technology effective when applied to a machine using the same.

  In relatively large machines such as construction machines, vehicles, and work machines, the operator or driver (hereinafter referred to as the operator) always knows the situation of the person around the machine for the safety of the person around the machine. It is necessary to avoid danger each time. One of the important information for the operator to know the situation of the person around the machine is the voice uttered by the person around.

  It is assumed that a microphone is installed outside the machine in order to pick up the voices of the surrounding people and the collected sounds are presented to the operator so that the operator can grasp the situation of the surrounding people. The sound picked up by the microphone includes not only the sounds of the surrounding people but also the engine sounds, machine driving sounds, excavation sounds, etc. that accompany machine operation. Only need to be extracted and presented to the operator.

  If a sound source separation technique using a plurality of microphones (microphone arrays) is used, it is possible to extract only sound coming from a specific position. However, there are the following two problems.

  First, in sound source separation, the problem is that it is necessary to specify the position where the voice is extracted, that is, the position where the person exists. For example, a sound source separation method based on position estimation assuming sparsity (for example, Patent Document 1) performs sound source separation by applying a filter with a designated extraction position as a target sound source position and the other as a disturbing sound source position. For this reason, it is necessary to specify the position. In addition, there is a technique called blind sound source separation that extracts the sound of each sound source without specifying the position of the sound source, but even in that case, which sound is to be extracted from among the plurality of obtained acoustic signals. The problem remains to determine if there was.

  The second problem is that there is a trade-off between “accuracy” of sound source separation and filter adaptation time. The accuracy here means how close the extracted sound is to the sound of the original target sound source. In general, an adaptive method for extracting with high accuracy (for example, independent component analysis of Non-Patent Document 1) cannot apply a filter only with an instantaneous input signal, and an operator grasps the situation of surrounding people. However, it is not possible to make a decision to avoid danger (hereinafter, “instantaneous” means that the time from when the sound is presented until the operator performs the danger avoidance action is sufficiently shorter).

  On the other hand, there are sound source separation algorithms that can be extracted using only an instantaneous input signal (for example, binary masking in Non-Patent Document 2), but the accuracy is generally low and noise is mixed. It is difficult for an operator to recognize what a person is talking about. There is also a problem that the operator is always exposed to residual noise without being separated.

  In order to achieve both real-time processing and separation accuracy, there is a method of selecting the independent component analysis and binary masking based on the volume difference according to the situation (for example, Patent Document 2). Patent Document 2 shows an example in which selection is performed based on the degree of convergence of a separation matrix for independent component analysis.

JP 2007-47427 A JP 2007-33825 A

T.A. Takatani, T .; Nishikawa, H .; Saruwatari, and K.A. Shikano, “Blind separation of binaural sound mixtures using SIMO-model-based independent component analysis,” ICASSP2004, vol. 4, pp. 113-116, 2004. O. Yilmaz and S.J. Rickard, “Blind separation of speed mixture via time-frequency masking,” IEEE Trans. Signal Process. , Vol. 52, no. 7, pp. 1830-1847, July 2004. M.M. Togami, T .; Sumioshi, and A.A. Amano, “Stepwise phase difference restoration method for sound source localization using multiple microphone pairs,” ICASP2007, vol. I, pp. 117-120, 2007.

  By the way, in the above-mentioned patent document 2, the merit to select on the basis of the degree of convergence is the stability that the separation accuracy does not decrease to less than the binary masking. In the present invention in which the safety of surrounding people is the most important, instantaneousness is necessary so that danger avoidance is necessary, but this problem depends on the invention of Patent Document 2 that places importance on stability of separation accuracy. It cannot be solved. Also, the problem of specifying the position to be extracted as described above cannot be solved.

  Therefore, the present invention has been made to solve the above-mentioned problems, and its typical purpose is to extract the voice of the person at the position to be extracted for the safety of the person around the machine and to avoid danger. It is an object of the present invention to provide an acoustic processing system for instantaneously extracting speech useful for a person.

  The above and other objects and novel features of the present invention will be apparent from the description of this specification and the accompanying drawings.

  Of the inventions disclosed in the present application, the outline of typical ones will be briefly described as follows.

  That is, a typical acoustic processing system includes a sound input unit including a plurality of microphones that collect sound, a risk calculation unit that calculates a risk associated with contact with a surrounding person or object due to the operation of the machine, A sound extraction unit that outputs a separation signal corresponding to the degree of risk calculated by the risk level calculation unit using the signal output from the sound input unit, and a sound that outputs the separation signal output from the sound extraction unit And an output unit. Furthermore, you may have the following characteristics.

  The sound extraction unit is composed of a plurality of sound source separation units with each position having a relatively high risk as an extraction position. The extraction method of each sound source separation unit is a method that can be extracted instantaneously when the risk of the corresponding extraction position is high, and a method that can extract with high accuracy when the risk of the extraction position is low.

  The degree of risk is calculated from the motion state of the machine and the detection result of the person position. The machine motion state is estimated based on sensor information or a machine operation signal installed in the work machine by the machine motion state estimation unit. The person detection is performed by combining the voice non-voice discrimination result and the moving object detection result based on the video. The sound non-speech discrimination includes a sound source position estimation unit that estimates a sound source position from a signal output from the sound input unit, and a voice non-speech discrimination unit that determines speech non-speech based on a sound source position output from the sound source position estimation unit. To achieve. The moving object detection is realized by a video input unit including one or more cameras such as a visible light camera or an infrared camera, and a moving object detection unit that detects a moving object based on an image output from the video input unit. Further, the sound source position estimation unit changes the estimation method according to the risk level for each position, and the moving object detection unit changes the detection method.

  A video output unit that displays video in accordance with the risk level, an external output sound generation unit that generates an external output sound to the outside of the machine based on the risk level, and an external that is generated by the external output sound generation unit An external sound output unit that outputs a direct output sound, and a machine control unit that controls the operation of the machine based on the degree of risk.

  Of the inventions disclosed in the present application, effects obtained by typical ones will be briefly described as follows.

  That is, according to a typical acoustic processing system, for the safety of a person around the machine, the acoustic processing for extracting the voice of the person at the position to be extracted and instantaneously extracting the voice useful for danger avoidance A system can be provided.

It is a figure which shows an example of the hardware constitutions of the sound processing system in Embodiment 1 of this invention. It is a figure which shows an example of the block configuration of the sound processing system in Embodiment 1 of this invention. It is a figure which shows an example of the block configuration of the sound input part shown in FIG. It is a figure which shows an example of the block configuration of the sound source position estimation part shown in FIG. It is a figure which shows an example of the block configuration of the moving body detection part shown in FIG. It is a figure which shows an example of the block configuration of the sound extraction part shown in FIG. In FIG. 2, it is a figure which shows an example of the data structure of the frequency domain signal Xf (f, (tau)) in a certain frame (tau). In FIG. 2, it is a figure which shows an example of a block configuration in case the method 2 which a sound source separation unit selects is the minimum dispersion | distribution beamformer by the adaptation based on sparsity. It is a flowchart which shows an example of the processing flow of the sound extraction part shown in FIG. It is a figure which shows an example of the block configuration of the sound processing system in Embodiment 3 of this invention. It is a figure which shows an example of the block configuration of the sound processing system in Embodiment 4 of this invention. 3 is a flowchart illustrating an example of a SPIRE algorithm in the sound source position estimation unit illustrated in FIG. 2. It is a figure which shows an example of the external appearance at the time of applying the sound processing system in Embodiment 1 of this invention to a construction machine.

  DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings, taking, as an example, an acoustic processing system integrated with a construction machine. Note that components having the same function are denoted by the same reference symbols throughout the drawings for describing the embodiment, and the repetitive description thereof will be omitted.

<Embodiment 1>
Hereinafter, Embodiment 1 of the present invention will be described with reference to FIGS. 1 to 9, 12, and 13.

  FIG. 1 is a diagram illustrating an example of a hardware configuration of a sound processing system according to Embodiment 1 of the present invention.

  The hardware configuration of the sound processing system 100 according to the present embodiment includes a microphone array 1011 to 101M, a speaker array 1021 to 102S, a visible light camera 1031 to 103A, an infrared camera 1041 to 104B, a microphone 105, a headphone 106, and A / D−. D / A converter 107, central processing unit 108, volatile memory 109, storage medium 110, image display device 111, audio cables 1141 to 114M, 1151 to 115S, 116, 117, monitor cable 118, digital cables 119, 1201 120A, 1211-121B, etc. The sound processing system 100 is integrated with a construction machine including a work machine 112, a machine operation input unit 113, and the like.

  The microphone arrays 1011 to 101M are a group of microphones that are mounted outside the construction machine and each array is composed of N microphones. The speaker arrays 1021 to 102S are a speaker group including S speakers 1021 to 102S mounted outside the construction machine.

  The visible light cameras 1031 to 103A are a visible light camera group mounted outside the construction machine. The infrared cameras 1041 to 104B are a group of infrared cameras mounted outside the construction machine.

  The microphone 105 is a microphone worn by the operator. The headphone 106 is a headphone worn by an operator.

  The A / D-D / A conversion device 107 converts the signal output from the microphone array 1011 to 101M and the signal output from the microphone 105 into digital data, and simultaneously converts the analog sound pressure to the speaker array 1021 to 102S and the headphone 106. This is an A / D-D / A converter that outputs a signal.

  The central processing unit 108 is a central processing unit that processes the output of the A / D-D / A conversion unit 107. The volatile memory 109 is a volatile memory that temporarily stores data of arithmetic processing in the central processing unit 108. The storage medium 110 is a storage medium that stores information such as programs. The image display device 111 is a display device that displays arithmetic processing information and images in the central processing unit 108.

  The audio cables 1141 to 114M are cables that connect the microphone arrays 1011 to 101M and the A / D / D / A converter 107. The audio cables 1151 to 115S are cables that connect the speaker arrays 1021 to 102S and the A / D / D / A converter 107. The audio cable 116 is a cable that connects the microphone 105 and the A / D / D / A converter 107. The audio cable 117 is a cable for connecting the headphones 106 and the A / D / D / A converter 107.

  The monitor cable 118 is a cable for connecting the image display device 111 and the central processing unit 108.

  The digital cable 119 is a cable that connects the A / D / D / A converter 107 and the central processing unit 108. The digital cables 1201 to 120A are cables that connect the visible light cameras 1031 to 103A and the central processing unit 108. The digital cables 1211 to 121B are cables that connect the infrared cameras 1041 to 104B and the central processing unit 108.

  The work machine 112 is a construction machine having an arm or the like. The machine operation input unit 113 is a part for inputting various operations of the construction machine.

  The hardware operation of the sound processing system 100 configured as described above is as follows.

  The sound pressure data output from the microphone arrays 1011 to 101M is sent to the A / D / D / A converter 107 via the audio cables 1141 to 114M. The sound pressure data from the microphone arrays 1011 to 101M is converted into digital sound pressure data by the A / D-D / A converter 107, respectively. In this conversion, conversion is performed by synchronizing the conversion timing between signals. The converted digital sound pressure data is sent to the central processing unit 108 through the digital cable 119, and the central processing unit 108 performs acoustic signal processing. The digital sound pressure data after the acoustic signal processing is sent to the A / D-D / A converter 107 via the digital cable 119. The digital sound pressure data from the central processing unit 108 is converted to analog sound pressure data by the A / D-D / A converter 107 and output from the headphones 106 via the audio cable 117.

  The digital sound pressure data X collected by the microphone arrays 1011 to 101M and sent to the central processing unit 108 includes voices of workers outside the work machine 112, engine sounds and arm drive sounds generated by the work machine 112, and the like. It is mixed with noise. In the central processing unit 108, digital sound pressure data X, image data VI obtained from the visible light cameras 1031 to 103A, image data II obtained from the infrared cameras 1041 to 104B, and operation signals obtained from the machine operation input unit 113. And the risk level H for each position is calculated based on the speed information of the work machine 112. The risk level H is stored in the volatile memory 109. Based on the risk level H, the central processing unit 108 changes the sound source position estimation method, further changes the moving object detection method, further sets the position with a relatively high risk level as the sound extraction position, and has a particularly high risk level. Sound extraction is performed for a position by a method that can be extracted instantaneously, and sound extraction is performed for a position with a low degree of danger by a method that can be extracted with high accuracy. The extracted signal Y is sent to the A / D / D / A converter 107 via the digital cable 119, converted into an analog signal, and output from the headphones 106 via the audio cable 117.

  The risk level H for each position stored in the volatile memory 109 is converted into an image by the central processing unit 108 and output from the image display device 111 via the monitor cable 118.

  The audio signal collected by the microphone 105 is converted into digital sound pressure data by the A / D-D / A converter 107 via the audio cable 116 and is sent to the central processing unit 108 via the digital cable 119. Entered. In addition, a directional filter using the speaker arrays 1021 to 102S is stored in advance in the storage medium 110 for each position where the directivity is directed. The digital sound pressure data is convolved by selecting a directional filter that directs the directivity to a position where the degree of risk H is relatively high, and multi-channel digital signal data is generated. The multi-channel digital signal data is input to the A / D-D / A converter 107 via the digital cable 119, and the A / D-D / A converter 107 converts it into a multi-channel analog signal. Are output from the speaker arrays 1021 to 102S through .about.115S.

  The central processing unit 108 controls the work machine 112 such as the type of movement, the movement speed, the type of movement, and the movement speed according to the degree of risk H.

  The digital cable 119 uses a USB cable or the like. As the digital cables 1201 to 120A and the digital cables 1211 to 121B, USB cables or LAN cables are used.

  FIG. 13 is a diagram illustrating an example of an external appearance when the sound processing system 100 according to the present embodiment is applied to a construction machine. FIG. 13 is a schematic view of the construction machine as viewed from above.

  In the example of FIG. 13, the construction machine includes a cabinet 13001, an engine unit 13002, an arm unit 13003, and the like. Microphone arrays 1011 to 1014 are arranged at the four corners outside the construction machine. An operator operates in the cabinet 13001.

  For example, when the present invention is not used, external sounds are hardly audible inside the cabinet 13001. In addition, the construction machine itself has noise sources such as the engine part 13002 and the arm part 13003. Even if the sound collected by the microphone arrays 1011 to 1014 is heard as it is, the voices of surrounding people buried in those noises are not heard. I can hardly hear. The present invention solves these problems.

  FIG. 2 is a diagram illustrating an example of a block configuration of the sound processing system 100 according to the present embodiment. The block configuration shown in FIG. 2 is a functional configuration by software realized by the central processing unit 108 shown in FIG. 1 reading out and executing a program stored in the storage medium 110. However, some components include the hardware configuration shown in FIG.

  The acoustic processing system 100 according to the present embodiment includes a sound input unit 201, a sound source position estimation unit 202 connected to the sound input unit 201, a sound extraction unit 203 connected to the sound input unit 201, and a sound source position estimation unit. The voice non-voice discrimination unit 204 connected to 202, the person detection unit 205 connected to the voice non-speech discrimination unit 204, and the person detection unit 205 are connected to the sound source position estimation unit 202 and the sound extraction unit 203. A risk calculation unit 206, a machine sensor input unit 207, and a machine sensor input unit 207 are connected to the machine motion state estimation unit 209, a visible light input unit 210, and an infrared input unit 211 connected to the risk calculation unit 206. And a visible light input unit 210, an infrared input unit 211, and a risk level calculation unit 206, a moving object detection unit 212 connected to the person detection unit 205, and a person The video output unit 213 connected to the output unit 205 and the risk level calculation unit 206, the operator voice input unit 215, and the output sound generation unit 216 for the outside connected to the operator voice input unit 215 and the risk level calculation unit 206. An external sound output unit 217 connected to the external output sound generation unit 216, a machine operation control unit 218 connected to the risk level calculation unit 206, and a sound output unit 219 connected to the sound extraction unit 203 And a machine operation input unit 221 connected to the machine motion state estimation unit 209.

  The voice non-voice discrimination unit 204 and the machine motion state estimation unit 209 use the machine size 208. The sound source position estimation unit 202 and the sound extraction unit 203 use information on the microphone arrangement 214. In the moving object detection unit 212, a camera projection matrix 220 is used.

  The main functions by software of the sound processing system 100 configured as described above (some components include a hardware configuration) are as follows.

  The sound input unit 201 is a functional unit including a plurality of microphones that collect sound. Details will be described later with reference to FIG. The sound source position estimation unit 202 is a functional unit that estimates a sound source position from a signal output from the sound input unit 201 or estimates a sound source position from a signal output from the sound extraction unit 203. The sound source position estimation unit 202 changes the estimation method based on the risk level for each position output by the risk level calculation unit 206. Details will be described later with reference to FIG. The sound extraction unit 203 is a functional unit that outputs a separation signal corresponding to the risk calculated by the risk calculation unit 206 using the signal output from the sound input unit 201 as an input. The sound extraction unit 203 includes a plurality of sound source separation units. Each sound source separation unit sets an extraction position according to the degree of danger, and the sound source separation unit changes the separation method according to the degree of risk. Details will be described later with reference to FIG.

  The speech non-speech determination unit 204 is a functional unit that determines speech non-speech based on the sound source position output from the sound source position estimation unit 202. The person detection unit 205 is a functional unit that detects a person position based on the voice / non-voice discrimination result output by the voice / non-voice discrimination unit 204. The person detection unit 205 performs person detection based on a signal output from the moving object detection unit 212.

  The risk level calculation unit 206 is a functional unit that calculates the level of risk associated with contact with surrounding people or objects due to machine operations. The risk level calculation unit 206 calculates the risk level for each position. Further, the risk level calculation unit 206 calculates the risk level based on the motion state output by the machine motion state estimation unit 209 or calculates the risk level based on the person position detection result output by the person detection unit 205. The machine motion state estimation unit 209 is a functional unit that estimates the motion state of the machine estimated based on sensor information or a machine operation signal installed in the machine.

  The video input unit includes a visible light input unit 210 and an infrared light input unit 211, and is a functional unit including a visible light camera or one or more cameras of an infrared camera. The moving object detection unit 212 is a functional unit that performs moving object detection based on the video output from the video input unit. In addition, the moving object detection unit 212 changes the detection method based on the risk level for each position output by the risk level calculation unit 206. Details will be described later with reference to FIG. The video output unit 213 is a functional unit that displays a video based on the risk level output by the risk level calculation unit 206.

  The external output sound generation unit 216 is a functional unit that generates an external output sound for the outside of the machine based on the risk level output by the risk level calculation unit 206. The external sound output unit 217 is a functional unit that outputs an external output sound generated by the external output sound generation unit 216.

  The machine operation control unit 218 is a functional unit that controls the operation of the machine based on the risk level output from the risk level calculation unit 206. The sound output unit 219 is a functional unit that outputs the separated signal output from the sound extraction unit 203.

  Below, the main function parts by the software of the sound processing system 100 will be described in detail.

  FIG. 3 shows an example of a block configuration of the sound input unit 201. The sound input unit 201 includes a multi-channel AD converter 301, a multi-channel frame processing unit 302, a multi-channel short-time frequency analysis unit 303, and the like. The multi-channel AD converter 301 is included in the A / D-D / A conversion device 107.

  In the sound input unit 201, the multichannel analog sound pressure data obtained from the microphone arrays 1011 to 101M is converted into digital sound pressure data x_11 (t) to x_MN (t) by the multichannel AD converter 301. t is a discrete time for each sampling period. The converted digital sound pressure data x_11 (t) to x_MN (t) is passed to the multi-channel frame processing unit 302.

  The multi-channel frame processing unit 302 changes x_ij (t) from t = τs to t = τs + F_s−1 to Xf_ij (t, τ) from t = 0 to t = F−1, respectively. Here, τ is called a frame index, and is incremented by 1 after the processing from the multi-channel frame processing unit 302 to the sound output unit 219 is completed. s is called a frame shift and means the number of samples shifted for each frame. F_s is called a frame size, and means the number of samples processed at one time for each frame. i is an index (1,..., M) indicating a microphone array number. j is an index (1,..., N) indicating a microphone number.

  Thereafter, Xf_ij (t, τ) is passed to the multi-channel short-time frequency analysis unit 303. The multi-channel short-time frequency analysis unit 303 performs a DC component cut and window processing such as a Hamming window, a Hanning window, and a Blackman window on Xf_ij (t, τ), and then performs a short-time Fourier transform on each frequency domain. Signal Xf_ij (f, τ). The frequency bin number here is F. Xf_ij (f, τ) in a certain frame τ has a data structure as shown in FIG. The frequency domain signal Xf_ij (f, τ) is sent to the sound source position estimation unit 202 and the sound extraction unit 203.

  FIG. 4 shows an example of a block configuration of the sound source position estimation unit 202. The sound source position estimation unit 202 includes frequency direction estimation units 4011 to 401M, a direction estimation integration unit 402, and the like.

  First, the direction estimator 401i for each frequency receives the sound arrival direction θ_i for each frequency index f with respect to the multi-channel frequency domain signals Xf_i1 (f, τ) to Xf_iN (f, τ) corresponding to one microphone array 101i. Estimate (f). When the number of microphone elements in the microphone array is two, θ is estimated by [Equation 1].

  Here, ρ (f, τ) is a phase difference between the input signals of the two microphone elements at the frame τ and the frequency index f. freq (f) is the frequency (Hz) of the frequency index f, and is calculated by [Equation 2].

However, _{F S} is the sampling rate of the A / D converter. d is the physical distance (m) between the two microphone elements. c is the speed of sound (m / s). Strictly speaking, the speed of sound changes depending on the temperature and the density of the medium, but is usually fixed to one value such as 340 m / s. Since the noise removal processing here may be performed separately for each time-frequency based on the above-mentioned assumption of “sparseness”, the time-frequency suffix (f, τ) is omitted hereinafter. It describes as.

  When the number of microphone elements in the microphone array is three or more, the direction can be calculated with high accuracy by the SPIRE algorithm (see Non-Patent Document 3). Also in the SPIRE algorithm, the same processing is performed separately for each time-frequency based on the above-described assumption of “sparseness”. FIG. 12 shows a flowchart of the SPIRE algorithm.

  First, in the SPIRE algorithm, the arrangement of microphone elements is read (S1201). Next, in the SPIRE algorithm, the microphone elements constituting each microphone pair are selected so that each microphone pair is composed of two microphone elements (S1202). At this time, it is desirable to divide the microphone interval between the two microphone elements constituting the microphone pair so as to be different for each microphone pair.

  Next, the SPIRE algorithm sorts each microphone pair in ascending order of the microphone interval and stores it in the microphone pair queue (S1203). Here, l is an index for specifying one microphone pair, l = 1 is a microphone pair with the shortest microphone interval, and l = L is a microphone pair with the longest microphone interval. A comparison operation is performed to determine whether the number of elements in the microphone pair queue is 0 (S1204). While the number of elements is not 0 (S1204-No), S1205 and S1206 described below are repeated.

That is, next, the process of reading one microphone pair 1 with the shortest interval from the microphone pair queue and removing it from the microphone pair queue is performed (S1205). In the subsequent phase difference estimation process, an integer n ₁ satisfying [Equation 3] is first found for the read l (S1206). Since the range surrounded by inequalities corresponds to 2π, only one solution can be found. Then, [Formula 4] is executed.

  Also, [Formula 5] is set as an initial value before the above processing is performed for l = 1. When S1205 and S1206 are repeated P times and the number of elements in the microphone pair queue becomes 0 (S1204-Yes), direction calculation is performed from the phase difference according to [Equation 6], and θ (f, τ) is calculated (S1207). .

Here, d _l is the distance between the microphone elements of the l th microphone pair.

  It is known that the estimation accuracy of sound source direction estimation increases as the microphone interval increases, but if the microphone interval is longer than the half wavelength of the signal for estimating the direction, one direction is specified from the phase difference between the microphones. It is known that there are two or more directions with the same phase difference (spatial aliasing). The SPIRE method includes a mechanism that selects a direction close to a sound source direction obtained at a short microphone interval from two or more estimated directions generated at a long microphone interval. Therefore, there is an advantage that the sound source direction can be estimated with high accuracy even with a long microphone interval that causes spatial aliasing.

  The direction estimation result θ_i (f, τ) output from the frequency direction direction estimation units 4011 to 401M is input to the direction estimation integration unit 402. According to [Expression 7], it is possible to obtain a position histogram h (p, τ) having a larger value as the position index p where the sound source exists.

  Here, according to the risk map data H (p, τ) calculated in the previous frame, if [Expression 8] obtained by thinning out the addition process of [Expression 7] is used, a position with a high risk is used. A position histogram can be calculated with high follow-up performance.

  The speech non-speech discrimination unit 204 generates a speech non-speech discrimination map v (p, τ) indicating the presence / absence of speech for each position p based on the position histogram h (p, τ) input from the sound source position estimation unit 202. judge. For speech non-speech discrimination, h (p, τ) is regarded as a speech signal with human noise present at the position p, noise estimation based on MCRA is performed, and then the input signal-to-noise ratio (post SNR) γ (p , Τ) may be discriminated by using a general algorithm such as a discriminating method [Equation 9], which is not an essential functional difference.

  Further, the calculation cost can be reduced by setting v (p, τ) to be always 0 with respect to p inside the machine based on the machine size 208. The voice / non-voice discrimination map v (p, τ) is sent to the person detection unit 205.

  The visible light input unit 210 including the visible light cameras 1031 to 103A sends the visible light image data VI to the moving object detection unit 212.

  The infrared input unit 211 including the infrared cameras 1041 to 104B sends the infrared image data II to the moving object detection unit 212.

  FIG. 5 shows an example of a block configuration of the moving object detection unit 212. The moving object detection unit 212 includes a background difference / interframe difference calculation unit 501, a body surface detection unit 502, a visual cone intersection calculation unit 503, and the like.

  Based on the visible light image data VI_1 to VI_A, the background difference / interframe difference calculation unit 501 calculates images EI_1 to EI_A in which object regions are extracted by background difference processing and interframe difference processing for each image. Based on the infrared image data II_1 to II_B, the body surface detection unit 502 calculates images BI_1 to BI_B in which pixel regions having high temperatures are extracted as body surface regions for the respective images. The visual cone intersection calculation unit 503 back-projects the respective visual cones of the object areas of the images EI_1 to EI_A and the body surface areas of the images BI_1 to BI_B into the three-dimensional space based on the camera projection matrix 220. The moving object existence map e (p, τ) is updated as in [Equation 11] for the region in which the visual volume intersects among the three-dimensional regions in which the visual fields intersect between the cameras obtained by [Equation 10].

Here, w _e also, depending on the calculated in the previous frame the risk map data H (p, tau), the use of the [number 12] obtained by thinning the back projection processing in the number 10, the moving object existence The followability becomes high with respect to a position having a high degree of risk in calculating the map e (p, τ).

The person detection unit 205 calculates the person detection map d (p, τ) from [Equation 13] based on the voice / non-voice discrimination map v (p, τ) and the moving object existence map e (p, τ). Here, _wv is a weighting coefficient of 0 or more and 1 or less.

  The machine sensor input unit 207 includes sensors such as a machine speedometer and a machine arm hydraulic pressure sensor, for example, and each sensor signal is set as a vector C (t) = (c_1 (t),..., C_Ω (t)). Output.

  The machine motion state estimation unit 209 obtains the three-dimensional position P_k (t) of each small part z_k from the machine dimension 208. Here, k (k = 1,..., K) is a part index. Further, the vector V () of the motion velocity V_k (t) of the small part z_k with respect to the set of the vector C (t) of the sensor signal and the vector P (t) = (P_1 (t),..., P_K (t)). It is assumed that a table of t) = (V_1 (t),..., V_K (t)) is stored in the storage medium 110 in advance. This table can be easily obtained by simulation at the time of design. With this table, the velocity V_k (t) of the small part z_k is obtained.

  Further, an operation signal μ (t) is obtained from the machine operation input unit 221. By storing a table of corresponding accelerations A (t) = (A_1 (t),..., A_k (t)) for combinations of the operation signals μ (t) and P (t), the operation signal μ From (t), the acceleration A_k (t) of the small part z_k is obtained. [Expression 14] The predicted position P (t + Δt) of the small part z_k at time t + Δt is obtained. Finally, a map g (p, t) of the shortest time required for contact is obtained from [Equation 15].

  The risk level calculation unit 206 is based on the person detection map d (p, τ) input from the person detection unit 205 and the map g (p, t) of the shortest contact time input from the machine motion state estimation unit 209. Thus, the risk map H (p, τ) is calculated from [Equation 16]. Here, ε and ν are appropriate constants.

  In the video output unit 213, the person detection map d (p, τ) and the risk map H (p, τ) are superimposed and presented.

  The sound extraction unit 203 extracts the extracted signal Yf (f) based on the frequency domain signals Xf_11 (f, τ) to Xf_MN (f, τ) and the risk map H (p, τ) input from the sound input unit 201. , Τ).

  FIG. 6 shows an example of a block configuration of the sound extraction unit 203. The sound extraction unit 203 includes an extraction direction selection unit 601, sound source separation units 6021 to 602R, a mixing unit 603, and the like.

  First, the extraction direction selection unit 601 sorts H (p, τ) of all the position indexes p, and determines the top R positions p_1 to p_R as extraction positions. The sound source separation units 6021 to 602R correspond to the extraction positions p_1 to p_R, respectively. A flowchart of the r-th sound source separation unit 602r (for example, 602R) is shown in FIG.

  In S901, the case is divided according to H (p_r, τ)> T_h or H (p_r, τ) ≦ T_h. When H (p_r, τ)> T_h with a high degree of risk H (p_r, τ) (S901—Yes), it is determined that particularly high speed is required, and a method that can be instantaneously extracted at S902 Select 1. The method 1 leaves the frequency component when the direction θ (f, τ) obtained for each frequency index by the direction estimation algorithm such as SPIRE described above overlaps the extraction position p_r, and the frequency component when the direction θ does not overlap. Binary masking in which the component is 0 may be used.

  On the other hand, when H (p_r, τ) ≦ T_h where the risk level H (p_r, τ) is relatively low (S901-No), it is determined that high-precision extraction is required for smooth communication. In step S903, the method 2, which is a method that can be instantaneously extracted, is selected.

  FIG. 8 shows an example of a block configuration in the case of a minimum dispersion beamformer by adaptation based on sparsity as an example of method 2. Method 2 has a detailed configuration of a target sound / noise separation unit 801, a target sound steering vector update unit 802, a noise covariance matrix update unit 803, a filter update unit 804, and a filter multiplication unit 805. This will be described with reference to FIG.

  Similar to the above-described binary masking, the target sound / noise separation unit 801 uses the direction θ (f, τ) obtained for each frequency index by the direction estimation algorithm, and the target sound signal X_des (Equation 17). f, τ) and X_int (f, τ). X_des (f, τ) is sent from the target sound / noise separation unit 801 to the target sound steering vector update unit 802. X_int (f, τ) is sent from the target sound / noise separation unit 801 to the noise covariance matrix update unit 803.

The target sound steering vector update unit 802 updates the target sound steering vector a (f, τ) = [a — 0 (f, τ),..., A_M−1 (f, τ)] ^{T based} on [Equation 18]. . However, γ _s is an appropriate constant parameter of 0 or more and less than 1. Of course, for the sake of stability, it may be updated only when | X_des_i (f, τ) | is sufficiently large.

The noise covariance matrix updating unit 803 updates the noise covariance matrix R (f, τ) based on [Equation 19]. However, X_int (f, τ) = [X_int_0 (f, τ),..., X_int_M−1 (f, τ)] ^T, and γ _n is an appropriate constant parameter of 0 or more and less than 1. Of course, for the sake of stability, it may be updated only when | X_int (f, τ) | is sufficiently large.

The filter update unit 804 calculates a filter w (f, τ) from the target sound steering vector a (f, τ) and the noise covariance matrix R (f, τ) based on [Equation 20]. However, γ _w is an appropriate constant parameter of 0 or more and less than 1.

Finally, the filter multiplier 805 converts the filter w (f, τ) to Xf (f, τ) = [Xf_0 (f, τ),..., Xf_M−1 (f, τ) based on [Equation 21]. By multiplying ^T , a signal Yf (f, τ) from which the sound coming from the designated direction is removed is obtained.

  In this example, the minimum dispersion beamformer based on sparsity is used for method 2, but ICA, which is another highly accurate extraction method, may be used for method 2. Since ICA uses high-order statistics, an audio signal of about several seconds is required for adaptation, and instantaneous extraction is difficult, but high-precision extraction is possible. In this example, only two methods 1 and 2 are selected and executed. However, the number of methods may be three or more, and may be selected and executed according to the degree of risk.

  The mixing unit 603 mixes the frequency domain signals output from the sound source separation units 6021 to 602R, and outputs an extraction signal Yf (f, τ).

  The frequency domain frame signal Yf (f, τ) calculated by the above procedure is sent to the sound output unit 219, where it is subjected to inverse FFT and converted to a time domain signal y (t, τ). y (t, τ) overlaps every frame period, is added, and is converted into y (t) subjected to the inverse of the window function, and y (t) is output from the headphone 106 via the DA conversion. Is done.

  The external output sound generation unit 216 selects a filter having the directivity of the speaker array at a position p_r where the H (p, τ) is large, based on the risk map H (p, τ). The voice signal input from the operator voice input unit 215 including the operator-side microphone 105 is multiplied by the filter to generate a multi-channel signal, and the speaker output 1021 via the DA conversion by the external sound output unit 217. -102S.

  The machine operation control unit 218 decelerates or stops the operation of the machine when the risk map H (p, τ) is very large with respect to a certain p.

According to the sound processing system in the present embodiment described above, the following effects can be obtained.
(1) Since the risk level calculation unit 206 calculates the risk level for each position, and the sound extraction unit 203 automatically selects a position with a high risk level as an extraction position, speech should be extracted for safety. It is possible to extract the voice of a person present at a high risk level.
(2) In the sound extraction unit 203, a sound source separation unit that selects a position with a high degree of danger as the extraction position is selected so that a method that can be extracted instantaneously is selected. . As a result, the operator can instantly avoid danger.
(3) In the sound extraction unit 203, the sound source separation unit having a position with a relatively low degree of risk as the extraction position selects a high-accuracy separation method, and therefore outputs extracted speech with little residual noise. As a result, the operator can recognize the content of the voices of the surrounding people, and further, a smooth conversation is possible between the operator and the surrounding people via the external sound output unit 217.
(4) Calculation for a position with a high degree of risk by the sound source position estimation unit 202 changing the estimation method and the moving object detection unit 212 changing the detection method according to the risk level calculated by the risk level calculation unit 206. Can be preferentially performed, and the frequency of calculation for a position with a low risk level can be reduced. Therefore, the update of the risk level calculation is shortened for a position with a high risk level that requires quick action by the operator.
(5) Since the degree of danger is visually presented to the video output unit 213 as a video, it is possible to avoid danger even when the operator cannot use hearing for some reason, such as when the operator is talking by telephone or wirelessly.
(6) Since the external sound output unit 217 outputs sound with directivity to a position with a high degree of danger, even in an environment where it is difficult to hear due to the noise of the machine, it is possible to call attention to persons around the machine it can.
(7) Since the machine operation control unit 218 urgently controls the machine itself to avoid danger when the degree of danger is high, there is a possibility that an accident can be avoided if the operator's avoidance decision is not in time.

<Embodiment 2>
The second embodiment of the present invention will be described below with reference to FIG.

  In the first embodiment, the example in which the r-th sound source separation unit 602r (for example, 602R) of the sound extraction unit 203 switches the method for each position has been described. However, in this embodiment, the method is switched for each position. Instead, this is an example applied to a configuration in which the method is switched only by time.

  According to the acoustic processing system in the present embodiment having such a configuration, in addition to the effects of the first embodiment, for example, when H (p, τ)> T_h for a certain p, Even when the method 1 is selected, it is possible to extract a time with a high degree of risk in real time and extract a time with a low degree of danger with high accuracy.

<Embodiment 3>
The third embodiment of the present invention will be described below with reference to FIG. FIG. 10 is a diagram illustrating an example of a block configuration of the sound processing system according to the present embodiment.

  The present embodiment is different from the first embodiment in the visible light input unit 210, the infrared input unit 211, the moving object detection unit 212, the video output unit 213, the operator voice input unit 215, and the output sound generation unit 216 for the outside. The external sound output unit 217, the machine operation control unit 218, and the camera projection matrix 220 are not provided.

  That is, as shown in FIG. 10, the sound processing system according to the present embodiment includes a sound input unit 201, a sound source position estimation unit 202, a sound extraction unit 203, a voice non-speech discrimination unit 204, and a person detection unit 205. A risk degree calculation unit 206, a machine sensor input unit 207, a machine motion state estimation unit 209, a sound output unit 219, a machine operation input unit 221, and the like. It has the same function.

According to the acoustic processing system in the present embodiment having such a configuration, the following effects (1) to (4) excluding (5) to (7) among the effects of the first embodiment. Can be obtained.
(1) Since the risk level calculation unit 206 calculates the risk level for each position, and the sound extraction unit 203 automatically selects a position with a high risk level as an extraction position, speech should be extracted for safety. It is possible to extract the voice of a person present at a high risk level.
(2) In the sound extraction unit 203, a sound source separation unit that selects a position with a high degree of danger as the extraction position is selected so that a method that can be extracted instantaneously is selected. . As a result, the operator can instantly avoid danger.
(3) In the sound extraction unit 203, the sound source separation unit having a position with a relatively low degree of risk as the extraction position selects a high-accuracy separation method, and therefore outputs extracted speech with little residual noise. Thereby, the operator can recognize the content of the voice of the surrounding person.
(4) The sound source position estimation unit 202 changes the estimation method according to the risk level for each position calculated by the risk level calculation unit 206, thereby preferentially calculating a position with a high risk level and having a low risk level. Since the frequency of calculation with respect to the position can be lowered, the update of the risk level calculation is shortened as the position has a high level of risk that requires quick action by the operator.

<Embodiment 4>
Embodiment 4 of the present invention will be described below with reference to FIG. FIG. 11 is a diagram illustrating an example of a block configuration of the sound processing system according to the present embodiment.

  The present embodiment is a configuration that does not further include the sound source position estimation unit 202, the voice / non-speech discrimination unit 204, and the person detection unit 205 as compared with the third embodiment.

  That is, as shown in FIG. 11, the sound processing system according to the present embodiment includes a sound input unit 201, a sound extraction unit 203, a risk level calculation unit 206, a machine sensor input unit 207, and a machine motion state estimation unit. 209, a sound output unit 219, a machine operation input unit 221, and the like, and each functional unit has the same function as in the first embodiment.

According to the acoustic processing system in the present embodiment having such a configuration, the following effects (1) to (3) other than (4) among the effects of the third embodiment can be obtained. it can.
(1) Even if the person detection unit is not provided, the risk level calculation unit 206 calculates the risk level for each position, and the sound extraction unit 203 automatically selects a position with a high risk level as the extraction position. Therefore, it is possible to extract the voice of a person who should be extracted for safety and is present at a high risk level.
(2) In the sound extraction unit 203, a sound source separation unit that selects a position with a high degree of danger as the extraction position is selected so that a method that can be extracted instantaneously is selected. . As a result, the operator can instantly avoid danger.
(3) In the sound extraction unit 203, the sound source separation unit having a position with a relatively low degree of risk as the extraction position selects a high-accuracy separation method, and therefore outputs extracted speech with little residual noise. Thereby, the operator can recognize the content of the voice of the surrounding person.

  As mentioned above, the invention made by the present inventor has been specifically described based on the embodiment. However, the present invention is not limited to the embodiment, and various modifications can be made without departing from the scope of the invention. Needless to say.

  For example, in the above-described embodiment, the configuration example in which the sound processing system is integrated with the construction machine has been described. However, the present invention is not limited to the construction machine but can be applied to general vehicles, work machines, and the like. is there.

  The acoustic processing system of the present invention relates to an acoustic processing technique suitable for an operator or driver operating a relatively large machine such as a construction machine, a vehicle, or a work machine to grasp the situation of a person around the machine. The present invention is applicable to a sound processing system suitable for safety of persons around the machine and a machine using the sound processing system.

DESCRIPTION OF SYMBOLS 100 ... Sound processing system, 1011-101M ... Microphone array, 1021-102S ... Speaker array, 1031-103A ... Visible light camera, 1041-104B ... Infrared camera, 105 ... Microphone, 106 ... Headphone, 107 ... A / D-D / A converter, 108 ... central processing unit, 109 ... volatile memory, 110 ... storage medium, 111 ... image display device, 112 ... work machine, 113 ... machine operation input unit, 1141 to 114M, 1151 to 115S, 116, 117 ... Audio cable, 118 ... Monitor cable, 119, 1201-120A, 1211-121B ... Digital cable,
DESCRIPTION OF SYMBOLS 201 ... Sound input part, 202 ... Sound source position estimation part, 203 ... Sound extraction part, 204 ... Voice non-speech discrimination part, 205 ... Person detection part, 206 ... Risk level calculation part, 207 ... Machine sensor input part, 208 ... Machine 209 ... mechanical motion state estimation unit, 210 ... visible light input unit, 211 ... infrared input unit, 212 ... moving object detection unit, 213 ... video output unit, 214 ... microphone arrangement, 215 ... operator voice input unit, 216 ... external output sound generation unit, 217 ... external sound output unit, 218 ... machine operation control unit, 219 ... sound output unit, 220 ... camera projection matrix, 221 ... machine operation input unit,
301 ... multi-channel AD converter, 302 ... multi-channel frame processing unit, 303 ... multi-channel short-time frequency analysis unit,
4011-401M ... Direction estimation unit for each frequency, 402 ... Direction estimation integration unit,
501 ... Background difference / interframe difference calculation unit, 502 ... Body surface detection unit, 503 ... Visual cone intersection calculation unit,
601 ... Extraction direction selection unit, 6021-602R ... Sound source separation unit, 603 ... Mixing unit,
801... Target sound / noise separator, 802... Target sound steering vector update unit, 803... Noise covariance matrix update unit, 804.
13001 ... Cabinet, 13002 ... Engine part, 13003 ... Arm part.

Claims (14)

A sound input unit composed of a plurality of microphones for collecting sound;
A risk level calculation unit for calculating a level of risk associated with contact with a person or object in the vicinity due to the operation of the machine;
A sound extraction unit that outputs a separation signal corresponding to the degree of risk calculated by the risk level calculation unit by using the signal output from the sound input unit;
A sound output unit that outputs a separation signal output from the sound extraction unit. The sound processing system according to claim 1,
The sound processing system, wherein the risk calculating unit calculates a risk for each position. The sound processing system according to claim 1 or 2,
The sound extraction unit includes a plurality of sound source separation units,
The sound processing system according to claim 1, wherein the plurality of sound source separation units set extraction positions according to the degree of risk. The sound processing system according to claim 3,
The sound processing system, wherein the sound source separation unit changes a separation method according to the degree of risk. The sound processing system according to claim 4,
A machine motion state estimation unit for estimating a motion state of the machine estimated based on sensor information or a machine operation signal installed in the machine;
The risk processing unit calculates the risk based on a motion state output from the mechanical motion state estimation unit. The sound processing system according to claim 5,
A sound source position estimating unit that estimates a sound source position from a signal output from the sound input unit;
A sound non-speech discrimination unit that discriminates speech non-speech based on a sound source position output by the sound source position estimation unit;
A person detection unit that detects a person position based on a voice non-voice discrimination result output by the voice non-speech discrimination unit;
The acoustic processing system, wherein the risk level calculation unit calculates the risk level based on a person position detection result output by the person detection unit. The sound processing system according to claim 5,
A sound source position estimating unit that estimates a sound source position from a signal output by the sound extraction unit;
A sound non-speech discrimination unit that discriminates speech non-speech based on a sound source position output by the sound source position estimation unit;
A person detection unit that detects a person position based on a voice non-voice discrimination result output by the voice non-speech discrimination unit;
The acoustic processing system, wherein the risk level calculation unit calculates the risk level based on a person position detection result output by the person detection unit. The sound processing system according to claim 7,
A video input unit composed of one or more cameras such as a visible light camera or an infrared camera;
A moving object detection unit that detects a moving object based on the video output from the video input unit;
The acoustic processing system, wherein the person detection unit detects a person based on a signal output from the moving object detection unit. The sound processing system according to claim 8.
The sound processing system, wherein the sound source position estimation unit changes an estimation method based on a risk level for each position output by the risk level calculation unit. The sound processing system according to claim 8 or 9,
The acoustic processing system, wherein the moving body detection unit changes a detection method based on a risk level for each position output by the risk level calculation unit. In the sound processing system according to any one of claims 1 to 10,
The acoustic processing system further comprising: a video output unit that displays video based on the risk level output by the risk level calculation unit. In the sound processing system according to any one of claims 1 to 11,
An external output sound generator for generating an external output sound to the outside of the machine based on the risk output by the risk calculator;
The sound processing system further comprising: an external sound output unit that outputs an external output sound generated by the external output sound generation unit. In the sound processing system according to any one of claims 1 to 12,
The acoustic processing system further comprising a machine operation control unit that controls the operation of the machine based on the risk level output by the risk level calculation unit.   A machine using the sound processing system according to claim 1.

Images