JP2022183848A - Speech recognition device, display device, and control method and program and storage medium therefor - Google Patents

Abstract

To provide a speech recognition device that enables operation commands for speech recognition control to be input even in a state in which a speech is acquired with sensitivity enhanced in a specific direction, a display device with the speech recognition device, and a control method, program and storage medium therefor.SOLUTION: Electronic binoculars have a speech processing part 305 which executes beam forming control capable of acquiring a speech input from a specific direction with higher sensitivity than a speech input from a different direction based upon a plurality of speech data obtained from a plurality of speech input parts 301 to 304, and outputs a corresponding control command when the speech data obtained from the plurality of speech input parts 301 to 304 correspond to a predetermined command. The speech processing part 305 determines whether or not the plurality of speech data correspond to the predetermined command when the beam forming control is effective and the plurality of speech data input from the plurality of speech input parts 301 to 304 meet predetermined conditions.SELECTED DRAWING: Figure 3

Description

The present invention relates to a speech recognition device, a display device having the speech recognition device, a control method thereof, a program, and a storage medium.

2. Description of the Related Art Electronic binoculars include a telephoto camera and a display unit that displays an image captured by the telephoto camera. Electronic binoculars sometimes record sound in addition to images. A microphone that has strong sensitivity in a specific direction (has directivity) is sometimes used to capture the sound of an object observed with a telephoto camera. Instead of using microphones with strong directivity, there is a method of acquiring sound from a specific direction using beamforming technology using multiple microphones.

Patent Literature 1 discloses beamforming control using a plurality of microphones. Patent Document 1 further describes a method of measuring the distance to a subject using a depth sensor and not controlling the directivity when there is no beamforming target, thereby preventing erroneous recognition of voice.

In addition, when voice can be input, the user (observer) can use voice recognition processing to give hands-free instructions to control the electronic binoculars.

U.S. Patent Application Publication No. 2015/0058003

However, in the conventional technology disclosed in the above-mentioned patent document, when the subject exists in the beamforming target range, the sensitivity to sound coming from a direction other than a specific direction decreases. At this time, even if the user near the electronic binoculars utters an utterance (input of an operation command) for operation by voice recognition processing, since the sensitivity to voice from a direction other than a specific direction has decreased, Voice recognition processing was not executed in some cases.

Accordingly, the present invention provides a speech recognition device, a display device, and the like that enable input of an operation command for speech recognition control even in a state in which speech is being acquired with increased sensitivity in a specific direction. The object is to provide a control method, a program, and a storage medium for

One form of the speech recognition device according to the present invention is to make speech input from a specific direction higher than speech input from other directions based on a plurality of speech data obtained from a plurality of speech input means. Control means for executing beamforming control that can be obtained with sensitivity; and voice for outputting a control signal corresponding to a command when voice data obtained from the plurality of voice input means corresponds to a predetermined command. and a recognition means, wherein the voice recognition means recognizes the plurality of voice data input from the plurality of voice input means when the beamforming control is effective and the plurality of voice data input from the plurality of voice input means satisfies a predetermined condition. It is characterized by determining whether or not the voice data corresponds to the predetermined command.

According to the speech recognition device, the display device, the control method thereof, the program, and the storage medium according to the present invention, even in a state in which speech is being acquired with increased sensitivity in a specific direction, speech recognition control can be performed. It becomes possible to input an operation command for

4 is a flow chart for implementing voice recognition performed by the electronic binoculars; 1 is an external view of electronic binoculars; FIG. 1 is a block diagram of electronic binoculars; FIG. 4 is a block diagram showing functional blocks of an audio processing unit; FIG. 4 is a timing chart regarding each signal in voice recognition control during beam forming control; FIG. 4 is a schematic diagram showing an example data structure of a voice command;

Preferred embodiments of the present invention are described in detail below with reference to the accompanying drawings. In this embodiment, electronic binoculars including a telephoto camera and a display section for displaying an image captured by the telephoto camera are exemplified as a speech recognition device and a display device including the speech recognition device. In addition, the apparatus to which the present invention is applied is not limited to the above. The present invention can be applied to any device capable of inputting voice with strong directivity in a specific direction using inputs from a plurality of sound collectors (microphones) and capable of executing device control by voice recognition. It is possible. For example, it is also applicable to devices such as personal computers (PCs), tablets, smartphones, and televisions.

Control by the electronic binoculars 100 according to the present embodiment will be described below with reference to FIGS. 1 to 6. FIG. In this embodiment, a scene is assumed in which a user using the electronic binoculars 100 observes a desired subject and controls the electronic binoculars 100 by voice recognition.

FIG. 1 is a flow chart for voice recognition performed by the electronic binoculars 100 . FIG. 2 is an external view of the electronic binoculars 100. As shown in FIG. FIG. 2A is an external perspective view of the electronic binoculars 100 as seen from the front side. FIG. 2B is an external perspective view of the electronic binoculars 100 as seen from the rear side.

As shown in FIG. 2 , electronic binoculars 100 have camera 101 , right-eye display 102 , left-eye display 103 , panning section 104 , tilting section 105 , gyro sensor 106 , operation member 107 and frame 110 . Further, the electronic binoculars 100 have an acceleration sensor 203 and audio input units 301-304.

A camera 101 is an imaging device that captures an image of an observation target. As indicated by arrows in FIG. 2A, by driving the panning section 104 and the tilting section 105, it is possible to rotate in the panning direction and the tilting direction. The panning section 104 and tilting section 105 are driven by built-in actuators. The camera unit 101 can be used by switching between two focal lengths of 100 mm and 400 mm (focal length in terms of 35 mm full size) by user operation. A user performs operations such as switching the focal length and turning ON/OFF the power of the electronic binoculars via the operation member 108 . A gyro sensor 106 is also provided for detecting head shake. When the focal length is 400 mm, both the optical zoom and the electronic zoom are used for photographing.

An image captured by the camera 101 (hereinafter referred to as a captured image) is cut out in a display range according to the angle of view set by the user, and a display image that the user observes is generated. The generated display image is displayed on displays 102 and 103, which are display units, in real time.

The camera 101 has an autofocus function, and automatically focuses on an observation target included within an observation range. The in-focus subject distance is uniquely determined by the stop position of a focusing lens (not shown) that is driven for focus adjustment. Therefore, if this relationship is held in the system in advance, the object distance can be detected from the stop position of the focusing lens stopped in autofocus. That is, the camera 101 also has a function of detecting the subject distance.

FIG. 3 is a block diagram of the electronic binoculars 100. As shown in FIG. A CPU 201 is a processor that controls each part of the electronic binoculars. The CPU 201 controls the camera 101 , right-eye display 102 , left-eye display 103 , gyro sensor 106 , operation member 107 , and camera rotation control mechanism 202 . The CPU 201 also controls the acceleration sensor 203 and the audio processing unit 305 .

The acceleration sensor 203 detects minute vertical and horizontal displacements of the electronic binoculars 100 .

Audio input units 301, 302, 303, and 304 are audio input units each including a microphone (hereinafter referred to as a microphone). The audio input units 301 to 304 convert audio signals into electrical signals and then into digital signals and output them. The audio input units 301 and 302 are arranged on a substantially straight line on the left side of the frame 110 of the electronic binoculars 100 . The audio input units 303 and 304 are arranged substantially in a straight line on the right side of the frame 110 of the electronic binoculars 100 . Beam forming control is performed by the audio processing unit 305 and the audio memory 306 using the input signals (audio signals) of the two audio input units arranged side by side on a straight line.

Beamforming control uses two or more omnidirectional microphones, without changing the placement of the microphones, and uses signal processing to sharpen the directivity of the microphones on the axis of the two microphones, emphasizing sound in a given direction. can do. Two sound input units are provided side by side in the front and back direction of the electronic binoculars 100, and beam forming control emphasizes the sound input from the front of the electronic binoculars 100, that is, directly in front of the user, that is, the wearer (observer). can be done. As a result, it is possible to emphasize the sound around the sound of the object that the wearer is viewing (capturing).

Based on voice commands input from voice input units 301 , 302 , 303 and 304 , voice processing unit 305 and voice memory 306 can perform voice recognition control. Voice commands corresponding to various controls of the photographing apparatus are stored in advance in the voice memory 306 . When voice data corresponding to the stored voice command is input from at least one of voice input units 301 to 304, voice processing unit 305 outputs a corresponding control command to CPU 201. FIG. By executing control according to the input control command by the CPU 201 , it is possible to issue an instruction to control the electronic binoculars 100 in the same manner as an instruction through the operation member 107 .

FIG. 6 is a schematic diagram showing an example data structure of a voice command stored in the voice memory 306. As shown in FIG. The voice processing unit 305 executes voice command recognition processing based on voice commands stored in the voice memory 306 .

The audio memory 306 has an audio data buffer area for successive rewriting and a command area having a predetermined audio command profile.

The audio data buffer area sequentially stores audio data from any one of the audio input units 301, 302, 303, and 304. FIG. The amount of audio data that can be buffered is determined according to the length of the determined audio command. It is set in advance so that voice data can be stored for a period in which a voice command to be judged can be input.

Voice commands include activation commands and operation commands.

The activation command is a voice command that instructs the start of control by voice recognition. Speech recognition processing is always executed while the speech recognition function is enabled, but the activation command is used by the user to explicitly instruct the start of control by speech recognition. It is desirable that the activation command be a unique command that is short or small in number of words and is unlikely to occur in a conversation. For example, a command calling the imaging device 100 such as "Hi, Camera" is assumed. Profile data corresponding to this registered word is stored as word profile A. FIG.

The operation command is a voice command for instructing execution of control of the electronic binoculars 100 by voice recognition. An operation command is composed of a short sentence or a plurality of words corresponding to the commanded control. Processing that can be controlled by voice recognition is assumed to be, for example, still image shooting, moving image shooting, tracking start, and the like. For example, the operation command B-1 is a voice command for instructing still image shooting, and "Snap it" is set. The operation command B-2 is a voice command for instructing the start of moving image shooting, and "Start Movie" is set. As for the operation command, a plurality of commands (operation commands B-1 to BN) are registered according to the process to be controlled.

The activation command and the operation command are stored separately for convenience, but they may be stored in the same word. In addition, it may be a mechanism in which words are rewritten depending on the destination or language compatibility, or the user can freely rewrite the words.

The voice processing unit 305 compares the voice data sequentially stored in the voice data buffer area with the activation command or operation command in the command area, and receives the voice data corresponding to the voice command whose degree of matching is equal to or greater than the threshold. determined to have been The voice processing unit 305 outputs to the CPU 201 a control command for instructing execution of control corresponding to the voice command.

Note that, in the electronic binoculars 100 of the present embodiment, voice commands related to beam forming control such as "Zoom Up" and "Zoom OUT" are held in a larger size than other commands, thereby achieving higher recognition accuracy than other commands. It is possible to react with

FIG. 1 is a flowchart of voice recognition control of the electronic binoculars 100 according to this embodiment. It shows the processing from controlling the electronic binoculars 100 by a voice command, executing and ending object sound amplification by beamforming control, and returning to normal photography again by a voice command. The control shown in this flowchart is assumed to be executed repeatedly with the voice recognition function of the electronic binoculars 100 enabled and the electronic binoculars 100 powered on.

In the electronic binoculars 100, when the user turns on the power with the operation member 107 when starting normal photographing, the camera 101 is activated and the photographed images are displayed on the displays 102 and 103 in real time. do. The electronic binoculars 100 have a configuration in which the focal length of the camera 101 can be switched between 100 mm and 400 mm, but immediately after the power is turned on, it is controlled at a wider angle of 100 mm so that the observation target can be easily found. .

When the user finds an observation target, the user switches the focal length to 400 mm via the operation member 107 . Autofocus (AF) and automatic exposure control (AE) are performed from the obtained image, and the user can start observing the subject through the image captured by the camera 101 .

Also, the user can use voice commands instead of the operation member 107 to control the focal length. At that time, the beam forming control described above can be used to emphasize sounds around the subject.

In S1001, the voice processing unit 305 determines whether or not a voice corresponding to the activation command has been input from at least one of the voice input units 301-304. If it is determined that the voice corresponding to the activation command has been input, the voice processing unit 305 transitions to the standby state for the operation command, and proceeds to S1002. If the voice corresponding to the activation command is not input, the process proceeds to S1020.

In S<b>1002 , the audio processing unit 305 acquires the frequency characteristic (f characteristic) from the input audio data, and stores it in the audio memory 306 .

In S1003, the voice processing unit 305 determines whether a voice corresponding to an operation command has been input from at least one of the voice input units 301-304. The voice processing unit 305 determines whether or not the voice input from at least one of the voice input units 301 to 304 matches any one of a plurality of operation commands pre-stored in the voice memory 306 . If a voice matching the operation command has been input, the process advances to S1004. If no voice matching the operation command is input, the process returns to S1003 to wait for input of voice corresponding to the operation command.

In S1004, the voice processing unit 305 notifies the CPU 201 of a control command instructing execution of control corresponding to the operation command corresponding to the input voice. The CPU 201 executes processing corresponding to the notified control command. Assume that a voice corresponding to the operation command "Zoom UP" is input. The CPU 201 controls the focal length of the camera 101 to 400 mm, and further executes processing for emphasizing sound from a specific direction by beamforming control using the sound input units 301 and 302 and the sound input units 303 and 304 . At this time, the direction in which the sensitivity is increased by beam forming control is the photographing direction of the camera 101 . That is, when the operation command "Zoom UP" is input, the imaging range of the camera 101 is expanded, and the sensitivity of voice input to the imaging direction is increased by beamforming control.

In S1005, the audio processing unit 305 determines whether or not audio data has been input from at least one of the audio input units 301-304. If no voice data is input, the process returns to S1005. If voice data has been input, the process proceeds to S1006.

In S1006, the audio processing unit 305 compares the phase difference of the audio data (input signal) input from each of the audio input units 301-304.

In S1007, the audio processing unit 305 compares the levels (volumes) of the audio data input from each of the audio input units 301-304.

In S1008, it is determined whether the voice data input to the voice input units 301 to 304 are equivalent. If a voice command is uttered by a user who is the wearer, the phase and level of the voice data input to the four voice input units 301 to 304 are almost the same, so it is difficult to determine whether the command is issued by the wearer. Used for primary judgment. If the phases and levels of the audio data input to the four audio input units 301-304 are substantially the same, the process proceeds to S1009. Otherwise, it is determined that the voice input is not for voice recognition control by the wearer, and the process returns to S1005.

At S1009, the frequency characteristics of the audio data input from each of the audio input units 301 to 304 are compared with the frequency characteristics held at S1002. If the frequency characteristics are the same, a secondary determination is made as to whether or not the wearer who uttered the activation command is the same as the wearer who input the operation command. S1007 is realized by the audio processing unit 305 processing each audio data temporarily held in the audio memory 306 from the audio input units 301 to 304 . If the frequency characteristics are the same, it is determined that the wearer who uttered the activation command is the same as the wearer who input the operation command, and the process proceeds to S1008. Otherwise, it is determined that the voice input is not for voice recognition control by the wearer, and the process returns to S1005.

In S1010, the CPU 201 determines whether or not the wearer has spoken at the timing when the audio data is input, based on the minute displacement in the vertical direction detected by the acceleration sensor 203 attached to the bridge portion of the electronic binoculars 100. If the minute displacement in the vertical direction (detection result) does not satisfy the displacement detection condition predetermined for speech detection, it is determined that the voice data input is not the wearer's speech, and the process returns to S1005. If the minute displacement in the vertical direction satisfies the displacement detection condition predetermined for speech detection, it is determined that the voice data is input by the wearer's speech, and the process proceeds to S1011.

At S1011, a command corresponding to the input voice data is detected. By performing the processing of S1006 to S1010, voice recognition control can be executed even during beamforming control in which the directivity of the voice input units 301 to 304 is not directed toward the wearer's mouth.

In S1011, it is determined whether or not the command is related to beamforming control. As shown in FIG. 6, in this embodiment, voice commands such as "ZoomUp" and "ZoomOUT" related to beamforming are held in a larger size than other commands.

In addition, after beamforming control, the user inevitably performs control related to beamforming control, such as canceling beamforming and recording the sound of the beamforming destination. Therefore, it is possible to respond with higher recognition accuracy by performing determination processing focusing on whether or not the command is related to beam forming control.

In S1012, the CPU 201 controls the electronic binoculars 100 according to the command detected in S1011. Also, the process returns to S1005.

Each process will be described in detail with reference to FIGS. 4 and 5. FIG. FIG. 4 is a block diagram showing functional blocks of the audio processing unit 305. As shown in FIG. The audio processor 305 has a beamforming controller 401 , a recording processor 402 , a speech recognition processor 403 , a phase comparison processor 404 , a sound pressure comparison processor 405 , and a specific band detector 406 .

The voice recognition processing unit 403 uses the voice memory 306 to compare the voice data stored in the voice data buffer with voice commands held in advance, and performs voice command recognition. In addition, a speech recognition processing unit 403, a phase comparison processing unit 404 that compares the phases of the inputs of each of the speech input units 301 to 304, a sound pressure comparison processing unit 405 that compares sound pressures, a specific band that detects a specific band A configuration is adopted in which the determination signal from the detection unit 406 is input as an enable signal.

The enable signal becomes effective when each comparison or detection process detects a predetermined value as will be described later, and the speech recognition processing section can be corrected. Also, the enable signal can be controlled to be enabled or disabled under the control of the CPU 201 .

The recording processing unit 402 can output the sound data of the band emphasized by the beamforming control unit 401 to the audio memory 306 .

FIG. 5 is a timing chart regarding each signal in voice recognition control during beamforming control shown in S1005 to S1011 in FIG.

When the acceleration displacement in the acceleration sensor 203 exceeds a certain value, the detection triggers the CPU 201 to send an enable signal to the audio processing unit 305 to enable each process of the audio processing unit 305 .

After that, the inputs of the audio input units 301 to 304 are compared by the phase comparison processing unit 404 and the sound pressure comparison processing unit 405 . The phase comparison processing unit 404 compares the phases of the audio data input to the audio input units 301 to 304 to determine whether the phases of the data are aligned. A predetermined threshold value held in the audio memory 306 is used for the comparison. When the phase difference of each data is within a predetermined threshold value, the inter-MIC phase difference comparison signal is validated.

Further, the sound pressure comparison processing unit 405 determines whether or not the sound pressure level input to the voice input units 301 to 304 is within a predetermined threshold range. When the sound pressure difference of each data is within a predetermined threshold value, the inter-MIC sound pressure level comparison signal is validated.

As described above, in the case of a voice command uttered by the user who is the wearer, the distance between each voice input unit and the user's mouth is almost the same, so the phase of the sound data input to the four voice input units Levels are almost the same. Both the inter-MIC phase difference comparison signal and the inter-MIC sound pressure level comparison signal become valid.

Further, the specific band detection unit 406 compares the frequency characteristics held in advance with the band of the uttered command, and makes secondary determination as to whether or not it is the same as the wearer who uttered the activation command. The band comparison is realized by having the audio processing unit 305 process each piece of audio data temporarily held in the audio memory 306 from the audio input units 301 to 304 . If the voice command is determined to be within the predetermined band, the specific band detection signal is activated.

After that, at T601, using the voice data and the three signals, the judgment as to whether or not it is an activation command is completed. If it is determined to be an activation command, the inter-MIC phase difference comparison signal, the inter-MIC sound pressure level comparison signal, and the specific band detection signal are valid.

A timing chart between T601 and T602 is a voice command detection chart when a control command is uttered. Since the processing and timing are the same as those of the activation command, the description is omitted.

By controlling as described above, the speech recognition device and the display device having the speech recognition device according to the present embodiment can perform speech recognition by the wearer even when voice data in a specific direction is amplified by beamforming control. Allows execution of control. In particular, by performing the displacement of the user's mouth, the phase difference comparison between MICs, the sound pressure level comparison between MICs, and the detection of a specific band, it is possible to return to normal photography by a voice command.

(Other examples)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.

100 electronic binoculars 101 camera 102 103 display 104 panning unit 105 tilt unit 106 gyro sensor 107 operation member 203 acceleration sensor 301, 302, 303, 304 voice input unit

Claims (11)

Executes beamforming control that can acquire voice input from a specific direction with higher sensitivity than voice input from other directions based on multiple voice data obtained from multiple voice input means. a control means for
voice recognition means for outputting a control command corresponding to a predetermined command when voice data obtained from the plurality of voice input means corresponds to the command;
has
When the beamforming control is effective and the plurality of speech data input from the plurality of speech inputting devices satisfies a predetermined condition, the speech recognition device performs the above-mentioned plurality of speech data according to the predetermined condition. A voice recognition device that determines whether or not it corresponds to a command. further comprising detection means for detecting displacement of the speech recognition device at the timing when the plurality of speech data are input;
The voice recognition means determines whether or not the plurality of voice data correspond to the predetermined command when the detection result of the detection means satisfies a predetermined displacement detection condition. The speech recognition device according to claim 1. The predetermined condition is characterized in that the frequency characteristics of the plurality of audio data match the frequency characteristics of the audio data of the wearer's utterance acquired before the beam forming control is executed. 3. The speech recognition device according to claim 1 or 2. 4. The speech recognition apparatus according to any one of claims 1 to 3, wherein said predetermined command includes a command for instructing activation of the device and a command for instructing control of the device. Further comprising imaging means for capturing an image,
The control means executes the beam forming control so that the sound input from the direction in which the imaging means captures the image is acquired with higher sensitivity than the sound input from other directions. 5. The speech recognition device according to any one of claims 1 to 4. In response to the voice recognition means detecting input of voice data corresponding to a command for expanding the imaging range of the imaging means while the beam forming control is not being executed,
6. The speech recognition apparatus according to claim 1, wherein said control means executes said beamforming control. While the beam forming control is being performed, the speech recognition means recognizes the plurality of speech data if the plurality of speech data do not satisfy the predetermined condition even if the plurality of speech data are input. 7. The speech recognition apparatus according to any one of claims 1 to 6, wherein it does not determine whether or not data corresponds to the predetermined command. imaging means for capturing an image;
display means for displaying the image;
a plurality of voice input means;
Based on the plurality of audio data obtained from the plurality of audio input means, it is possible to acquire the audio input from the imaging direction with higher sensitivity than the audio input from other directions. a control means for performing beamforming control;
voice recognition means for outputting a control command corresponding to a predetermined command when voice data obtained from the plurality of voice input means corresponds to the command;
has
When the beamforming control is effective and the plurality of speech data input from the plurality of speech inputting devices satisfies a predetermined condition, the speech recognition device performs the above-mentioned plurality of speech data according to the predetermined condition. A display device that determines whether or not it corresponds to a command. Executes beamforming control that can acquire voice input from a specific direction with higher sensitivity than voice input from other directions based on multiple voice data obtained from multiple voice input means. a control process for
a speech recognition step of outputting a control command corresponding to a predetermined command when the speech data obtained from the plurality of speech input means corresponds to the command;
has
In the speech recognition step, when the beamforming control is effective and the plurality of speech data input from the plurality of speech input means satisfies a predetermined condition, the plurality of speech data are determined in advance. A control method for a speech recognition device, characterized by determining whether or not it corresponds to a command. A program for causing a computer to function as each means of the speech recognition apparatus according to any one of claims 1 to 7. A computer-readable storage medium storing a program for causing a computer to function as each means of the speech recognition apparatus according to any one of claims 1 to 7.

Images