KR20240038532A - Method for operating singing mode and electronic device performing the same - Google Patents

Abstract

A method of operating a singing mode and an electronic device performing the same are disclosed. A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations include determining an operation mode of the wireless audio device (102; 202; 302) as one of a singing mode and a conversation mode based on the analysis result of the audio signal, and determining the operation mode of the wireless audio device (102; 202; 302) according to the determined mode. ;202;302) may include an operation of controlling the output signal.

Description

Singing mode operation method and electronic device performing the same {METHOD FOR OPERATING SINGING MODE AND ELECTRONIC DEVICE PERFORMING THE SAME}

Embodiments of the present invention relate to a method of operating a singing mode and an electronic device that performs the same.

Wireless audio devices such as earbuds are widely used. A wireless audio device can be wirelessly connected to an electronic device such as a mobile phone and output audio data received from the mobile phone. Since wireless audio devices are wirelessly connected to electronic devices, user convenience can be increased. Due to increased convenience, the time a user wears a wireless audio device may increase.

A wireless audio device may be worn on the user's ears. When wearing a wireless audio device, it may be difficult for the user to hear external sounds. The wireless audio device can output ambient sound so that the wearer of the wireless audio device can hear external sounds. For example, a wireless audio device can provide surrounding sounds to the user by outputting sound received by the microphone of the wireless audio device in real time.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode of the wireless audio device (102; 202; 302) as one of a singing mode and a conversation mode based on an analysis result of the audio signal. The plurality of operations may include controlling an output signal of the wireless audio device (102; 202; 302) according to the determined mode. The conversation mode may be a mode for outputting at least part of the ambient sound included in the audio signal, and the singing mode may be a mode for outputting at least a part of the ambient sound and media included in the audio signal.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode for the audio signal of the wireless audio device (102; 202; 302) as a singing mode. The plurality of operations may include controlling an output signal of the wireless audio device (102; 202; 302) according to the singing mode. The singing mode may be a mode that outputs at least part of the surrounding sounds and media included in the audio signal.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode of the wireless audio device (102; 202; 302) for the audio signal as either a singing mode or a conversation mode based on an analysis result of the audio signal. . The plurality of operations may include outputting at least a portion of the ambient sound included in the audio signal when the determined mode is the conversation mode. The plurality of operations may include outputting at least a portion of ambient sounds and media included in the audio signal when the determined mode is the singing mode. The plurality of operations may include an operation of deactivating the singing mode when a singing voice is not detected in the ambient sound for more than a specified time in the singing mode.

Figure 1 is a block diagram showing an integrated intelligence system according to an embodiment.
Figure 2 is a block diagram showing an integrated intelligence system according to one embodiment.
FIG. 3 illustrates a communication environment between a wireless audio device and an electronic device according to an embodiment.
Figure 4 shows a block diagram of electronic devices and wireless audio devices according to one embodiment.
Figure 5 shows front and rear views of a first wireless audio device according to one embodiment.
Figure 6 shows a block diagram of a wireless audio device according to one embodiment.
Figure 7 is a block diagram showing the configuration of a wireless audio device according to an embodiment.
FIG. 8 is a flowchart illustrating an operation of a wireless audio device controlling an output signal according to an embodiment.
FIG. 9 is a flowchart illustrating an operation of a wireless audio device controlling an output signal according to one of a singing mode and a conversation mode, according to an embodiment.
Figure 10 is a schematic diagram of a similarity determination module according to one embodiment.
Figure 11 is a schematic diagram of a singing mode module according to one embodiment.
12A and 12B are examples of screens output on a display of an electronic device according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the attached drawings. In the description with reference to the accompanying drawings, identical components will be assigned the same reference numerals regardless of the reference numerals, and overlapping descriptions thereof will be omitted.

Figure 1 is a block diagram showing an integrated intelligence system according to an embodiment.

Referring to Figure 1, an integrated intelligence system in one embodiment includes a first electronic device 101 (e.g., a user terminal), a second electronic device 102 (e.g., any device including earbuds or a microphone), and an intelligent system. It may include a server 100 and a service server 103.

According to the illustrated embodiment, the first electronic device 101 includes a communication interface 110, an input/output (I/O) interface 120, at least one processor 130, and/or a memory 140. ) may include. The components listed above may be operatively or electrically connected to each other.

In one embodiment, communication interface 110 connects to an external device (e.g., intelligent server 100) via first network 199 (e.g., any network including a cellular network and/or wireless local area network (WLAN)). ), or connected to the service server 103) to transmit and receive data. The communication interface 110 may support data transmission and reception with an external device (e.g., the second electronic device 102) through the second network 198 (e.g., a short-range wireless communication network).

In one embodiment, I/O interface 120 uses input/output devices (not shown) (e.g., microphones, speakers, and/or displays) to receive user input, process received user input, and /Or the results processed by the processor 130 may be output.

In one embodiment, the processor 130 may be electrically connected to the communication interface 110, the I/O interface 120, and/or the memory 140 to perform designated operations. The processor 130 may perform a designated operation by executing a program (or one or more instructions) stored in the memory 140. For example, the processor 130 may receive a user's voice input (eg, user speech) through the I/O interface 120. For example, the processor 130 may receive the user's voice input received by the second electronic device 102 through the communication interface 110. The processor 130 may transmit the voice input received through the communication interface 110 to the intelligent server 100.

In one embodiment, the processor 130 may receive a result corresponding to the voice input from the intelligent server 100. For example, the processor 130 may receive a plan corresponding to the voice input and/or a result calculated using the plan from the intelligent server 100. The processor 130 may receive a request from the intelligent server 100 to obtain information (eg, parameters) necessary to create a plan corresponding to the voice input. The processor 130 may transmit the necessary information to the intelligent server 100 in response to the request.

In one embodiment, the processor 130 may output the results of executing a specified operation according to the plan visually, tactilely, and/or vocally through the I/O interface 120. For example, the processor 130 may sequentially display execution results of a plurality of operations on the display. As an example, the processor 130 may display only partial results of executing a plurality of operations (eg, the result of the last operation) on the display. The processor 130 may transmit an execution result or a partial execution result to the second electronic device 102 through the second network 198, thereby providing feedback through the second electronic device 102.

In one embodiment, processor 130 may recognize voice input to perform limited functions. For example, the processor 130 may execute an intelligent app (or voice recognition app) to process voice input in response to a designated voice input (e.g., wake up!). The processor 130 may provide a voice recognition service through an intelligent app (or application program). The processor 130 transmits voice input to the intelligent server 100 through the intelligent app. , a result corresponding to the voice input can be received from the intelligent server 100.

According to one example, the second electronic device 102 includes a communication interface 111, an input/output (I/O) interface 121, at least one processor 131, and/or a memory 141. It can be included. The components listed above may be operatively or electrically connected to each other. In one example, the second electronic device 102 may be a set of a plurality of electronic devices (eg, a left earbud and a right earbud).

In one embodiment, the communication interface 111 may support connection with an external device (eg, the first electronic device 101) through the second network 198. The I/O interface 121 uses an input/output device (not shown) (e.g., at least one microphone, at least one speaker, and/or button) to receive user input, process the received user input, or And/or the results processed by the processor 131 may be output.

In one embodiment, the processor 131 may be electrically connected to the communication interface 111, the I/O interface 121, and/or the memory 141 to perform a designated operation. The processor 131 may perform a designated operation by executing a program (or one or more instructions) stored in the memory 141. For example, the processor 131 may receive a user's voice input (eg, user speech) through the I/O interface 121. In one example, the processor 131 may perform voice activity detection (VAD) using at least one sensor (not shown) of the second electronic device 102. The processor 131 may detect the user's speech of the second electronic device 102 using an acceleration sensor (not shown) and/or a microphone.

In one embodiment, the processor 131 may transmit the received voice input to the first electronic device 101 through the second network 198 using the communication interface 111.

In one embodiment, the processor 131 may receive a result corresponding to the voice input from the first electronic device 101 through the second network 198. For example, the processor 131 may receive data (e.g., text data) corresponding to a result corresponding to a voice input from the first electronic device 101. The processor 131 may output the received result through the I/O interface 121.

In one embodiment, processor 131 may recognize voice input to perform limited functions. For example, the processor 131 may request the first electronic device 101 to execute an intelligent app (or voice recognition app) to process the voice input in response to a specified voice input (e.g., wake up!). there is.

The intelligent server 100 of one embodiment may receive a user's voice input from the first electronic device 101 through the first network 199. The intelligent server 100 may convert audio data corresponding to the received voice input into text data. The intelligent server 100 may generate at least one plan for performing a task corresponding to the user's voice input based on text data. The intelligent server 100 may transmit the generated plan or a result according to the generated plan to the first electronic device 101 through the first network 199.

The intelligent server 100 of one embodiment includes a front end 160, a natural language platform 150, a capsule database 190, an execution engine 170, and/or an end user interface (180).

In one embodiment, the front end 160 may receive a voice input received by the first electronic device 101 from the first electronic device 101 . The front end 160 may transmit a response corresponding to the voice input to the first electronic device 101.

In one embodiment, the natural language platform 150 includes an automatic speech recognition module (ASR module) 151, a natural language understanding module (NLU module) 153, and a planner module. module) (155), a natural language generator module (NLG module) (157), and/or a text to speech module (TTS module) (159).

In one embodiment, the automatic voice recognition module 151 may convert voice input received from the first electronic device 101 into text data. The natural language understanding module 153 may determine the user's intent and/or parameters based on text data of the voice input.

In one embodiment, the planner module 155 may generate a plan using the intent and parameters determined by the natural language understanding module 153. The planner module 155 may determine a plurality of domains required to perform the task based on the determined intention. The planner module 155 may determine a plurality of operations included in each of the plurality of domains determined based on intention. The planner module 155 may determine parameters required to execute the determined plurality of operations or result values output by executing the plurality of operations. Parameters and result values may be defined as concepts of a specified type (or class). Accordingly, the plan may include a plurality of operations and/or a plurality of concepts determined by the user's intention. The planner module 155 may determine the relationship between a plurality of operations and a plurality of concepts in a stepwise (or hierarchical) manner. For example, the planner module 155 determines a plurality of operations based on the user's intention based on a plurality of concepts (e.g., parameters required for execution of a plurality of operations, and results output by execution of the plurality of operations). The execution order can be determined. The planner module 155 may generate a plan that includes association information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 155 may create a plan using information (eg, at least one capsule) stored in the capsule database 190, which stores a set of relationships between concepts and operations.

In one embodiment, the planner module 155 may generate a plan based on an artificial intelligence (AI) system. For example, an artificial intelligence system may include a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), and/or a recurrent neural network ( It may be a recurrent neural network (RNN)), or a combination of the above, or it may be another artificial intelligence system. The planner module 155 may select a plan corresponding to a user request from a set of predefined plans or generate a plan in real time in response to a user request.

In one embodiment, the natural language generation module 157 may change designated information into text form. Information changed to text form may be in the form of natural language speech. The text-to-speech conversion module 159 can convert information in text form into information in voice form.

In one embodiment, the capsule database 190 may store information about the relationship between a plurality of concepts and operations corresponding to a plurality of domains (eg, applications). The capsule database 190 may store at least one capsule 191 or 193 in CAN (concept action network) format. For example, the capsule database 190 may store operations for processing tasks corresponding to the user's voice input and parameters necessary for the operations in CAN format. A capsule may include a plurality of action objects (or action information) and/or concept objects (or concept information) included in the plan.

In one embodiment, execution engine 170 may use the generated plan to produce results. The end user interface 180 may transmit the calculated result to the first electronic device 101.

According to one embodiment, some functions (eg, natural language platform 150) or all functions of the intelligent server 100 may be implemented in the first electronic device 101. For example, the first electronic device 101 includes a natural language platform separate from the intelligent server 100, or the natural language platform 150 of the intelligent server 100 (e.g., automatic speech recognition module 151, natural language understanding module At least some of the operations of (153), planner module (155), natural language generation module (157), and/or text-to-speech conversion module (159) may be directly performed.

The service server 103 in one embodiment may provide a designated service (eg, food ordering or hotel reservation) to the first electronic device 101. The service server 103 may be a server operated by a third party. The service server 103 may communicate with the intelligent server 100 and/or the first electronic device 101 through the first network 199. The service server 103 can communicate with the intelligent server 100 through a separate connection. The service server 103 provides information for generating a plan corresponding to the voice input received by the first electronic device 101 (e.g., operation information and/or concept information for providing a designated service) to the intelligent server 100. can be provided to. The provided information may be stored in the capsule database 190. The service server 103 may provide result information according to the plan received from the first electronic device 101 to the intelligent server 100.

Figure 2 is a block diagram showing an integrated intelligence system according to one embodiment.

Referring to FIG. 2, the integrated intelligence system includes a first electronic device 201 (e.g., the first electronic device 101 in FIG. 1) and a second electronic device 202 (e.g., the second electronic device in FIG. 1 (e.g., 102)) and an intelligent server 200 (eg, the intelligent server 100 of FIG. 1). The first electronic device 201 and the intelligent server 200 are connected to each other through a network and can transmit and receive data. The first electronic device 201 and the second electronic device 202 are connected to each other through a local area network and can transmit and receive data. According to one embodiment, the integrated intelligence system may consist of a single device or multiple devices. For example, each device may include substantially the same or similar functional configuration, and the configuration of one device may be replaced with the configuration of another device.

According to one embodiment, the intelligent server 200 may include the entire configuration or at least a partial configuration of the intelligent server 100 shown in FIG. 1. For example, the intelligent server 200 may include the natural language platform 150 and/or the capsule database 190 of the intelligent server 100 of FIG. 1 . However, the configuration of the intelligent server 200 is not limited to that shown in FIG. 2, and includes at least some components of the natural language platform 250 (e.g., automatic speech recognition module 251, natural language understanding module 253, planner module ( 255), natural language generation module 257, and/or text-to-speech conversion module 259) may be omitted, and some components of the intelligent server 100 of FIG. 1 (e.g., front end 160, execution engine ( 170) and/or an end user interface 180) may be further included.

According to one embodiment, the first electronic device 201 may include a natural language platform 260 and/or a capsule database 280. The natural language platform 260 includes an automatic speech recognition (ASR) module (261), a natural language understanding (NLU) module (263), a planner module (265), and a natural language generation module (NLG). (natural language generator) module) 267, and/or a text to speech (TTS) module 269. The automatic speech recognition module 261, the natural language understanding module 263, the planner module 265, the natural language generation module 267, and the text-to-speech conversion module 269 are the automatic speech recognition module 151 and the natural language of FIG. 1, respectively. It may perform substantially the same or similar functions as the understanding module 153, the planner module 155, the natural language generation module 157, and the text-to-speech conversion module 159.

According to one embodiment, the capsule database 280 may perform substantially the same or similar functions as the capsule databases 190 and 290 of the intelligent servers 100 and 200. The capsule database 280 may store information about relationships between a plurality of operations and a plurality of concepts included in the plan generated by the planner module 265. For example, the capsule database 280 may store at least one capsule 281 or 283.

According to one embodiment, a first electronic device 201 (e.g., natural language platform 260 and/or capsule database 280) and an intelligent server 200 (e.g., natural language platform 250 and/or capsule database ( 290)) may perform at least one function (or operation) in conjunction with each other, or may perform at least one function (or operation) independently. For example, the first electronic device 201 may perform voice recognition on its own without transmitting the received user's voice input to the intelligent server 200. As an example, the first electronic device 201 may convert a voice input received through the automatic voice recognition module 261 into text data. The first electronic device 201 can transmit the converted text data to the intelligent server 200. The intelligent server 200 may determine the user's intention and/or parameters from text data through the natural language understanding module 253. The intelligent server 200 generates a plan through the planner module 255 based on the determined intent and parameters and transmits it to the first electronic device 201, or transmits the determined intent and parameters to the first electronic device 201 A plan can be created through the planner module 265 of the first electronic device 201. The planner module 265 of the first electronic device 201 may generate at least one plan for performing a task corresponding to a voice input using information stored in the capsule database 280.

As an example, the first electronic device 201 converts the voice input received through the automatic voice recognition module 261 into text data, and determines the user's intention and/or information based on the text data through the natural language understanding module 263. Parameters can be determined. The first electronic device 201 generates a plan through the planner module 265 based on the determined intent and parameters, or transmits the determined intent and parameters to the intelligent server 200 to plan the planner module 255 of the intelligent server 200. ), you can create a plan. For example, if the first electronic device 201 does not include the planner module 265 and/or the capsule database 280, the first electronic device 201 may generate a plan through the intelligent server 200. there is.

As an example, the first electronic device 201 detects a speech pattern that is difficult to learn in the automatic speech recognition module 261 or the natural language understanding module 263, and the voice input corresponding to the detected speech pattern is sent to the intelligent server 200. It can be transmitted to and processed by the automatic voice recognition module 251 or the natural language understanding module 253 of the intelligent server 200.

Embodiments of the present disclosure are not limited to the above-described examples. For example, the first electronic device 201 may process the received voice input only within the terminal and even calculate a result corresponding to the voice input. For example, the first electronic device 201 and the intelligent server 200 not only divide voice input into modules and process them, but also process them collaboratively between corresponding modules. For example, the natural language understanding module 263 of the first electronic device 201 and the natural language understanding module 253 of the intelligent server 200 operate together to produce one result value (user intent and/or parameters). can do.

According to one embodiment, the second electronic device 202 includes an automatic speech recognition (ASR) module 262 and/or a text to speech (TTS) module 264. It can be included. The automatic speech recognition module 262 and the text-to-speech conversion module 264 may perform substantially the same or similar functions as the automatic speech recognition module 151 and the text-to-speech conversion module 159 of FIG. 1 , respectively.

According to one embodiment, the first electronic device 201 and the second electronic device 202 perform at least one function (or operation) in conjunction with each other, or each independently performs at least one function (or or action) can also be performed. For example, the second electronic device 202 may perform voice recognition for voice input using the automatic voice recognition module 262. The second electronic device 202 may perform a function corresponding to voice input based on voice recognition. For example, the second electronic device 202 may transmit a command corresponding to the recognized voice command to the first electronic device 201. The second electronic device 202 may output data received from the first electronic device 201. For example, the second electronic device 202 may convert data received from the first electronic device 201 into voice using the text-to-speech conversion module 264 and output the converted voice.

FIG. 3 illustrates a communication environment between a wireless audio device and an electronic device according to an embodiment.

Referring to FIG. 3, according to one embodiment, the electronic device 301 is at least partially the same or similar to the first electronic device 101 shown in FIG. 1 and the first electronic device 201 shown in FIG. 2. It includes components, and at least some of them may perform the same or similar functions. In addition, and the wireless audio device 302 (e.g., the first wireless audio device 302-1 and/or the second wireless audio device 302-2) includes the second electronic device 102 and At least some of the second electronic device 202 shown in FIG. 2 may include the same or similar components, and at least some of them may perform the same or similar functions. Hereinafter, unless otherwise specified, the term wireless audio device 302 refers to a first wireless audio device 302-1, a second wireless audio device 302-2, or the first and second wireless audio devices 302. -1, 302-2). The electronic device 301 may include, for example, a user terminal such as a smartphone, tablet, desktop computer, or laptop computer. The wireless audio device 302 may include, but is not limited to, wireless earphones, headsets, earbuds, or speakers. The wireless audio device 302 may include various types of devices (eg, hearing aids or portable audio devices) that receive audio signals and output the received audio signals. The term “wireless audio device” is intended to distinguish it from the electronic device 301, and “wireless audio device” may be referred to as an electronic device, wireless earphone, earbud, true wireless stereo (TWS), or earset. .

For example, the electronic device 301 and the wireless audio device 302 may perform wireless communication in a short distance according to a Bluetooth network defined by the Bluetooth™ special interest group (SIG). The Bluetooth network may include, for example, a Bluetooth legacy network or a Bluetooth low energy (BLE) network. According to one embodiment, the electronic device 301 and the wireless audio device 302 may perform wireless communication through one of a Bluetooth legacy network or a BLE network, or may perform wireless communication through two networks.

According to one embodiment, the electronic device 301 functions as a primary device (e.g., master device), and the wireless audio device 302 functions as a secondary device (e.g., slave device). can be performed. The number of devices performing the role of secondary devices is not limited to the example shown in FIG. 3. According to one embodiment, the role of the primary device or secondary device may be determined in an operation in which a link (eg, 305, 310, and/or 315) between devices is created. According to another embodiment, one device (e.g., the first wireless audio device 302-1) among the first wireless audio device 302-1 and the second wireless audio device 302-2 is the primary device. performs the role, and another device may perform the role of the secondary device.

According to one embodiment, the electronic device 301 may transmit a data packet containing content such as text, audio, image, or video to the wireless audio device 302. At least one of the electronic device 301 and the wireless audio device 302 may also transmit a data packet to the electronic device 301. For example, when music is played on the electronic device 301, the electronic device 301 connects the wireless audio device 302 to a link created (e.g., first link 305 and/or second link 310). A data packet containing content (e.g., music data) can be transmitted. For example, at least one of the wireless audio devices 302 may transmit a data packet including content (eg, audio data) to the electronic device 301 through the created link. When the electronic device 301 transmits a data packet, the electronic device 301 may be referred to as a source device, and the wireless audio device 302 may be referred to as a sink device.

According to one embodiment, the electronic device 301 creates or establishes a link with at least one device (302-1 and/or 302-2) of the wireless audio device 302 in order to transmit a data packet ( establish) can be done. For example, the electronic device 301 may establish a first link 305 with the first wireless audio device 302-1 and/or a second link with the second wireless audio device 302-2 based on the Bluetooth or BLE protocol. A link 310 can be created. In one embodiment, the electronic device 301 may communicate with the first wireless audio device 302-1 through a first link 305 with the first wireless audio device 302-1. In this case, for example, the second wireless audio device 302-2 may be set to monitor the first link 305. For example, the second wireless audio device 302-2 can receive data transmitted by the electronic device 301 through the first link 305 by monitoring the first link 305.

According to one embodiment, the second wireless audio device 302-2 may monitor the first link 305 using information associated with the first link 305. Information associated with the first link 305 may include address information (e.g., the Bluetooth address of the primary device of the first link 305, the Bluetooth address of the electronic device 301, and/or the first wireless audio device 302- 1) Bluetooth address), piconet (e.g., topology) clock information (e.g., CLKN (clock native) of the primary device of the first link 305), logical transport (LT) address information ( Example: information allocated by the primary device of the first link 305), used channel map information, link key information, service discovery protocol (SDP) information (e.g., first link It may include service and/or profile information associated with (305), and/or supported feature information.

Figure 4 shows a block diagram of electronic devices and wireless audio devices according to one embodiment.

Referring to FIG. 4, according to one embodiment, the electronic device 301 includes a processor 420 (e.g., processor 130 of FIG. 1), a memory 430 (e.g., memory 140 of FIG. 1), It may include a first communication circuit 491, a display 460, and/or a second communication circuit 492. The processor 420 may be operatively connected to the memory 430, the display 460, the first communication circuit 491, and the second communication circuit 492. The memory 430 may store one or more instructions that, when executed, cause the processor 420 to perform various operations of the electronic device 301. The second communication circuit 492 may be configured to support wireless communication based on a Bluetooth protocol (eg, Bluetooth legacy and/or BLE). The first communication circuit 491 may be configured to support communication based on wireless communication standards (eg, cellular and/or Wi-Fi) other than the Bluetooth protocol. The electronic device 301 may further include components not shown in FIG. 4 . For example, the electronic device 301 may further include an audio input/output device and/or a housing.

According to one embodiment, the electronic device 301 may be connected to the first wireless audio device 302-1 through a first link 305. For example, the electronic device 301 and the first wireless audio device 302-1 may communicate in units of time slots set based on the clock of the primary device of the first link 305. The electronic device 301 may be connected to the second wireless audio device 302-2 through a second link 310. For example, the electronic device 301 may establish the second link 310 after connection with the first wireless audio device 302-1. In one example, second link 310 may be omitted.

According to one embodiment, the first wireless audio device 302-1 includes a processor 521 (e.g., processor 131 in FIG. 1), a memory 531 (e.g., memory 141 in FIG. 1), and a sensor circuit. 551, may include an audio output circuit 571, an audio reception circuit 581, and/or a communication circuit 591.

According to one embodiment, the processor 521 may be operatively connected to the sensor circuit 551, the communication circuit 591, the audio output circuit 571, the audio reception circuit 581, and the memory 531. there is.

According to one embodiment, the sensor circuit 551 may include at least one sensor. The sensor circuit 551 may detect information about the wearing state of the first wireless audio device 302-1, the wearer's biometric information, and/or movement. For example, the sensor circuit 551 may include a proximity sensor for detecting a wearing state, a biometric sensor (e.g., a heart rate sensor) for detecting biometric information, and/or a motion sensor (e.g., an acceleration sensor) for detecting movement. You can. In one embodiment, the sensor circuit 551 may further include at least one of a bone conduction sensor or an acceleration sensor. In another embodiment, the acceleration sensor may be placed close to the skin to detect bone conduction. For example, an acceleration sensor may be set to detect tremor information in kHz units using sampling in kHz units, which is relatively higher than general motion sampling. The processor 521 can perform voice identification, voice detection, tap detection, and/or wearing detection in a noisy environment by using tremor centered on a significant axis (x, y, z axis) among the tremor information of the acceleration sensor. You can.

According to one embodiment, the audio output circuit 571 may be set to output sound. The audio receiving circuit 581 may include one or multiple microphones. The audio receiving circuit 581 may be configured to detect an audio signal using one or multiple microphones. In one embodiment, each of the plurality of microphones may correspond to a different audio reception path. For example, when the audio receiving circuit 581 includes a first microphone and a second microphone, the audio signal acquired by the first microphone and the audio signal obtained by the second microphone may be referred to as different audio channels. . The processor 521 may acquire audio data using at least one microphone among a plurality of microphones connected to the audio reception circuit 581. For example, the processor 521 may dynamically select or determine at least one microphone for acquiring audio data among a plurality of microphones. The processor 521 may acquire audio data by performing beamforming using a plurality of microphones. The memory 531 may store one or more instructions that, when executed, cause the processor 521 to perform various operations of the first wireless audio device 302-1.

According to one embodiment, the processor 521 may acquire audio data using at least one of the audio receiving circuit 581 or the sensor circuit 551. For example, the processor 521 may acquire audio data using one or more microphones connected to the audio receiving circuit 581. The processor 521 may acquire audio data by detecting vibration corresponding to the audio signal using the sensor circuit 551. For example, the processor 521 may acquire audio data using at least one of a motion sensor, a bone conduction sensor, or an acceleration sensor. The processor 521 may be configured to process (e.g., noise suppression, noise removal, or echo cancellation) audio data acquired through various paths (e.g., at least one of the audio receiving circuit 581 or the sensor circuit 551). there is.

According to one embodiment, the first wireless audio device 302-1 may further include a component not shown in FIG. 4. For example, the first wireless audio device 302-1 may further include an indicator, an input interface, and/or a housing.

According to one embodiment, the second wireless audio device 302-2 includes a processor 522 (e.g., processor 131 in FIG. 1), a memory 532 (e.g., memory 141 in FIG. 1), and a sensor. It may include circuitry 552, audio output circuitry 572, audio reception circuitry 582, and/or communication circuitry 592.

According to one embodiment, processor 522 may be operatively coupled to communication circuitry 592, audio output circuitry 572, audio reception circuitry 582, and memory 532.

According to one embodiment, the sensor circuit 552 may detect information about the wearing state of the second wireless audio device 302-2, the wearer's biometric information, and/or movement information. For example, the sensor circuit 552 may include a proximity sensor for detecting a wearing state, a biometric sensor (e.g., a heart rate sensor) for detecting biometric information, and/or a motion sensor (e.g., an acceleration sensor) for detecting movement. You can. In one embodiment, the sensor circuit 552 may further include at least one of a bone conduction sensor or an acceleration sensor. In another embodiment, the acceleration sensor may be placed close to the skin to detect bone conduction. For example, an acceleration sensor may be set to detect tremor information in kHz units using sampling in kHz units, which is relatively higher than general motion sampling. The processor 522 can perform voice identification, voice detection, tap detection, and/or wearing detection in a noisy environment by using tremor centered on a significant axis (out of x, y, and z axes) among the tremor information of the acceleration sensor. You can.

According to one embodiment, the audio output circuit 572 may be set to output sound. Audio receiving circuitry 582 may include one or multiple microphones. The audio receiving circuit 582 may be configured to detect an audio signal using one or multiple microphones. In one embodiment, each of the plurality of microphones may correspond to a different audio reception path. For example, when the audio receiving circuit 582 includes a first microphone and a second microphone, the audio signal acquired by the first microphone and the audio signal obtained by the second microphone may be referred to as different audio channels. . The processor 522 may acquire audio data by performing beamforming using a plurality of microphones.

According to one embodiment, the memory 532 may store one or more instructions that, when executed, cause the processor 522 to perform various operations of the second wireless audio device 302-2.

According to one embodiment, the processor 522 may acquire audio data using at least one of the audio receiving circuit 582 or the sensor circuit 552. For example, the processor 522 may acquire audio data using one or more microphones connected to the audio receiving circuit 582. The processor 522 may acquire audio data by detecting vibration corresponding to the audio signal using the sensor circuit 552. For example, the processor 522 may acquire audio data using at least one of a motion sensor, a bone conduction sensor, or an acceleration sensor. Processor 522 may be configured to process (e.g., noise suppression, noise removal, or echo cancellation) audio data acquired through various paths (e.g., at least one of audio receiving circuitry 582 or sensor circuitry 552). there is.

In one embodiment, the second wireless audio device 302-2 may further include a component not shown in FIG. 4. For example, the second wireless audio device 302-2 may further include an indicator (e.g., I/O interface 121 in FIG. 1), an audio input device, an input interface, and/or a housing. there is.

Figure 5 shows front and rear views of a first wireless audio device according to one embodiment.

With reference to Figure 5, the structure of the first wireless audio device 302-1 is described. For convenience of explanation, redundant descriptions are omitted, but the second wireless audio device 302-2 may have a structure substantially the same or similar to that of the first wireless audio device 302-1.

In one embodiment, reference numeral 501 depicts a front view of the first wireless audio device 302-1. The first wireless audio device 302-1 may include a housing 510. The housing 510 may form at least a portion of the exterior of the first wireless audio device 302-1. It may include a button 513 and a plurality of microphones 581a and 581b disposed on the first side of the housing 510 (eg, the side facing the outside of the ear when worn). Button 513 may be set to receive user input (eg, touch input or push input). The first microphone 581a and the second microphone 581b may be included in the audio receiving circuit 581 of FIG. 4. The first microphone 581a and the second microphone 581b may be arranged to detect sound in a direction toward the outside of the user when the first wireless audio device 302-1 is worn. The first microphone 581a and the second microphone 581b may be referred to as external microphones. The first microphone 581a and the second microphone 581b can detect sound from outside the housing 510. For example, the first microphone 581a and the second microphone 581b can detect sounds generated around the first wireless audio device 302-1. The sound of the surrounding environment detected by the first wireless audio device 302-1 may be output through the speaker 570. In one embodiment, the first microphone 581a and the second microphone 581b may be microphones for collecting noise for a noise cancellation function (e.g., active noise cancellation (ANC)) of the first wireless audio device 302-1. . Additionally, the first microphone 581a and the second microphone 581b may be microphones for collecting ambient sounds (eg, transparency function or ambient awareness function) of the first wireless audio device 302-1. For example, the first microphone 581a and the second microphone 581b may include various types of microphones, including electronic condenser microphones (ECM) and micro electro mechanical system (MEMS) microphones. A wing tip 511 may be coupled to the circumference of the housing 510. At least a portion of the wing tip 511 may be formed of an elastic material. The wing tip 511 may be detached from the housing 510 or attached to the housing 510. The wingtip 511 can improve the wearability of the first wireless audio device 302-1.

According to one embodiment, reference numeral 502 depicts a rear view of the first wireless audio device 302-1. A first electrode 514, a second electrode 515, a proximity sensor 550, a third microphone 581c disposed on the second side of the housing 510 (e.g., the side facing the user when worn), and It may include a speaker 570. The speaker 570 may be included in the audio output circuit 571 of FIG. 4. The speaker 570 can convert electrical signals into sound signals. The speaker 570 can output sound to the outside of the first wireless audio device 302-1. For example, the speaker 570 can convert an electrical signal into a sound that can be perceived by the user and output it. At least a portion of the speaker 570 may be disposed inside the housing 510. The speaker 570 may be coupled to the eartip 512 through one end of the housing 510. The eartip 512 may be formed in a cylindrical shape with a hollow interior. For example, when the eartip 512 is combined with the housing 510, sound (audio) output from the speaker 570 may be transmitted to an external object (e.g., a user) through the hollow of the eartip 512. there is.

According to one embodiment, the first wireless audio device 302-1 may include a sensor 551a (e.g., an acceleration sensor, a bone conduction sensor, and/or a gyro sensor) disposed on the second side of the housing 510. You can. The location and shape of the sensor 551a shown in FIG. 5 are illustrative, and the embodiments of this document are not limited thereto. For example, the sensor 551a may be placed inside the housing 510 and not exposed to the outside. The sensor 551a may be located in a position that, when worn, may contact the wearer's ear or in a portion of the housing 510 that contacts the wearer's ear.

According to one embodiment, the eartip 512 may be formed of an elastic material (or flexible material). The eartip 512 can assist the first wireless audio device 302-1 to be inserted in close contact with the user's ear. For example, the ear tips 512 may be made of silicone material. At least one area of the eartip 512 may be deformed according to the shape of an external object (eg, the shape of the ear kernel). According to various embodiments of the present invention, the eartip 512 may be formed by a combination of at least two of silicone, foam, and plastic materials. For example, the area of the ear tip 512 that is inserted into the user's ear and comes into contact with it may be made of a silicone material, and the area where the housing 510 is inserted may be made of a plastic material. The eartip 512 may be detached from or attached to the housing 510 . The first electrode 514 and the second electrode 515 are connected to an external power source (eg, a case) and can receive an electrical signal from the external power source. The proximity sensor 550 can be used to detect the user's wearing state. The proximity sensor 550 may be placed inside the housing 510. The proximity sensor 550 may be disposed so that at least a portion is exposed to the exterior of the first wireless audio device 302-1. The first wireless audio device 302-1 may determine whether the user of the first wireless audio device 302-1 is wearing the device based on data measured by the proximity sensor 550. For example, proximity sensor 550 may include an IR sensor. The IR sensor may detect whether the housing 510 is in contact with the user's body, and the first wireless audio device 302-1 may detect the first wireless audio device 302-1 based on the detection of the IR sensor. You can decide whether to wear it or not. The proximity sensor 550 is not limited to an IR sensor and may be implemented using various types of sensors (eg, an acceleration sensor or a gyro sensor). The third microphone 581c may be arranged to detect sound in a direction toward the user when the first wireless audio device 302-1 is worn. The third microphone 581c may be referred to as an internal microphone.

Figure 6 shows a block diagram of a wireless audio device according to one embodiment.

Referring to Figure 6, according to one embodiment, components of wireless audio device 302 may include example software modules. For example, the above configurations include a first wireless audio device (e.g., the first wireless audio device 302-1 in FIGS. 3 to 5) or a second wireless audio device (e.g., the second wireless audio device in FIGS. 3 and 4). It can be implemented by an audio device 302-2). At least some of the above configurations may be omitted. At least some of the above configurations may be implemented as one software module. The division of the configurations is logical, and any program, thread, application, or code that performs the same function may correspond to the configurations.

According to one embodiment, the pre-processing module 610 includes a first audio receiving circuit (e.g., the audio receiving circuit 581 or 582 in FIG. 5) and a second audio receiving circuit (e.g., the audio receiving circuit 581 or 582 in FIG. 7). 2 Preprocessing of the received sound (audio) (or audio signal) can be performed using the audio receiving circuit 583). The second audio receiving circuit 583 may be included in a wireless audio device (e.g., the first wireless audio device 302-1 and the second wireless audio device 302-2 in FIG. 5). The second audio receiving circuit 583 may receive an audio signal (eg, a reference signal) from an electronic device (eg, the electronic device 301 of FIG. 5). The reference signal may correspond to media played on the electronic device 301. For example, the preprocessing module 610 may remove echo of the acquired audio signal using an acoustic echo canceller (AEC) 611. The preprocessing module 610 can reduce noise of the acquired audio signal using noise suppression (NS) 612. The pre-processing module 610 may use a high pass filter (HPF) 613 to reduce signals in a designated band of the acquired audio signal. The preprocessing module 610 can change the sampling rate of the audio input signal using the converter 614. For example, the converter 614 may be set to perform downsampling or upsampling on an audio input signal. The preprocessing module 610 may selectively apply at least one of the AEC 611, NS 612, HPF 613, or converter 614 to the audio signal.

According to one embodiment, the phase determination module 620 controls the wireless audio devices 302-1 and 302-2 based on one or more of whether media is played on the electronic device 301 and information associated with the electronic device 301. may decide to enter one of the first mode change phase and the second mode change phase. Information related to the electronic device 301 may include one or more of environmental information of the electronic device 301, location information of the electronic device 301, and information about devices in the vicinity of the electronic device 301.

According to one embodiment, the first mode change phase may be for determining a change to one of the operation modes, the singing mode and the conversation mode. The second mode change phase may be for determining the change to conversation mode.

According to one embodiment, the dialogue mode module 625 may determine activation and deactivation of the dialogue mode. For example, the conversation mode module 625 may use the first voice activity detection (VAD) 621 to detect whether the wearer (eg, user) of the wireless audio device 302 speaks. The conversation mode module 625 can use the second VAD 622 to detect whether the wearer speaks or whether an outsider speaks. The conversation mode module 625 can identify and/or specify the wearer's speech section through the first VAD (621). The conversation mode module 625 can identify and/or specify the speech section of an outsider through the first VAD 621 and the second VAD 622. For example, the conversation mode module 625 identifies the utterance section of an outsider by excluding the section in which the wearer's utterance is identified through the first VAD 621 among the sections in which the utterance is identified through the second VAD 622. and/or may be specified. The conversation mode module 625 can determine whether to run or deactivate the voice agent using the first VAD 621, the second VAD 622, and the conversation mode function 623.

According to one embodiment, the conversation mode module 625 can detect whether the user speaks and whether an outsider speaks using the first VAD 621 and the second VAD 622. In one example, the conversation mode module 625 uses the audio signal preprocessed by the preprocessing module 610 or the audio signal not processed by the preprocessing module 610 to process the first VAD 621 or the second VAD 622. ) can be performed at least one of the following. Referring to FIG. 4, the wireless audio device 302 can receive an audio signal using the audio reception circuits 581 and 582. The wireless audio device 302 may detect movement of the wireless audio device 302 using sensor circuits 551 and 552 (e.g., a motion sensor, an acceleration sensor, and/or a gyro sensor). For example, when an audio signal (e.g., a voice signal) of a specified size or greater is detected in a designated band (e.g., a human vocal range band), the wireless audio device 302 may detect a voice signal from the audio signal. If specified movement is detected simultaneously or substantially simultaneously while the audio signal is being detected, the wireless audio device 302 may detect user utterance (e.g., wearer utterance) based on the audio signal. For example, the designated movement may be a movement detected by the wireless audio device 302 due to speech by the wearer of the wireless audio device 302. For example, movement due to the wearer's speech may be transmitted to the motion sensor, acceleration sensor, and/or gyro sensor in the form of movement or vibration. Movement caused by the wearer's speech may flow into the motion sensor, acceleration sensor, and/or gyro sensor in a similar way to input from a bone conduction microphone. The wireless audio device 302 can obtain information about the activation time and end time of the wearer's speech based on designated movement and voice signals. If no specified movement is detected while the audio signal is being detected, simultaneously or substantially simultaneously, the wireless audio device 302 may detect an outsider utterance (e.g., an utterance from a person other than the wearer (e.g., an outsider or an opponent)) based on the audio signal. ) can be detected. The wireless audio device 302 can obtain information about the activation time and end time of an outsider's speech based on designated movement and voice signals. The conversation mode module 625 stores information about the activation and end time of the user or an outsider's utterance in a memory (e.g., memories 531 and 532 in FIG. 4), and based on the information stored in the memories 531 and 532, You can decide to enable or disable conversation mode.

For example, the first VAD 621 and the second VAD 622 may be serial processes. When a voice signal is detected using the second VAD 622, the wireless audio device 302 detects movement using a motion sensor (e.g., an acceleration sensor and/or a gyro sensor), so that the voice signal responds to the user's utterance. You can identify what you are doing.

For example, the first VAD 621 and the second VAD 622 may be parallel processes. For example, the first VAD 621 may be set to detect user utterance independently from the second VAD 622. The second VAD 622 may be set to detect voice signals regardless of whether the user speaks.

For example, the wireless audio device 302 may use different microphones to detect the user's speech and the speech of an outsider. The wireless audio device 302 may use an external microphone (eg, the first microphone 581a and the second microphone 581b in FIG. 5) to detect an outsider's speech. The wireless audio device 302 may use an internal microphone (eg, the third microphone 581c in FIG. 5) to detect the user's speech. When using an internal microphone, the electronic device 302 can determine whether the wearer speaks based on a voice signal and movement information based on the internal microphone. The wireless audio device 302 may determine whether the wearer speaks based on the voice signal received through the sensor input to detect the user's speech. Signals flowing into the sensor input may include at least one acceleration sensor and gyro sensor input.

According to one embodiment, the conversation mode module 625 may determine activation of the conversation mode using the first VAD 621 and/or the second VAD 622. In the conversation mode OFF state, the conversation mode module 625 can determine whether to activate the conversation mode. For example, the conversation mode module 625 may determine activation of the conversation mode when the user's speech is maintained for a specified time interval (eg, more than L frames, where L is a natural number). For another example, the conversation mode module 625 may determine activation of the conversation mode when the other party's speech is maintained for a specified time period after deactivating the user's speech.

According to one embodiment, the conversation mode module 625 may determine whether to maintain or deactivate the conversation mode using the first VAD 621 and/or the second VAD 622. In the conversation mode ON state, the conversation mode module 625 can determine whether to maintain or deactivate the conversation mode. For example, during conversation mode, the conversation mode module 625 may determine to deactivate the conversation mode if a voice signal is not detected for a specified time period. During the conversation mode, the conversation mode module 625 may determine whether to maintain the conversation mode when a voice signal is detected within a specified time interval from the deactivation of the previous voice signal.

According to one embodiment, conversation mode module 625 may determine activation and/or deactivation of conversation mode based on conversation mode function 623. Conversation mode function 623 may detect conversation mode activation and/or deactivation based on user input. For example, the user input may include the user's voice command, the user's touch input, or the user's button input.

According to one embodiment, the conversation mode module 625 may determine the length of the designated time period based on surrounding sounds. For example, the conversation mode module 625 may determine the length of the designated time section based on at least one of the sensitivity of background noise of sound acquired using an external microphone, the SNR value, or the type of noise. In a noisy environment, the conversation mode module 625 may increase the length of the designated time interval.

According to one embodiment, the conversation mode module 625 may determine whether to activate and/or deactivate the conversation mode based on the user's voice command. In one example, the voice agent module 630 may detect a user's voice command instructing activation of the conversation mode, and transmit information instructing activation of the conversation mode to the conversation mode function 623 in response to detection of the voice command. there is. The voice command instructing to activate the conversation mode may include a wake-up utterance (e.g., Hi Bixby) and a voice command to wake up the voice agent. For example, the voice command may take the form of “Hi Bixby, activate conversation mode.” As another example, a voice command instructing to activate conversation mode may have a form such as “Activate conversation mode,” which does not include a wake-up utterance. When the conversation mode function 623 receives information indicating activation of the conversation mode from the voice agent module 630, the conversation mode module 625 may determine activation of the conversation mode. In one example, the voice agent module 630 detects a voice command instructing the user to deactivate the conversation mode, and transmits information indicating deactivation of the conversation mode to the conversation mode function 623 in response to detection of the voice command. You can. For example, a voice command instructing to deactivate the conversation mode may include a wake-up utterance and a voice command to wake up the voice agent. The voice command may take the form of “Hi Bixby, disable conversation mode.” For example, a voice command instructing to deactivate conversation mode may have a form such as “Deactivate conversation mode,” which does not include a wake-up utterance. When the conversation mode function 623 receives information indicating deactivation of the conversation mode from the voice agent module 630, the conversation mode module 625 may determine to deactivate the conversation mode.

According to one embodiment, the conversation mode module 625 may determine activation and/or deactivation of the conversation mode based on the user's touch input. For example, the electronic device 301 may provide an interface for controlling the conversation mode of the wireless audio device 302. Through the interface, the electronic device 301 may receive a user's input for activating or deactivating the conversation mode. When a user input indicating activation of the conversation mode is received, the electronic device 301 may transmit a signal indicating activation of the conversation mode to the wireless audio device 302. If the conversation mode function 623 obtains information indicating activation of the conversation mode from the signal, the conversation mode module 625 can determine activation of the conversation mode. When a user input indicating deactivation of the conversation mode is received through the interface, the electronic device 301 may transmit a signal indicating deactivation of the conversation mode to the wireless audio device 302. If the conversation mode function 623 obtains information from the signal indicating deactivation of the conversation mode, the conversation mode module 625 can determine deactivation of the conversation mode.

According to one embodiment, when the conversation mode module 625 determines whether to activate or deactivate the conversation mode, the wireless audio device 302 transmits a signal to the electronic device 301 indicating that the conversation mode has been determined to be activated or deactivated. can do. The electronic device 301 may provide information indicating that activation or deactivation of the conversation mode obtained from the signal has been determined through an interface for controlling the conversation mode of the wireless audio device 302.

According to one embodiment, the conversation mode module 625 may determine activation and/or deactivation of the conversation mode based on the user's button input. For example, the wireless audio device 302 may include at least one button (eg, button 513 in FIG. 5). The conversation mode feature 623 can be configured to detect designated input on a button (e.g., a double tap or long press). When an input indicating activation of the conversation mode is received through a button, the conversation mode module 625 can determine activation of the conversation mode. When an input indicating deactivation of the conversation mode is received through a button, the conversation mode module 625 can determine deactivation of the conversation mode.

According to one embodiment, conversation mode function 623 may be configured to interact with voice agent module 630. For example, the conversation mode function 623 may obtain information from the voice agent module 630 indicating whether the utterance is for a voice agent call. For example, the wearer's speech that continues for more than a specified time may be detected by the first VAD (621). In this case, the conversation mode module 625 can use the conversation mode function 623 to identify whether the wearer's utterance is in response to a call from a voice agent. If the conversation mode function 623 uses the voice agent module 630 to confirm that a voice agent call has been made due to the corresponding utterance, the conversation mode module 625 may ignore the corresponding utterance. For example, even if the corresponding utterance continues for more than a specified time, the conversation mode module 625 may not decide to activate the conversation mode based on the corresponding utterance alone. For example, the voice agent module 630 may identify a voice command instructing activation of the conversation mode from the corresponding utterance. In this case, the voice agent module 630 transmits a signal indicating activation of the conversation mode to the conversation mode module 625, and the conversation mode module 625 can determine activation of the conversation mode. That is, in this case, the conversation mode module 625 may determine activation of the conversation mode based on the instruction of the voice agent module 630, rather than the length of the utterance itself.

According to one embodiment, the conversation mode module 625 may determine deactivation of the conversation mode based on the operating time of the conversation mode. For example, after a certain amount of time has elapsed after the conversation mode is turned on, the conversation mode module 625 may determine to deactivate the conversation mode.

According to one embodiment, the singing mode module 627 may determine activation and deactivation of the singing mode. In the first mode change phase, the singing mode module 627 activates the singing mode based on whether the analysis result of the audio signal received by the wireless audio devices (302-1, 302-2) satisfies the activation conditions of the singing mode. and deactivation can be decided. The activation conditions for the singing mode may be classified according to the sensitivity level of the electronic device 301 among the first sensitivity level, second sensitivity level, and third sensitivity level.

According to one embodiment, the activation condition according to the first sensitivity level may include a condition regarding whether a singing voice among ambient sounds is continuously detected for a predetermined time. The activation condition according to the second sensitivity level may include a condition regarding the acoustic similarity between the singing voice included in the surrounding sound and the media. Ambient sounds and media may be included in the audio signal. The activation condition according to the third sensitivity level may include a condition regarding the similarity between the singing voice included in the ambient sound and the lyrics included in each media.

According to one embodiment, the activation condition of the singing mode may include activation conditions according to all levels below the sensitivity level of the electronic device 301. For example, when the sensitivity level of the electronic device 301 is the second sensitivity level, the activation condition of the singing mode includes activation conditions according to the first sensitivity level and the second sensitivity level, and the sensitivity of the electronic device 301 When the level is the third sensitivity level, the activation condition of the singing mode may include activation conditions according to the first sensitivity level, the second sensitivity level, and the third sensitivity level.

According to one embodiment, singing mode module 627 may determine activation and/or deactivation of conversation mode. The singing mode module 627 may detect singing mode activation and/or deactivation based on user input. For example, the user input may include the user's voice command, the user's touch input, or the user's button input.

According to one embodiment, the singing mode module 627 may determine the length of the designated time section based on surrounding sounds. For example, the singing mode module 627 may determine the length of the designated time section based on at least one of the sensitivity of background noise of sound acquired using an external microphone, the value of SNR, or the type of noise. In a noisy environment, the singing mode module 627 may increase the length of the designated time interval.

According to one embodiment, the singing mode module 627 may determine activation and/or deactivation of the singing mode based on the user's voice command. In one example, the voice agent module 630 detects a voice command instructing the user to activate the singing mode, and transmits information instructing the activation of the singing mode to the singing mode module 627 in response to detection of the voice command. there is. The voice command instructing to activate the singing mode may include a wake-up utterance (e.g., Hi Bixby) and a voice command to wake up the voice agent. For example, the voice command may take the form of “Hi Bixby, activate singing mode.” As another example, a voice command instructing to activate singing mode may have a form such as “Activate singing mode,” which does not include a wake-up utterance. When the singing mode module 627 receives information indicating activation of the singing mode from the voice agent module 630, the singing mode module 627 may determine activation of the singing mode. In one example, the voice agent module 630 detects a voice command instructing the user to deactivate the singing mode, and transmits information indicating deactivation of the singing mode in response to detection of the voice command to the singing mode module 627. You can. For example, a voice command instructing to deactivate the singing mode may include a wake-up utterance and a voice command for waking up the voice agent. The voice command may take the form of “Hi Bixby, disable singing mode.” For example, a voice command instructing to deactivate singing mode may have a form such as “Deactivate singing mode,” which does not include a wake-up utterance. When the singing mode module 627 receives information indicating deactivation of the singing mode from the voice agent module 630, the singing mode module 627 may determine deactivation of the singing mode.

According to one embodiment, the singing mode module 627 may determine activation and/or deactivation of the singing mode based on the user's touch input. For example, the electronic device 301 may provide an interface for controlling the singing mode of the wireless audio device 302. Through the interface, the electronic device 301 may receive a user's input for activating or deactivating the singing mode. When a user input indicating activation of the singing mode is received, the electronic device 301 may transmit a signal indicating activation of the singing mode to the wireless audio device 302. When the singing mode module 627 obtains information indicating activation of the singing mode from the signal, the singing mode module 627 may determine activation of the singing mode. When a user input indicating deactivation of the singing mode is received through the interface, the electronic device 301 may transmit a signal indicating deactivation of the singing mode to the wireless audio device 302. If the singing mode module 627 obtains information indicating deactivation of the singing mode from the signal, the singing mode module 627 may determine deactivation of the singing mode.

According to one embodiment, when the singing mode module 627 determines activation or deactivation of the singing mode, the wireless audio device 302 transmits a signal to the electronic device 301 indicating that activation or deactivation of the singing mode has been determined. can do. The electronic device 301 may provide information indicating that activation or deactivation of the singing mode obtained from the signal has been determined through an interface for controlling the singing mode of the wireless audio device 302.

According to one embodiment, the singing mode module 627 may determine activation and/or deactivation of the singing mode based on the user's button input. For example, the wireless audio device 302 may include at least one button (eg, button 513 in FIG. 5). Singing mode module 627 can be configured to detect designated input on a button (e.g., double tap or long press). When an input instructing activation of the singing mode is received through a button, the singing mode module 627 may determine activation of the singing mode. When an input instructing deactivation of the singing mode is received through a button, the singing mode module 627 may determine deactivation of the singing mode.

According to one embodiment, the singing mode module 627 may be configured to interact with the voice agent module 630. For example, the singing mode module 627 may obtain information indicating whether the utterance is for a voice agent call from the voice agent module 630. For example, the wearer's speech that continues for more than a specified time may be detected by the first VAD (621). In this case, the singing mode module 627 can identify whether the wearer's utterance is in response to a voice agent's call. When the singing mode module 627 confirms using the voice agent module 630 that a voice agent call has been made by the corresponding utterance, the singing mode module 627 may ignore the corresponding utterance. For example, even if the singing voice included in the corresponding utterance continues for more than a specified time, the singing mode module 627 may not decide to activate the singing mode based on the corresponding utterance alone. For example, the voice agent module 630 may identify a voice command instructing activation of the singing mode from the corresponding utterance. In this case, the voice agent module 630 transmits a signal indicating activation of the singing mode to the singing mode module 627, and the singing mode module 627 may determine activation of the singing mode. That is, in this case, the singing mode module 627 may determine activation of the singing mode based on the instructions of the voice agent module 630, rather than the activation condition of the singing mode.

According to one embodiment, the singing mode module 627 may determine deactivation of the singing mode in the singing mode. For example, the singing mode module 627 determines deactivation of the singing mode when the analysis result of the audio signal received by the wireless audio devices (302-1, 302-2) in the singing mode no longer satisfies the activation conditions. You can. As another example, the singing mode module 627 may determine deactivation of the singing mode based on whether media is playing and information associated with the electronic device 301. In this case, the singing mode module 627 determines that media is no longer played on the electronic device 301 or that the singing mode is not needed according to information associated with the electronic device 301, thereby determining deactivation of the singing mode. .

According to one embodiment, the wireless audio devices (302-1, 302-2), in the singing mode, track the singing voice included in the surrounding sound using the singing mode module 627, and provide the user with information about the singing voice and media. We can provide guidance on this. For example, the wireless audio devices 302-1 and 302-2 may provide guide information about the media to the user when the user selects to provide a song guide or when the similarity between the singing voice and the media is low. Guide information about the media may include main melody information that can be sung along with the media (e.g., a song), a beat, or lyrics to be played in the next verse of the song. Guide information about media may be output as low-pitched audio through TTS generation through the wireless audio device 302, or may be displayed as visual information on the screen of the electronic device 301.

According to one embodiment, the voice agent module 630 may include a wake-up speech recognition module 631 and a voice agent control module 632. In one example, the voice agent module 630 may further include a voice command recognition module 633. The wake-up speech recognition module 631 may acquire an audio signal using the audio reception circuits 581 and 582 and recognize a wake-up speech (eg, Hi Bixby) from the audio signal. When the wake-up speech recognition module 631 recognizes a designated voice command, it can control the voice agent using the voice agent control module 632. For example, the voice agent control module 632 may transmit a received voice signal to the electronic device 301 and receive a task or command corresponding to the voice signal from the electronic device 301. For example, when a voice signal instructs volume adjustment, the electronic device 301 may transmit a signal instructing volume adjustment to the wireless audio device 302. The voice command recognition module 633 can obtain an audio signal using the audio reception circuits 581 and 582 and recognize a designated voice command from the audio signal. In one embodiment, the designated voice utterance may include a voice command for controlling the conversation mode (eg, activating the conversation mode, deactivating the conversation mode). The voice command recognition module 633 can perform a function corresponding to the designated voice command when the designated voice command is recognized even without recognition of the wake-up utterance. For example, when the voice command recognition module 633 recognizes the utterance of a designated command such as “deactivate conversation mode” or “deactivate singing mode,” it provides a signal instructing the electronic device 301 to deactivate the conversation mode or singing mode. can be transmitted. For example, the voice command recognition module 633 may perform a function corresponding to a designated voice command without interaction with a voice agent. The electronic device 301 may perform sound control of the wireless audio device 302, which will be described later, in response to a signal indicating deactivation of a specific mode (eg, conversation mode, singing mode).

According to one embodiment, the conversation mode module 625 may transmit a decision about the conversation mode (eg, deactivating the conversation mode or activating the conversation mode) to the conversation mode control module 655. The conversation mode control module 655 may control functions of the wireless audio device 302 according to activation and/or deactivation of the conversation mode. For example, the conversation mode control module 655 may control the output signal of the wireless audio device 302 using the sound control module 640 according to activation and/or deactivation of the conversation mode.

According to one embodiment, the singing mode module 627 may transmit a decision about the singing mode (eg, deactivation of the singing mode or activation of the singing mode) to the singing mode control module 657. The singing mode control module 657 may control the functions of the wireless audio device 302 according to activation and/or deactivation of the singing mode. For example, the singing mode control module 657 may control the output signal of the wireless audio device 302 using the sound control module 640 according to activation and/or deactivation of the singing mode.

For example, the sound control module 640 may include an active noise cancellation (ANC) control module 641 and an ambient sound control module 642. The ANC control module 641 may be set to acquire ambient sounds and perform noise removal based on the ambient sounds. For example, the ANC control module 641 may acquire ambient sounds using an external microphone and perform noise removal using the acquired ambient sounds. The ambient sound control module 642 can be set to provide ambient sounds to the wearer. For example, the ambient sound control module 642 may be set to provide ambient sound by acquiring ambient sound using an external microphone and outputting the acquired ambient sound using the speaker of the wireless audio device 302. .

According to one embodiment, when the conversation mode is activated, the conversation mode control module 655 can control the output signal of the wireless audio device 302 using the sound control module 640. For example, conversation mode control module 655 may disable ANC and activate ambient sound in response to activation of conversation mode. For another example, when music is being output from the wireless audio device 302, the conversation mode control module 655 reduces the volume level of the music being output by a certain percentage or more in response to activation of the conversation mode, or sets it to maximum mute. You can. The user of the wireless audio device 302 can hear surrounding sounds more clearly by activating the conversation mode.

According to one embodiment, when the conversation mode is deactivated, the conversation mode control module 655 can control the output signal of the wireless audio device 302 using the sound control module 640. For example, the conversation mode control module 655 may restore the ANC settings and/or ambient sound settings to settings prior to activation of the conversation mode and disable ambient sounds in response to deactivation of the conversation mode. For example, before activating the conversation mode, the conversation mode control module 655 may store ANC settings and/or ambient sound settings in the memories 531 and 532. When the conversation mode is deactivated, the conversation mode control module 655 may activate or deactivate ANC and/or ambient sound according to the ANC settings and/or ambient sound settings stored in the memories 531 and 532.

As another example, the conversation mode control module 655 may restore the output signal of the wireless audio device 302 to a setting prior to activation of the conversation mode in response to deactivation of the conversation mode. For example, if music is being output from the wireless audio device 302 before activating the conversation mode, the conversation mode control module 655 may store the music output signal settings in the memories 531 and 532. When the conversation mode is deactivated, the conversation mode control module 655 can restore the music output signal to the music output signal settings stored in the memories 531 and 532. In conversation mode, the conversation mode control module 655 can reduce or mute the media output volume to a specified value according to settings. In conversation mode, the wireless audio device 302 may output a voice agent's notification (e.g., a response to a user utterance) independently of the volume of the conversation mode. For example, in conversation mode, the wireless audio device 302 may output a voice agent's notification (eg, TTS-based response) at a specified volume value.

According to one embodiment, the conversation mode control module 655 may control the output signal using the sound control module 640 during operation in the conversation mode. For example, conversation mode control module 655 may control ANC and/or the intensity of ambient sounds. The conversation mode control module 655 can control the gain value of the surrounding sound to amplify the intensity of the surrounding sound. The conversation mode control module 655 may amplify only the section where a voice is present or a frequency band corresponding to the voice from the surrounding sound. In conversation mode, conversation mode control module 655 can reduce the intensity of ANC. The conversation mode control module 655 can control the output volume of the audio signal.

Tables 1 and 2 below show examples of sound control of the conversation mode control module 655 according to conversation mode activation (eg, ON) and deactivation (eg, OFF).

[Table 1]

Referring to Table 1, the wearer of the wireless audio device 302 may be listening to music using the wireless audio device 302. For example, the wireless audio device 302 can output music while performing ANC. For example, the wireless audio device 302 may output music at a first volume. Depending on activation of the conversation mode, the conversation mode control module 655 may activate ambient sound and deactivate ANC. In this case, the conversation mode control module 655 may reduce the volume of music being output below a specified value or by a specified ratio. For example, the conversation mode control module 655 may reduce the volume of music being output to a second value in conversation mode. Upon deactivation of the conversation mode, the conversation mode control module 655 may restore settings related to the output signal. For example, talk mode control module 655 can activate ANC and disable ambient sounds. Additionally, the conversation mode control module 655 may increase the volume of the music being output to the first value.

[Table 2]

Referring to Table 2, the wearer of the wireless audio device 302 may be listening to music using the wireless audio device 302. For example, the wireless audio device 302 may output music without applying ANC. For example, the wireless audio device 302 may output music at a first volume. Depending on activation of the conversation mode, the conversation mode control module 655 may activate ambient sound and keep ANC in a deactivated state. In this case, the conversation mode control module 655 may reduce the volume of music being output below a specified value or by a specified ratio. For example, the conversation mode control module 655 may reduce the volume of music being output to a second value in conversation mode. Upon deactivation of the conversation mode, the conversation mode control module 655 may restore settings related to the output signal. For example, talk mode control module 655 may keep ANC disabled and disable ambient sounds. Additionally, the conversation mode control module 655 may increase the volume of the music being output to the first value.

In the examples of Tables 1 and 2, the wireless audio device 302 is described as disabling ambient sound when the conversation mode is not set, but embodiments of the present document are not limited thereto. For example, the wireless audio device 302 may activate ambient sound according to the user's settings even when the conversation mode is not set.

According to one embodiment, the singing mode control module 657 may transmit a decision about the singing mode (e.g., deactivation of the singing mode or activation of the singing mode) to the singing mode control module 657. there is. The singing mode control module 657 may control the functions of the wireless audio device 302 according to activation and/or deactivation of the singing mode. For example, the singing mode control module 657 may control the output signal of the wireless audio device 302 using the sound control module 640 according to activation and/or deactivation of the singing mode.

According to one embodiment, the surrounding situation recognition module 660 acquires an audio signal using an audio receiving circuit (e.g., the audio receiving circuit 581 and the second audio receiving circuit 582 in Figures 14), Based on audio signals, the surrounding situation can be recognized and the environment of the surrounding situation can be classified. The surrounding situation recognition module 660 may include an environment classification module 661 and a device search module 663 around the user. The surrounding situation awareness module 660 may obtain at least one of background noise, signal to noise ratio (SNR), or type of noise from the audio signal. The surrounding situation recognition module 660 may further obtain sensor information from a sensor circuit (eg, sensor circuits 551 and 552 in FIG. 4). Sensor information may include Wi-Fi information and/or BLE information, and GPS information.

According to one embodiment, the environment classification module 661 may detect the environment based on the intensity of background noise, SNR, or type of noise. For example, the environment classification module 661 compares the environmental information stored in the memories 531 and 532 with at least one of the intensity of background noise, SNR, or type of noise to determine the environmental information of the wireless audio device 302. It can be calculated.

According to one embodiment, the user's surrounding device discovery module 663 uses sensor information to detect the surroundings of a wireless audio device (e.g., the first wireless audio device 302-1, the second wireless audio device 302-2). Information about the device in can be calculated. For example, the user surrounding device search module 663 may use sensor information to calculate the type and distribution of surrounding devices in the environment where the wireless audio devices 302-1 and 302-2 are located. For another example, the user's surrounding device discovery module 663 may obtain location information about the user of the wireless audio devices 302-1 and 302-2 using sensor information. The user's surrounding device search module 663 maps one or more of environmental information corresponding to the utterance, location information, and information about peripheral devices of the electronic device 301 with the mode used for the utterance, and maps the pattern of the mapped mode. can be analyzed.

According to one embodiment, when either the conversation mode or the singing mode is activated, the surrounding situation recognition module 660 may control the output signal based on the identified environment. The surrounding situation awareness module 660 may control surrounding sounds based on the intensity and/or SNR of background noise. For example, the surrounding situation awareness module 660 may determine the total output of surrounding sounds, amplification of the voice band among surrounding sounds, or amplification of a specified sound (eg, alarm or siren) among surrounding sounds.

For example, the situation awareness module 660 can determine the intensity of ANC. For example, the surrounding situation recognition module 660 can adjust parameters (eg, coefficients) of a filter for ANC.

According to one embodiment, the surrounding situation recognition module 660 may control either a conversation mode or a singing mode based on the identified environment. For example, the surrounding situation recognition module 660 may activate either a conversation mode or a singing mode based on the identified environment. If the surrounding situation recognition module 660 determines that the user is in an environment where he or she needs to hear surrounding sounds, it can activate the conversation mode using the conversation mode control module 655 and provide surrounding sounds to the user according to the conversation mode. . For example, when a user is in a dangerous environment (e.g., an environment where a siren sound is detected), the surrounding situation recognition module 660 may activate the conversation mode.

According to one embodiment, the electronic device 301 may display an interface on the display 360 that instructs to deactivate or activate any one of the conversation mode and the singing mode. The electronic device 301 may provide an interface in synchronization with either the conversation mode or the singing mode of the wireless audio device 302. When the electronic device 301 determines to deactivate or activate any one of the conversation mode and the singing mode, or when the electronic device 301 determines the deactivation or activation of any one of the conversation mode and the singing mode from the wireless audio device 302 The interface can be displayed when an indicating signal is received. For example, when one of the conversation mode and the singing mode is activated, the electronic device 301 may display a first interface including information indicating that one of the conversation mode and the singing mode is set. The first interface may include an interface for controlling output signal settings in any one of conversation mode and singing mode. For example, when one of the conversation mode and the singing mode is deactivated, the electronic device 301 may display a second interface including information indicating that one of the conversation mode and the singing mode is deactivated. The electronic device 301 may display a first interface and a second interface on the execution screen of an application (eg, a wearable application) for controlling the wireless audio device 302.

According to one embodiment, the conversation mode module 625 may determine activation and deactivation of the conversation mode based further on whether the device is worn. For example, when the wireless audio device 302 is worn by a user, the conversation mode module 625 may activate the conversation mode based on a user's (eg, wearer's) utterance or user input. If the wireless audio device 302 is not worn by the user, the conversation mode module 625 may not activate the conversation mode even if the user's speech is detected.

For example, the first wireless audio device 302-1 and the second wireless audio device 302-2 may each include components of the wireless audio device 302 shown in FIG. 5. Each of the first wireless audio device 302-1 and the second wireless audio device 302-2 may be set to determine whether to activate either the conversation mode or the singing mode. According to one embodiment, when the first wireless audio device 302-1 or the second wireless audio device 302-2 determines activation of either the conversation mode or the singing mode, the first wireless audio device 302- 1) and the second wireless audio device 302-2 may be set to operate in either conversation mode or singing mode. For example, the first wireless audio device 302-1 or the second wireless audio device 302-2, which has decided to activate either the conversation mode or the singing mode, is connected to another wireless audio device and/or electronic device 301. It can be set to transmit a signal instructing activation of any one of conversation mode and singing mode. According to one embodiment, when both the first wireless audio device 302-1 and the second wireless audio device 302-2 decide to activate the conversation mode, the first wireless audio device 302-1 and the second wireless audio device 302-2 The wireless audio device 302-2 may be set to operate in either conversation mode or singing mode. For example, the first wireless audio device 302-1 or the second wireless audio device 302-2, which has decided to activate either the conversation mode or the singing mode, may cause the other wireless audio device to activate either the conversation mode or the singing mode. Check whether activation of one is determined, and if both wireless audio devices determine activation of either conversation mode or singing mode, the first wireless audio device 302-1 and the second wireless audio device 302-2 It can operate in either conversation mode or singing mode. For another example, the first wireless audio device 302-1 or the second wireless audio device 302-2, which has decided to activate either the conversation mode or the singing mode, activates the conversation mode or the singing mode with the electronic device 301. A signal indicating activation of any one of the signals may be transmitted. When the electronic device 301 receives a signal instructing to activate either the conversation mode or the singing mode from both the first wireless audio device 302-1 and the second wireless audio device 302-2 within a specified time, The first wireless audio device 302-1 and the second wireless audio device 302-2 may transmit signals to operate in either a conversation mode or a singing mode.

According to one embodiment, the similarity determination module 670 may detect information about the singing voice among ambient sounds included in the audio signal based on the characteristics of the singing voice. The similarity determination module 670 may extract a main signal for the surrounding sound included in the audio signal and a main signal for the reference signal corresponding to the media included in the audio signal. Based on the main signal and the singing voice, the acoustic similarity and the similarity of the lyrics between the media and the singing voice can be calculated. The similarity determination module 670 may output the similarity to the singing mode module 627 to determine activation of the singing mode when the similarity exceeds a predetermined threshold.

A method of determining activation, maintenance, and/or deactivation of any one of the conversation mode and the singing mode may be referred to by the contents described later in relation to FIGS. 7 to 12B.

Figure 7 is a block diagram showing the configuration of a wireless audio device according to an embodiment.

Referring to FIG. 7, according to one embodiment, the wireless audio device 302 includes a sensor circuit (e.g., sensor circuits 551 and 552 in FIG. 4) and an audio output circuit (e.g., audio output circuit 571 in FIG. 4). , 572)), audio reception circuit (e.g., the first audio reception circuits 581 and 582 and the second audio reception circuit 583 in FIG. 4), preprocessing module 610, phase determination module 620, conversation mode module 625, singing mode module 627, voice agent module 630, sound control module 640, conversation mode control module 655, singing mode control module 657, surrounding situation recognition module 660, and a similarity determination module 670.

According to one embodiment, wireless audio device 302 may provide multiple operating modes to a user of wireless audio device 302 based on components. The plurality of operation modes may include normal mode, dialogue mode, and singing mode. A plurality of operation modes are activated alternatively, and two or more operation modes cannot be activated simultaneously.

According to one embodiment, the normal mode may be the default mode of the wireless audio device 302. The conversation mode is an ambient sound included in the audio signal detected by the wireless audio device 302 in order to smoothly have a conversation with a speaker other than the user while using (e.g., wearing) the wireless audio device 302. ) may be a mode that outputs at least part of. The singing mode may be a mode that outputs at least some of the surrounding sounds and media included in the audio signal to optimally assist the user's music listening experience.

According to one embodiment, an audio receiving circuit (eg, the first audio receiving circuits 581 and 582 and the second audio receiving circuit 583) may detect an audio signal. The audio signal may include ambient sounds of the wireless audio device 302 and reference signals corresponding to media played on the electronic device 301. For example, the first audio receiving circuits 581 and 582 receive ambient sound (e.g., a conversation between a user and a speaker other than the user, a singing voice) of the electronic device 301, and receive the second audio. The circuit 583 may receive a reference signal from the electronic device 301.

According to one embodiment, the pre-processing module 610 detects an audio signal using an audio receiving circuit (e.g., the first audio receiving circuits 581 and 582, the second audio receiving circuit 583). Distortion of the audio signal can be improved by performing preprocessing.

According to one embodiment, the phase determination module 620 may obtain whether the electronic device 301 is playing media. For example, it is possible to obtain whether media is played on the electronic device 301, the type of media, and whether there are lyrics through media playback app information received from the electronic device 301. For another example, the phase determination module 620 may obtain whether to play media based on a reference signal. The phase determination module 620 may determine that media is being played when the size of the reference signal is greater than or equal to a predetermined size and flows in for more than a predetermined time.

According to one embodiment, the phase determination module 620 may obtain information related to the electronic device 301 from one or more of the surrounding situation recognition module 660 and the sensor circuit 551. Information related to the electronic device 301 may include one or more of environmental information of the electronic device 301, location information of the electronic device 301, and information about devices in the vicinity of the electronic device 301.

According to one embodiment, the environmental information is based on the intensity of background noise, SNR, or type of noise obtained by the surrounding situation awareness module 660 (e.g., environment classification module 661) from the audio signal and the preprocessed audio signal. It may have been created by doing so.

According to one embodiment, the location information of the electronic device 301 and information about devices around the electronic device 301 are obtained from a sensor circuit (e.g., WIFI, BLE, UWB, GPS, ACC, gyro sensor, etc.) It may be obtained from collected sensor information. Alternatively, the location information of the electronic device 301 and information about devices in the vicinity of the electronic device 301 may be obtained by the surrounding situation recognition module 660 (e.g., the user's surrounding device search module 663) utilizing sensor information. This may have been calculated.

According to one embodiment, the phase determination module 620 controls the wireless audio devices 302-1 and 302-2 based on one or more of whether media is played on the electronic device 301 and information associated with the electronic device 301. may decide to enter one of the first mode change phase and the second mode change phase. The first mode change phase may be for determining a change in the operating mode to one of the singing mode and the conversation mode. The second mode change phase may be for determining the change to conversation mode.

For example, when the number of peripheral devices of the electronic device 301 is less than a predetermined number, a small noise environment is detected based on the audio signal, or a user pre-registered position for the singing mode is detected, the first mode change phase You can enter.

According to one embodiment, the phase determination module 620 may learn the user's usage pattern using a user's usage pattern model. The phase determination module 620 may enter the first mode change phase according to the user's usage pattern of the singing mode. For example, when the phase determination module 620 determines that the user is located in an environment that is substantially the same or similar to an environment in which the user frequently sings based on the user's usage pattern, the phase determination module 620 may enter the first mode change phase. . The user's usage pattern may be specified by one or more of whether the electronic device 301 plays media and information related to the electronic device 301. Information related to the electronic device 301 may include environmental information (e.g., type and size of surrounding noise), location information, and the type and number of surrounding devices.

According to one embodiment, the conversation mode module 625 detects a conversation between the user of the wireless audio device 302 and a speaker other than the user in the first mode change phase and the second mode change phase to activate and deactivate the conversation mode. can be decided.

According to one embodiment, the conversation mode module 625 does not start the singing mode by the singing mode module 627 in the first mode change phase, and the voice signal corresponding to the user's utterance is set to a designated time interval (e.g. If it is maintained for more than L frames (L is a natural number), activation of the conversation mode can be determined. For another example, the conversation mode module 625 does not initiate the singing mode by the singing mode module 627 in the first mode change phase, and after deactivating the user's speech, the voice signal corresponding to the other party's speech is designated. If maintained for the time interval, activation of the conversation mode can be determined.

According to one embodiment, in the second mode change phase, the conversation mode module 625 switches to the conversation mode when the voice signal corresponding to the user's utterance is maintained for a specified time interval (e.g., L or more frames, L is a natural number). You can decide to activate it. For another example, in the second mode change phase, the conversation mode module 625 may determine activation of the conversation mode if, after deactivating the user's speech, the voice signal corresponding to the other party's speech is maintained for a designated time period.

According to one embodiment, conversation mode module 625 may be configured to interact with voice agent module 630. For example, the conversation mode module 625 may obtain information indicating activation of the conversation mode from the voice agent module 630. That is, in this case, the singing mode module 627 may determine activation of the singing mode based on the instructions of the voice agent module 630, rather than the activation condition of the singing mode.

According to one embodiment, the singing mode module 627 may determine activation and deactivation of the singing mode by detecting the user's singing voice in the first mode change phase. The singing mode module 627 may determine activation and deactivation of the singing mode in priority over the conversation mode module 625 in the first mode change phase.

According to one embodiment, the singing mode module 627 determines whether the analysis result of the audio signal and the pre-processed audio signal received through the phase determination module 620 satisfies the activation conditions of the singing mode in the first mode change phase. Based on this, it is possible to determine whether to activate or deactivate the singing mode. The activation condition of the singing mode may be that the user of the electronic device 301 is classified according to the sensitivity level of the electronic device 301 among the first sensitivity level, the second sensitivity level, and the third sensitivity level.

According to one embodiment, the activation condition according to the first sensitivity level may include a condition regarding whether a singing voice among ambient sounds is continuously detected for a predetermined time. The activation condition according to the second sensitivity level may include a condition regarding the acoustic similarity between the singing voice included in the surrounding sound and the media. Ambient sounds and media may be included in the audio signal. The activation condition according to the third sensitivity level may include a condition regarding the similarity between the singing voice included in the ambient sound and the lyrics included in each media.

According to one embodiment, the singing mode module 627 detects whether the singing voice received from the similarity determination module 670 is detected, and sets a sensitivity level (e.g., a first sensitivity level, a second sensitivity level) based on the similarity between the media and the singing voice. It is possible to determine whether the activation conditions are satisfied according to the level, third sensitivity level). The singing mode module 627 may determine activation of the singing mode when the activation conditions are satisfied.

According to one embodiment, the activation condition of the singing mode may include activation conditions according to all levels below the sensitivity level of the electronic device 301. For example, when the sensitivity level of the electronic device 301 is the second sensitivity level, the activation condition of the singing mode includes activation conditions according to the first sensitivity level and the second sensitivity level, and the sensitivity of the electronic device 301 When the level is the third sensitivity level, the activation condition of the singing mode may include activation conditions according to the first sensitivity level, the second sensitivity level, and the third sensitivity level.

According to one embodiment, the singing mode module 627 may be configured to interact with the voice agent module 630. For example, the singing mode module 627 may obtain information indicating activation of the singing mode from the voice agent module 630. That is, in this case, the singing mode module 627 may determine activation of the singing mode based on the instructions of the voice agent module 630, rather than the activation condition of the singing mode.

According to one embodiment, the singing mode module 627 may determine deactivation of the singing mode in the singing mode. For example, the singing mode module 627 determines deactivation of the singing mode when the analysis result of the audio signal received by the wireless audio devices (302-1, 302-2) in the singing mode no longer satisfies the activation conditions. You can. As another example, the singing mode module 627 may determine deactivation of the singing mode based on whether media is playing and information associated with the electronic device 301. In this case, the singing mode module 627 determines that media is no longer played on the electronic device 301 or that the singing mode is not needed according to information associated with the electronic device 301, thereby determining deactivation of the singing mode. .

According to one embodiment, the voice agent module 630 may transmit a signal indicating activation of the conversation mode or singing mode to the conversation mode module 625 or the singing mode module 627. Accordingly, the conversation mode module 625 or the singing mode module 627 may determine activation of the conversation mode or singing mode.

According to one embodiment, the sound control module 640 can control the output signal of the wireless audio device 302 according to the conversation mode or singing mode by the conversation mode control module 655 or the singing mode control module 657. there is. The sound control module 640 may transmit an output signal to the audio output circuit 571 so that the output signal is output (e.g., played) through the audio output circuit 571.

According to one embodiment, the conversation mode control module 655 may control the output signal of the wireless audio device 302 using the sound control module 640. The conversation mode control module 655 may output at least part of the ambient sound included in the audio signal in conversation mode. For example, in the conversation mode, the conversation mode control module 655 may change the volume of at least part of the ambient sound to the first gain and output it.

According to one embodiment, the singing mode control module 657 may control the output signal of the wireless audio device 302 using the sound control module 640. The singing mode control module 657 may output at least a portion of the surrounding sounds and media included in the audio signal in the singing mode. For example, the singing mode control module 657 may change the volume of at least part of the ambient sound to a second gain in the singing mode and output it.

According to one embodiment, the similarity determination module 670 may detect information about the singing voice among ambient sounds included in the audio signal based on the characteristics of the singing voice. The similarity determination module 670 may extract a main signal for the surrounding sound included in the audio signal and a main signal for the reference signal corresponding to the media included in the audio signal. Based on the main signal and the singing voice, the acoustic similarity and the similarity of the lyrics between the media and the singing voice can be calculated. The similarity determination module 670 may output the similarity to the singing mode module 627 to determine activation of the singing mode when the similarity exceeds a predetermined threshold.

FIG. 8 is a flowchart illustrating an operation of a wireless audio device controlling an output signal according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 810 to 830 may be understood as being performed by a processor (e.g., processors 521 and 522 of FIG. 4) of a wireless audio device (e.g., wireless audio device 302 of FIG. 3). .

Operations 810 to 830 may be used to explain an operation in which a wireless audio device according to an embodiment controls an output signal according to either a singing mode or a conversation mode.

In operation 810, a wireless audio device (eg, wireless audio device 302 of FIG. 3) may detect an audio signal. Audio signals may include ambient sounds. The audio signal may include a reference signal corresponding to media played on the electronic device 301.

In operation 820, the wireless audio device 302 may determine the operation mode of the wireless audio device 302 to be one of a singing mode and a conversation mode based on the analysis result of the audio signal. The conversation mode may be a mode that outputs at least part of the surrounding sounds, and the singing mode may be a mode that outputs at least a part of the surrounding sounds and media.

In operation 830, the wireless audio device 302 may control the output signal of the wireless audio device 302 according to the determined mode. The wireless audio device 302 may change the volume of at least part of the surrounding sound to a first gain in conversation mode and output it, and change the volume of at least part of the surrounding sound to the second gain in singing mode and output it.

FIG. 9 is a flowchart illustrating an operation of a wireless audio device controlling an output signal according to one of a singing mode and a conversation mode, according to an embodiment.

In the following embodiments, each operation may be performed sequentially, but is not necessarily performed sequentially. For example, the order of each operation may be changed, and at least two operations may be performed in parallel.

According to one embodiment, operations 910 to 990 may be understood as being performed by a processor (e.g., processors 521 and 522 of FIG. 4) of a wireless audio device (e.g., wireless audio device 302 of FIG. 3). .

Operations 910 to 990 relate to operations in which the wireless audio device according to an embodiment controls the output signal according to any one of the singing mode and the conversation mode, with the use of both the conversation mode and the singing mode set to ON. You can.

In one embodiment, when the wireless audio device 302 determines that there are no lyrics in the media based on the media information received from the electronic device 301, the wireless audio device 302 sets the sensitivity level to only one of the first sensitivity level and the second sensitivity level. It can be limited.

In operation 910, the wireless audio device (e.g., the wireless audio device 302 in FIG. 3) may determine to enter one of the first mode change phase and the second mode change phase. The wireless audio device 302 performs one of the first mode change phase and the second mode change phase based on one or more of whether media is played on the electronic device 301 and information associated with the electronic device 301. You can decide to enter. Information related to the electronic device 301 may include one or more of environmental information of the electronic device 301, location information of the electronic device 301, and information about devices in the vicinity of the electronic device 301.

For example, the wireless audio device 302 may be directed to a location where the user frequently sings, as media is being played or the singing mode has been activated more than a predetermined number of times at the user's current location in the wireless audio device 302. If confirmed, if the number of peripheral devices of the electronic device 301 is less than a predetermined number, if a small noise environment is detected based on the audio signal, or if a user pre-registered position for the singing mode is detected, to the first mode change phase You can decide to enter.

The wireless audio device 302 may perform operation 920 if it decides to enter the first mode change phase, and may perform operation 960 if it decides to enter the second mode change phase. The first mode change phase may be for determining a change to one of the operating modes, the singing mode and the conversation mode. The second mode change phase may be for determining the change to conversation mode.

In operation 920, the wireless audio device 302 satisfies an activation condition (e.g., a first singing mode activation condition) according to the first sensitivity level based on the audio signal detected by the wireless audio device 302 and the preprocessed audio signal. You can decide whether to do it or not. The first singing mode activation condition may include a condition regarding whether a singing voice among ambient sounds included in the audio signal is continuously detected for a predetermined time.

For example, the wireless audio device 302 satisfies the second singing mode activation condition when a singing voice is maintained for a specified time interval (e.g., N frames or more, N is a natural number) among the surrounding sounds included in the audio signal. It can be judged that The singing voice may include one or more of a singing voice and a humming voice.

The wireless audio device 302 may perform operation 930 when the first singing mode activation condition is satisfied, and may perform operation 970 when the first singing mode activation condition is not satisfied.

In operation 930, the wireless audio device 302 may determine whether the sensitivity level of the electronic device 301 is greater than 1. The sensitivity level of the electronic device 301 may be a sensitivity level preset by the user, or, if not preset by the user, may be a sensitivity level set as default (eg, a first sensitivity level). The wireless audio device 302 may perform operation 940 if the sensitivity level of the electronic device 301 is greater than 1, and may perform operation 980 if the sensitivity level of the electronic device 301 is 1 or less.

In operation 940, the wireless audio device 302 satisfies an activation condition according to the second sensitivity level (e.g., a second singing mode activation condition) based on the audio signal detected by the wireless audio device 302 and the preprocessed audio signal. You can decide whether to do it or not. The second singing mode activation condition may include conditions regarding the acoustic similarity between the singing voice included in the surrounding sound and the media. Ambient sounds and media may be included in the audio signal.

For example, the wireless audio device 302 may compare a singing voice among ambient sounds included in the audio signal with a reference signal corresponding to media played on the electronic device 301. According to the comparison result, the wireless audio device 302 is configured to perform a second wireless audio device if the acoustic similarity between the singing voice and the reference signal exceeds a predetermined threshold or the pattern matching similarity between the singing voice and the reference signal exceeds a predetermined threshold. It can be determined that the singing mode activation conditions are satisfied.

The wireless audio device 302 may perform operation 950 when the second singing mode activation condition is satisfied, and may perform operation 970 when the second singing mode activation condition is not satisfied.

In operation 950, the wireless audio device 302 may determine whether the sensitivity level of the electronic device 301 is greater than 2. The wireless audio device 302 may perform operation 960 when the sensitivity level of the electronic device 301 is greater than 2, and may perform operation 980 when the sensitivity level of the electronic device 301 is 2 or less.

In operation 960, the wireless audio device 302 satisfies an activation condition according to the third sensitivity level (e.g., a third singing mode activation condition) based on the audio signal detected by the wireless audio device 302 and the preprocessed audio signal. You can decide whether to do it or not. The third singing mode activation condition may include a condition regarding the similarity between the singing voice included in the surrounding sound and the lyrics included in each media.

For example, the wireless audio device 302 may compare a singing voice among ambient sounds included in the audio signal with a reference signal corresponding to media played on the electronic device 301. According to the comparison result, the wireless audio device 302 is a third song if the similarity regarding the lyrics (e.g., similarity in length of lyrics, similarity in content of lyrics) between the singing voice and the reference signal exceeds a predetermined threshold. It can be determined that the mode activation conditions are satisfied.

The wireless audio device 302 may perform operation 980 when the third singing mode activation condition is satisfied, and may perform operation 970 when the third singing mode activation condition is not satisfied.

In operation 970, the wireless audio device 302 determines whether a voice signal corresponding to an utterance of a user (or a speaker other than the user) included in the audio signal is detected during a specified time interval (e.g., L frames or more, where L is a natural number). can do. The wireless audio device 302 may perform operation 990 when a voice signal is detected for a specified time period or longer, and may perform operation 910 when a voice signal is not detected for a designated time period or longer.

In operation 980, the wireless audio device 302 may control the output signal of the wireless audio device 302 according to the singing mode. The wireless audio device 302 may change the volume of at least part of the ambient sound to the second gain and output the volume in the singing mode. For example, in the singing mode, the wireless audio device 302 may change the volume of a singing voice among ambient sounds to a second gain and change the volume of a reference signal corresponding to media in accordance with the second gain. The volume of the reference signal corresponding to the media can be changed to a level of gain that allows the user to monitor both signals when the wireless audio device 302 outputs (e.g., plays) a singing voice of the second gain. .

In the singing mode, the wireless audio device 302 sets the activation conditions for the singing mode according to the sensitivity level of the wireless audio device 302 (e.g., a first singing mode activation condition, a second singing mode activation condition, and a third singing mode activation condition). ) is not satisfied, the singing mode can be deactivated. Alternatively, the wireless audio device 302 may perform a mode change phase in a second mode change phase based on one or more of whether media is played on the electronic device 301 and information associated with the electronic device 301. If you decide to enter , you can disable singing mode. When the singing mode is deactivated, the wireless audio device 302 may restore gain settings for ambient sounds and reference signals before the singing mode was activated.

At operation 990, the wireless audio device 302 may control the output signal of the wireless audio device 302 according to the conversation mode. In conversation mode, the wireless audio device 302 may change the volume of at least part of the ambient sound to the first gain and output it. For example, the wireless audio device 302 may disable ANC in conversation mode and change the volume of ambient sounds to the first gain. For another example, when media is being played on the wireless audio device 302 in conversation mode, the wireless audio device 302 may reduce the volume of the reference signal corresponding to the media by a certain percentage or set it to mute at the maximum. A user of the wireless audio device 302 can more clearly hear conversation content included in ambient sounds in conversation mode.

Figure 10 is a schematic diagram of a similarity determination module according to one embodiment.

Referring to FIG. 10, according to one embodiment, the similarity determination module 670 includes a main part extraction module 1010, a singing voice detection module 1020, a calculation module 1030, a lyrics recognition module 1040, and a melody/vocal It may include a model 1050, a lyrics model 1060, and a weight model 1070.

According to one embodiment, the singing voice detection module 1020 receives an audio signal from an audio receiving circuit (e.g., the audio receiving circuits 581, 582, and 583 in FIG. 7), and receives an audio signal from a preprocessing module (e.g., the preprocessing module in FIG. 7). (610)). The singing voice detection module 1020 can detect information about the singing voice among the ambient sounds included in the audio signal based on the characteristics of the singing voice. For example, unlike general voices, singing voices are characterized by a long fixed pitch and a short pause period. Pitch refers to the pitch of the sound, and pause refers to the section in which the voice is not played. The singing voice detection module 1020 can detect information about the singing voice through signal processing-based pitch/melody estimation or learning-based various deep learning classifiers, based on the characteristics of the singing voice. Information about whether a specific section (e.g., frame) of the ambient sound or reference signal is a singing voice, information of the detected signal (e.g., acoustic information), and information about whether a specific section (e.g., frame) of the ambient sound or reference signal is a singing voice. It may include one or more probability information about the degree of proximity to .

According to one embodiment, the singing voice detection module 1020 may further utilize main part information of surrounding sounds to detect the singing voice. The main part information of the surrounding sound may be about the main melody or vocal received from the main part extraction module 1010.

According to one embodiment, the singing voice detection module 1020 may be activated when determining an activation condition according to a sensitivity level higher than the first sensitivity level. A wireless audio device (eg, the wireless audio device 302 of FIG. 3) may use the singing voice detection module 1020 to determine whether the activation condition according to the first sensitivity level is satisfied.

According to one embodiment, the main part extraction module 1010 receives an audio signal from an audio receiving circuit (e.g., the audio receiving circuits 581, 582, and 583 in FIG. 7) and receives an audio signal from a pre-processing module (e.g., the pre-processing module in FIG. 7). A pre-processed audio signal can be received from 610). The main part extraction module 1010 extracts a main part signal for the surrounding sound included in the audio signal and a main part signal for the reference signal corresponding to the media included in the audio signal. The main part extraction module 1010 can extract either the main melody or the vocal as a main part signal based on media information. The media information may be related to whether the media includes lyrics. Media The information may be obtained from an electronic device (eg, the electronic device 301 of FIG. 3).

According to one embodiment, the main part extraction module 1010 can extract the main part signal for the surrounding sound and the main part signal for the reference signal using the melody/vocal model 1050. If the media does not include lyrics according to media information, the main part extraction module 1010 may extract the main part signal using a melody model among the melody/vocal models 1050. If the media includes lyrics according to media information, the main part extraction module 1010 may extract the main part signal using the vocal model among the melody/vocal models 1050.

According to one embodiment, the melody model among the melody/vocal models 1050 may be learned with the input being a media without lyrics (e.g., a performance song) or a feature of the media, and the target output being the main melody of the media. . Among the melody/vocal models 1050, the vocal model can be learned with the input being media with lyrics or features of the media, and the target output being the main vocal of the media.

According to one embodiment, the calculation module 1030 may calculate acoustic similarity between the media and the singing voice based on the main signal and the singing voice. The main part signal may include the main part signal of the reference signal and the main part signal of the singing voice. For the singing voice detected from the surrounding sound obtained from the vpu (voice pickup unit), the calculation module 1030 applies bandwidth extension to the singing voice to compensate for the low frequency resolution of the vpu (voice pickup unit) signal, and then acoustically Similarity can be calculated or acoustic similarity can be calculated only for singing voices corresponding to the vpu signal band.

According to one embodiment, the calculation module 1030 may calculate acoustic similarity based on melody characteristics (e.g., octave, pitch, duration, etc.) or vocal characteristics (e.g., pitch, prosody, etc.). The calculation module 1030 may calculate acoustic similarity by reflecting variations in melody characteristics and vocal characteristics, considering the case where the user does not sing accurately. For example, the calculation module 1030 may calculate acoustic similarity by reflecting the dynamic margin of the melody characteristics and vocal characteristics. Dynamic margin may refer to the range in which variations in melody characteristics and vocal characteristics occur.

According to one embodiment, the calculation module 1030 may calculate the similarity between main part signals by performing pattern matching between main part signals extracted through a hidden markov model (HMM), deep learning, template, etc. Additionally, the calculation module 1030 may obtain a text pattern by first converting the melody or vocal from the main signal into an octave (e.g., CDCCDEF) and secondarily converting it into a text pattern. The calculation module 1030 may calculate similarity by comparing text patterns.

According to one embodiment, the similarity determination module 670 outputs the similarity to the singing mode module 657, so that the singing mode module (e.g., the singing mode module 627 of FIGS. 6 and 7) determines that the similarity is a predetermined standard value. If it exceeds the (threshold), activation of the singing mode can be determined. The condition may correspond to an activation condition according to the second sensitivity level. Similarity can be calculated as a score between 0 and 1, with 1 being a perfect match and 0 being a mismatch.

According to one embodiment, the calculation module 1030 and the weight module 1070 may be activated when an activation condition according to a sensitivity level equal to or higher than the second sensitivity level is determined. The wireless audio device (e.g., the wireless audio device 302 in FIG. 3) may use the calculation module 1030 to determine whether the activation condition according to the second sensitivity level is satisfied.

According to one embodiment, the lyrics recognition module 1040 may recognize lyrics included in the main signal using a lyrics model (eg, ASR for lyrics model). For example, the lyrics recognition module 1040 may calculate the similarity of the length of the lyrics and the similarity of the contents of the lyrics between main signals through a method such as word error rate (WER).

The lyrics recognition module 1040 calculates the similarity based on the similarity of the length of the lyrics and the similarity of the content of the lyrics, so that it can be known that the user is singing even if the user sings a part of the lyrics with different words or omits it. . The lyrics recognition module 1040 can output a WER value or a normalized value of similarity to lyric length between 0 and 1.

According to one embodiment, when each syllable of a specific word in lyrics is uttered for a long time (e.g., "I remember you"), insertion of syllables becomes frequent, so insertion of overlapping syllables is removed (e.g., "I remember you"). After transforming the main part signal into "), the similarity in the length of the lyrics and the similarity in the content of the lyrics between the main part signals can be calculated.

According to one embodiment, the weight module 1070 may receive acoustic similarity between the media and the singing voice from the calculation module 1030. The acoustic similarity may include the similarity between the singing voice detected from the ambient sound obtained from the vpu and the reference signal, and the similarity between the singing voice detected from the ambient sound obtained from the microphone and the reference signal. The weighting module 1070 may adjust the final similarity value by assigning weights between similarity values. For example, if it is determined that there is a lot of noise in the surrounding environment and the surrounding sound acquired from the microphone has a lot of noise, a relatively larger weight can be applied to the similarity between the singing voice detected from the surrounding sound obtained from the vpu and the reference signal. there is.

According to one embodiment, the weight module 1070 may receive the similarity of lyrics between main signals from the lyrics recognition module 1040. The weighting module 1070 may calculate the final similarity by assigning weights to the length of the detection section of the singing voice, the similarity between the main part signals, the lyric recognition rate and recognition length of the main part signals, etc. The weight module 1070 may transmit the final similarity to the singing mode module 627. The singing mode module 627 may use the final similarity to determine whether the activation condition according to the second sensitivity level and the activation condition according to the third sensitivity level are satisfied.

According to one embodiment, the lyrics recognition module 1040 may be activated when an activation condition according to the third sensitivity level is determined. A wireless audio device (e.g., the wireless audio device 302 in FIG. 3) may use the lyrics recognition module 1040 to determine whether the activation condition according to the third sensitivity level is satisfied.

Figure 11 is a schematic diagram of a singing mode module 627 according to one embodiment.

Referring to FIG. 11, according to one embodiment, the singing mode module 627 may include a singing mode activation module 1110, a gain calculation module 1130, and a guide generation module 1140. The singing mode module 627 may determine activation of the singing mode based on the components and calculate a gain for controlling the output signal in the singing mode. The singing mode module 627 can create a guide to optimize the user's music listening experience in singing mode.

According to one embodiment, the singing mode activation module 1110 may determine whether the singing mode activation condition according to the sensitivity level of the electronic device 301 is satisfied. When the gain calculation module 1130 determines that the singing mode activation module 1110 satisfies the activation conditions, the audio signal detected by the wireless audio device (e.g., the wireless audio device 302 in FIG. 3) is included in the ambient sound. You can compare the intensity of the singing voice with the external noise. The gain calculation module 1130 may calculate the appropriate volume of the singing voice and media included in the audio signal based on the comparison result. For example, the appropriate volume of media may be the minimum volume within which a user can hear the media, and the appropriate volume of a singing voice may be a volume that allows a user to monitor along with the media. The gain calculation module 1130 may reflect the volume for the singing mode set in advance by the user. The gain calculation module 1130 may transmit the appropriate volume for each of the media and singing voice to the singing mode control module (e.g., the singing mode control module 657 in FIGS. 6 and 7).

According to one embodiment, the guide creation module 1140 may generate a guide that optimizes the user's music listening experience in singing mode and provide the guide to the user. For example, the guide creation module 1140 may provide guide information about the media to the user when the user selects to provide a song guide or when the similarity between the singing voice and the media is low. Guide information about the media may include main melody information that can be sung along with the media (e.g., a song), a beat, or lyrics to be played in the next verse of the song. Guide information about media may be output as low-pitched audio through TTS generation through the wireless audio device 302, or may be displayed as visual information on the screen of the electronic device 301.

According to one embodiment, the operation of the singing mode module 627 (eg, activating/deactivating the singing mode and providing a guide) may be performed through the voice agent module 630.

According to one embodiment, when a plurality of wireless audio devices 302 are connected to one electronic device 301 or when a plurality of wireless audio devices 302 share each other's playing music through music sharing, etc. You can also activate singing mode. In this case, users of a plurality of wireless audio devices 302 can simultaneously monitor each other's singing voices while listening to a song.

12A and 12B are examples of screens output on a display of an electronic device according to an embodiment.

Referring to FIGS. 12A and 12B, according to one embodiment, the electronic device 301 includes a user interface for setting a singing mode of a wireless audio device (e.g., the audio device 302 of FIG. 3). It can be displayed on the execution screen of . For example, the user may enter the mode determination phase described above in FIG. 9 by turning the singing mode setting 1200 on the interface to ON. Additionally, the user interface may include a setting 1210 regarding the accuracy level that is activated when the singing mode is ON. The interface may include settings for a plurality of sensitivity levels as detailed items of the accuracy level settings (1210). For example, settings regarding a plurality of sensitivity levels may include settings regarding a first sensitivity level 1220, a second sensitivity level 1230, and a third sensitivity level 1240.

According to one embodiment, if the user does not change the settings regarding the sensitivity level, the sensitivity level may be set to the first sensitivity level by default.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode of the wireless audio device (102; 202; 302) as one of a singing mode and a conversation mode based on an analysis result of the audio signal. The plurality of operations may include controlling an output signal of the wireless audio device (102; 202; 302) according to the determined mode. The conversation mode may be a mode for outputting at least part of the ambient sound included in the audio signal, and the singing mode may be a mode for outputting at least a part of the ambient sound and media included in the audio signal.

The determining operation is based on one or more of whether or not to play media on an electronic device (101;201;301) connected to the wireless audio device (102;202;302) and information associated with the electronic device (101;201;301). A first mode change phase for determining a change of the operation mode to any one of the singing mode and the conversation mode based on, and a second mode change phase for determining a change to the conversation mode It may include an operation to enter any one of the following.

Information associated with the electronic device (101;201;301) includes environmental information of the electronic device (101;201;301), location information of the electronic device (101;201;301), and information related to the electronic device (101;201;301). ;301) may contain one or more pieces of information about devices in the vicinity.

The determining operation may include determining the operation mode to be one of the above based on whether the analysis result satisfies the activation condition of the singing mode in the first mode change phase.

The activation conditions of the singing mode may be classified according to the sensitivity level of the electronic device (101; 201; 301) among the first sensitivity level, second sensitivity level, and third sensitivity level.

The activation condition according to the first sensitivity level may include a condition regarding whether a singing voice among the surrounding sounds is continuously detected for a predetermined time.

Activation conditions according to the second sensitivity level may include conditions regarding acoustic similarity between the singing voice included in the ambient sound and the media.

The activation condition according to the third sensitivity level may include a condition regarding the similarity between the singing voice included in the ambient sound and the lyrics included in each of the media.

The controlling operation includes changing the volume of at least some of the ambient sounds to a first gain and outputting them in the conversation mode, and changing the volume of at least some of the ambient sounds to a second gain in the singing mode to output them. It can be included.

The activation conditions of the singing mode may include activation conditions according to all levels below the sensitivity level of the electronic device (101; 201; 301).

The plurality of operations may further include an operation of deactivating the singing mode when the activation conditions of the singing mode are not satisfied in the singing mode.

The plurality of operations may further include an operation of providing information about the singing voice by tracking the singing voice included in the ambient sound in the singing mode.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode for the audio signal of the wireless audio device (102; 202; 302) as a singing mode. The plurality of operations may include controlling an output signal of the wireless audio device (102; 202; 302) according to the singing mode. The singing mode may be a mode that outputs at least part of the surrounding sounds and media included in the audio signal.

A wireless audio device (102;202;302) according to an embodiment includes a memory (141;531;532) including instructions, is electrically connected to the memory (141;531;532), and executes the instructions. It may include processors (131; 521; 522) for. When the instructions are executed by the processor (131;521;522), the processor (131;521;522) may perform a plurality of operations. The plurality of operations may include detecting an audio signal. The plurality of operations may include determining an operation mode of the wireless audio device (102; 202; 302) for the audio signal as either a singing mode or a conversation mode based on an analysis result of the audio signal. . The plurality of operations may include outputting at least a portion of the ambient sound included in the audio signal when the determined mode is the conversation mode. The plurality of operations may include outputting at least a portion of ambient sounds and media included in the audio signal when the determined mode is the singing mode. The plurality of operations may include an operation of deactivating the singing mode when a singing voice is not detected in the ambient sound for more than a predetermined time in the singing mode.

The determining operation is based on one or more of whether or not to play media on an electronic device (101;201;301) connected to the wireless audio device (102;202;302) and information associated with the electronic device (101;201;301). A first mode change phase for determining a change of the operation mode to any one of the singing mode and the conversation mode based on, and a second mode change phase for determining a change to the conversation mode It may include an operation to enter any one of the following.

Information associated with the electronic device (101;201;301) includes environmental information of the electronic device (101;201;301), location information of the electronic device (101;201;301), and information related to the electronic device (101;201;301). ;301) may contain one or more pieces of information about devices in the vicinity.

The determining operation may include determining the operation mode to be one of the above based on whether the analysis result satisfies the activation condition of the singing mode in the first mode change phase.

The activation conditions of the singing mode may be classified according to the sensitivity level of the electronic device (101; 201; 301) among the first sensitivity level, second sensitivity level, and third sensitivity level.

The controlling operation includes changing the volume of at least some of the ambient sounds to a first gain and outputting them in the conversation mode, and changing the volume of at least some of the ambient sounds to a second gain in the singing mode to output them. It can be included.

The plurality of operations may further include an operation of providing information about the singing voice by tracking the singing voice included in the ambient sound in the singing mode.

Electronic devices according to embodiments disclosed in this document may be of various types. Electronic devices may include, for example, portable communication devices (e.g., smartphones), computer devices, portable multimedia devices, portable medical devices, cameras, wearable devices, or home appliances. Electronic devices according to embodiments of this document are not limited to the above-described devices.

The embodiments of this document and the terms used herein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various changes, equivalents, or replacements of the embodiments. In connection with the description of the drawings, similar reference numbers may be used for similar or related components. The singular form of a noun corresponding to an item may include one or more of the above items, unless the relevant context clearly indicates otherwise. As used herein, “A or B”, “at least one of A and B”, “at least one of A or B”, “A, B or C”, “at least one of A, B and C”, and “A Each of phrases such as “at least one of , B, or C” may include any one of the items listed together in the corresponding phrase, or any possible combination thereof. Terms such as "first", "second", or "first" or "second" may be used simply to distinguish one component from another, and to refer to that component in other respects (e.g., importance or order) is not limited. One (e.g., first) component is said to be “coupled” or “connected” to another (e.g., second) component, with or without the terms “functionally” or “communicatively.” When mentioned, it means that any of the components can be connected to the other components directly (e.g. wired), wirelessly, or through a third component.

The term "module" used in embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as logic, logic block, component, or circuit, for example. It can be used as A module may be an integrated part or a minimum unit of the parts or a part thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

One embodiment of the present document is one or more instructions stored in a storage medium (e.g., built-in memory 136 or external memory 138) that can be read by a machine (e.g., electronic device 101). It may be implemented as software (e.g., program 140) including these. For example, a processor (e.g., processor 120) of a device (e.g., electronic device 101) may call at least one command among one or more commands stored from a storage medium and execute it. This allows the device to be operated to perform at least one function according to the at least one instruction called. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. A storage medium that can be read by a device may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain signals (e.g. electromagnetic waves), and this term refers to cases where data is semi-permanently stored in the storage medium. There is no distinction between temporary storage cases.

According to one embodiment, the method according to the embodiments disclosed in this document may be provided and included in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or on two user devices (e.g. It can be distributed (e.g. downloaded or uploaded) directly between smart phones) or online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

According to one embodiment, each component (e.g., module or program) of the above-described components may include a single or plural entity, and some of the plurality of entities may be separately placed in other components. there is. According to one embodiment, one or more of the above-described corresponding components or operations may be omitted, or one or more other components or operations may be added. Alternatively or additionally, multiple components (eg, modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar manner as those performed by the corresponding component of the plurality of components prior to the integration. . According to embodiments, operations performed by a module, program, or other component may be executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations may be executed in a different order, or omitted. Alternatively, one or more other operations may be added.

301: Electronic device
302: wireless audio device

Claims (20)

In a wireless audio device (102;202;302),
memory containing instructions (141;531;532); and
A processor (131;521;522) electrically connected to the memory (141;531;532) and configured to execute the instructions.
Including,
When the instructions are executed by the processor (131;521;522), the processor (131;521;522) performs a plurality of operations,
The plurality of operations are,
An operation of detecting an audio signal;
Based on the analysis result of the audio signal, determining an operation mode of the wireless audio device (102; 202; 302) as one of a singing mode and a conversation mode; and
An operation of controlling the output signal of the wireless audio device (102; 202; 302) according to the determined mode.
Including,
The conversation mode is,
A mode that outputs at least part of the ambient sound included in the audio signal,
The singing mode is,
A wireless audio device (102; 202; 302) in a mode that outputs at least a portion of ambient sounds and media included in the audio signal.
According to paragraph 1,
The determining operation is,
Based on one or more of whether media is played on an electronic device (101;201;301) connected to the wireless audio device (102;202;302) and information associated with the electronic device (101;201;301), the operation Entering one of a first mode change phase for determining a change in mode to one of the singing mode and the conversation mode, and a second mode change phase for determining a change to the conversation mode. action
A wireless audio device (102;202;302), including.
According to any one of paragraphs 1 and 2,
Information related to the electronic device (101; 201; 301) is:
One or more of environmental information of the electronic device (101;201;301), location information of the electronic device (101;201;301), and information about devices in the vicinity of the electronic device (101;201;301) A wireless audio device (102;202;302), including.
According to any one of claims 1 to 3,
The determining operation is,
In the first mode change phase, determining the operation mode to be one of the above based on whether the analysis result satisfies the activation condition of the singing mode.
A wireless audio device (102;202;302), including.
According to any one of claims 1 to 4,
The activation conditions for the singing mode are,
A wireless audio device (102;202;302) that is classified according to the sensitivity level of the electronic device (101;201;301) among a first sensitivity level, a second sensitivity level, and a third sensitivity level.
According to any one of claims 1 to 5,
Activation conditions according to the first sensitivity level are:
Conditions regarding whether singing voices among the above ambient sounds are continuously detected for a specified period of time
A wireless audio device (102;202;302), including.
According to any one of claims 1 to 6,
Activation conditions according to the second sensitivity level are:
Conditions regarding the acoustic similarity of the singing voice included in the ambient sound and the media
A wireless audio device (102;202;302), including.
According to any one of claims 1 to 7,
Activation conditions according to the third sensitivity level are:
Conditions regarding the similarity between the singing voice included in the ambient sound and the lyrics included in each of the media
A wireless audio device (102;202;302), including.
According to any one of claims 1 to 8,
The controlling operation is,
An operation of changing the volume of at least a portion of the ambient sound to a first gain in the conversation mode and outputting the volume, and changing the volume of at least a portion of the ambient sound to a second gain in the singing mode and outputting the sound.
A wireless audio device (102;202;302), including.
According to any one of claims 1 to 9,
The activation conditions for the singing mode are,
A wireless audio device (102;202;302), comprising activation conditions according to all levels below the sensitivity level of the electronic device (101;201;301).
According to any one of claims 1 to 10,
The plurality of operations are,
In the singing mode, the operation of deactivating the singing mode when the activation conditions of the singing mode are not satisfied.
Further comprising a wireless audio device (102;202;302).
According to any one of claims 1 to 11,
The plurality of operations are,
An operation of providing information about the singing voice by tracking the singing voice included in the surrounding sound in the singing mode.
Further comprising a wireless audio device (102;202;302).
In a wireless audio device (102;202;302),
memory containing instructions (141;531;532); and
It is electrically connected to the memory (141; 531; 532) and includes a processor (131; 521; 522) for executing the instructions,
When the instructions are executed by the processor (131;521;522), the processor (131;521;522) performs a plurality of operations,
The plurality of operations are,
detecting an audio signal;
determining an operation mode for the audio signal of the wireless audio device (102; 202; 302) as a singing mode; and
An operation of controlling the output signal of the wireless audio device (102; 202; 302) according to the singing mode.
Including,
The singing mode is,
A wireless audio device (102; 202; 302) in a mode that outputs at least a portion of ambient sounds and media included in the audio signal.
In a wireless audio device (102;202;302),
memory containing instructions (141;531;532); and
It is electrically connected to the memory (141; 531; 532) and includes a processor (131; 521; 522) for executing the instructions,
When the instructions are executed by the processor (131;521;522), the processor (131;521;522) performs a plurality of operations,
The plurality of operations are,
detecting an audio signal;
An operation of determining an operation mode of the wireless audio device (102; 202; 302) for the audio signal as either a singing mode or a conversation mode based on an analysis result of the audio signal;
When the determined mode is the conversation mode, outputting at least a portion of an ambient sound included in the audio signal;
When the determined mode is the singing mode, outputting at least a portion of ambient sounds and media included in the audio signal; and
In the singing mode, an operation of deactivating the singing mode when a singing voice is not detected in the surrounding sound for more than a predetermined time
A wireless audio device (102;202;302), including.
According to clause 14,
The determining operation is,
The operation mode based on one or more of whether media is played on an electronic device (101;201;301) connected to the wireless audio device (102;202;302) and information associated with the electronic device (101;201;301) Entering any one of a first mode change phase for determining a change to any one of the singing mode and the conversation mode, and a second mode change phase for determining a change to the conversation mode. movement
A wireless audio device (102;202;302), including.
According to any one of claims 14 and 15,
Information related to the electronic device (101; 201; 301) is:
One or more of environmental information of the electronic device (101;201;301), location information of the electronic device (101;201;301), and information about devices in the vicinity of the electronic device (101;201;301) A wireless audio device (102;202;302), including.
According to any one of claims 14 to 16,
The determining operation is,
In the first mode change phase, determining the operation mode to be one of the above based on whether the analysis result satisfies the activation condition of the singing mode.
A wireless audio device (102;202;302), including.
According to any one of claims 14 to 17,
The activation conditions for the singing mode are,
A wireless audio device (102;202;302) that is classified according to the sensitivity level of the electronic device (101;201;301) among a first sensitivity level, a second sensitivity level, and a third sensitivity level.
According to any one of claims 14 to 18,
The controlling operation is,
An operation of changing the volume of at least some of the surrounding sounds to a first gain and outputting them in the conversation mode, and changing the volume of at least some of the surrounding sounds to a second gain in the singing mode and outputting them.
A wireless audio device (102;202;302), including.
According to any one of claims 14 to 19,
The plurality of operations are,
An operation of providing information about the singing voice by tracking the singing voice included in the surrounding sound in the singing mode.
Further comprising a wireless audio device (102;202;302).

Images