TWI847219B - Audio system with dynamic target listening spot and ambient object interference cancelation - Google Patents

Abstract

A sound system is proposed, dynamically playing optimized audio signals based on user position. A sensor circuits dynamically senses a target space to generate a field environment information. First speaker and second speaker are arranged for audio playback. A host device recognizes a user in the field environment information, and determine the user position corresponding to the target space, and adaptively designate the user position as a target listening spot. A sensor circuit contains a camera capturing a field environment image out of the target space. A control circuit utilizes a user interface circuit to perform a configuration procedure which determines an ambient object with location, size and acoustic attribute information. The control circuit performs a channel based compensation operation on the target listening spot to generate optimized first channel audio signal and second channel audio signal.

Description

A sound system that dynamically adjusts the target listening point and eliminates interference from environmental objects

This application involves audio processing technology, which is actually an audio system that can dynamically adjust the playback effect according to changes in the sound field space.

The existing audio system includes multiple speakers arranged around a target space to form an ambient sound field environment. Each speaker can output the corresponding channel audio. When configuring the ambient sound field environment, the installer of the audio system usually designates a central area of the target space as the best listening point as the basis for installing multiple speakers. When multiple speakers play multiple channel audios at the same time, users at the best listening point can get an immersive listening effect.

However, in real environments, the user's listening effect is easily affected by various variables. For example, in traditional audio systems, the range of the best listening point is limited to a region. When the user moves to an area outside the best listening point, although the multi-channel audio output by the audio system can still be heard, the listening effect of the multi-channel audio at the user's position may have been greatly reduced or completely ineffective. In addition, the room layout, furniture position and material in the target space are all environmental objects that may interfere with the listening effect. For example, sofas, windows, and curtains absorb or reflect part of the sound energy, distorting the audio channels received at the best listening point.

In other words, traditional audio systems cannot dynamically adjust the position of the best listening point, and users are forced to limit their movements to adjust to the best listening point, which is inconvenient. On the other hand, the audio of each channel may be distorted by the interference of environmental objects, making the range of the best listening point more limited or even disappearing. In this way, the sound field environment built at a high cost loses its meaning.

In view of this, how to make the audio system dynamically adjust the best listening point as the user moves and eliminate the interference of environmental objects in the target space is a problem to be solved.

This specification provides an embodiment of an audio system that can dynamically optimize the playback effect based on the user's position, wherein the audio system includes a sensor circuit, a first speaker, a second speaker, and a host device. The sensor circuit is configured to dynamically sense a target space and generate sound field environment information. The first speaker and the second speaker are configured to play audio. The host device is coupled to the sensor circuit, the first speaker, and the second speaker, and includes an identification circuit, a control circuit, and an audio transmission circuit. The identification circuit is configured to identify a user from the sound field environment information and determine a user position of the user in the target space. The control circuit is coupled to the identification circuit and configured to dynamically assign the user position as a target listening point. The audio transmission circuit is coupled to the control circuit, the first speaker and the second speaker, and is configured to transmit audio. The sensor circuit includes a camera configured to capture a sound field environment image of the target space. The recognition circuit analyzes the sound field environment image to obtain spatial configuration information and acoustic property information of an environmental object in the target space. The control circuit performs a channel floor compensation operation based on the target listening point and the spatial configuration information and acoustic property information of the environmental object to generate a first channel audio and a second channel audio optimized for the target listening point. Finally, the control circuit outputs the first channel audio and the second channel audio to the corresponding first speaker and the second speaker respectively through the audio transmission circuit.

The present specification provides an embodiment of an audio system, which can dynamically optimize the playback effect according to the user's position, wherein the audio system includes a sensor circuit, a first speaker, a second speaker, and a host device. The sensor circuit is configured to dynamically sense a target space and generate a sound field environment information. A first speaker and a second speaker are configured to play audio. The host device is coupled to the sensor circuit, the first speaker, and the second speaker, and includes an identification circuit, a control circuit, and an audio transmission circuit. The identification circuit is configured to identify a user from the sound field environment information and determine a user position of the user in the target space. The control circuit is coupled to the identification circuit and configured to dynamically assign the user position as a target listening point. The audio transmission circuit is coupled to the control circuit, the first speaker and the second speaker, and is configured to transmit audio. The sensor circuit includes a camera, which is configured to capture a sound field environment image of the target space. The recognition circuit analyzes the sound field environment image to obtain spatial configuration information and acoustic property information of an environmental object in the target space. The control circuit corresponds the target space to an object base space, and establishes a compensation sound source object in the object base space according to the environmental object. Metadata of the compensation sound source object includes: the coordinate position, size, and reflectivity and absorptivity of the environmental object to sound. The control circuit performs an object-based compensation operation based on the target listening point and the metadata to offset the interference of the environmental object on the target listening point and generate a first channel audio and a second channel audio optimized for the target listening point. The control circuit outputs the first channel audio and the second channel audio to the corresponding first speaker and the second speaker respectively through the audio transmission circuit.

This specification provides an embodiment of an audio system that can dynamically optimize the playback effect based on the user's position, wherein the audio system includes a sensor circuit, a first speaker, a second speaker, and a host device. The sensor circuit is configured to dynamically sense a target space and generate sound field environment information. The first speaker and the second speaker are configured to play audio. The host device is coupled to the sensor circuit, the first speaker, and the second speaker, and includes an identification circuit, a control circuit, an audio transmission circuit, and a human-machine interface circuit. The identification circuit is configured to identify a user from the sound field environment information and determine a user position of the user in the target space. The control circuit is coupled to the identification circuit and is configured to dynamically assign the user position as a target listening point. The audio transmission circuit is coupled to the control circuit, the first speaker and the second speaker and is configured to transmit audio. The human-machine interface circuit is coupled to the control circuit and is configured to run a configuration program to obtain spatial configuration information and acoustic property information of an environmental object in the target space. The control circuit performs a channel floor compensation operation based on the target listening point and the spatial configuration information and acoustic property information of the environmental object to generate a first channel audio and a second channel audio optimized for the target listening point. The control circuit outputs the first channel audio and the second channel audio to the corresponding first speaker and the second speaker respectively through the audio transmission circuit.

This specification provides an embodiment of an audio system that can dynamically optimize the playback effect based on the user's position, wherein the audio system includes a sensor circuit, a first speaker, a second speaker, and a host device. The sensor circuit is configured to dynamically sense a target space and generate sound field environment information. The first speaker and the second speaker are configured to play audio. The host device is coupled to the sensor circuit, the first speaker, and the second speaker, and includes an identification circuit, a control circuit, an audio transmission circuit, and a human-machine interface circuit. The identification circuit is configured to identify a user from the sound field environment information and determine a user position of the user in the target space. The control circuit is coupled to the identification circuit and is configured to dynamically assign the user position as a target listening point. The audio transmission circuit is coupled to the control circuit, the first speaker and the second speaker and is configured to transmit audio. The human-machine interface circuit is coupled to the control circuit and is configured to run a configuration program to obtain spatial configuration information and acoustic property information of an environmental object in the target space. The control circuit corresponds the target space to an object base space and establishes a compensation sound source object in the object base space in accordance with the environmental object. Metadata of the compensation sound source object includes: the coordinate position, size, and reflectivity and absorptivity of the environmental object to sound. The control circuit performs an object-based compensation operation based on the target listening point and the metadata to offset the interference of the environmental object on the target listening point and generate a first channel audio and a second channel audio optimized for the target listening point. The control circuit outputs the first channel audio and the second channel audio to the corresponding first speaker and the second speaker respectively through the audio transmission circuit.

One of the advantages of the above embodiment is that the audio system can dynamically track the user's position through sensors and continuously optimize the playback effect for the user's position. The user does not need to move to a fixed listening position in order to obtain the best experience.

Another advantage of the above embodiment is that the audio system can identify environmental objects in the target space and adjust the channel audio accordingly to offset the interference of the environmental objects.

Other advantages of the present invention will be explained in more detail with the following description and diagrams.

100: audio system

110: first speaker

112: First channel audio (first channel signal)

120: Second speaker

122: Second channel audio (second channel signal)

130: Host device

131: Storage circuit

132: Control circuit

133: User interface circuit

134: Recognizer circuit

135: Audio transmission circuit

136: Communication circuit

140: Sensor circuit

150: User equipment

160: Remote database

170: target space

171: first position

172: Second position

173: Movement trail

175: Ambient object

180: User

202~218: Operation

312~316: Operation

410: Operation

600: target space

601: first location

602: Second location (first location)

610: Camera (camera)

620: Infrared sensor

630: Radio detector

700: target space

800: target space

902~910: Operation

1002~1012: Operation

1100: target space

1103: Object movement trail

1105: Virtual audio object

1110: first speaker

1120: second speaker

1130: Third speaker

1140: fourth speaker

P0: original point

P1: first position

P1’: new first position

P2: second position

1200: target space

1201: target listening spot

1203: Movement trail

1210: first speaker

1212: first channel output

1220: second speaker

1222: Second channel output

1230: Third speaker

1240: fourth speaker

1250: fifth speaker

1252: fifth channel output

1260: Sixth speaker

1262: Sixth channel output

1304~1312: Operation

Figure 1 is a functional block diagram of an audio system of an embodiment of the present invention.

Figure 2 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

Figure 3 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

Figure 4 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

Figure 5 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

FIG6 is a schematic diagram of a target space of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the position of the best listening point.

FIG7 is a schematic diagram of a target space of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the absorption rate of environmental objects.

FIG8 is a schematic diagram of a target space of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the reflectivity of environmental objects.

Figure 9 is a flowchart of the host device identifying an object in an embodiment of the present invention.

FIG10 is a flow chart of an audio processing method of an embodiment of the present invention, illustrating an embodiment of calculating the output compensation value based on the positional relationship of environmental objects.

FIG11 is a schematic diagram of a target space of the present invention, used to illustrate an embodiment of optimizing the sound field by object-based compensation operation.

FIG12 is a schematic diagram of a target space of the present invention, used to illustrate an embodiment of optimizing the sound field by object-based compensation operation.

Figure 13 is a flowchart of the object base compensation operation of an embodiment of the present invention.

The following will be used in conjunction with relevant drawings to illustrate the embodiments of the present invention. In the drawings, the same reference numerals represent the same or similar elements or method flows.

FIG1 is a functional block diagram of an audio system 100 according to an embodiment of the present invention.

The audio system 100 is mainly composed of a host device 130 and multiple speakers. The host device 130 can control multiple speakers to play audio. The host device 130 can be a computer host, a barebone, an embedded system, or a customized digital audio processing device. The host device 130 includes a communication circuit 136, which enables the host device 130 to be connected to a user device 150 by wire or wirelessly, and serves as an input channel for audio source signals or data.

The user device 150 may be a mobile phone, a computer, a TV stick, a game console, or other audio source providing device, which provides music or sound streaming to the host device 130 through the communication circuit 136. Furthermore, the audio system 100 may utilize the communication circuit 136 to cooperate with the user device 150 or other multimedia devices to form a home theater system with both video and audio functions. For example, the target space 170 may also include a projection screen, a screen or a display (not shown), which is controlled by the user device 150 to display images. For another example, the user device 150 may be a head-mounted virtual reality device. The user 180 can stand in the target space 170 and see the screen through the user device 150, and the host device 130 can be controlled by the user device 150 to play the audio synchronously with the screen. The communication circuit 136 in this embodiment can be, but is not limited to, a high-definition multimedia interface (HDMI), a digital transmission interface (Sony/Philips Digital Interface Format; SPDIF), a wireless local area network module, an Ethernet module, a short-wave radio frequency transceiver, or an evolved application of Bluetooth Low Energy (BLE) version 4 or version 5, or a universal serial bus.

The host device 130 also includes an audio transmission circuit 135 for connecting multiple speakers and outputting multiple channels of audio for the speakers to play. The host device 130 controls the multiple speakers through the audio transmission circuit 135 in a one-way digital or analog output or a two-way synchronous communication protocol. The connection between the audio transmission circuit 135 and each speaker can be a wired interface, a wireless interface, or a combination of the two. The wired interface can be, but is not limited to, a composite audio and video terminal, a digital transmission interface, or a high-definition multimedia interface. The wireless interface can be, but is not limited to, a wireless local area network, a short-wave radio frequency transceiver, or an evolved application of Bluetooth low energy version 4 or version 5. In a further derived embodiment, since the audio transmission circuit 135 and the communication circuit 136 are both functionally positioned as interfaces for connecting to external components, they can be integrated into a multifunctional bidirectional transmission interface module in the derived implementation. The audio transmission circuit 135 and the communication circuit 136 use various public standard transmission technologies to achieve connection and transmission between components, which can increase the future functional expansion of the audio system 100 and reduce the replacement cost when components are damaged.

The target space 170 in FIG. 1 can be understood as a three-dimensional space in which the user 180 can use the audio system 100. Each speaker can be configured at a different position in the target space 170 to play a corresponding channel audio. The surround configuration of multiple speakers can create an ambient sound field environment in a target space 170. There are many standard specifications for the number and configuration of speakers. For example, in a 5.1-channel surround sound system, two front speakers, a center speaker, two surround channel speakers, and a subwoofer are included to create an ambient sound field space in a manner of surrounding a target listening point, and together play sound to the target listening point. In a 7.1-channel ambient sound system, further configuring a pair of rear surround channel speakers behind the target listening point can provide a more three-dimensional sound field effect. In recent years, new specifications such as 5.1.2 channels and 7.2.2 channels have also emerged, which include more speakers and channel configurations in specific directions, and can achieve more realistic "panoramic sound", "sky sound effects" or "floor sound effects" and other effects. In order to facilitate the description of the technical features of the sound system 100 of this embodiment, only the first speaker 110 and the second speaker 120 are shown in FIG1 as representatives. Among them, the first speaker 110 receives and plays the first channel audio 112 provided by the host device 130, and the second speaker 120 receives and plays the second channel audio 122 provided by the host device 130. It must be understood that, in practice, the audio system 100 of this embodiment is not limited to only two speakers, but can be applied to 2.1-channel, 4.1-channel, 5.1-channel, 7.2-channel or more channel configurations. Each speaker in the target space 170 may have different audio output specifications. For example, some speakers are good at outputting heavy bass, and some speakers are good at outputting mid-high frequencies. The host device 130 can plan various sound field environments with different characteristics in the target space 170 according to different speaker specifications.

The term "channel" referred to in the specification and patent application refers to various physical channels and logical channels. Logical channels refer to the audio data stream transmitted within the system, while physical channels refer to the signal source used by each speaker to play. In the present embodiment, the first channel audio 112 and the second channel audio 122 played by each speaker are physical channels, which can be the result of down-mixing one or more logical channels. For example, a pair of headphones has only two speakers, but the sound effects generated by multiple applications can be heard at the same time. In other words, the sound effect data of multiple applications can be down-mixed into two physical channels by the system and played as audible sound through two speakers. Therefore, the first channel audio 112 and the second channel audio 122 in this embodiment are not limited to audio signals containing only a single logical channel, but can also be audio signals generated by mixing multiple logical channels according to a predetermined ratio.

In FIG. 1 , a first speaker 110 and a second speaker 120 are arranged on both sides of a target space 170 to play sound to a target listening point in the target space 170. The target listening point can be understood as the position where the playback effect of the audio system 100 is optimized. In some audio systems, the target listening point is also called the listening sweet spot. In most cases, the target listening point is usually located in a specific area of the target space 170, such as a center point, an axis, a tangent plane, or an equivalent volume center of multiple speakers. In FIG. 1 , the target listening point of the target space 170 is represented by a first position 171 where a user 180 is located. When the user 180 moves from the first position 171 to the second position 172 along the moving track 173, the listening effect received by the user 180 deviates because the user 180 moves away from the first speaker 110 and approaches the second speaker 120. The traditional audio system cannot track the movement of the user 180 and adjust the listening effect received by the second position 172 accordingly. The solution proposed in this embodiment will be described in detail later.

On the other hand, the target space 170 usually includes some environmental objects 175, such as sofas, tables, curtains, walls, ceilings, and floors. These environmental objects 175 will produce different degrees of interference reactions on the sounds played by the first speaker 110 and the second speaker 120 depending on the material, size, and position. For example, a cloth sofa or curtain will absorb sound, and a marble floor or wall will reflect sound. In other words, the presence of environmental objects 175 will affect the first channel audio 112 and the second channel audio 122 received by the target listening point. Conventional audio systems do not have the ability to identify environmental objects 175 in the target space 170, nor do they have the function of compensating the first channel audio 112 and the second channel audio 122 according to the size, material, and position of the environmental objects 175. The audio system 100 of this embodiment can calculate and eliminate the interference of all environmental objects 175 in the target space 170 on the first channel audio 112 and the second channel audio 122. For the convenience of explanation, FIG. 1 of this embodiment only shows one environmental object 175 to explain the operation of the audio system 100. However, it must be understood that the target space 170 of FIG. 1 is not limited to only one environmental object 175. The solution to the interference of environmental object 175 will be described in detail later.

The host device 130 of the present embodiment further includes a storage circuit 131. The storage circuit 131 may include a non-volatile memory for storing the relevant operating system, application software or firmware required for the operation of the host device 130. The storage circuit 131 may also include a volatile memory for use as a computing memory for the control circuit 132. The host device 130 of the present embodiment further includes a control circuit 132. The control circuit 132 may be a central processing unit, a digital signal processor, or a microcontroller. The control circuit 132 may read the pre-stored operating system, software or firmware from the storage circuit 131 to control the host device 130, the first speaker 110 and the second speaker 120 to perform audio playback operations. Furthermore, the host device 130 of this embodiment uses the control circuit 132 to perform a series of sound field compensation operations to dynamically optimize the playback effect and solve the shortcomings that traditional audio systems cannot overcome.

In order to dynamically optimize the playback effect at the target listening point, the audio system 100 of this embodiment includes a sensor circuit 140, which is configured to dynamically sense a target space 170 and generate sound field environment information. The sensor circuit 140 can be a component located outside the host device 130 and coupled to the host device 130. The sensor circuit 140 can be a combination of one or more of a camera 610, an infrared sensor 620, and a wireless detector 630. The type of sound field environment information captured by the sensor circuit 140 can have different combinations depending on the implementation method of the sensor circuit 140. For example, the sound field environment information may be a combination of one or more of images, pictures, thermal imaging, and radio wave imaging of the user and environmental objects. In one embodiment, the sensor circuit 140 is disposed around the target space 170. It is understood that although only one sensor circuit 140 is shown in FIG. 1 , in practice, the audio system 100 may include multiple sets of sensor circuits 140, which are respectively disposed at different positions around the target space 170 to obtain more accurate sound field environment information.

In the host device 130 of the present embodiment, an identification circuit 134 is included, which is coupled to the sensor circuit 140. The identification circuit 134 can identify key information that affects the sound field from the sound field environment information, so that the control circuit 132 can dynamically adjust the first channel audio 112 and the second channel audio 122 played from the first speaker 110 and the second speaker 120. For example, the identification circuit 134 can identify a user from the sound field environment information and determine a user position of the user in the target space. Since the sound field environment information provided by the sensor circuit 140 can be a combination of multiple different types, the identification circuit 134 can also implement different identification technology solutions accordingly. For example, when the sound field environment information is an image, the recognition circuit 134 can use artificial intelligence recognition technology to identify the user in the image. Through the application of artificial intelligence, after the recognition circuit 134 analyzes the user in the image, it can further locate the user's head, face, and even ear position. If the sensor circuit 140 can provide diversified information such as three-dimensional images with spatial depth, infrared thermal imaging, or wireless signals, it will help the recognition circuit 134 obtain more accurate recognition results.

In order to calculate the degree of interference caused by the environmental object 175 to the sound field environment, the host device 130 needs the spatial configuration information and acoustic property information of the environmental object 175. The spatial configuration information may include the size, position, shape, and various appearance characteristics of the environmental object 175. The acoustic property information may include material-related characteristics such as the absorption rate, reflectivity, and resonance frequency of sound. In one embodiment, when recognizing the sound field environment information, the recognition circuit 134 may further recognize the spatial configuration information of the environmental object 175 in the target space 170 from the sound field environment information and search for the acoustic property information. In order to recognize the environmental object, an object database is required. In one embodiment, the storage circuit 131 in the host device 130 can also be used to store an object database. The object database may include various appearance feature information for identifying environmental objects, as well as various acoustic property information corresponding to each environmental object. For example, when the host device 130 needs to calculate the degree of interference caused by an environmental object 175 to the sound field environment, the object name of the environmental object 175 can be analyzed through the identification circuit 134 first, and then the host device 130 reads the storage circuit 131 to find the corresponding absorption rate and reflectivity of the environmental object 175.

In practice, the identification circuit 134 can be a custom-made processor chip, which is used in conjunction with the existing operating system, software or firmware in the storage circuit 131 to perform the artificial intelligence identification function. The identification circuit 134 can also be one of the cores or thread circuits of the control circuit 132, which executes the existing artificial intelligence software product in the storage circuit 131 to realize the identification function. The identification circuit 134 can also be a memory module of a specific artificial intelligence software product, which is executed by the control circuit 132 to complete the identification function.

The human-machine interface circuit 133 in the host device 130 allows the user to control the operation of the host device 130. The human-machine interface circuit 133 may include a display screen, buttons, a turntable, or a touch screen, which allows the user to perform basic control functions of the audio system 100, such as adjusting the volume, playing, fast forwarding and rewinding, etc. In one embodiment, the control circuit 132 may also execute a configuration program through the human-machine interface circuit 133 to allow the user to set various sound field scenarios, or tell the host device 130 the spatial configuration information of the environmental objects 175 in the target space 170. For example, in the configuration program, the control circuit 132 uses the human-machine interface circuit 133 to receive object configuration data input by the user, such as the object name, type, size and position of one or more environmental objects 175. After the control circuit 132 obtains the spatial configuration information, it searches the object database stored in the storage circuit 131 for the corresponding absorptivity and reflectivity for subsequent sound field compensation operations. In a further derived embodiment, the human-machine interface circuit 133 may also be provided by the user device 150. The user may use the user device 150 to operate the configuration program, and finally the user device 150 transmits the setting result to the control circuit 132 via the communication circuit 136.

The host device 130 can also be connected to a remote database 160 via the communication circuit 136. In a further embodiment, the object database originally stored by the storage circuit 131 can also be stored via the remote database 160. When the host device 130 needs to calculate the degree of interference caused by an environmental object 175 to the sound field environment, it can first analyze the sound field environment information through the recognition circuit 134 to obtain an object feature value, and then use the communication circuit 136 to access the remote database 160 to find an environmental object 175 that meets the object feature value, and obtain the sound field attribute information of the environmental object 175. The remote database 160 may be a server located in the cloud or other systems, and is connected to the host device 130 via a wired or wireless two-way network communication technology. In addition to providing a search function, the remote database 160 may also accept the upload of updated data to continuously expand the database content. For example, the host device 130 may communicate with the remote database 160 using Structured Query Language (SQL).

Based on the system architecture of Figure 1, the audio system 100 proposed in this application can achieve at least the following technical effects. First, the audio system 100 can dynamically track the user's position as a target listening point. The audio system 100 can also dynamically obtain the spatial configuration information of environmental objects as a basis for optimizing the sound field effect. Finally, the audio system 100 dynamically compensates the speaker output based on the user's position and the spatial configuration information of environmental objects to eliminate object interference and optimize the listening effect at the target listening point. The implementation method of dynamically tracking the user's position can adopt multiple technical solutions such as cameras, infrared sensors, or wireless positioning. The implementation method of obtaining the spatial configuration information of environmental objects can be automatic or manual. For example, the audio system 100 can use a camera to capture images and perform artificial intelligence recognition, or allow the user to manually input the on-site environmental conditions through a configuration program. The implementation method of compensating the speaker output can be based on several different algorithms. For example, this manual introduces the channel base algorithm and the object base algorithm.

The following is an example of an audio system 100 dynamically tracking the user's position, using a camera to obtain the sound field environment configuration, and using channel floor compensation to compensate the speaker output using FIG. 2.

Figure 2 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart of FIG. 2, the process in the column belonging to a specific device represents the process performed by the specific device. For example, the part marked in the "sensor circuit" column is the process performed by the sensor circuit 140; the part marked in the "host device" column is the process performed by the host device 130; the part marked in the "speaker" column is the process performed by the first speaker 110 and/or the second speaker 120; and so on. The aforementioned logic is also applicable to other subsequent flowcharts.

In process 202, the sensor circuit 140 dynamically senses the target space to generate sound field environment information. In this embodiment, the sound field environment information can be optical, thermal, or electromagnetic wave information in the target space 170. For example, the sensor circuit 140 can include a camera that continuously shoots video of the target space 170 in a video recording manner, or periodically captures still photos of the target space 170 in a photo taking manner. In another embodiment, the sensor circuit 140 may also include an infrared sensor that is configured to capture thermal imaging data in the target space. The thermal imaging data generated by the infrared sensor, in addition to containing information about the depth of the space, is also extremely sensitive to temperature changes, and is therefore particularly suitable for tracking the user's position. In another embodiment, the sensor circuit 140 may also include a wireless detector, which is set in the target space and detects the wireless signal of an electronic device. When a user holds an electronic device, the wireless detector can detect the beacon time difference or wireless signal strength of the electronic device as an auxiliary means to track the user's location. The electronic device can be the user's own mobile phone, a specially made beacon generator, a head-mounted virtual reality device, a game controller, or a remote control of the audio system 100. It can be understood that this embodiment does not limit the number of sensor circuits 140, nor does it limit the use of only one sensing scheme at a time. For example, the audio system 100 of this embodiment can use multiple sensor circuits 140 to work together from different locations, or use one or more cameras, infrared sensors, and wireless detectors at the same time. In this way, the host device 130 can obtain more complete sound field environment information and obtain more accurate recognition results in subsequent procedures.

In process 204, the sensor circuit 140 transmits the sensed sound field environment information to the host device 130. The sensor circuit 140 can continuously transmit data, such as video, or periodically return static data. The frequency of the sensor circuit 140 transmitting data can be adaptively determined based on the amount of information in the sound field environment information, the tracking accuracy requirements, and the computing power of the host device 130. The sensor circuit 140 and the host device 130 can be connected through a dedicated line or through a communication circuit 136. In a further derived embodiment, the sensor circuit 140 can share the audio transmission circuit 135 with the speaker, so as to transmit the sound field environment information to the host device 130 through the audio transmission circuit 135.

In process 206, the host device 130 determines the user's position based on the sound field environment information received from the sensor circuit 140. The recognition circuit 134 in the host device 130 can perform a recognition program on the sound field environment information, such as applying artificial intelligence. As the sensing scheme of the sensor circuit 140 is different, the recognition algorithm of the recognition circuit 134 is also correspondingly different. It can be understood that the target space 170 and the user's position can be represented in two-dimensional space or three-dimensional space. If only a single sensor circuit 140 is implemented in the audio system 100, at least the position information in two-dimensional space can be perceived. If the number of sensor circuits 140 is increased or a mixed multi-sensing scheme is used in the implementation of the audio system 100, depth information in three-dimensional space can be obtained to more accurately determine the user's position or the user's head position. In one embodiment, the recognition circuit 134 can dynamically identify the user's head position, facial direction, or ear position based on the sound field environment image captured by the camera. In another embodiment, the recognition circuit 134 can analyze the movement trajectory of the thermal imaging data generated by the infrared sensor to dynamically determine the position of the user 180. For another example, the recognition circuit 134 can dynamically locate the coordinate value of the electronic device in the target space 170 based on the characteristics of the wireless signal detected by the wireless detector. By using the coordinate value, the control circuit 132 can further estimate the user's ear position.

In process 208, after the recognition circuit 134 in the host device 130 analyzes the user's position, the control circuit 132 in the host device 130 dynamically assigns the user's position as a target listening point. For the convenience of describing subsequent embodiments, the target space 170 is described as a two-dimensional coordinate space or a three-dimensional coordinate space, and the target listening point can be represented as a coordinate value in the target space 170. Depending on the layout of multiple speakers, the range of the target listening point can be more than a single point, but can also be a surface, or a three-dimensional area with length, width, and height. For example, after the recognition circuit 134 analyzes the user's head position or ear position, the control circuit 132 can assign the head position or ear position as the target listening point. The control circuit 132 will perform subsequent compensation operations so that the playback effect obtained by the target listening point is not affected by the user's movement. In practice, the control circuit 132 compensates the listening effect obtained by the target listening by adjusting the first channel audio 112 and the second channel audio 122. It can be understood that the process 208 may be executed dynamically as the user's position changes. Therefore, the process 208 is not limited to being executed in the order shown in Figure 2. In other words, the target listening point can be updated in real time as the user's position changes. The specific adjustment operation will be described later.

In process 210, the recognition circuit 134 in the host device 130 further recognizes the sound field environment information provided by the sensor circuit 140, and obtains the spatial configuration information of the environmental objects in the target space 170. In other words, the sound field environment information provided by the sensor circuit 140 can be used not only to determine the user's position, but also to determine the various environmental objects 175 in the target space 170. In one embodiment, after the camera in the sensor circuit 140 captures a sound field environment image of the target space 170, the recognition circuit 134 analyzes the sound field environment image and recognizes one or more environmental objects 175 and the spatial configuration information of these environmental objects 175 from the target space 170. The spatial configuration information includes the size, position, shape, and appearance characteristics of the environmental objects 175. The recognition circuit 134 can also determine the acoustic property information of each environmental object 175, such as the absorption rate and reflectivity of sound, through artificial intelligence calculation or database retrieval. In a further derived embodiment, the recognition circuit 134 can also determine the application scene category of the target space 170 based on the sound field environment image. The application scene category can include theater, living room, bathroom, outdoor, etc. If the host device 130 knows the application scene category of the target space 170, it can more quickly identify the environmental objects 175 in the target space 170 and reduce misjudgment. The relevant embodiment will be described in Figure 9.

In process 212, the control circuit 132 in the host device 130 can calculate the degree to which the playback effect of the speaker on the target listening point is affected by environmental objects. The playback effect of a speaker on the target listening point can be defined as the equivalent volume (Equal loudness) or sound pressure level (SPL) received from the speaker at the target listening point. The ISO226 standard defines an equal sound curve (Fletcher-Munson Curve), which explains that the equivalent volume perceived by the user in different sub-bands actually corresponds to different sound pressure values. In one embodiment, the control circuit 132 can use the equal sound curve as a standard reference benchmark for the playback effect to calculate the sound pressure values received by the target listening point in various situations. The control circuit 132 can use the spatial configuration information and acoustic property information of the environmental object 175 to evaluate the interference caused by the environmental object 175 to the target listening point, so as to further calculate the method of eliminating the interference. The impact of the spatial configuration information and property information of the environmental object 175 includes many scenarios. For example, the larger the volume of the environmental object 175, the greater the interference coefficient on the target listening point may be. Whether the position of the environmental object 175 blocks the user 180 and the speaker also determines the degree of influence on the speaker. Depending on the material, the environmental object 175 may absorb sound or bounce sound. Therefore, the control circuit 132 needs to select corresponding parameters or formulas for different acoustic properties to calculate the degree of influence on the speaker.

In process 214, the control circuit 132 in the host device 130 uses a channel floor compensation operation to calculate the output compensation value required for the channel audio of each speaker. The channel floor compensation operation is calculated separately for each channel audio when judging the playback effect on the target listening point. For example, a first channel audio 112 played by a first speaker 110 among multiple speakers may be interfered by an environmental object 175 and lose energy before the first channel audio 112 is transmitted to the target listening point through the air. Changes in the position of the target listening point will also affect the sound pressure value generated by the first channel audio 112 at the target listening point. Through the channel floor compensation operation, the control circuit 132 can calculate the sound pressure value change of the first channel audio 112 at the target listening point. The control circuit 132 of this embodiment adds an output compensation value to the first channel audio 112 to offset the sound pressure value change, so that the first channel audio 112 received by the target listening point is restored to the state before being affected. In other words, the output compensation value has the same value as the sound pressure value change, but the opposite positive and negative polarity.

In process 216, the control circuit 132 adjusts and outputs the channel audio to the speaker according to the output compensation value. Since the adjusted channel audio has offset the displacement effect of the user 180 in the target space 170 and the interference caused by the environmental objects 175, the listening effect felt by the user 180 remains consistent. Taking the first speaker 110 and the second speaker 120 in the target space 170 as an example, the control circuit 132 calculates and adjusts the sound pressure values of different sub-bands in the first channel audio 112 and the second channel audio 122, thereby offsetting the equivalent volume deviation felt by the user 180 due to movement. On the other hand, the control circuit (132) compensates the first channel audio 112 and the second channel audio 122 accordingly according to the change in the sound pressure value caused by the position, size, and acoustic property information of the environmental object 175 at the target listening point.

In process 218, each speaker receives the channel audio from the host device 130 via the audio transmission circuit 135. Taking the first speaker 110 and the second speaker 120 in the target space 170 as an example, the control circuit 132 outputs the first channel audio 112 and the second channel audio 122 to the corresponding first speaker 110 and the second speaker 120 via the audio transmission circuit 135. Therefore, the first speaker 110 and the second speaker 120 play the adjusted first channel audio 112 and the second channel audio 122 correspondingly, so that the target listening point where the user 180 is located obtains an optimized listening effect. For ease of explanation, only two speakers and one environmental object 175 are shown in the embodiment of the target space 170 in FIG. 1. However, it is understood that in practice, the host device 130 may include more than two speakers, and the number of environmental objects 175 is not limited to one. In further derived embodiments, the audio range that each speaker is good at outputting may be different. For example, some speakers are mid-high speakers, and some speakers are subwoofers. When adjusting the channel audio, the control circuit 132 can further adjust the corresponding output first channel audio 112 and second channel audio 122 according to the characteristics of different speakers.

FIG. 3 below illustrates an example of an audio system 100 dynamically tracking the user's position, using a camera to obtain the sound field environment configuration, and using object-based compensation to compensate the speaker output.

Figure 3 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart of FIG. 3, the process in the column belonging to a specific device represents the process performed by the specific device. For example, the part marked in the "sensor circuit" column is the process performed by the sensor circuit 140; the part marked in the "host device" column is the process performed by the host device 130; the part marked in the "speaker" column is the process performed by the first speaker 110 and/or the second speaker 120; and so on. The aforementioned logic is also applicable to other subsequent flowcharts.

The processes 202, 204, 206, 208 and 210 in FIG. 3 are the same as those in the previous embodiment and will not be repeated for the sake of space saving.

When the audio system 100 of the present embodiment completes process 210, the control circuit 132 has tracked the position of the user 180 and assigned it as the target listening point, and has also obtained the spatial configuration information of one or more environmental objects 175 in the target space 170. The subsequent process then describes the object-based compensation operation to adjust the channel audio of each speaker.

The object-based acoustic system originated from the mixing technology of virtual reality, which can use a limited number of physical speakers to simulate the effect of the movement of the sound source object. Some existing software products, such as Dolby Atmos, Spatial Audio Workstation, or Digital Spatial Reality, are all object-based acoustic systems. Users can define the movement trajectory of the sound source object in a virtual space through a human-computer interface. The object-based system can use physical speakers to simulate the sound effects of the sound source object in the virtual space. Users at the target listening point can truly feel the sound source object moving in the space.

The object-based acoustic system is based on array operations of a large number of acoustic parameters. Each sound source object has a metadata that describes the type, position, size (length, width and height), divergence, etc. of the sound source object. After the object-based array operation, the sound represented by a sound source object will be assigned to one or more speakers to play together, and each speaker plays a part of the sound of the sound source object. In other words, the object-based array operation can use multiple speakers to simulate the spatial effect of a sound source object. The embodiment of Figure 3 proposes an object-based compensation operation based on the object-based acoustic system to solve the traditional playback effect problem.

In process 312, the control circuit 132 in the host device 130 establishes an object-based compensating sound source object according to the environmental object 175. In practice, the control circuit 132 first maps the target space 170 to an object-based space in the virtual reality, and then establishes a compensating sound source object in the object-based space according to the environmental object 175 to produce a sound source effect that offsets the environmental object 175. For the user 180 at the target listening point, the existence of the environmental object 175 can also be analogized to a sound source object. In actual application scenarios, the environmental object 175 may reflect the sound emitted by a speaker to the target listening point. Environmental object 175 may also block or absorb part of the sound, causing the sound emitted by a speaker to the target listening point to be attenuated. In other words, after the control circuit 132 of this embodiment compares the environmental object 175 to a sound source object, it can correspondingly establish a negative sound source object with an opposite sound source effect in the base space of the object as a means of offsetting interference. The sound source effect described in this embodiment can be the sound pressure value, equivalent volume, or gain value generated for the target listening point.

In process 314, the host device 130 substitutes the compensation sound source object into the object-based compensation operation to generate channel audio. The object-based compensation operation can utilize the object-based array operation module in the existing object-based acoustic product to perform a large number of array operations related to acoustic interaction based on the metadata of the sound source object. For example, a metadata of the compensation sound source object includes: the coordinate position, size, and reflectivity and absorption rate of the environmental object 175 to sound. The control circuit 132 performs an object-based compensation operation based on the target listening point and the metadata to offset the interference of the environmental object 175 on the target listening point and generate the first channel audio 112 and the second channel audio 122 optimized for the target listening point.

In one embodiment, the object base compensation operation is performed on multiple sub-bands. Due to the characteristics of sound transmission, the sound pressure value on each sub-band has different effects on the equivalent volume. Taking the effect of the first channel audio 112 generated by the first speaker 110 on the environmental object 175 as an example, the control circuit 132 of this embodiment can calculate a sound source effect generated by the environmental object 175 being activated by the first channel audio 112 on multiple sub-bands according to the coordinate position, size, and reflectivity and absorption rate of the environmental object 175. Then the control circuit 132 establishes the compensation sound source object according to the sound source effect. In this embodiment, the compensation sound source object is established in correspondence with the environment object 175, wherein the metadata has the same coordinate position, size, and sound reflectivity and absorption rate as the environment object 175, but the positive and negative signs of the sound source effect generated are opposite to those of the environment object 175.

It is known that the audible range of human ears is between 20 Hz and 20000 Hz. This embodiment can divide the audible range of human ears into multiple sub-band intervals and compensate them separately. The interval size of each sub-band can be an exponential interval. For example, the exponential interval with a base of 10 can divide the sound signal into multiple sub-band ranges such as 10Hz to 100Hz, 100Hz to 1000Hz, and 1000Hz to 10000Hz. In other embodiments, the exponential interval can also be divided with a base of 2 or 4 according to the requirements of the precision of the playback quality. In the equalizer in the field of audio processing, there is already a processing technology for cutting multiple sub-bands, which will not be explained in depth here.

After the control circuit 132 obtains the negative sound source effect of the compensation sound source object, an object base compensation operation is performed to mix the negative sound source effect into the first channel audio 112 and the second channel audio 122 in a proportion determined by the mixing operation result, thereby offsetting the interference of the environmental object 175 on the target listening point. The object base compensation operation will be described in detail in the embodiments of Figures 11 to 13.

In process 316, the host device 130 outputs the first channel audio 112 and the second channel audio 122 to the first speaker 110 and the second speaker 120 in accordance with the calculation result of process 314. Process 316 of FIG. 3 is different from process 216 of the embodiment of FIG. 2. FIG. 2 calculates the compensation value for the existing channel audio to adjust the existing channel audio. When the control circuit 132 performs the object base compensation operation, it directly calculates the channel audio corresponding to each speaker at once according to all the metadata. The object base compensation operation mixes the interference component that needs to be offset or compensated into the channel audio in the form of a compensation sound source. In other words, because the channel audio includes the compensation sound source emitted by the compensation sound source object, the user 180 cannot feel the influence caused by the existence of the environmental object 175 at the target listening point.

As can be seen from process 316, the object-based compensation operation converts the target listening point and environmental objects into the metadata of the object-based acoustic system and establishes the compensation sound source object, which simplifies the calculation process of eliminating interference and optimizing the playback effect. It should be understood that the audio system 100 of this embodiment can use the sensor circuit 140 to track the position of the user 180 in real time or periodically to dynamically update the target listening point. The object-based compensation operation performed by the control circuit 132 can also synchronously update all the metadata related to the relative position of the target listening point in the target space 170 as the target listening point changes.

The process 218 in FIG. 3 is the same as that in the previous embodiment, and will not be repeated for the sake of saving space.

The following is an example of how the audio system 100 dynamically tracks the user's position, runs a configuration program to obtain the sound field environment configuration, and uses a channel floor compensation operation to compensate the speaker output.

Figure 4 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart of FIG. 4 , the process in the column belonging to a specific device represents the process performed by the specific device. For example, the part marked in the "sensor circuit" column is the process performed by the sensor circuit 140; the part marked in the "host device" column is the process performed by the host device 130; the part marked in the "speaker" column is the process performed by the first speaker 110 and/or the second speaker 120; and so on. The aforementioned logic is also applicable to other subsequent flowcharts.

Processes 202, 204, 206, and 208 in FIG. 4 are the same as those in the previous embodiment, and will not be repeated for the sake of space saving.

When the audio system 100 of the present embodiment completes process 210, the control circuit 132 has tracked the position of the user 180 and assigned it as the target listening point, and has also obtained the spatial configuration information of one or more environmental objects 175 in the target space 170. The next process to be performed is to use the object-based algorithm to adjust the channel audio of each speaker.

In order to eliminate interference in the sound field environment, the audio system 100 needs to obtain the spatial configuration information of various environmental objects 175 in the target space 170.

In process 410, the control circuit 132 in the host device 130 can run a configuration program to obtain the spatial configuration information of one or more environmental objects 175 in the target space 170. In the previous embodiment, the host device 130 uses the sound field environment information captured by the sensor circuit 140 to automatically identify the spatial configuration information of the environmental object 175. When running the configuration program, the host device 130 can use a human-machine interface circuit 133 to interact with the user, allowing the user to manually input the spatial configuration information of the environmental object 175. The human-machine interface circuit 133 can provide a screen and an input method, allowing the user to define the spatial configuration information of various objects in the target space 170 in a two-dimensional plane or a three-dimensional stereogram. The spatial configuration information of the environmental object 175 may include the relative position, size, name, and material type of the environmental object 175 in the target space 170. In a further derived embodiment, the user 180 may inform the host device 130 of the application scene category to which the current target space 170 belongs through the human-machine interface circuit 133. In different application scenes, such as open outdoor spaces, theater spaces, or bathrooms, the common types of environmental objects 175 are also different, and the sound field atmosphere felt by the user is also different. Optimizing the sound field for different application scenes is also one of the important functions of the audio system 100.

Different types of materials have different acoustic properties. When the host device 130 runs the configuration program, it further queries an object database based on the object name or material type input by the user to obtain acoustic property information of the environmental object 175, such as the absorption rate or reflectivity of sound. In this way, the host device 130 can calculate the degree to which the playback effect of each speaker on the target listening point is affected by the environmental object 175 based on the aforementioned spatial configuration information and acoustic property information in the subsequent process 212. In a further derived embodiment, the host device 130 can use the corresponding object database preferentially based on the application scene category of the target space 170 to more quickly identify the environmental object 175 in the target space 170. The relevant embodiment will be illustrated in Figure 9.

Processes 212, 214, 216 and 218 in FIG. 4 are the same as those in the previous embodiment and will not be repeated for the sake of space saving.

The embodiment of FIG. 4 illustrates that the audio system 100 can not only dynamically track the user's position, but also allow the user 180 to set the spatial configuration information of the environmental object 175 in the target space 170 through a configuration program. The configuration program provides a channel for manual input to make up for the lack of recognition function. In addition to assisting the host device 130 to make more accurate judgments through active input, the user also has the opportunity to deliberately specify different application scene categories according to his or her preferences, or deliberately set up imaginary virtual sound source objects to change the playback effect. The host device 130 will operate with channel base compensation, and calculate the output compensation value corresponding to each speaker according to the spatial configuration information of the environmental object 175 in the target space 170.

The following is an example of how the audio system 100 dynamically tracks the user's position, runs a configuration program to obtain the sound field environment configuration, and uses object-based compensation to compensate for the speaker output.

Figure 5 is a flow chart of a dynamic sound optimization method according to an embodiment of the present invention.

In the flowchart of FIG. 5 , the process in the column belonging to a specific device represents the process performed by the specific device. For example, the part marked in the "sensor circuit" column is the process performed by the sensor circuit 140; the part marked in the "host device" column is the process performed by the host device 130; the part marked in the "speaker" column is the process performed by the first speaker 110 and/or the second speaker 120; and so on. The aforementioned logic is also applicable to other subsequent flowcharts.

The processes 202, 204, 206, 208 and 210 in FIG. 5 are the same as those in the previous embodiment and will not be repeated for the sake of space saving.

Similar to the embodiment of FIG. 4 , the embodiment of FIG. 5 runs the same process 410 as FIG. 4 in order to eliminate interference in the sound field environment.

In process 410, the host device 130 runs a configuration program to obtain spatial configuration information of one or more environmental objects 175 in the target space 170. In the embodiment of FIG. 4 , it is illustrated that the host device 130 can receive spatial configuration information of the environmental objects 175 manually input by the user through the human-machine interface circuit 133. In a further derivative embodiment, the host device 130 can also use the communication circuit 136 to receive spatial configuration information transmitted from the user equipment 150 or other devices. For example, the user equipment 150 can be a mobile phone running an application to provide functions similar to the human-machine interface circuit 133. The application allows the user to define the scope and size of the target space 170, the position of each speaker relative to the target space 170, the position, size, name and type of various environmental objects 175, and even the position of the user 180 himself. The application can also communicate with the control circuit 132 through the communication circuit 136 to perform various playback operations, such as play, pause, fast forward, adjust the volume, etc. In addition, the user can set the application scene category of the target space 170 through the human-machine interface circuit 133, so that the host device 130 produces a diversified playback effect on the target space 170.

In a further derived embodiment, the user device 150 connected to the host device 130 may be a virtual reality device or a game console. The user device 150 generates an audio source signal for the host device 130 to play. The audio source signal may include virtual objects moving in a virtual reality space, such as an airplane or a fire-breathing dragon. The user device 150 can transmit the metadata of these virtual objects to the host device 130, which becomes part of the environmental object space configuration information of the target space 170. In other words, the host device 130 can use the object-based acoustic system to process virtual objects and physical objects equally. Through the object base compensation operation, the host device 130 can make the user feel that there is a virtual object in the target space 170, and can also make the user not feel the interference of a physical object in the target space 170. The implementation of the object base compensation operation is further described in the embodiments of Figures 11 to 13.

When the audio system 100 of the present embodiment completes process 410, the control circuit 132 has tracked the position of the user 180 and assigned it as the target listening point, and has also obtained the spatial configuration information of one or more environmental objects 175 in the target space 170. Then in processes 312 to 316, the host device 130 uses the object-based algorithm to adjust the channel audio of each speaker. Since processes 312 to 316, as well as process 218, are the same as the previous embodiment, they will not be repeated for the sake of saving space.

The embodiment of FIG. 5 illustrates that in addition to dynamically tracking the user's position, the audio system 100 also allows the user 180 to set the spatial configuration information of the environmental object 175 in the target space 170 through a configuration program. The configuration program can be integrated with the existing virtual reality technology to receive the spatial configuration information of the virtual object. The audio system 100 converts the physical environmental objects and virtual objects into metadata with the same format, and then applies all the metadata to the object base array operation module of the existing object base acoustic system to perform object base compensation operations. In this way, the control circuit 132 does not need to develop additional operation modules for different objects, which can reduce costs and improve execution efficiency.

The following uses Figure 6 to illustrate several implementations of the sensor circuit and the compensation algorithm for the channel base.

FIG6 is a schematic diagram of a target space 600 of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the position of the best listening point.

The audio system 100 of the present application uses a sensor circuit 140 to dynamically sense the target space 600 to generate sound field environment information. The sound field environment information mainly includes the position of the user 180, and may also include the spatial configuration information of environmental objects. There are many options for the technical solution of dynamic sensing. For example, the sensor circuit 140 can be a combination of one or more of a camera 610, an infrared sensor 620, and a wireless detector 630, which are respectively configured at different positions around the target space 600, providing sound field environment information with spatial depth to help the recognition circuit 134 and the control circuit 132 in the host device 130 to track the position of the user 180 more efficiently. Thus, the recognition circuit 134 utilizes the sound field environment information provided by the sensor circuit 140 to not only recognize the user's 180 position, but also the direction of the face, the position of the ears, and even the gestures or body postures. The control factors that can be applied to adjust the sound field become richer. For example, focus detection, sleep detection, gesture control, etc.

In the target space 600 of FIG. 6 , a first speaker 110 and a second speaker 120 are configured. The channel base compensation operation can calculate the output compensation value for each speaker separately. Under the default situation, the target listening point is located at the center of the target space 600, that is, the first position 601 in FIG. 6 . The distance between the first position 601 and the first speaker 110 and the second speaker 120 is also R1. At this time, the first channel audio 112 and the second channel audio 122 played by the first speaker 110 and the first channel audio 112 are also in the default state, and no compensation processing is required for the position.

When the user 180 moves from the first position 601 to the second position 602 along the moving trajectory 173, the sensor circuit 140 detects the new position of the user 180 and assigns the target listening point of the audio system 100 to the second position 602. At this time, the distance between the user 180 and the first speaker 110 changes to R2, and the distance between the user 180 and the second speaker 120 changes to R2'. For the user 180, the first speaker 110 becomes farther away, so the received first channel audio 112 is attenuated due to the distance. In contrast, the second speaker 120 becomes closer, and the received second channel audio 122 is enhanced. In other words, the strength of the first channel audio 112 and the second channel audio 122 received at the second position 602 has lost balance. This embodiment uses a channel-based algorithm to restore the listening effect received by the second position 602 to the same default state as the first position 601. In other words, the control circuit 132 compensates the first channel audio 112 and the second channel audio 122 output by the first speaker 110 and the second speaker 120 to offset the listening effect deviation caused by the movement of the user 180. The target space 600 shown in FIG. 6 is not limited to a multi-speaker environment configured horizontally. In a three-dimensional sound field environment configured with upper and lower speakers, the problem of distance deviation also occurs. For example, if the user changes from a standing position to a sitting position, he will move away from the upper speaker and closer to the lower speaker.

In order to obtain a better compensation effect, this embodiment uses equivalent loudness as the calculation standard. For example, this embodiment can calculate the sound pressure value that needs to be compensated at the target listening point according to the equal sound curve defined by the ISO226; 2003 protocol. Each channel audio is divided into multiple sub-bands and processed separately. In addition, as the distance between the user and the speaker is different, the sound field formula used is also different. Since the equal sound curve defines a linear relationship between equivalent volume and sound pressure value, and the equivalent volume and the "gain value" in "decibels" have a linear corresponding relationship. Therefore, this embodiment is not limited to adjusting in any of the units of equivalent volume, sound pressure value, or gain value.

In the audio system 100, the space where sound is transmitted due to air vibration is called the sound field. Due to the existence of reflection, the sound field in a closed room can be divided into multiple types. (1) Near Field: When the user is 180 degrees closer to the sound source, the physical influence of the sound source (such as pressure, displacement, vibration) will enhance the sound. (2) Reflected Field: The sound is reflected by an object and causes a wave superposition effect. (3) Free Field: The sound field is not disturbed by the aforementioned near field and reflected field. The above reflected field and free field can be collectively referred to as the far field.

In many current audio systems, the near and far sound fields are defined differently. For example, assuming that R is the distance between the speaker and the user (meters), L is the width of the speaker (meters), and λ is the representative wavelength of a sub-band signal (meters), the conditions for satisfying the far sound field include the following types: R>>λ/2π (1)

R>>L (2)

R>>πL ² /2λ (3)

Take the first speaker 110 in FIG. 1 as an example. When the distance between the target listening point and the first speaker 110 is greater than a specific ratio of the wavelength of the sub-band signal or the size of the first speaker 110, the audio system 100 determines that the sound field type is a far sound field. When the distance between the target listening point and the first speaker 110 is less than the wavelength of the sub-band signal or the specific ratio of the size of the first speaker 110, the sound field type is determined to be a near sound field. In a simpler implementation, the audio system 100 can define twice the value (2λ) of the wavelength corresponding to the central frequency of a sub-band signal as the dividing point between the far sound field and the near sound field of the sub-band signal.

In the far-field sound field, the relationship between the change in sound pressure value of a sub-band signal received by the user 180 from the speaker and the change in distance is as follows: SPL2 = SPL1-20 log ₁₀ (R2/R1) (4)

Among them, SPL2 is the sound pressure value of the sub-band signal received at the new position, SPL1 is the sound pressure value of the sub-band signal received at the original position, R2 is the distance between the new position and the speaker, and R1 is the distance between the original position and the speaker.

From formula (4), we can see that the difference between SPL1 and SPL2 is the part that needs to be compensated for the speaker.

SPL2'=SPL2+20 log ₁₀ (R2/R1)=SPL1 (5)

Among them, SPL2' is the sound pressure value of the sub-band signal received at the new position after compensation. From formula (5), it can be seen that this embodiment compensates the changed part.

In the near sound field, the relationship between the sound pressure value change of the sub-band signal received by the user 180 from the speaker and the distance change is as follows: SPL2 = SPL1-10 log ₁₀ (R2/R1) (6)

SPL2'=SPL2+20 log ₁₀ (R2/R1)=SPL1 (7)

From formulas (6) and (7), we can see that the sound attenuation change rate in the near sound field is gentler than that in the far sound field, while the other calculation logics are the same.

It is understandable that the above formula may have exceptions when encountering some special circumstances. For example, when the user 180 moves from the first position 601 to the second position 602 and approaches the second speaker 120, the distance between the user 180 and the second speaker 120 decreases from R1 to R2', which may make the calculation result of formula (7) become a negative value. However, the sub-band signal output by the second speaker 120 cannot be a negative value, and the minimum can only be reduced to the lowest audible value of the human ear. For example, the sound pressure value of the sub-band signal output by the second speaker 120 is made zero. On the other hand, when the user 180 moves from the first position 601 to the second position 602 and moves away from the first speaker 110, the distance between the user 180 and the first speaker 110 increases from R1 to R2. The maximum output limit of the first speaker 110 may not be able to satisfy formula (5). At this time, the audio system 100 may issue an over-limit reminder to the user 180.

The embodiment of FIG. 6 highlights the following advantages. Through the compensation algorithm of the channel basis, the user's best listening point is not affected by movement. The calculation method of the channel basis is simple and efficient, and can be applied in most target spaces 600.

FIG6 has already explained the sound compensation method based on the movement of the user 180. FIG7 is used below to explain the sound compensation method based on the environmental object 175. The acoustic property information of the environmental object 175 includes the reflectivity and absorptivity of the sound. This embodiment uses an appropriate calculation method to calculate the acoustic impact of the environmental object according to the spatial configuration information of the environmental object 175.

FIG. 7 is a schematic diagram of a target space 700 of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the absorption rate of environmental objects.

FIG. 7 shows that in a target space 700, an environmental object 175 is located between a first speaker 110 and a user 180. For example, the environmental object 175 may be a sofa or a pillar. In this case, the environmental object 175 may cause the user 180's listening effect to be attenuated due to shielding. In other words, the sound pressure value received by the user 180 from the first speaker 110 will be shielded or absorbed. When the control circuit 132 interprets this layout condition through the spatial configuration information, it uses the absorption rate of the environmental object 175 to calculate the extent to which the playback effect of the first speaker 110 at the target listening point (the position of the user 180) is affected by the environmental object 175, so as to determine the equivalent volume, sound pressure value or gain value that the first channel audio 112 needs to output.

In one embodiment, the sound loss absorbed by the environmental object 175 can be calculated based on the sound pressure value received by the environmental object 175 from the first speaker 110: _At [n]=R[n]* _SPLt (8)

Wherein, n represents the number of the sub-band. That is, the first channel audio 112 output by the first speaker 110 can be cut into multiple sub-bands for calculation respectively. _At [n] represents the gain value of the nth sub-band detected at the time point t. R [n] represents the absorption rate of the nth sub-band. SPL _t represents the sound pressure value from the first speaker 110 to the environmental object 175 at the tth time point. The time point t can represent the time difference of the sound being transmitted from the first speaker 110 to the environmental object 175.

As can be seen from formula (8), _At [n] represents the gain value of the first channel audio 112 absorbed by the environmental object 175 in the nth sub-band, and also represents the output compensation value required for the nth sub-band of the first channel audio 112. Therefore, when the control circuit 132 generates the first channel audio 112 through the first speaker 110, the gain value of the nth sub-band of the first channel audio 112 is increased by the gain value _At [n].

There may be multiple scenarios when the environmental object 175 is located between the first speaker 110 and the user 180. In this embodiment, whether the visual line between the first speaker 110 and the user 180 is blocked is the main basis, or the visual line between the first speaker 110 and the ear of the user 180 is further used as the judgment standard. It can be understood that SPL _t itself is a function related to the distance and time between the environmental object 175 and the first speaker 110, and the influence of the calculated _At [n] on the user 180 is a function related to the distance and time between the environmental object 175 and the user 180. After considering the arrangement conditions at different angles and the distance relationship, various nonlinear correlations are involved. The present application does not limit the derivative changes of formula (8), for example, other weight coefficients, parameters, and offset correction values may be added according to the implementation. For example, a sofa may be placed between the user 180 and the first speaker 110. Although the sofa does not block the line of sight, it may still affect the sound pressure value received by the user 180 from the first speaker 110. The control circuit 132 can use the interpolation method or other correction formulas according to formula (8) to make the compensation result more in line with the requirements.

FIG8 is a schematic diagram of a target space 800 of the present invention, used to illustrate an embodiment of calculating the audio adjustment amount based on the reflectivity of environmental objects.

FIG8 shows a user 180 located between a first speaker 110 and an environmental object 175 in a target space 800. The environmental object 175 may be a wall, a ceiling, or a floor. In this case, the environmental object 175 will bounce the first channel audio 112 output by the first speaker 110 to the user 180. In other words, the sound pressure value received by the user 180 from the first speaker 110 will be superimposed or interfered. When the control circuit 132 interprets this layout condition through the spatial configuration information, it uses the reflectivity of the environmental object 175 to calculate the extent to which the playback effect of the first speaker 110 at the target listening point (the position of the user 180) is affected by the environmental object 175, so as to determine the equivalent volume, sound pressure value or gain value that the first channel audio 112 needs to output.

In this embodiment, the impact of the environmental object 175 can also be calculated according to formula (8), but R[n] is changed to represent the reflectivity of the environmental object 175 in the nth sub-band.

The calculation result _At [n] of formula (8) may represent the component of the first channel audio 112 reflected by the environmental object 175 to the user 180 in the nth sub-band. Therefore, when the control circuit 132 generates the first channel audio 112 through the first speaker 110, it may appropriately reduce the gain value of the first channel audio 112, so that the total sound pressure value received by the user 180 from the first speaker 110 and the environmental object 175 is maintained at a preset level.

Similar to the embodiment of FIG. 7 , there may be multiple variations of the situation in which the user 180 in FIG. 8 is located between the first speaker 110 and the environmental object 175 . This embodiment is mainly based on whether the visual line of sight between the first speaker 110 and the environmental object 175 is blocked by the user 180 . However, in practice, the walls, ceilings, and floors have a reflective effect regardless of their angles. Therefore, the calculation formula of this embodiment is not limited to formula (8), and other nonlinear compensation calculation methods may be further derived based on the arrangement and proximity relationship. For example, the target space 800 can be classified into different application scenarios, such as a living room, a study, a bathroom, a theater, or outdoors, due to the characteristics of the walls, ceilings, floor materials, and the size, shape, and layout of the room. The host device 130 may first classify the application scenarios to which the target space 800 belongs, and then respectively adopt corresponding parameters or formulas.

The embodiments of FIG. 7 and FIG. 8 highlight the following advantages. Through the channel floor compensation operation, the influence of the environmental object 175 on the listening effect of the user 180 is offset. The channel floor compensation operation can flexibly apply different object acoustic properties according to the configuration of the environmental objects, and can effectively deal with the optimization problems of various complex environments.

In summary, the identification circuit 134 can receive data from the sensor circuit 140 and identify the position of the user 180 in the target space 170, so that the control circuit 132 dynamically assigns the position of the user 180 as the target listening point. The compensation made by the control circuit 132 for the movement of the target listening point has been described in the embodiment of FIG6 and formulas (4) to (7). The compensation made by the control circuit 132 for the interference of the environmental object 175 has been described in FIGS. 7 to 8 and formula (8). These two compensation operations can be performed separately and applied to the channel audio. In other words, the optimized channel audio finally output includes the compensation value made for the movement of the target listening point and also includes the compensation made for the interference of the environmental object 175.

The recognition circuit 134 recognizes the position of the user 180 based on the sound field environment information captured by the sensor circuit 140. The recognition process may also include the recognition of the application scene to help accelerate the subsequent operation of the control circuit 132. The following is a description of the process of the host device 130 recognizing objects based on the application scene category with FIG. 9.

FIG9 is a flowchart of the host device 130 identifying an object in an embodiment of the present invention. The acoustic properties of environmental objects appearing in different application scenarios usually have significant group correlations, and the sound field reflection coefficients caused by the surrounding environmental materials or room sizes are also different. Therefore, distinguishing the application scene categories in advance helps the audio system 100 improve the efficiency of sound field optimization. It can be understood that each process in FIG9 is executed by the host device 130, but it is not limited to being executed by a single circuit or module therein, and can also be the coordinated operation of multiple circuits.

In process 902, the host device 130 obtains the application scene category of the target space 170. The host device 130 can obtain the application scene category in several different ways. In one embodiment, the recognition circuit 134 in the host device 130 can determine an applicable application scene category based on the sound field environment information provided by the sensor circuit 140 when recognizing the sound field environment information. In another embodiment, the control circuit 132 in the host device 130 obtains the spatial configuration information of the environmental object by running a configuration program through the human-machine interface circuit 133, and also obtains the application scene category defined by the user 180 through the configuration program. In a further derived embodiment, the control circuit 132 in the host device 130 can obtain relevant information of the application scenario category from a user device 150 through the communication circuit 136.

In process 904, in order to speed up the query of environmental objects and improve accuracy, the host device 130 preferentially selects relevant object databases according to the application scenario category. The object database is usually a pre-established data set, which can be provided by a variety of different channels. For example, the storage circuit 131 in the host device 130 can pre-store one or more object databases corresponding to different application scenarios. In another embodiment, the host device 130 can be connected to a remote database 160 using the communication circuit 136. The remote database 160 can include multiple object databases corresponding to different application scenarios. Each object database contains appearance feature information and acoustic attribute information of multiple environmental objects.

After the host device 130 obtains the application scene category in process 902, it can preferentially select and use an object database related to the application scene category from the storage circuit 131 or the remote database 160 for subsequent environmental object recognition. In one embodiment, the recognition circuit 134 analyzes the sound field environment information provided by the sensor circuit 140 to obtain one or more object appearance feature information, and searches the object database based on the object appearance feature information to recognize the environmental object that meets the object appearance feature information, including the name, absorptivity and reflectivity. In another embodiment, the control circuit 132 executes a configuration program and uses the human-machine interface circuit 133 to obtain the name of an environmental object. The control circuit 132 searches the object database according to the name of the environmental object 175 to obtain the absorptivity and reflectivity corresponding to the environmental object.

In a further derived embodiment, the parameters used in the search process can be multi-combined. For example, in the process of analyzing the sound field environment information, the recognition circuit 134 can obtain the external characteristics of the environmental object 175, such as material, size, shape, etc. The recognition circuit 134 transmits these external characteristic information to the object database for multi-condition cross-comparison, and obtains a list of candidate objects sorted according to the matching score. If the information of the application scene category is used as the search condition in the process of searching the object database, it will help to narrow the possible range, speed up the recognition, and improve the accuracy.

In process 906, the control circuit 132 searches the object database selected in process 904 for the absorptivity and reflectivity of the environmental object. In practice, the acoustic property information of the environmental object stored in the object database is not limited to being divided into multiple independent object databases for storage. The object database can be a relational database, including multiple fields connected together in the form of correlation coefficients. For example, the fields of the object database can include object name, application scene category, material, absorptivity, reflectivity, and even external features such as shape, color, and gloss. The field value corresponding to each environmental object is not limited to a one-to-one relationship, but can be one-to-many or many-to-one. The value stored in each field is not necessarily an absolute value, but a range value or a probability value. In a further derivative implementation, the object database can be an adaptive database that can be machine-learned and continuously iterated and corrected. The user 180 can train the object database by feeding back the preferred setting value through the human-machine interface circuit 133.

In process 908, the control circuit 132 adjusts the channel audio in multiple sub-bands according to the search results and configuration status of the environmental objects. The acoustic properties of the environmental objects 175 may have significant differences in different frequency bands. For example, a sofa may absorb a large amount of high-frequency signals but does not affect the penetration of low-frequency signals. Therefore, the absorption rate or reflectivity found from the object database can be an array value corresponding to multiple sub-bands, or a frequency response curve. The interval size or segmentation method of the sub-bands can be determined according to design requirements and is not limited in this embodiment. The control circuit 132 adjusts the gain values of the channel audio signals in multiple sub-bands respectively, which can be compared to the concept of an equalizer or a filter in practice. In other words, the control circuit 132 can implement an equalizer for each speaker in the audio system 100, and customize the equalizer according to the output compensation value calculated in the above embodiment, so that the corresponding channel audio signals are adjusted. A further embodiment of calculating the output compensation value will be described in FIG. 10.

In process 910, the control circuit 132 outputs the adjusted channel audio to the corresponding speaker through the audio transmission circuit 135. The implementation of the audio transmission circuit 135 has been introduced in FIG. 1 and will not be described in detail here.

The embodiment of FIG. 9 highlights the following advantages. The operation of object recognition can refer to the application scene category (automatic recognition or manual input) to increase recognition efficiency. The object database adopts an extensible architecture, and the recognition capability is continuously enhanced in the long term with the feedback of cloud big data services and machine learning. The audio system 100 can apply the concept of equalizer to divide the channel audio into multiple sub-bands for separate processing, so that the final synthesized sound quality is effectively improved.

The following FIG. 10 further illustrates how the control circuit 132 calculates the output compensation value of each channel based on the spatial configuration information of the environmental object 175.

FIG10 is a flow chart of an audio processing method of an embodiment of the present invention, illustrating an embodiment of calculating an output compensation value based on the position relationship of environmental objects. The process of FIG10 is mainly executed by the control circuit 132 in the host device 130.

In process 1002, the control circuit 132 determines the relative positional relationship between the environmental objects, the target listening point, and the speaker. Multiple speakers and multiple environmental objects 175 in the target space 170 can be arranged and combined with the target listening point to form multiple sets of positional relationships. Each set of positional relationships includes a speaker, an environmental object 175, and the target listening point. The control circuit 132 checks and determines each positional relationship combination in the target space 170 and calculates the corresponding output compensation value. The following takes one set of positional relationships in the audio system 100 as an example to explain the compensation method of the control circuit 132 for the interference caused by an environmental object 175 to a speaker at the target listening point.

In process 1004, the control circuit 132 determines whether the environmental object 175 is between the target listening point and the speaker. The position of the environmental object 175 in the target space 170 can also be obtained by the identification circuit 134, or by the human-machine interface circuit 133 through a configuration program. After integrating the above information, the control circuit 132 can determine the relative position relationship between each environmental object 175, the target listening point, and each speaker, and perform corresponding compensation operations for each speaker. The situation to be determined by process 1004 is the situation shown in Figure 7. If the situation meets the requirements, process 1008 is performed. If the situation does not meet the requirements, process 1006 is performed.

In process 1006, the control circuit 132 determines whether the target listening point is located between the environmental object 175 and the speaker. The situation to be determined in process 1006 is the situation shown in FIG8. If the situation meets the requirements, process 1010 is performed. If the situation does not meet the requirements, process 1012 is performed.

In process 1008, the control circuit 132 uses the absorption rate of the environmental object 175 to calculate the output compensation value of the channel audio. In a preferred embodiment, the output compensation value of the channel audio of the speaker is calculated separately for multiple sub-bands. For detailed calculations, please refer to the target space 700 and formula (8) in Figure 7. The control circuit 132 can search the absorption rate of the environmental object 175 from the object database and substitute it into formula (8) to obtain the output compensation value.

In process 1010, the control circuit 132 uses the reflectivity of the environmental object 175 to calculate the output compensation value of the channel audio. Referring to the target space 800 of FIG8 and formula (8), the control circuit 132 can search the reflectivity of the environmental object 175 from the object database and substitute it into formula (8) to obtain the output compensation value.

It can be understood that the output compensation value calculated based on the absorptivity of the environmental object 175 may amplify the gain value, sound pressure value, or equivalent volume of the adjusted channel audio to compensate for the absorbed energy. Conversely, the output compensation value calculated based on the reflectivity of the environmental object 175 may reduce the gain value, sound pressure value, or equivalent volume of the adjusted channel audio to balance the reflected energy. In other words, the output compensation values calculated based on the absorptivity and reflectivity are usually opposite in sign.

In process 1012, if the environmental object 175 does not meet the conditions of process 1004 or process 1006, the control circuit 132 may determine that the environmental object 175 is located at a position that will not affect the speaker's playback of the target listening point. In this case, the control circuit 132 may not calculate the impact of the environmental object 175 on the speaker and the target listening point for the set of positional relationships. However, it should be understood that a target space 170 usually includes multiple speakers. The environmental object 175 will not affect the playback of one of the speakers to the target listening point, but it may still affect the playback of other speakers to the target listening point. In other words, the control circuit 132 needs to perform the process of Figure 10 for each set of positional relationships in the target space 170.

In some specific cases, the existence of the environmental object 175 can be directly ignored. For example, if the reflectivity or absorptivity of the environmental object 175 to sound is less than a specific threshold, it means that its existence in the target space 170 can be ignored. On the other hand, if the control circuit 132 determines that the volume of the environmental object 175 is less than a specific size, the existence of the environmental object 175 can also be ignored.

In a further derived embodiment, if more than one user is detected in the target space 170, the determination of the target listening point may be based on the position center point of multiple users, or may be selectively based on the position of one of the users. As for the users who are not selected as the target listening point, the host device 130 may analogize them to environmental objects and process them according to the embodiments of Figures 7 and 8.

The embodiment of FIG. 10 highlights the following advantages. The embodiment of FIG. 10 continues the processing method of FIG. 7 and FIG. 8, simplifying the complex environmental problem into multiple linear relationship problems and solving them separately. For the environmental object 175 in a specific case, the complexity of the simplified calculation can also be ignored.

FIG11 is a schematic diagram of a target space 1100 of the present invention, used to illustrate an embodiment of optimizing the sound field by object-based compensation operation.

In the target space 1100, a plurality of speakers are included, such as a first speaker 1110, a second speaker 1120, a third speaker 1130, and a fourth speaker 1140. When the sound system 100 operates based on object base compensation, the control circuit 132 logically regards the target space 1100 as a spatial coordinate system. The spatial coordinate can be a two-dimensional plane coordinate or a three-dimensional plane coordinate. For ease of explanation, FIG. 11 illustrates the two-dimensional plane coordinate including an X-axis and a Y-axis.

In the target space 1100, the user 180 is located at the origin P0. The control circuit 132 assigns the user 180 as the target listening point. As described in the embodiment of FIG. 3 , the object-based acoustic system is based on array operations of a large number of acoustic parameters. Each sound source object has metadata that describes the type, position, size (length, width, and height), divergence, etc. of the sound source object. After object-based operations, the sound represented by a sound source object will be assigned to one or more speakers and played together, with each speaker playing a portion of the sound of the sound source object. In other words, the object-based acoustic system can use multiple speakers to simulate the physical presence of a sound source object. For example, through the object-based compensation operation, the user 180 at the target listening point can hear a virtual sound source object 1105 moving from the first position P1 to the new first position P1' along the moving trajectory 1103.

The object-based compensation operation of this embodiment can optimize the channel audio output by all speakers for the target listening point. The object-based compensation operation uses the array operation module in the existing object-based acoustic system to parameterize various distance factors and sound field categories, and can perform operations similar to formulas (4) to (7). For the audio system 100, the host device 130 only needs to apply the position information of the user 180 to the object-based compensation operation to optimize the channel audio output by all speakers for the target listening point.

In one embodiment, the control circuit 132 may define the target listening point as the origin of the entire spatial coordinate system. When the user 180 moves, the entire spatial coordinate system moves with the origin. In other words, the position of the virtual sound source object 1105 relative to the origin remains unchanged. When the control circuit 132 plays the effect of the virtual sound source object 1105 through the object base compensation operation, the relative position of the virtual sound source object 1105 felt by the user 180 will not change with the movement of the user 180.

In the target space 1100 of this embodiment, there may be an environmental object 175, which will have a substantial impact on the listening effect of the user 180. The control circuit 132 can obtain the spatial configuration information of the environmental objects in the target space 1100 through the process 902 of Figure 9, and know that the environmental object 175 is located at the second position P2. When the user 180 moves, the origin of the entire spatial coordinate system changes with the user 180. Although the environmental object 175 does not move, its relative position to the origin changes. Therefore, it can be understood that in the spatial coordinate system after the movement, the coordinate value of the environmental object 175 moves in the opposite direction.

In order to offset the interference of the environmental object 175 to the user 180, the control circuit 132 of the present embodiment establishes a compensation sound source object of the object base according to the environmental object 175. The metadata of the compensation sound source object includes: the coordinate position, size, and reflectivity and absorptivity of the environmental object 175 to sound. The reflectivity and absorptivity of the environmental object 175 to sound can be obtained by the process 906 of Figure 9. The compensation sound source object will be regarded as a negative sound source object of the environmental object 175 and applied to the object base compensation operation, becoming a virtual sound source that can offset the environmental object 175.

It is understandable that the essence of the compensating sound source object is the negative sound source object corresponding to the environmental object 175, and its location overlaps with the environmental object 175, so it is not marked separately in Figure 11. In addition, the four-speaker configuration of the target space 1100 is only an example. In the actual application of the audio system 100, the number of speakers can be more, even including a stereo configuration of upper and lower speakers. This description does not limit other possible configurations.

The embodiment of FIG11 illustrates the advantages of the object-based compensation operation. The control circuit 132 converts the information of the target space 1100 into the form of a spatial coordinate system, which can simplify the complex multi-object interaction operation into an array operation of metadata. The position of the moving user 180 is set as the origin of the spatial coordinate system, so that the processing of the virtual object is completely unaffected by the movement of the user 180, thereby simplifying the operation process. This embodiment also proposes the concept of compensating the sound source object, directly applying the object-based compensation operation to offset the interference of environmental objects, eliminating the complex multi-channel interaction operation.

The following uses Figure 12 to illustrate the simplicity of object-based compensation operation and possible derivative applications.

FIG. 12 is a schematic diagram of a target space 1200 of the present invention, used to illustrate an embodiment of optimizing the sound field by object-based compensation operation.

The target space 1200 may include a plurality of speakers, such as a first speaker 1210, a second speaker 1220, a third speaker 1230, a fourth speaker 1240, a fifth speaker 1250, and a sixth speaker 1260, arranged in a long strip sound field. Each speaker corresponds to an ID. When the user 180 is located at the first position P1, the metadata of a virtual sound source object (not shown) is mapped to the IDs of the first speaker 1210 and the second speaker 1220. After the control circuit 132 performs the object base compensation operation, the first speaker 1210 and the second speaker 1220 will play the first channel output 1212 and the second channel output 1222, so that the user 180 can feel the existence of the virtual sound source object. When the user 180 moves to the second position P2 along the moving trajectory 1203, the control circuit 132 recalculates the target listening point and maps the metadata of the virtual sound source object to the fifth speaker 1250 and the sixth speaker 1260. After the control circuit 132 performs the object base compensation operation, the fifth speaker 1250 and the fifth speaker 1250 play the fifth channel output 1252 and the sixth channel output 1262, so that the user 180 feels that the virtual sound source object still exists around the user 180 and does not leave with the movement of the user 180.

This embodiment mainly illustrates the flexible application and simplicity of object base compensation operation. In many special cases, only a small amount of calculation is required to optimize the sound field. For example, if the user 180 is in a spherical sound field, the control circuit 132 only needs to perform rotation coordinate calculations to allow the user 180 to experience a consistent sound field effect when facing various directions.

The following is a summary of the basic logic of the control circuit 132 when performing object base compensation operation with FIG. 13.

FIG13 is a flowchart of the object-based compensation operation of an embodiment of the present invention, illustrating the concept of establishing a compensation sound source object.

In process 1304, the control circuit 132 establishes a corresponding compensation sound source object according to the environmental object 175. For the user 180 located at the target listening point, the existence of the environmental object 175 is a physical sound source. The environmental object 175 may reflect the sound emitted by a speaker to the target listening point. The environmental object 175 may also block or absorb part of the sound, causing the sound emitted by a speaker to the target listening point to be attenuated. The compensation sound source object is a negative sound source object established for the environmental object 175. When the host device 130 substitutes the compensation sound source object into the object base compensation operation to generate a channel audio, the presence of the environmental object 175 can be eliminated. The specific details of the object-based calculation itself can continue to use the calculation method of the existing object-based acoustic products, using the metadata of the sound source object to perform a large number of related array operations. For example, a metadata of the compensation sound source object includes: the coordinate position, size, and reflectivity and absorption rate of the environmental object 175 to the sound.

In process 1306, the control circuit 132 calculates the sound source effect of the compensation sound source object. In this embodiment, the compensation sound source object is established in correspondence with the environment object 175, wherein the metadata has the same coordinate position, size, and reflectivity and absorption rate of the sound as the environment object 175, but the sound source effect generated is the inverse gain value of the environment object 175.

The embodiment of FIG. 13 can also refer to the calculation similar to that of FIG. 7 and FIG. 8. Formula (8) can be derived into Formula (9), which calculates the gain value passively generated by the environmental object 175 according to the sound pressure value received by the environmental object 175 from the first speaker 110: _At [m][n] = R [n] * SPL _t [m] (9)

Wherein, m represents the speaker number, and n represents the sub-band number. _At [m] [n] represents the gain value generated by the n-th sub-band being affected by the m-th speaker. R [n] represents the absorption rate of the n-th sub-band. SPL _t [m] represents the sound pressure value from the m-th speaker received by the environmental object 175 at the t-th time point. The time point t may represent the time difference of the sound being transmitted from the speaker to the environmental object 175. If the time difference is greater than a non-negligible range, it indicates that there is an echo in the target space 170.

From formula (9), we can see that the calculation result corresponding to each environmental object includes a gain value array of multiple speakers and multiple sub-bands at a time point. The sound source effect of the compensation sound source object is the negative value of the gain value array. In other words, the object-based compensation operation based on formula (9) includes an array operation of interactively arranged and combined parameters of multiple dimensions. For the sake of convenience, the following is an explanation of the gain value corresponding to one speaker and one sub-band at a time point.

The embodiment of FIG. 13 is similar to the embodiments of FIG. 7 and FIG. 8 in that the present embodiment can use appropriate calculation methods to calculate the acoustic impact of the environmental object according to the spatial configuration information of the environmental object 175. For example, if the target listening point is between a speaker and the line of sight of the environmental object 175, the control circuit 132 calculates the sound source effect of the compensation sound source object according to the reflectivity of the environmental object 175. Conversely, if the environmental object 175 is between the target listening point and the line of sight of the speaker, the control circuit 132 calculates the sound source effect of the compensation sound source object according to the absorption rate of the environmental object 175.

For example, when an environmental object 175 absorbs the sound from a speaker, the volume effect received by the target listening point is reduced. At this time, the control circuit 132 creates a virtual sound source object that will produce a corresponding volume effect at the coordinate position of the environmental object 175 as compensation. Conversely, if an environmental object 175 reflects the sound of a speaker, the target listening point receives too much volume. At this time, the control circuit 132 creates a virtual sound source object with a negative gain value at the coordinate position of the environmental object 175.

It is understandable that the visible line is defined as a straight line connecting two objects in space. Since objects have a certain volume and area, the volume may be very large, and the situation where the visible line is blocked may include partial blockage and complete blockage. This embodiment can be based on formula (9), and then multiplied by different weight coefficients or added with different offset corrections depending on various scenarios.

In process 1308, the control circuit 132 mixes the sound source effect of the compensation sound source object into the channel audio so that the corresponding speaker plays. When the control circuit 132 performs the object base compensation operation, it can process complex object corresponding array operations to mix the multiple sound source signals assigned to each speaker into the corresponding one-channel audio. After applying the object base compensation operation, the volume effect received at the target listening point will include the sound source effect generated by the compensation sound source object. In this way, the interference caused by the environmental object 175 can be effectively offset by the compensation sound source object.

In process 1310, the control circuit 132 determines whether the target listening point has moved to a new position. As described in process 208, the audio system 100 can continuously track the movement of the user 180 and update the target listening point. If the target listening point has moved, process 1312 is performed. Otherwise, the playback operation of process 1308 is continued.

In process 1312, the control circuit 132 updates the metadata of the compensation sound source object. In the present embodiment, the control circuit 132 establishes the object base space with the target listening point as a coordinate origin. If the target listening point moves to a new position, the control circuit 132 assigns the new position as the new coordinate origin of the object base space. The position difference between the new coordinate origin and the original coordinate origin can be represented as a movement vector. The spatial coordinate value of the environmental object 175 relative to the target listening point will also change inversely with the movement vector. The control circuit 132 then updates the metadata of the compensation sound source object corresponding to the environmental object 175 according to the movement vector. In a further embodiment, all speakers in the object base space can also be regarded as a kind of object, with corresponding ID, metadata and coordinate values.

In another embodiment, the sound system 100 is not limited to using the target listening point as the coordinate origin. The sound system 100 can also use a fixed reference point as the origin of the object base space. When the relative position of the sound source object in the object base space changes, the control circuit 132 updates the coordinate value in the metadata of the sound source object accordingly.

When process 1312 is completed, control circuit 132 repeats process 1308.

The embodiment of FIG. 13 illustrates the advantages of the object-based compensation operation. The control circuit 132 converts the information of the target space 1100 into the form of a spatial coordinate system, which can simplify the complex multi-object interaction operation into an array operation of metadata. The position of the moving user 180 is set as the origin of the spatial coordinate system, so that the processing of the virtual object is completely unaffected by the movement of the user 180, thereby simplifying the operation process. This embodiment also proposes the concept of compensating the sound source object, directly applying the object-based compensation operation to offset the interference of environmental objects, eliminating the complex multi-channel interaction operation.

In a further derived embodiment, if the host device 130 itself does not have the object-based mixing computing capability, the control circuit 132 can provide a channel mapping function by executing software so that the object-based computing result can be correctly mapped to each speaker.

In summary, the present application proposes an audio system 100 that can dynamically track the user's position to optimize the sound field, and can also intelligently eliminate interference caused by environmental objects. The means of tracking the user's position can be the individual use or combined use of a variety of methods such as cameras, infrared, or wireless detectors. The spatial configuration information of the environmental objects 175 in the target space 170 can be obtained by identifying the images captured by the camera, or can be manually input by the user. The way to optimize the sound field can be a channel-based compensation operation or an object-based compensation operation. When calculating the impact of the environmental object 175 on the target listening point, the relative position relationship between the environmental object 175 and the speaker and the target listening point can also be considered, and different calculation methods can be adopted. When using the object-based compensation operation, the control circuit 132 establishes a corresponding compensation sound source object for each environmental object 175, so that the channel audio generated by the final mix eliminates the interference caused by the environmental object 175 on the target listening point.

Certain terms are used in the specification and patent application to refer to specific components, and technicians in the field may use different terms to refer to the same components. This specification and patent application do not use differences in names as a way to distinguish components, but use differences in components' functions as the basis for distinction. The term "including" mentioned in the specification and patent application is an open term and should be interpreted as "including but not limited to". In addition, the term "coupled" includes any direct and indirect connection means. Therefore, if the text describes that a first component is coupled to a second component, it means that the first component can be directly connected to the second component through electrical connection or wireless transmission, optical transmission and other signal connection methods, or indirectly connected to the second component electrically or by signal through other components or connection means.

The description method of "and/or" used in the manual includes any combination of one or more of the listed items. In addition, unless otherwise specified in the manual, any singular term also includes the plural meaning.

The above are only the preferred embodiments of the present invention. All equivalent changes and modifications made according to the claims of the present invention shall fall within the scope of the present invention.

100: Audio system

110: First Speaker

112: First channel audio

120: Second speaker

122: Second channel audio

130: Host device

131: Storage circuit

132: Control circuit

133: Human-machine interface circuit

134: Identify circuits

135: Audio transmission circuit

136: Communication circuit

140:Sensor circuit

150: User equipment

160: Remote database

170: Target space

171: First position

172: Second position

173:Moving track

175: Environmental objects

180:User

Claims (8)

A sound system (100) is capable of dynamically optimizing a playback effect according to a user's position, comprising: a sensor circuit (140) configured to dynamically sense a target space (170) to generate sound field environment information, wherein the sound field environment information includes a user's position in the target space (170); a first speaker (110) and a second speaker (120) configured to play audio; A host device (130) coupled to the sensor circuit (140), the first speaker (110) and the second speaker (120), comprising: an identification circuit (134) configured to identify the user's position in the target space from the sound field environment information; and a control circuit (132) coupled to the identification circuit (134) configured to dynamically assign the user's position as a target listening point. an audio transmission circuit (135), coupled to the control circuit (132), the first speaker (110) and the second speaker (120), configured to transmit audio; and a human-machine interface circuit (133), coupled to the control circuit (132), configured to be controlled by the control circuit (132) to run a configuration program to obtain spatial configuration information and acoustic properties of an environmental object in the target space (170). The control circuit (132) is configured to generate a first channel audio signal (112) and a second channel audio signal (122) optimized for the target listening point based on the target listening point, the spatial configuration information and the acoustic property information of the environmental object. The control circuit (132) outputs the first channel audio (112) and the second channel audio (122) to the corresponding first speaker (110) and the second speaker (120) respectively through the audio transmission circuit (135); wherein the channel base compensation operation includes: splitting the first channel audio (112) emitted by the first speaker (110) into a plurality of sub-band signals; judging whether a sound field type generated by the sub-band signal at the target listening point belongs to a near sound field or a far sound field according to the wavelength of a sub-band signal among the plurality of sub-band signals and the distance between the target listening point and the first speaker (110); and judging whether a sound field type generated by the sub-band signal at the target listening point belongs to a near sound field or a far sound field when the distance between the target listening point and the first speaker (110) is greater than the wavelength of a sub-band signal among the plurality of sub-band signals and the distance between the target listening point and the first speaker (110). When the distance between the target listening point and the first speaker (110) is greater than a specific ratio of the wavelength of the sub-band signal or the size of the first speaker (110), the sound field type is determined to be a far sound field; when the distance between the target listening point and the first speaker (110) is less than the specific ratio of the wavelength of the sub-band signal or the size of the first speaker (110), the sound field type is determined to be a near sound field; when the control circuit (132) determines that the distance between the position of the target listening point and the first speaker (110) changes from a first distance (R1) to a second distance (R2), and the sound field type belongs to the far sound field, a far sound field formula is used to calculate the sound pressure value of the sub-band signal; wherein the far sound field formula includes: SPL'=SPL+20 log ₁₀ (R2/R1) wherein SPL is the sound pressure value of the sub-band signal before adjustment, SPL' is the sound pressure value of the sub-band signal after adjustment, R1 is the first distance, and R2 is the second distance; and wherein, if the sound pressure value of the sub-band signal after adjustment is less than zero, the control circuit (132) makes the sound pressure value of the sub-band signal after adjustment zero. The audio system (100) as claimed in claim 1, wherein the channel floor compensation operation further comprises, when the control circuit (132) determines that the distance between the position of the target listening point and the first speaker (110) changes from the first distance (R1) to the second distance (R2), and the sound field type belongs to the near sound field, using a near sound field formula to calculate the sound pressure value of the sub-band signal; wherein the near sound field formula comprises: SPL'=SPL+10 log ₁₀ (R2/R1) wherein SPL is the sound pressure value of the sub-band signal before adjustment, SPL' is the sound pressure value of the sub-band signal after adjustment, R1 is the first distance, and R2 is the second distance; and wherein, if the sound pressure value of the sub-band signal after adjustment is less than zero, the control circuit (132) makes the sound pressure value of the sub-band signal after adjustment zero. The audio system (100) as described in claim 1, wherein the spatial configuration information of the environmental object includes the position, size and appearance characteristics of the environmental object, and the acoustic property information of the environmental object includes the reflectivity and absorptivity of the sound; wherein when the control circuit (132) generates the first channel audio, if the target listening point is located between the first speaker (110) and the line of sight of the environmental object, the control circuit (132) calculates the first speaker (110) at the target listening point according to the reflectivity of the environmental object. The control circuit (132) is used to calculate the degree to which the playing effect at the target listening point is affected by the environmental object, so as to determine the sound pressure value of the first channel audio signal; and wherein, when the control circuit (132) generates the first channel audio signal, if the environmental object is located between the target listening point and the visual line of the first speaker (110), the control circuit (132) calculates the degree to which the playing effect of the first speaker (110) at the target listening point is affected by the environmental object according to the absorption rate of the environmental object, so as to determine the sound pressure value of the first channel audio signal. The audio system (100) as described in claim 1, wherein the sensor circuit (140) includes a camera (610) configured to capture a sound field environment image of the target space (170); wherein the recognition circuit (134) dynamically recognizes the head position, face direction, or ear position of the user based on the sound field environment image captured by the camera (610) to determine the user's position. The audio system (100) as described in claim 1, wherein the sensor circuit (140) further comprises an infrared sensor (620) configured to capture thermal imaging data in the target space; wherein the recognition circuit (134) analyzes the movement trajectory of the thermal imaging data to dynamically determine the user's position. The audio system (100) as described in claim 1, wherein the sensor circuit (140) further comprises a wireless detector (630) disposed in the target space and detecting a wireless signal of an electronic device; wherein the identification circuit (134) dynamically locates the position of the electronic device according to the characteristics of the wireless signal detected by the wireless detector (630); and wherein the identification circuit (134) dynamically determines the user's position according to the position of the electronic device. The audio system (100) as claimed in claim 1, wherein the host device (130) further comprises a storage circuit (131) coupled to the control circuit (132) and configured to store one or more object databases, wherein each object database corresponds to an application scene category and comprises appearance feature information and acoustic property information of a plurality of environmental objects; wherein the human-machine interface circuit (133) When the configuration program is run under the control of the control circuit (132), an application scene category of the target space (170) is also obtained; and wherein the control circuit (132) preferentially selects and uses an object database related to the application scene category from the storage circuit (131) according to the application scene category to identify the environmental object and search for the acoustic property information of the environmental object. The audio system (100) as claimed in claim 1, wherein the host device (130) further comprises a communication circuit (136) coupled to the control circuit (132) and configured to be controlled by the control circuit (132) to connect to a remote database (160) corresponding to the application scenario category; the remote database (160) is configured to store one or more object databases, wherein each object database corresponds to an application scenario category and includes external information of multiple environmental objects. The human-machine interface circuit (133) is controlled by the control circuit (132) to run the configuration program, and also obtains an application scene category of the target space (170); and the control circuit (132) preferentially selects an object database related to the application scene category from the remote database (160) according to the application scene category to identify the environmental object and search for the acoustic property information of the environmental object.