CN106686520A - Multi-channel audio system capable of tracking user and equipment with multi-channel audio system - Google Patents

Multi-channel audio system capable of tracking user and equipment with multi-channel audio system Download PDF

Info

Publication number
CN106686520A
CN106686520A CN201710001369.8A CN201710001369A CN106686520A CN 106686520 A CN106686520 A CN 106686520A CN 201710001369 A CN201710001369 A CN 201710001369A CN 106686520 A CN106686520 A CN 106686520A
Authority
CN
China
Prior art keywords
speaker
relative
user
sound
relative position
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN201710001369.8A
Other languages
Chinese (zh)
Other versions
CN106686520B (en
Inventor
张芊
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nanjing Horizon Robotics Technology Co Ltd
Original Assignee
Nanjing Horizon Robotics Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nanjing Horizon Robotics Technology Co Ltd filed Critical Nanjing Horizon Robotics Technology Co Ltd
Priority to CN201710001369.8A priority Critical patent/CN106686520B/en
Publication of CN106686520A publication Critical patent/CN106686520A/en
Application granted granted Critical
Publication of CN106686520B publication Critical patent/CN106686520B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S3/00Systems employing more than two channels, e.g. quadraphonic

Abstract

The invention discloses a multi-channel audio system capable of tracking a user and equipment with the multi-channel audio system. The multi-channel audio system comprises multiple loudspeakers, an image sensor, a sound sensor and a controller; a predetermined first relative position is located between the image sensor and the sound sensor, and the controller comprises a loudspeaker position detection unit, a user position detection unit, a position fusion unit, a delay compensation calculating unit and a delay adjusting unit, wherein the loudspeaker position detection unit is used for determining second relative positions of the loudspeakers on the basis of predetermined audio signals which are detected by the sound sensor and come from the loudspeakers, the user position detection unit is used for determining a third relative position of the user on the basis of a user image detected by the image sensor, the position fusion unit is used for fusing the first relative position, the second relative positions and the third relative position to determine a fourth relative position of the user to the loudspeakers, the delay compensation calculating unit is used for calculating delay compensation values of the loudspeakers on the basis of the fourth relative position, and the delay adjusting unit is used for adjusting the phase positions of audio signals of the loudspeakers according to the delay compensation values.

Description

The multi-channel audio system and the equipment including it of user can be tracked
Technical field
Present invention relates in general to sound reproduction, more specifically it relates to a kind of multi-channel audio system, it can track use Family, so as to persistently provide optimal hearing effect for user.
Background technology
Typical multi-channel audio system, including 2.0 systems, 2.1 systems, 5.1 systems, 6.1 systems, 7.1 systems etc., all Stereophonic effect can be provided the user using its multiple sound channel.However, in order to realize optimal hearing effect, for sound equipment System is placed with certain requirement.
Figure 1A illustrates putting for 2.0 or 2.1 sound systems, and Figure 1B illustrates putting for 5.1 sound systems.For 2.0 or 2.1 System, as shown in Figure 1A, left channel loudspeaker (or audio amplifier) 101, right channel loudspeaker 102 and the three of user 110 form equilateral Triangle, and left channel loudspeaker 101 and the user oriented 110 of right channel loudspeaker 102.Because the sensing of subwoofer Property it is not strong, so it is put can not show in figure relatively flexibly.For 5.1 systems, as shown in Figure 1B, except front left channel Speaker 101, right front channels speaker 102 and the three of user 110 are formed outside equilateral triangle, and center loudspeaker 103 is placed in The dead ahead of user 110, and center loudspeaker 103 is equal to the length of side of aforementioned equilateral triangle with a distance from user 110.Additionally, Left subsequent channel speaker 104 and rear right channel speaker 105 are respectively placed in the left back and right back of user 110.Equally, Figure 1B In eliminate subwoofer.In addition to 2.0 systems, 2.1 systems and 5.1 systems, for 6.1 systems, 7.1 systems etc. Having put accordingly requirement.When sound system is put according to such requirement, the position that user 110 is located is commonly referred to as " emperor Throne ", i.e., most preferably listen phoneme, and in the position optimal audition experience can be obtained.
It can be seen that, in order to realize optimal hearing effect, need the position for performing complexity to sound system to put.Even if additionally, As requested putting speaker, emperor position region be also it is limited, when user not in emperor position region when, or when user because When moving and leaving emperor position, then audio quality still can be caused because of the interference between sound to be greatly lowered.
Application for a patent for invention CN201510112555.X proposes a kind of audio reproduction technology synthesized based on sound field ripple, its According to the frequency and wavelength of audio frequency every audio signal all the way is carried out to divide coding and uniform treatment, and closed Jing after diffraction process Into output, to avoid the interference between different speakers, so as to realize the improvement of the audio quality outside emperor position.Although the party Case can to a certain extent by weaken interfere to improve emperor position region beyond hearing effect, but its still suffer from it is some Problem.First, the frequency dividing of audio signal is encoded, diffraction and synthesis are processed etc. inevitably affects fidelity, is caused Audio is deteriorated.In addition, although the program can to a certain extent improve the audio of the exterior domain of emperor position, but it still can not Realize the hearing effect suitable with emperor position.
Accordingly, it is desired to provide a kind of sound effect system, it can neatly be put, even and if user in diverse location, Such as position outside emperor position, it can also provide the user optimal hearing effect.
The content of the invention
One aspect of the present invention is to provide a kind of multi-channel audio system, and it can track user, from regardless of whether with Whether family can provide the user optimal audition impression in emperor position.
Another aspect of the present invention also resides in a kind of multi-channel audio system of offer, and it can adjust prolonging for each speaker Slow and gain, such that it is able to flexibly be put to each speaker.
According to an exemplary embodiment, a kind of multi-channel audio system may include multiple speakers, imageing sensor, sound Sensor and controller, have the first predetermined relative position, institute between described image sensor and the sound transducer Stating controller includes:Loudspeaker position detector unit, be configured to that the sound transducer detects from the plurality of The predetermined audio signal of speaker is determining second relative position of the plurality of speaker relative to the sound transducer;With Family position detection unit, is configured to user images that described image sensor detects to determine user relative to the figure As the third phase of sensor is to position;Co-factor propagation unit, is configured to fusion first relative position, the described second relative position Put and the third phase is to position, so that it is determined that fourth relative position of the user relative to each speaker;Delay compensation meter Unit is calculated, is configured to the 4th relative position to calculate the delay compensation value of the audio signal for each speaker; And postpone adjustment unit, the phase place of the audio signal for being configured to adjust each speaker according to the delay compensation value.
In one example, described image sensor and the sound transducer are arranged in the plurality of speaker On, so as to have the second predetermined relative position between the sound transducer and one speaker.
In one example, the sound transducer includes microphone array, and the loudspeaker position detector unit is configured to Direction of each speaker relative to the microphone array is determined based on sound localization, and is sent based on each speaker Predetermined audio signal and the microphone array receive the time difference between the audio signal to determine each speaker phase For the distance of the microphone array.
In one example, the controller also includes:Gain compensation computing unit, is configured to the described 4th relative position Put to calculate the gain compensation value of the audio signal for each speaker;And gain adjustment unit, it is configured to according to described Gain compensation value is adjusting the gain of the audio signal of each speaker.
The another exemplary embodiment of the present invention provides a kind of equipment including above-mentioned multi-channel audio system.The equipment Can be television set or automobile etc..
The another exemplary embodiment of the present invention provides a kind of method that sound is reappeared using multi-channel audio system, institute Multi-channel audio system is stated including multiple speakers, methods described includes:Receive what each speaker sent from sound transducer Predetermined audio signal, and the second relative position of each speaker is determined based on received predetermined audio signal;From figure As sensor receive user image, and determine the third phase of user to position based on received user images;To described Second relative position, the third phase are to the first relative position predetermined between position and sound transducer and imageing sensor Put and merged, to determine fourth relative position of the user relative to each sensor;And based on the 4th relative position It is determined that the delay compensation value of the audio signal for each speaker, and raised one's voice adjusting each according to the delay compensation value The phase place of the audio signal of device.
In one example, described image sensor and the sound transducer are arranged in the plurality of speaker On, so as to have the second predetermined relative position between the sound transducer and one speaker.
In one example, determining the second relative position of each speaker includes:Determine each based on sound localization to raise Sound device and sends predetermined audio signal and the sound and passes relative to the direction of the sound transducer based on each speaker Sensor receives the time difference between the audio signal to determine distance of each speaker relative to the sound transducer.
In one example, methods described also includes:The sound for each speaker is determined based on the 4th relative position The gain compensation value of frequency signal, and according to the gain compensation value adjusting the gain of the audio signal of each speaker.
The another exemplary embodiment of the present invention provides a kind of computer program, including computer program instructions, institute Stating computer program instructions makes the computing device said method when being run by processor.
Description of the drawings
Figure 1A illustrates the schematic putting position figure of 2.0 or 2.1 traditional sound systems.
Figure 1B illustrates the schematic putting position figure of 5.1 traditional sound systems.
Fig. 2 illustrates the schematic diagram of the multi-channel audio system according to an exemplary embodiment of the invention.
Fig. 3 illustrates the functional block diagram of the controller of the Fig. 2 according to an exemplary embodiment of the invention.
Fig. 4 illustrates the flow chart of sound reproduction process according to an embodiment of the invention.
Fig. 5 illustrates the example of the equipment with sound system according to an embodiment of the invention.
Fig. 6 illustrates the example of another equipment with sound system according to an embodiment of the invention.
Specific embodiment
The exemplary embodiment of the present invention is described below with reference to accompanying drawings.
Fig. 2 illustrates the schematic diagram of the multi-channel audio system 200 according to an exemplary embodiment of the invention.Although figure 2 illustrate only two speakers, but should be understood that the principle of the present invention also may be used on having more by explained below The multi-channel audio system of multi-loudspeaker, such as but not limited to 2.1 systems, 5.1 systems, 6.1 systems, 7.1 systems etc..
As shown in Fig. 2 multi-channel audio system 200 includes the first speaker 210 and the second speaker 220.Typically, example Such as in 2.0 systems, in the first speaker 210 and the second speaker 220 can be left channel loudspeaker, and another can Think right channel loudspeaker.Two speakers 210 and 220 may be coupled to controller 230 to be controlled by.Although Fig. 2 is illustrated Controller 230 is embodied as single unit, it will be understood that, controller 230 can also be implemented in multi-channel audio system 200 In any one speaker, including subwoofer (if any), or can also be arranged on what is be connected with speaker In equipment, power amplifier, computer etc..
With continued reference to Fig. 2, multi-channel audio system 200 also includes sound transducer 201 and imageing sensor 202.Sound Sensor 201 is preferably microphone array, and it includes the multiple mikes arranged by predetermined pattern.Imageing sensor 202 can be with It is that the shooting of monocular cam, binocular camera or more mesh is first-class.Preferably, imageing sensor 202 can be binocular camera shooting Head.
In the embodiment shown in Figure 2, sound transducer 201 and imageing sensor 202 have been all disposed within the first speaker On 210, therefore for convenience, the first speaker 210 can be referred to as main loudspeaker, by other speakers, such as second Speaker 220, referred to as secondary speaker.However, it is understood that main loudspeaker can be any one in multi-channel audio system 200 Speaker, including but not limited to subwoofer.Preferably, main loudspeaker can be able to easily see raising one's voice for user Device.It should also be understood that sound transducer 201 and imageing sensor 202 also may be provided in other equipment, or for example it is embodied as list Only module.Sound transducer 201 and imageing sensor 202 are preferably provided in same equipment or module so that therebetween With known predetermined phase position.As being appreciated that by explained below, from being easy to for raising level of integrated system For the angle for using, it is preferred that sound transducer 201 and imageing sensor 202 are arranged on some speaker.
Fig. 3 illustrates the functional block diagram of the controller 230 of the Fig. 2 according to an exemplary embodiment of the invention.As shown in figure 3, Controller 230 may include loudspeaker position detector unit 231, customer location detector unit 232, Co-factor propagation unit 233, delay Compensation calculation unit 234, gain compensation computing unit 235, delay adjustment unit 236 and gain adjustment unit 237.Below will ginseng The operation of all parts in controller 230 is described according to Fig. 2 and 3.
Loudspeaker position detector unit 231 can be used for the position of each speaker for determining sound system 200, it is concrete and Speech, speaker relative to sound transducer microphone array 201 in other words position, including distance and direction.Specifically, such as Shown in Fig. 2, each speaker 210,220 of sound system 200 can be controlled to send predetermined audio signal, for example, make a reservation for The audio signal of frequency and waveform, and sound transducer 201, such as microphone array, can receive each speaker and send The audio signal.Loudspeaker position detector unit 231 can be processed the audio signal that microphone array is received, base The direction of each speaker relative to microphone array of sound source is determined as in sound localization technology.Based on microphone array Sound localization method include steerable beam formed technology, High-Resolution Spectral Estimation technology, the skill based on sound wave reaching time-difference Art etc., is used equally in the present invention determine direction of each speaker relative to microphone array.During being reached based on sound wave Between difference technology as a example by, its principle be determine from sound source sound wave reach a pair of mikes time difference, it corresponds to one Hyperboloid, using multipair mike multiple hyperboloids are can determine that, their common factor is exactly the position that sound source is located at.The present invention People has found, especially for far field system, the error of the sound source distance determined based on microphone array and sound source location technology compared with Greatly, therefore in an embodiment of the present invention, merely with sound localization technology determining each speaker relative to microphone array Direction.
Additionally, loudspeaker position detector unit 231 is also connect by determining that speaker sends audio signal with microphone array Receive the time difference of audio signal to determine distance of the speaker away from microphone array.The electricity in speaker and microphone array The speed of signal can consider much larger than the velocity of sound, therefore, it is possible to be readily determined time and Mike that speaker sends audio signal To the time of audio signal, the difference of the two is multiplied by SVEL to wind array received, you can obtain speaker away from microphone array it Between distance.So as to loudspeaker position detector unit 231 can determine position of each speaker relative to microphone array 201 Put, including direction and distance.
It should be noted that when as shown in Fig. 2 microphone array 201 is arranged on one of speaker such as main loudspeaker 210, Main loudspeaker 210 can be fixed and known relative to the position of microphone array, it will therefore be appreciated that raising in above-mentioned utilization When sound device position detection unit 231 determines the position of each speaker, it is convenient to omit the step related to main loudspeaker 210.
Customer location detector unit 232 can be used to detect the position of user that specifically, user is relative to imageing sensor 202 position, including direction and distance.As shown in Fig. 2 imageing sensor 202, preferably binocular camera, can catch use The image at family 110.Customer location detector unit 232 can receive the view data from imageing sensor 202, and utilize image Technology of identification, such as recognition of face, head-shoulder contour identification, human body contour outline identification etc. are identifying user, and further basis Image information determines the direction at user place and distance.At some in particular cases, for example when multiple users are identified, user The centre position of such as multiple users can be defined as customer location by position detection unit 232, or only be recognized according to face The position of register user;When any user is not identified, customer location detector unit 232 can using default location as with Family position, etc..These strategies can determine according to actually used scene.
As described above, when loudspeaker position detector unit 231 determines each speaker relative to sound transducer 201 Position (hereinafter referred to as the second relative position), and customer location detector unit 232 determines user relative to imageing sensor After 202 position (hereinafter referred to as third phase is to position), Co-factor propagation unit 233 can melt to these position datas Close.Relative position specifically, known to Co-factor propagation unit 233 between sound transducer 201 and imageing sensor 202 (under Text is referred to as the first relative position), such as sound transducer 201 and imageing sensor 202 can be embodied in same module or It is implemented on same speaker, therefore first relative position is known and fixed.Therefore, based on sound transducer 201 Second relative position of the first relative position, each speaker and imageing sensor 202 between relative to sound transducer 201 Put and user relative to imageing sensor 202 third phase to position, Co-factor propagation unit 233 can determine that user is relative In the relative position (hereinafter referred to as the 4th relative position) of each speaker, such as the user 110 shown in Fig. 2 raises relative to first Sound device 210 apart from R1 and direction a and user 110 relative to the second speaker 220 apart from R2 and direction b.Additionally, position Integrated unit 233 is put it may also be determined that the relative distance and direction between each speaker, such as the first speaker 210 and second The distance between speaker 220 R3 and direction c.In a further exemplary embodiment, Co-factor propagation unit 233 can also only determine User relative to each speaker distance, i.e. the 4th relative position only include user relative to each speaker distance.
After the 4th relative position is determined, you can by delay compensation computing unit 234 and gain compensation computing unit 235 delay compensation values and gain compensation value for calculating the audio signal for each speaker respectively.As shown in Fig. 2 user 110 The first speaker of distance 210 apart from R1 likely differ from the second speaker of user distance 220 apart from R2, cause two to raise one's voice The sound wave that device is simultaneously emitted by arrives at user in the different time, and arrives at the sound wave of user and have different due to distance difference Strength retrogression, the audition for causing user experiences variation.In an embodiment of the present invention, made by delay compensation and gain compensation The sound wave for obtaining different speakers reaches user and with identical gain simultaneously.
In one example, delay compensation computing unit 234 can obtain distance of each speaker relative to user, it is determined that Ultimate range therein, then adjust each speaker postpone so that the sound wave of all speakers arrive at time of user with Ultimate range speaker is identical.For example, the example for Fig. 2, the distance users 110 of the second speaker 220 apart from R2 more than the The distance users 110 of one speaker 210 apart from R1, then delay compensation computing unit 234 can be calculated according to equation 1 below For the delay compensation value of the first speaker 210.
Δ=ROUND (Fs (R2-R1)/v) formula 1
Wherein, v represents the velocity of sound, and Fs represents sample frequency, and ROUND is the function of the integer for being rounded up to nearest.From public affairs Formula 1 is as can be seen that Δ represents that the sound wave of the second speaker 220 arrives at the time of user 110 and supports than the sound wave of the first speaker 210 Up to the sampling number of the time delay of user 110.
Then, postponing adjustment unit 236 can adjust the sound of each speaker according to the delay compensation value Δ for being calculated The phase place of frequency signal.For example, the example for Fig. 2, then by the phase delay delta of the audio signal of the first speaker 210 so that It arrives at user simultaneously with the audio signal of the second speaker 210.For the second speaker 220, then because it has the latest The sound wave arrival time, then without adjustment.
Additionally, gain compensation computing unit 235 can obtain distance of each speaker relative to user, determine therein Minimum range, then adjusts the gain of each speaker so that the intensity when sound wave of all speakers arrives at user each other one Intensity when causing, and user is arrived at corresponding to the sound wave of the speaker of minimum range.For example, the example for Fig. 2, first raises The distance users 110 of sound device 210 apart from R1 less than the second speaker 220 distance users 110 apart from R2, then gain compensation is calculated Unit 235 can be calculated for the gain compensation value of the second speaker 220 according to equation 2 below.
G=ea(R2-R1)Formula 2
Wherein, a is the aerial attenuation quotient of sound wave.Then, gain adjustment unit 236 can be according to the increasing for being calculated Beneficial offset g is adjusting the gain of the audio signal of each speaker.For example, the example for Fig. 2, then by the second speaker 220 gain is increased to g times of former yield value.For the first speaker 210 with beeline, then because calculated Gain compensation value g is equal to one, so its gain is constant.
As shown in figure 3, audio signal L and audio signal R for the second speaker 220 for the first speaker 210 After through postponing adjustment and Gain tuning, it is sent to corresponding speaker and is converted into acoustic signals.Although giving above The specific example for postponing adjustment and Gain tuning is gone out, however, it is understood that those skilled in the art can also adopt other modes To be adjusted, as long as so that the sound wave that each speaker sends finally arrives at user and arrives at use with the roughly the same time Having the change of corresponding intensity, all these adjustment modes during family all should fall within the scope of the invention.
In the embodiment shown in Fig. 2 and Fig. 3, the audio signal sent by using each speaker is raised recognizing each The relative position of sound device, and the position of user is determined by image recognition.This is because, the inventors discovered that, although can be with The position of each speaker and user is easily determined by image recognition, but in most application scenarios, very Hardly possible captures the position of each speaker and user by a photographic head, and is arranged on diverse location, tool when adopting When there are two photographic head in the different visuals field, the complexity of system, such as complexity of the aspect such as installation, wiring can be greatly increased. Therefore, the present inventor determines customer location merely with image recognition, and the position of each speaker is determined using voice signal Put, then the position of each speaker and user is carried out by the known relative position between imageing sensor and sound transducer Fusion such that it is able to easily solve foregoing problems so that the solution of the present invention can easily be employed.
Additionally, the present invention adjusts the delay and gain of each speaker by tracking the position of user, no matter therefore with At what position, or even when user is in mobile, the sound system can track the position of user at family, and for user Optimal audition experience is provided.Additionally, the solution of the present invention to the audio signal of each speaker due to being adjusted, institute With insensitive to the position of speaker, each speaker is neatly put such that it is able to be adapted to space, reduce or avoid Because space limits to the infringement of audition experience.
Fig. 4 illustrates the flow chart of sound reproduction process 300 according to an embodiment of the invention.Below in conjunction with Fig. 1 to Fig. 4 To illustrate sound reproduction process 300.
First, in step S301, each speaker can be controlled to send predetermined audio signal.In an embodiment In, for example, each speaker can successively send identical or different frequency, the audio signal of waveform.Due to electrical signal form Speed of the audio signal in the circuit of sound system can consider much larger than the velocity of sound, therefore in this step can also be easily Determine that each speaker sends the time of the audio signal (in other words, the audio signal of electric form being converted to into acoustic signals). Meanwhile, microphone array 201 can receive or detect the predetermined audio signal that each speaker sends.
In step S302, the loudspeaker position detector unit 231 of controller 230 can be detected using microphone array 201 To the audio signal that sends of each speaker determining the position of each speaker.For example, it is fixed using previously described sound source Position technology, it may be determined that direction of each speaker relative to microphone array;Sound and microphone array are sent using speaker Row detect time difference between sound, it may be determined that each speaker relative to microphone array distance, it is possible thereby to really Determine position of each speaker relative to microphone array, i.e., previously described second relative position.
Additionally, in step S303, it is possible to use imageing sensor 202, such as binocular camera are catching environment picture number According to.In step s 304, the view data that imageing sensor 202 is captured can be by the customer location detector unit of controller 230 232 are processed, such as by image recognition technology come the user in identifying user capture images;And if recognizing user, then Position of the user relative to imageing sensor 202 is further determined that, i.e., previously described third phase is to position.When being not detected by During any user, in one example, customer location detector unit 232 can not export third phase to position data so that control Device 230 does not carry out any regulation to each speaker, or in another example, customer location detector unit 232 can be exported The third phase of acquiescence is to position data so that controller 230 is adjusted using the default data to each speaker.Work as inspection When measuring multiple users, in one example, customer location detector unit 232 can detect the position of each user, then export The data of the center of such as these positions, to position, or can be exported in that middle user's as third phase Position data.
It should be understood that because fixation is typically compared in the position of speaker, and user may move, so detection is raised one's voice The step of device position S301, S302 execution frequency can be less than detection customer location the step of S303, S304 execution frequency. For example, the relative position of a speaker can be detected only when sound system 200 starts, and with predetermined frequency in system operation Rate repeatedly detects the relative position of user.Additionally, when the relative position of speaker is not detected, sound transducer 201 can be with It is turned off to save power consumption.
Next, in step S305, Co-factor propagation unit 233 can be with execution position mixing operation.Specifically, position Integrated unit 233 can receive the second relative position and third phase that determine in abovementioned steps to position, and using known to it Sound transducer 201 and imageing sensor 202 between relative position (i.e. the first relative position) come determine user relative to The relative position of each speaker, it is referred to as the 4th relative position.In one embodiment, Co-factor propagation unit 233 is at least true Determine distance of the user 110 relative to each speaker.
Based on the 4th relative position, i.e. user relative to each speaker distance, you can postponed and gain-adjusted. For example, in step S306, the delay compensation value of each speaker is calculated, and each speaker is calculated in step S307 Gain compensation value, is then carried out accordingly according to the offset for being calculated in step S308 to the audio signal of each speaker Adjust.These steps are not repeated herein description above in relation to having been described in the description of Fig. 3.
It should be understood that raising adjusting each in some embodiments, it is also possible to be based only upon user relative to the position of speaker The delay of the audio signal of sound device;In some embodiments, it is also possible to as described above adjustment postpones and both gains.
Fig. 5 illustrates the example of the equipment with sound system according to an embodiment of the invention.Equipment 400 can be car , such as domestic sedan car 400, it has sound system 410.As illustrated, sound system 410 can include multiple speakers 401st, 402,403,404,405 and 406, wherein speaker 401 and 402 can be mounted in the preposition high pitch loudspeaker in front, Speaker 403 and 404 can be mounted in middle part and put woofer, and speaker 405 and 406 can be mounted in rear portion Rearmounted high pitch loudspeaker.It should be understood that the quantity of the speaker of sound system 410, arrangement etc. are not limited to example shown in the drawings.
Sound system 410 may also include for example many mesh of sound transducer such as microphone array 407 and imageing sensor and take the photograph As 408 and controller 409, they and previously described sound transducer 201, imageing sensor 202 and controller 230 It is similar, description is not repeated herein.
In the vehicle 400 of Fig. 5, although each speaker 401 to 406 is installed in fixed position, but due to car Interior space is irregular, and the quantity of general vehicle-mounted loudspeaker is more, it is thus possible to is difficult to obtain each by measuring and raises one's voice The accurate location of device.In one embodiment, it is possible to use sound transducer 407, method as described previously determines like that each Speaker because the position is fixed, therefore can be surveyed only relative to the position of sound transducer 407 after speaker is installed Amount once, is then stored in measurement result in for example vehicle-mounted memorizer (not shown).
Imageing sensor 408 is used to determine the position of user as described previously.In one embodiment, when except driving When also there is other users (user i.e. on passenger-seat) outside the person of sailing (user i.e. on driver's seat), only will can drive Member is identified as user, when there is no driver, then the passenger of other positions can be identified as into user, then controller 409 Carry out appropriate speaker for the user for being recognized to adjust.
In further embodiments, except with photographic head come identifying user in addition to, it is also possible to receiving user's input specify Position.For example, user can specify certain position of in-car to be used as customer location by liquid crystal display (not shown), then control Device 409 can carry out appropriate regulation according to the position specified to each speaker.
By the sound system using the present invention, when user is on different seats, or have because of conditioned seat position When mobile, can obtain optimal audition and enjoy.
Fig. 6 illustrates the example of another equipment 500 with sound system according to an embodiment of the invention.Equipment 500 can To be such as television set, it has left channel loudspeaker 501, right channel loudspeaker 502, sound transducer 503, imageing sensor 504 and controller 505, they can be similar with element 210,220,201,202 and 230 shown in Fig. 2 respectively, here No longer repeat one by one.
From unlike previously described embodiment, in television set 500, the position of two speakers 501,502 is fixed And it is easier measurement or determines in design, and sound transducer 503 and imageing sensor 504 are also all disposed within electricity Depending on the housing of machine, therefore in certain embodiments, it is convenient to omit it is previously described determine each speaker position the step of, But only then previously described delay and gain are carried out to each speaker come the position of identifying user by imageing sensor Adjust.
In further embodiments, television set 500 may be connected to external loudspeaker (not shown) by audio interface. Now, then position of the external loudspeaker relative to sound transducer 503 can be determined according to previously described method, and then External loudspeaker is postponed with reference to customer location and gain-adjusted.
The principle or embodiment of the present invention is also implemented as computer program, and it is stored in computer-readable In medium, and including computer program instructions.These computer program instructions when being run by processor, can perform before The method of description.These computer programs can also be by transmission on Internet or distribution.For example, the computer program can be with It is audio driven or control program, it can be installed on the electronic equipments such as notebook, mobile phone, flat board, these equipment itself Just there is imageing sensor and sound transducer.Therefore, when these electronic equipments are connected to boom box system, then may be used Optimal audition experience is provided the user to perform previously described those methods.
The ultimate principle of the present invention is described above in association with exemplary embodiment, it should be noted however that in the application In the advantage, advantage, the effect that refer to etc. be only exemplary rather than limiting, it is impossible to think that these advantages, advantage, effect etc. are the application Each embodiment must be provided simultaneously with.In addition, detail disclosed above merely to the effect of example and be easy to reason The effect of solution, and it is unrestricted, and above-mentioned details does not limit the invention to realize using above-mentioned concrete details.
The device that is related in the present invention, device, equipment, the block diagram of system only illustratively the example of property and are not intended to The mode that requirement or hint must be illustrated according to square frame is attached, arranges, configures.As it would be recognized by those skilled in the art that , can be connected, be arranged by any-mode, configure these devices, device, equipment, system.Shown in the flow chart that the application is provided Method be not limited to the declaration order of flow chart, but, multiple steps can be performed in a different order, or two or More multi-step can be performed simultaneously, what these changes will be apparent to those skilled in the art.Such as " including ", " bag Containing ", the word of " having " etc. be open vocabulary, refer to " including but not limited to ", and can be with its used interchangeably.It is used here Vocabulary "or" and " and " refer to vocabulary "and/or", and can be with its used interchangeably, unless it be not such that context is explicitly indicated.This In the vocabulary " such as " that used refer to phrase " such as, but not limited to ", and can be with its used interchangeably.
It may also be noted that in the apparatus and method of the application, each part or each step can be to decompose and/or weigh Combination nova.These decompose and/or reconfigure the equivalents that should be regarded as the application.
The above description of disclosed aspect is provided so that any person skilled in the art can make or using this Application.Various modifications in terms of these are readily apparent to those skilled in the art, and here is defined General Principle can apply to other aspect without deviating from scope of the present application.Therefore, the application is not intended to be limited to Aspect shown in this, but according to the widest range consistent with the feature of principle disclosed herein and novelty.
In order to purpose of illustration and description has been presented for above description.Additionally, this description is not intended to the reality of the application Apply example and be restricted to form disclosed herein.Although already discussed above multiple exemplary aspects and embodiment, this area skill Art personnel will be recognized that its some modification, modification, change, addition and sub-portfolio.

Claims (11)

1. a kind of multi-channel audio system, including multiple speakers (210,220), imageing sensor (202), sound transducer (201) and controller (230), there is the predetermined first relative position between described image sensor and the sound transducer Put, the controller includes:
Loudspeaker position detector unit (231), be configured to that the sound transducer detects raises one's voice from the plurality of The predetermined audio signal of device is determining second relative position of the plurality of speaker relative to the sound transducer;
Customer location detector unit (232), is configured to user images that described image sensor detects to determine user Relative to described image sensor third phase to position;
Co-factor propagation unit (233), is configured to fusion first relative position, second relative position and described the Three relative positions, so that it is determined that fourth relative position of the user relative to each speaker;
Delay compensation computing unit (234), is configured to sound of the 4th relative position to calculate for each speaker The delay compensation value of frequency signal;And
Postpone adjustment unit (236), the phase of the audio signal for being configured to adjust each speaker according to the delay compensation value Position.
2. multi-channel audio system as claimed in claim 1, wherein, described image sensor and the sound transducer are arranged On in the plurality of speaker, so as to have predetermined the between the sound transducer and one speaker Two relative positions.
3. multi-channel audio system as claimed in claim 1, wherein, the sound transducer includes microphone array, described Loudspeaker position detector unit is configured to sound localization to determine side of each speaker relative to the microphone array To, and predetermined audio signal is sent based on each speaker and the microphone array is received between the audio signal Time difference is determining distance of each speaker relative to the microphone array.
4. multi-channel audio system as claimed in claim 1, wherein, the controller also includes:
Gain compensation computing unit (235), is configured to sound of the 4th relative position to calculate for each speaker The gain compensation value of frequency signal;And
Gain adjustment unit (237), the increasing of the audio signal for being configured to adjust each speaker according to the gain compensation value Benefit.
5. a kind of equipment of the multi-channel audio system including described in any one of Claims 1-4.
6. equipment as claimed in claim 5, wherein, the equipment is television set or automobile.
7. a kind of method that sound is reappeared using multi-channel audio system, the multi-channel audio system includes multiple raising one's voice Device, methods described includes:
The predetermined audio signal that each speaker sends is received from sound transducer, and is believed based on received predetermined audio Number determine the second relative position of each speaker;
From imageing sensor receive user image, and determine the third phase para-position of user based on received user images Put;
To second relative position, the third phase to predetermined between position and sound transducer and imageing sensor First relative position is merged, to determine fourth relative position of the user relative to each sensor;And
The delay compensation value of the audio signal for each speaker is determined based on the 4th relative position, and according to described Delay compensation value is adjusting the phase place of the audio signal of each speaker.
8. method as claimed in claim 7, wherein, described image sensor and the sound transducer are arranged on the plurality of On in speaker, so as to have the predetermined second relative position between the sound transducer and one speaker Put.
9. method as claimed in claim 7, wherein it is determined that the second relative position of each speaker includes:
Direction of each speaker relative to the sound transducer is determined based on sound localization, and based on each speaker Send predetermined audio signal and the sound transducer receives the time difference between the audio signal and raises one's voice determining each Distance of the device relative to the sound transducer.
10. method as claimed in claim 7, also includes:
The gain compensation value of the audio signal for each speaker is determined based on the 4th relative position, and according to described Gain compensation value is adjusting the gain of the audio signal of each speaker.
A kind of 11. computer programs, including computer program instructions, the computer program instructions are being run by processor When make described in any one of described computing device claim 7 to 10 method.
CN201710001369.8A 2017-01-03 2017-01-03 The multi-channel audio system of user and the equipment including it can be tracked Active CN106686520B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201710001369.8A CN106686520B (en) 2017-01-03 2017-01-03 The multi-channel audio system of user and the equipment including it can be tracked

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201710001369.8A CN106686520B (en) 2017-01-03 2017-01-03 The multi-channel audio system of user and the equipment including it can be tracked

Publications (2)

Publication Number Publication Date
CN106686520A true CN106686520A (en) 2017-05-17
CN106686520B CN106686520B (en) 2019-04-02

Family

ID=58848857

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201710001369.8A Active CN106686520B (en) 2017-01-03 2017-01-03 The multi-channel audio system of user and the equipment including it can be tracked

Country Status (1)

Country Link
CN (1) CN106686520B (en)

Cited By (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107888857A (en) * 2017-11-17 2018-04-06 青岛海信电器股份有限公司 For the method for adjustment of sound field, device and separate television in separate television
CN107920307A (en) * 2017-12-07 2018-04-17 北海威德电子科技有限公司 A kind of device and method of sound equipment adjust automatically sound output phase
CN108055633A (en) * 2017-12-06 2018-05-18 维沃移动通信有限公司 A kind of audio frequency playing method and mobile terminal
CN108347673A (en) * 2018-03-30 2018-07-31 上海与德科技有限公司 A kind of control method of intelligent sound box, device, storage medium and intelligent sound box
CN108495234A (en) * 2018-04-19 2018-09-04 北京微播视界科技有限公司 Multichannel audio processing method, device and computer readable storage medium
CN108551610A (en) * 2018-05-11 2018-09-18 四川斐讯信息技术有限公司 A kind of intelligent sound box and its audio effect generating method
WO2019041178A1 (en) * 2017-08-30 2019-03-07 深圳魔耳智能声学科技有限公司 Sound playback method and device, and readable storage medium
CN109466467A (en) * 2017-09-08 2019-03-15 哈曼国际工业有限公司 Sound tuning based on adjustable seat positioning
CN109495800A (en) * 2018-10-26 2019-03-19 成都佳发安泰教育科技股份有限公司 A kind of audio dynamic collect system and method
CN109862503A (en) * 2019-01-30 2019-06-07 北京雷石天地电子技术有限公司 A kind of method and apparatus of loudspeaker delay adjust automatically
CN110049429A (en) * 2019-05-10 2019-07-23 苏州静声泰科技有限公司 A kind of trailing type dynamic solid sound system for audio-visual equipment
CN110996238A (en) * 2019-12-17 2020-04-10 杨伟锋 Binaural synchronous signal processing hearing aid system and method
CN111225319A (en) * 2018-11-23 2020-06-02 广达电脑股份有限公司 Loudspeaker system and loudspeaker control method using same
WO2020119220A1 (en) * 2018-12-12 2020-06-18 深圳市冠旭电子股份有限公司 Smart loudspeaker box playing method and device and smart loudspeaker box
CN111332173A (en) * 2018-12-18 2020-06-26 奥迪股份公司 Automatic sound system
CN111757241A (en) * 2019-03-11 2020-10-09 深圳市冠旭电子股份有限公司 Sound effect control method and device, sound box array and wearable device
CN111937413A (en) * 2018-04-09 2020-11-13 索尼公司 Information processing apparatus, method, and program
CN112083379A (en) * 2020-09-09 2020-12-15 成都极米科技股份有限公司 Audio playing method and device based on sound source positioning, projection equipment and medium
CN112118527A (en) * 2019-06-19 2020-12-22 华为技术有限公司 Multimedia information processing method, device and storage medium
CN112136331A (en) * 2017-06-08 2020-12-25 Dts公司 Correction for loudspeaker delay
CN112153525A (en) * 2020-08-11 2020-12-29 广东声音科技有限公司 Positioning method and system for multi-loudspeaker panoramic sound effect
CN112312278A (en) * 2020-12-28 2021-02-02 汉桑(南京)科技有限公司 Sound parameter determination method and system
CN112383655A (en) * 2020-11-02 2021-02-19 Oppo广东移动通信有限公司 Electronic device, sound enhancement method for electronic device, and storage medium
CN112492455A (en) * 2020-11-27 2021-03-12 广东美的制冷设备有限公司 Control method and operation control device of electronic equipment and electronic equipment
CN113542984A (en) * 2021-06-09 2021-10-22 荣耀终端有限公司 Stereo sound implementation system, method, electronic device, and storage medium
CN113608641A (en) * 2020-06-18 2021-11-05 深圳市冠旭电子股份有限公司 Method and device for adjusting display position of curved screen, intelligent sound box and storage medium
CN113709631A (en) * 2020-05-22 2021-11-26 宏碁股份有限公司 Surround sound system and method for applying surround sound technology to electronic contest seats
CN113852905A (en) * 2021-09-24 2021-12-28 联想(北京)有限公司 Control method and control device
CN115426589A (en) * 2022-11-03 2022-12-02 荣耀终端有限公司 Audio signal adjusting method and electronic equipment
US11895466B2 (en) 2020-12-28 2024-02-06 Hansong (Nanjing) Technology Ltd. Methods and systems for determining parameters of audio devices

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102547533A (en) * 2010-11-05 2012-07-04 索尼公司 Acoustic control apparatus and acoustic control method
CN102572676A (en) * 2012-01-16 2012-07-11 华南理工大学 Real-time rendering method for virtual auditory environment
CN104581543A (en) * 2013-10-10 2015-04-29 三星电子株式会社 Audio system, method of outputting audio, and speaker apparatus
WO2015108824A1 (en) * 2014-01-18 2015-07-23 Microsoft Technology Licensing, Llc Enhanced spatial impression for home audio

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN102547533A (en) * 2010-11-05 2012-07-04 索尼公司 Acoustic control apparatus and acoustic control method
CN102572676A (en) * 2012-01-16 2012-07-11 华南理工大学 Real-time rendering method for virtual auditory environment
CN104581543A (en) * 2013-10-10 2015-04-29 三星电子株式会社 Audio system, method of outputting audio, and speaker apparatus
WO2015108824A1 (en) * 2014-01-18 2015-07-23 Microsoft Technology Licensing, Llc Enhanced spatial impression for home audio

Cited By (45)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112136331A (en) * 2017-06-08 2020-12-25 Dts公司 Correction for loudspeaker delay
WO2019041178A1 (en) * 2017-08-30 2019-03-07 深圳魔耳智能声学科技有限公司 Sound playback method and device, and readable storage medium
CN109466467A (en) * 2017-09-08 2019-03-15 哈曼国际工业有限公司 Sound tuning based on adjustable seat positioning
CN107888857A (en) * 2017-11-17 2018-04-06 青岛海信电器股份有限公司 For the method for adjustment of sound field, device and separate television in separate television
CN108055633A (en) * 2017-12-06 2018-05-18 维沃移动通信有限公司 A kind of audio frequency playing method and mobile terminal
CN107920307A (en) * 2017-12-07 2018-04-17 北海威德电子科技有限公司 A kind of device and method of sound equipment adjust automatically sound output phase
CN108347673A (en) * 2018-03-30 2018-07-31 上海与德科技有限公司 A kind of control method of intelligent sound box, device, storage medium and intelligent sound box
US11337022B2 (en) 2018-04-09 2022-05-17 Sony Corporation Information processing apparatus, method, and program
CN111937413A (en) * 2018-04-09 2020-11-13 索尼公司 Information processing apparatus, method, and program
CN108495234A (en) * 2018-04-19 2018-09-04 北京微播视界科技有限公司 Multichannel audio processing method, device and computer readable storage medium
CN108551610A (en) * 2018-05-11 2018-09-18 四川斐讯信息技术有限公司 A kind of intelligent sound box and its audio effect generating method
CN109495800A (en) * 2018-10-26 2019-03-19 成都佳发安泰教育科技股份有限公司 A kind of audio dynamic collect system and method
CN111225319A (en) * 2018-11-23 2020-06-02 广达电脑股份有限公司 Loudspeaker system and loudspeaker control method using same
CN111225319B (en) * 2018-11-23 2022-03-15 广达电脑股份有限公司 Loudspeaker system and loudspeaker control method using same
WO2020119220A1 (en) * 2018-12-12 2020-06-18 深圳市冠旭电子股份有限公司 Smart loudspeaker box playing method and device and smart loudspeaker box
CN111314821A (en) * 2018-12-12 2020-06-19 深圳市冠旭电子股份有限公司 Intelligent sound box playing method and device and intelligent sound box
CN111332173A (en) * 2018-12-18 2020-06-26 奥迪股份公司 Automatic sound system
CN111332173B (en) * 2018-12-18 2023-05-09 奥迪股份公司 Automatic sound system
CN109862503A (en) * 2019-01-30 2019-06-07 北京雷石天地电子技术有限公司 A kind of method and apparatus of loudspeaker delay adjust automatically
CN111757241A (en) * 2019-03-11 2020-10-09 深圳市冠旭电子股份有限公司 Sound effect control method and device, sound box array and wearable device
CN110049429A (en) * 2019-05-10 2019-07-23 苏州静声泰科技有限公司 A kind of trailing type dynamic solid sound system for audio-visual equipment
CN112118527A (en) * 2019-06-19 2020-12-22 华为技术有限公司 Multimedia information processing method, device and storage medium
CN110996238A (en) * 2019-12-17 2020-04-10 杨伟锋 Binaural synchronous signal processing hearing aid system and method
CN113709631B (en) * 2020-05-22 2023-01-17 宏碁股份有限公司 Surround sound system and method for applying surround sound technology to electronic contest seats
CN113709631A (en) * 2020-05-22 2021-11-26 宏碁股份有限公司 Surround sound system and method for applying surround sound technology to electronic contest seats
CN113608641A (en) * 2020-06-18 2021-11-05 深圳市冠旭电子股份有限公司 Method and device for adjusting display position of curved screen, intelligent sound box and storage medium
CN113608641B (en) * 2020-06-18 2024-01-16 深圳市冠旭电子股份有限公司 Method and device for adjusting display position of curved screen, intelligent sound box and storage medium
CN112153525A (en) * 2020-08-11 2020-12-29 广东声音科技有限公司 Positioning method and system for multi-loudspeaker panoramic sound effect
CN112083379B (en) * 2020-09-09 2023-10-20 极米科技股份有限公司 Audio playing method and device based on sound source localization, projection equipment and medium
CN112083379A (en) * 2020-09-09 2020-12-15 成都极米科技股份有限公司 Audio playing method and device based on sound source positioning, projection equipment and medium
CN112383655A (en) * 2020-11-02 2021-02-19 Oppo广东移动通信有限公司 Electronic device, sound enhancement method for electronic device, and storage medium
CN112383655B (en) * 2020-11-02 2022-07-12 Oppo广东移动通信有限公司 Electronic device, sound enhancement method for electronic device, and storage medium
CN112492455A (en) * 2020-11-27 2021-03-12 广东美的制冷设备有限公司 Control method and operation control device of electronic equipment and electronic equipment
CN112492455B (en) * 2020-11-27 2022-05-27 广东美的制冷设备有限公司 Control method and operation control device of electronic equipment and electronic equipment
CN112312278A (en) * 2020-12-28 2021-02-02 汉桑(南京)科技有限公司 Sound parameter determination method and system
CN113207059A (en) * 2020-12-28 2021-08-03 汉桑(南京)科技有限公司 Sound parameter determination method and system
CN113207059B (en) * 2020-12-28 2023-05-16 汉桑(南京)科技股份有限公司 Sound parameter determining method and system
CN113207060B (en) * 2020-12-28 2023-07-18 汉桑(南京)科技股份有限公司 Sound parameter determining method and system
CN113207060A (en) * 2020-12-28 2021-08-03 汉桑(南京)科技有限公司 Sound parameter determination method and system
CN112312278B (en) * 2020-12-28 2021-03-23 汉桑(南京)科技有限公司 Sound parameter determination method and system
US11895466B2 (en) 2020-12-28 2024-02-06 Hansong (Nanjing) Technology Ltd. Methods and systems for determining parameters of audio devices
CN113542984B (en) * 2021-06-09 2022-04-15 荣耀终端有限公司 Stereo sound implementation system, method, electronic device, and storage medium
CN113542984A (en) * 2021-06-09 2021-10-22 荣耀终端有限公司 Stereo sound implementation system, method, electronic device, and storage medium
CN113852905A (en) * 2021-09-24 2021-12-28 联想(北京)有限公司 Control method and control device
CN115426589A (en) * 2022-11-03 2022-12-02 荣耀终端有限公司 Audio signal adjusting method and electronic equipment

Also Published As

Publication number Publication date
CN106686520B (en) 2019-04-02

Similar Documents

Publication Publication Date Title
CN106686520B (en) The multi-channel audio system of user and the equipment including it can be tracked
US8996367B2 (en) Sound processing apparatus, sound processing method and program
US8204248B2 (en) Acoustic localization of a speaker
CN102771144B (en) Apparatus and method for direction dependent spatial noise reduction
CN104136299B (en) For the system, method and the device that in car, sound are led
EP2859772B1 (en) Wind noise detection for in-car communication systems with multiple acoustic zones
EP2984852B1 (en) Method and apparatus for recording spatial audio
EP3189521B1 (en) Method and apparatus for enhancing sound sources
EP2850611B1 (en) Noise dependent signal processing for in-car communication systems with multiple acoustic zones
CN107004425B (en) Enhanced conversational communication in shared acoustic spaces
US20120020480A1 (en) Systems, methods, and apparatus for enhanced acoustic imaging
US8615392B1 (en) Systems and methods for producing an acoustic field having a target spatial pattern
EP3669780B1 (en) Methods, devices and system for a compensated hearing test
JP2003264883A (en) Voice processing apparatus and voice processing method
US20230134271A1 (en) Method for Audio Processing
US11682414B1 (en) Adjusting audio transparency based on content
Scharrer et al. Sound field classification in small microphone arrays using spatial coherences
US11721353B2 (en) Spatial audio wind noise detection
JP2003224898A (en) Apparatus and method for enhancing articulation of loudspeaking
CN111163411B (en) Method for reducing influence of interference sound and sound playing device
JP2001337694A (en) Method for presuming speech source position, method for recognizing speech, and method for emphasizing speech
Bozzoli et al. Directivity balloons of real and artificial mouth simulators for measurement of the Speech Transmission Index
JP7320663B1 (en) In-vehicle adaptive sound quality output method, device, storage medium, and in-vehicle sound system
WO2022201456A1 (en) Information presentation device, information presentation method, and information presentation program
US11671752B2 (en) Audio zoom

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant