CN112558302B

CN112558302B - Intelligent glasses for determining glasses posture and signal processing method thereof

Info

Publication number: CN112558302B
Application number: CN202011441683.6A
Authority: CN
Inventors: 童伟峰; 张亮
Original assignee: Bestechnic Shanghai Co Ltd
Current assignee: Bestechnic Shanghai Co Ltd
Priority date: 2020-12-08
Filing date: 2020-12-08
Publication date: 2022-12-20
Anticipated expiration: 2040-12-08
Also published as: CN112558302A

Abstract

The present disclosure relates to smart glasses for determining a glasses pose parameter. The smart glasses include a first camera unit, a second camera unit, and a posture determination unit. The first camera unit is configured to take an image of the eyes of the wearer; a second image pickup unit configured to pick up an image of an external environment; and an attitude determination unit configured to: determining a pose-related parameter of the smart glasses based on at least a series of images of the external environment captured by the second camera unit. Therefore, the perception sound source of the 3D audio signal can be kept stable, 3D recording based on eyeball tracking is achieved, and system power consumption is reduced.

Description

Intelligent glasses for determining glasses posture and signal processing method thereof

Technical Field

The present disclosure relates to smart glasses and a signal processing method thereof, and more particularly, to smart glasses for determining a posture of the glasses and a signal processing method thereof.

Background

Spectacles are one of the most commonly used items. With social progress and improvement of living standard of people, intelligent glasses such as Bluetooth glasses and AR glasses gradually enter people's lives. After the user wears the intelligent glasses, the functions of listening to 3D audio, watching videos, experiencing virtual reality games and the like can be achieved by the aid of the intelligent glasses, and great convenience is brought to the user. However, currently, little research has been done on how smart glasses of various configurations sense the movement of the wearer. Often smart glasses rely on expensive dedicated sensors to detect gestures, so that flat smart glasses typically do not provide gesture detection functionality, which impacts the user experience. As the application range of the smart glasses is expanded, such as 3D vision and hearing feeling, the lack of detection of the posture of the smart glasses may seriously affect the reality of the user's perception. In addition, the high power consumption of the intelligent glasses greatly influences the use duration of the equipment and the wearing experience of the user.

Disclosure of Invention

The present disclosure is provided to solve the above-mentioned problems occurring in the prior art.

There is a need for smart glasses that can effectively use a series of images of the external environment captured by an external environment imaging unit, which is a member that smart glasses are usually equipped with, to determine posture-related parameters, that is, the series of images of the external environment can be used alone to determine posture-related parameters, or can be used in cooperation with other information to determine posture-related parameters with higher accuracy.

According to a first aspect of the present disclosure, there is provided smart glasses for determining a posture of glasses, which may include a first camera unit, a second camera unit, and a posture determination unit. Wherein the first imaging unit is configured to capture an image of an eye of the wearer; a second image pickup unit configured to pick up an image of an external environment; the posture determining unit is configured to determine a posture-related parameter of the smart glasses based on at least a series of images of the external environment captured by the second imaging unit.

According to a second aspect of the present disclosure, a signal processing method of smart glasses is provided. The signal processing method may comprise capturing an image of the eye of the wearer. In addition to capturing images of the wearer's eyes, images of the external environment may also be captured. Determining a pose-related parameter of the smart glasses based at least on a series of captured images of an external environment.

With the smart glasses according to various embodiments of the present disclosure, it is possible to effectively determine the posture-related parameters using a series of images of the external environment captured by a member that is typically equipped with the external environment imaging unit, that is, the series of images of the external environment can be used alone to determine the posture-related parameters (in the case where the accuracy of the required posture-related parameters is low), and can also be used in cooperation with other information to determine the posture-related parameters with higher accuracy (in the case where the accuracy of the required posture-related parameters is high).

Drawings

In the drawings, which are not necessarily drawn to scale, like reference numerals may describe similar parts throughout the different views. The drawings illustrate various embodiments generally by way of example and not by way of limitation, and together with the description and claims serve to explain the disclosed embodiments. The same reference numbers will be used throughout the drawings to refer to the same or like parts, where appropriate. Such embodiments are illustrative, and are not intended to be exhaustive or exclusive embodiments of the present apparatus or method.

Fig. 1 shows a schematic structural diagram of smart glasses according to an embodiment of the present disclosure.

Fig. 2 shows a configuration schematic of smart glasses according to an embodiment of the present disclosure.

Fig. 3 illustrates a flow chart of a method of glasses pose determination for smart glasses according to an embodiment of the present disclosure.

Fig. 4 illustrates a flowchart of a method for 3D audio playback using glasses pose information of smart glasses according to an embodiment of the present disclosure.

Fig. 5 shows a flowchart of a method for 3D recording based on point of regard information according to an embodiment of the present disclosure.

Fig. 6 shows a schematic diagram of a flow of detecting image activity of an eye image and adjusting the frame rate/resolution of each imaging unit accordingly according to an embodiment of the present disclosure.

Detailed Description

The following detailed description is provided to enable those skilled in the art to better understand the technical solutions of the present disclosure, with reference to the accompanying drawings and specific embodiments. Embodiments of the disclosure are described in further detail below with reference to the figures and the detailed description, but the disclosure is not limited thereto.

Fig. 1 shows a schematic structural diagram of smart glasses according to an embodiment of the present disclosure. As shown in fig. 1, the smart glasses may include a first image capturing unit 101, which may be configured to capture an image of the eyes of the wearer, which may be implemented using an image sensor, an eye-tracking camera, or the like.

In some embodiments, the smart glasses may further include a second camera unit 102. The second camera unit 102 may be an external image sensor or a camera, and may be disposed on the frame as shown in fig. 1, or may be disposed at another position, and is configured to: an image of the external environment is captured. Note that "image" herein may include a still image and a moving image (i.e., video).

As an example, the pose determination unit 2011 and the audio processing unit 2012 of the smart glasses and other arithmetic processing may be carried on the processing unit 201 (that is, implemented as an SOC). As an example, the pose determination unit 2011 and the audio processing unit 2012, as well as other computational processes (such as those shown in fig. 2), may also employ other implementations. Each unit in the present disclosure may be hardware, software, or a combination of hardware and software as necessary. In case the units are implemented by software, computer executable instructions may be stored on the respective memories, which when executed by the respective processors implement the functions of the respective units. The various processors may be implemented as any of an FPGA, ASIC, DSP chip, SOC (system on chip), MPU (e.g., without limitation Cortex), etc. The processor may be communicatively coupled to the memory and configured to execute computer-executable instructions stored therein. The memory may include Read Only Memory (ROM), flash memory, random Access Memory (RAM), dynamic Random Access Memory (DRAM) such as Synchronous DRAM (SDRAM) or Rambus DRAM, static memory (e.g., flash memory, static random access memory), etc., on which computer-executable instructions are stored in any format. In some embodiments, computer-executable instructions may be accessed by a processor, read from a ROM or any other suitable storage location, and loaded into RAM for execution by the processor.

The pose determination unit 2011 may be configured to determine pose-related parameters of the smart glasses based on at least a series of images of the external environment captured by the second camera unit 102. Specifically, in some embodiments, the smart glasses may also include an IMU 103. The IMU 103 is a device that measures the three-axis attitude angles (or angular rates) and acceleration of an object. Generally, an IMU 103 includes three single-axis accelerometers and three single-axis gyroscopes, the accelerometers detect acceleration signals of the object in three independent axes of the carrier coordinate system, and the gyroscopes detect angular velocity signals of the carrier relative to the navigation coordinate system, and measure angular velocity and acceleration of the object in three-dimensional space, and then solve the attitude of the object. In this case, the pose determination unit 2011 may determine the pose-related parameters of the smart glasses only by using the solution of the angular velocity, acceleration, and the like output by the IMU 103.

In some smart eyewear, the IMU 103 may not be installed because of cost or size constraints. In this case, the posture-related parameters of the smart glasses can be easily determined based only on a series of images of the external environment captured by the second camera unit 102. For example, the wearer may be instructed in advance to make a preset pose and capture a reference image of the external environment, and a series of subsequently captured actual images of the external environment may be registered with the reference image to calculate a spatial transformation of the former with respect to the latter, thereby solving for the pose-related parameters of the smart glasses. For another example, a fixed marker may be set in the external environment and its reference orientation determined in advance when the wearer makes a preset gesture, at least part of the marker may be recognized from a series of subsequently taken actual images of the external environment, and a transition between the recognized actual orientation of the marker and the reference orientation may be determined, whereby the gesture-related parameters of the smart glasses may be resolved. Therefore, the intelligent glasses do not need to depend on professional sensors such as the IMU 103, and the posture related parameters of the intelligent glasses can be determined in real time as long as the common second camera unit 102 is arranged, so that good audio-visual experience is provided for a wearer based on the real-time posture related parameters. This is particularly true where the wearer's posture is relatively stable.

In some smart eyewear, the IMU 103 may be mounted, for example, in a single-sided eyewear portion, or in both left and right-sided eyewear. The mounting position of the IMU 103 is not limited to this, and may be mounted at various positions such as a temple, a frame, and the like. In this manner, the IMU 103 may be utilized to determine the pose-related parameters of the smart glasses, and the images of a series of external environments captured by the second image capturing unit 102 may be fully utilized to calibrate their zero offset. Specifically, the IMU 103 is characterized in that the change of the eyeglass pose can be accurately obtained in a short time, but is easily affected by the zero offset in a long time, and the image captured by the second camera unit 102 is slightly jittered but is not affected by the zero offset in a long time, so that the IMU 103 can be used to calibrate the zero offset. The second camera unit 102 may acquire images at different times, the images may be related to the posture of the glasses and may reflect the change of the posture of the glasses, and the zero offset of the IMU 103 may be calibrated by means of SLAM (simultaneous localization and mapping) technology based on the images. By determining the posture-related parameters by the IMU 103 and calibrating the zero offset using the image of the external environment captured by the second imaging unit 102, the posture-related parameters of the smart glasses excellent in real-time can be obtained with significantly higher accuracy.

As shown in fig. 1, in some embodiments, the smart glasses may include a pair of speakers 301, respectively disposed on the left and right sides, so as to implement a stereo playing effect by playing audio signals for the left and right ears, respectively. Typically, the smart glasses may also include at least one microphone (microphone set 104 is shown in fig. 1 as an example) for receiving sound. In some embodiments, for subsequent signal acquisition and processing, a microphone set including a plurality of microphones disposed at different positions of the lens frame may also be provided, for example, 2-8 microphones disposed at certain intervals. In some embodiments, the microphone set 104 may have a beam forming component for beam forming according to the attention direction of the wearer (i.e. the direction of the wearer's gaze point), so as to optimize the sound reception effect of the microphone set 104 for the attention direction, reduce the mixing of noise or other direction noise, and optimize the signal-to-noise ratio of the audio signal collected by the microphone 104.

The audio processing unit 2012 may be configured to perform various processing on the audio signals to be played by the speakers 301 in response to the gesture-related parameters, or perform various processing on the original audio signals collected by the microphones 104 in response to the gaze point, etc., as needed, to enhance the 3D listening experience of the wearer. In particular, the audio processing unit 2012 may be configured to: processing the 3D audio signal to be played based on the pose-related parameters such that a perceptual sound source of the 3D audio signal remains stable. For another example, the audio processing unit 2012 may be configured to: the audio signals collected by the microphone set 104 are processed to obtain 3D recording signals based on the orientation of the wearer's gaze point relative to the left and right ears of the wearer.

In some embodiments, the smart glasses may also have a video display unit 302, which may be configured to display video content. Although the video display unit 302 is shown only for the right glasses in fig. 1, it should be understood that this may also be provided for the left glasses part, which is not described herein. In some embodiments, the video display unit 302 may include a video projection source (i.e., the video source to be played). For example, the video display unit 302 may further include an imaging screen to reflect the image projection source into the corresponding eye. For another example, the video display unit 302 may inject the image projection source into the corresponding eye using an optical waveguide.

The required information interaction between different parts inside the unit shown in the figure, such as the loudspeaker 301, or other different units, can be realized through communication connection or direct wire connection, and the communication connection includes but is not limited to bluetooth, WIFI, radio frequency, wired transmission, and the like.

Fig. 2 shows a schematic configuration of smart glasses according to an embodiment of the present disclosure. As shown in fig. 2, the smart glasses may be mainly divided into a sensing unit 10, a control unit 20, and an input/output unit 30.

The sensing portion 10 may contain a series of units for the smart eyewear wearer and external environmental perception, including a first camera unit 101, a second camera unit 102, an Inertial Measurement Unit (IMU) 103, and a microphone set 104. The series of units may be mounted directly on the smart glasses or may be connected to the smart glasses via a Mobile Industry Processor Interface (MIPI) 303. Wherein the first camera unit 101 may be configured to take images of the eyes of the wearer, the taken images of the eyes being used for image activity detection and further for determining the point of regard of the smart glasses wearer. The second camera unit 102 in the sensing part 10 may be configured to capture an image of the external environment, and the captured image of the external environment may be used to generate the posture related parameters of the smart glasses, or the zero offset of the IMU 103 may be calibrated to obtain more accurate posture related parameters. The IMU 103 in the sensing portion 10 may be configured to detect three-axis attitude angles (or angular rates) and attitude parameters such as acceleration of the smart glasses. The IMU 103 may also include a magnetometer. The microphone set 104 in the sensing part 10 may be configured to collect an audio signal and perform beamforming in the direction of the gaze point of the smart eyewear wearer using the collected audio signal and information such as the gaze point of the wearer for the generation of a 3D recording signal. The number of microphones used for 3D recording is at least 1. When beamforming is performed in the direction of the gaze point of the wearer of the smart glasses, the number of microphones is at least 2, and optionally, in order to enhance the beamforming effect, more than 2 microphones, for example, 4 or 8 microphones, may be configured, and may be spaced at different positions of the frame as required.

The control section 20 shown in fig. 2 may contain a series of units for signal processing and control of the smart glasses, including a processing unit 201 comprising a pose determination unit 2011 configured to determine pose related parameters of the smart glasses based on at least a series of images of the external environment captured by the second imaging unit 102. Alternatively, in some embodiments, when the smart glasses are configured with the IMU 103, the pose determination unit 2011 may also determine pose related parameters of the glasses based on output information of the IMU 103. In some other embodiments, the pose determination unit 2011 may also process the image from the second camera unit 102 and the output information of the IMU 103 at the same time to generate pose related parameters for the smart glasses. The processing unit 201 may further include an audio processing unit 2012 configured to: processing a 3D audio signal to be played based on the posture related parameters of the intelligent glasses so as to enable a perception sound source of the 3D audio signal to be stable; and processing the audio signals collected by the microphones 104 to obtain a 3D recording signal based on the orientation of the wearer's gaze point relative to the left and right ears. Although it is shown in fig. 2 that the processing unit 201 is provided separately from the eye tracking unit 202, the calibration unit 203, and the image-activity detection unit 204, this is merely an example. In particular, the audio processing unit 2012 can also be used to perform audio processing that is conventional for smart glasses, in addition to performing the special processing of the various embodiments of the present disclosure. The audio processing unit 2012 may be implemented by a combination of software and hardware. The pose determination unit 2011 and the calibration unit 203 may be implemented by respective processors by storing computer executable instructions on a memory. The eye tracking unit 202 may also be implemented by a corresponding processor by storing computer executable instructions on a memory. In contrast, the image activity detection unit 204 may be implemented by a separate hardware circuit, and is not dependent on other processors, and when the image activity detection unit 204 operates, other processors may not operate or operate in a low power consumption mode, so that power consumption may be significantly reduced. The image activity detection unit 204 may be configured to: the images of the eyes of the wearer captured by the first imaging unit 101 at different times are detected for image motion, and when image motion is detected, the operation of the eye tracking unit 202 is started.

The eye tracking unit 202 may be configured to determine the gaze point of the wearer based on the image of the eye captured by the first imaging unit 101. The eye tracking unit 202 may identify eye images and calculate eye motion vectors, obtain eye motion coordinates, and the like. The eyeball tracking unit 202 can be utilized to obtain not only the gaze point of the eyeball, but also various movements of the eye, such as blinking, rotating the eyeball, and the like. Various interactions may be accomplished based on the various eye movements detected. For example, blinking actions may be used as confirmation of menus and dialog boxes. The method comprises the steps that a menu button and/or a dialog box button are/is virtually displayed at a certain space by using intelligent glasses, and a wearer can watch the menu button and the dialog box button for a certain time to act as a key action of the button.

In some embodiments, the state of the eye tracked via the eye tracking unit 202, including but not limited to various eye movements, gaze points, and the like, may work in various ways in conjunction with other components to automatically respond to the actual needs implied by the wearer through the various movements of the eye and gaze points.

The gaze point information may be fed to any one or more of the audio processing unit 2012, the microphone set 104, and the second camera unit 102 for corresponding processing. As will be described in detail below in connection with fig. 5.

The calibration unit 203 may be configured to: the zero Bias (Bias) of the IMU 103 is calibrated based on a series of images of the external environment captured by the second imaging unit 102, improving the accuracy of its output data.

The input/output section 30 may include a series of means for inputting various media signals into the smart glasses or outputting processed audio/video signals, including: speakers 301 configured to play the processed 3D audio signals, typically symmetrically disposed on the left and right ear sides of the smart glasses; a video display unit 302, which may be configured to display video content. The video display unit 302 may include a video projection source (i.e., a video source to be played). For example, the video display unit 302 may further include an imaging screen to reflect the image projection source into the corresponding eye. For another example, the video display unit 302 may inject the image projection source into the corresponding eye using an optical waveguide. A Mobile Industry Processor Interface (MIPI) 303 configured as smart glasses having a MIPI interface, wherein one MIPI interface can be connected to multiple sensors (including the first camera unit 101, the second camera unit 102, and the IMU 103, for example) and the video display unit 302, and can also be connected to other peripherals.

Fig. 3 illustrates a flow chart of a method of glasses pose determination for smart glasses according to an embodiment of the present disclosure. When a user wears smart glasses to watch video with 3D audio, for example, when playing VR or AR games, and the head of the wearer makes inevitable movements, it is desirable that the perceived sound source is reasonable and stable, such as: the method is characterized in that a wearer feels a gunshot sound transmitted from a certain direction in a game scene, or a footstep sound is transmitted from a far distance to a near distance from a certain direction, when the wearer turns suddenly, the position and the direction of the glasses are changed, and the position and the direction of a loudspeaker on the glasses are also changed, but the gunshot sound and the footstep sound are not suddenly changed according to the game scene, so that the audio to be played is required to be processed according to posture parameters such as the position, the direction and the angle of the glasses, and the sound source perceived by the wearer is kept stable. Therefore, the pose-related parameters of the smart glasses must first be determined. As shown in fig. 3, there are three methods for obtaining the parameter.

In the first method, in some embodiments, the intelligent glasses are not provided with the IMU 103, and the pose determination unit 2011 determines the pose-related parameters of the glasses by only using the external environment video captured by the second camera unit 102, and using an image processing algorithm, for example, calculating the correlations between different frames of images, finding the changes of the positions and/or locations of the glasses, and the like.

Second, in some embodiments, the smart eyewear is configured with an IMU 103, where the IMU 103 is a device that measures the three-axis attitude (or angular rate) and acceleration of the object. Generally, an IMU 103 includes three single-axis accelerometers and three single-axis gyroscopes, the accelerometers detect acceleration signals of the object in three independent axes of the carrier coordinate system, and the gyroscopes detect angular velocity signals of the carrier relative to the navigation coordinate system, and measure angular velocity and acceleration of the object in three-dimensional space, and then solve the attitude of the object. In this case, the pose determination unit 2011 may determine the pose related parameters of the smart glasses only by using the solution of the angular velocity, acceleration, and the like output by the IMU.

When this method is implemented, when the IMU 103 works for a long time, a zero Bias (Bias) is generated, that is: even if the IMU 103 remains stationary, there is still a small output, which can severely affect the accuracy of the lens pose parameters output by the IMU 103. The image of the external environment captured by the second imaging unit 102 may have slight jitter due to noise, but may not have accumulated zero offset effect for a long time, and may be used to calibrate the zero offset of the IMU 103. In some embodiments, the calibration unit 203 in the IMU assembly may be used to acquire the image of the external environment captured by the second imaging unit 102 at a certain time interval (or at a time when the determination is needed), so as to calibrate the zero offset of the IMU 103, and the pose determination unit 2011 may process the output of the calibrated IMU assembly, so as to obtain more accurate pose related parameters of the smart glasses. That is, the IMU 103 together with the calibration unit 203 constitutes an IMU component for ensuring accuracy and real-time of its detection data.

A third method, in some embodiments, when the smart glasses are configured with the IMU 103 and the smart glasses have higher requirements for self-pose and external environment information processing, the image of the external environment captured by the second image capturing unit 102 and the output information of the IMU 103 may be used in combination, for example, an information fusion method is used in the pose determination unit 2011, so as to obtain a more optimal pose related parameter of the smart glasses.

Fig. 4 shows a flowchart of a method for 3D audio playback using glasses pose information of smart glasses according to an embodiment of the present disclosure. As shown in fig. 4, the audio processing unit 2012 processes the 3D audio information to be played from the audio/video signal source 302 by using the pose parameter information of the glasses output by the pose determining unit 2011 in fig. 3, for example, the above-mentioned processing may adopt a Head Related Impulse Response (HRTF) model or a Binaural Room Impulse Response (BRIR). The processed 3D audio signals are output through the loudspeaker 301, undesirable sound source changes caused by changes of the position and the orientation of the intelligent glasses loudspeaker 301 due to activities of a wearer are eliminated, the stability of the sound source perceived by the wearer is maintained, and the audio-visual experience of a glasses user is improved.

Fig. 5 shows a flowchart of a method of 3D recording based on point of regard information according to an embodiment of the present disclosure. As shown in fig. 5, when the eye tracking unit 202 starts to work, the eye image of the smart glasses wearer is obtained from the first image capturing unit 101, and the image of the external environment is obtained from the second image capturing unit 102 in cooperation with the map information, so as to obtain the information of the gaze point of the wearer. In another embodiment, when the eye tracking unit 202 starts to operate, an eye image of the smart glasses wearer is acquired from the first image capturing unit 101, eye tracking is performed on the eye image, and the direction and distance of the gaze point of the wearer relative to the wearer are obtained. This gaze point information may be fed to the second camera unit 102 for determining the imaging focus of the image taken by the second camera unit 102 of the external environment, improving the clarity of the taking of the location where the wearer's gaze point is located, i.e. the location where the wearer is paying attention to. In some embodiments, the first camera unit 101 acquires an eye image of one eye of the smart glasses wearer; another camera unit (not shown) acquires an eye image of another eye of the smart-glasses wearer; the eyeball tracking unit processes the two eye images to obtain the information of the gaze point of the wearer, such as the direction and distance of the gaze point relative to the wearer.

In some embodiments, the microphone set 104 including the plurality of microphones may have beamforming means therein to perform beamforming with the gaze point information of the wearer, thereby optimizing the sound reception effect of the microphone set 104 with respect to the gaze point and reducing the mixing in of noise or other directional noise, optimizing the signal-to-noise ratio of the audio signals collected by the microphone set 104.

The audio processing unit 2012 obtains the position and the orientation of the gaze point of the eyeball of the wearer relative to the left and right ears of the wearer or the wearer by using the audio signal after the beam forming optimization or the audio signal acquired by a single microphone and combining the gaze point information of the wearer and the pose parameter information of the glasses output by the pose determining unit 2011 in fig. 3, and obtains the 3D recording signal according to an HRTF (Head Related estimated Impulse Response) model or a Binaural Room Impulse Response (BRIR). The 3D sound recording signal obtained in this way can clearly reproduce the stereoscopic perception effect of the sound emitted at the listening gaze point at the left and right ears.

Fig. 6 illustrates a flow of detecting image activity by an eye image and adjusting the frame rate/resolution of each imaging unit accordingly according to an embodiment of the present disclosure. In some embodiments, the smart glasses may further include an image activity detection unit 204. The image activity detection unit 204 may be implemented by a hardware circuit, and may perform image activity detection on an image of an eye of a wearer, such as an activity condition of an eyeball, captured by the first image capturing unit 101 (step 601), and when the image activity detection unit is in operation (the determination result of step 601 is negative, that is, when no image activity is detected yet), the processing unit 201 (including the posture determination unit 2011 and the audio processing unit 2012) may not operate or operate in a low power consumption mode (for example, operate at a very low operating frequency, 1mhz,2mhz,6.5mhz,13mhz, and the like), and the eyeball tracking unit 202 may be turned off to save power consumption. Thereby, the data amount of the video content processed thereby can be reduced while further reducing the power consumption, so that the image motion can be detected more quickly.

When the image motion detection unit 204 detects image motion (the determination result in step 601 is affirmative, indicating that image motion is detected), the image motion detection unit 204 outputs a shutdown instruction signal 602 to the present unit, and shuts down the operation of the image motion detection unit 204, thereby reducing the power consumption of the unit. In case of detecting image activity, the image activity detection unit 204 may further output a turn-on indication signal (603 c) to the eye tracking unit 202 to turn on the eye tracking unit 202 or increase its operating frequency to achieve accurate real-time detection of eye activity. In some embodiments, the image activity detection unit 204 may further output an indication signal to the first image capturing unit 101 to switch the first image capturing unit to the second frame rate and the second resolution (step 603 a), output an indication signal to the second image capturing unit 102 to switch the second image capturing unit to the second frame rate and the second resolution (step 603 b), and output an indication signal to the processing unit 201 to switch the processing unit to the second operating frequency (step 603 d), in case an image activity is detected. Therefore, finer and more real-time shooting of the eye images can be achieved, and corresponding requirements can be met. For example, when the wearer needs to register with the pupil image, the wearer needs to capture a finer eye image with the first imaging unit 101, and can operate at a higher second resolution.

The above description is intended to be illustrative, and not restrictive. For example, the above-described examples (or one or more versions thereof) may be used in combination with each other. For example, other embodiments may be utilized by those of ordinary skill in the art upon reading the foregoing description. In addition, in the foregoing detailed description, various features may be grouped together to streamline the disclosure. This should not be interpreted as an intention that a disclosed feature not claimed is essential to any claim. Rather, inventive subject matter may lie in less than all features of a particular disclosed embodiment. Thus, the following claims are hereby incorporated into the detailed description as examples or embodiments, with each claim standing on its own as a separate embodiment, and it is contemplated that these embodiments may be combined with each other in various combinations or permutations. The scope of the invention should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. A smart eyewear, the smart eyewear comprising:

a first imaging unit configured to capture an image of an eye of a wearer;

a second image pickup unit configured to pick up an image of an external environment;

an attitude determination unit configured to: determining a pose-related parameter of the smart glasses based on at least a series of images of the external environment captured by the second camera unit;

an eye tracking unit configured to: determining a gaze point of a wearer based on an image of an eye captured by the first imaging unit, determining an imaging focus of the second imaging unit based on the gaze point of the wearer;

an image motion detection unit configured to perform image motion detection on the image of the eye; when image activity is detected, the eye tracking unit is turned on and instructs the first imaging unit to operate at a second resolution and/or a second frame rate, wherein the second resolution is higher than a resolution when no image activity is detected and the second frame rate is higher than a frame rate when no image activity is detected.

2. The smart eyewear of claim 1, further comprising:

an audio processing unit configured to: processing a 3D audio signal to be played based on the attitude related parameters so as to enable a perception sound source of the 3D audio signal to be stable; and

a speaker configured to play the processed 3D audio signal.

3. The smart eyewear of claim 2, further characterized in that the smart eyewear further comprises at least one microphone configured to capture audio signals;

the audio processing unit is further configured to: processing the audio signals collected by the at least one microphone based on the orientation of the wearer's gaze point relative to the wearer's left and right ears to obtain 3D recorded signals.

4. The smart eyewear of claim 3, wherein the at least one microphone is further configured to: beam shaping is performed in the direction of the wearer's gaze point.

5. The smart glasses according to claim 1, wherein the image activity detection unit is implemented by a hardware circuit.

6. The smart eyewear of claim 1 or claim 2, further comprising an Inertial Measurement Unit (IMU) configured for detecting pose-related parameters of the smart eyewear.

7. The smart eyewear of claim 6, further comprising a calibration unit configured to: calibrating a zero offset of the Inertial Measurement Unit (IMU) based on a series of images of the external environment captured by the second camera unit.

8. A signal processing method of smart glasses, the signal processing method comprising:

capturing an image of a wearer's eye;

shooting an image of an external environment;

determining pose-related parameters of the smart glasses based at least on a series of captured images of the external environment;

determining a gaze point of a wearer based on an image of an eye of the wearer captured, determining an imaging focus of an image capturing an external environment based on the gaze point of the wearer;

performing image motion detection on the image of the eye of the photographing wearer; in case image activity is detected, eye tracking is turned on and images of the wearer's eyes are taken at a second resolution and/or a second frame rate, and the second resolution is higher than the resolution when no image activity is detected, the second frame rate is higher than the frame rate when no image activity is detected.

9. The signal processing method of claim 8, further comprising:

processing a 3D audio signal to be played based on the posture-related parameters so that a perceptual sound source of the 3D audio signal remains stable; and

and playing the processed 3D audio signal.

10. The signal processing method of claim 8, further comprising:

acquiring an audio signal with at least one microphone;

processing the audio signals collected by the at least one microphone based on the orientation of the wearer's gaze point relative to the wearer's left and right ears to obtain 3D recorded signals.

11. The signal processing method of claim 10, further comprising:

beamforming the at least one microphone in a direction of the wearer's gaze point.

12. The signal processing method according to claim 8 or 9, characterized in that the signal processing method further comprises:

detecting a pose-related parameter of the smart eyewear with an Inertial Measurement Unit (IMU).

13. The signal processing method of claim 12, further comprising:

calibrating a zero offset of the Inertial Measurement Unit (IMU) based on a series of images of an external environment.