TW201427418A - Sensing apparatus and sensing method - Google Patents

Abstract

A sensing apparatus includes an infrared light generating device, an image sensing unit, a processing circuit and a control circuit. The image sensing unit is arranged for detecting a first infrared light signal reflected from an object to generate a first sensing signal when the infrared light generating device is activated, and detecting a second infrared light signal reflected from the object to generate a second sensing signal when the infrared light generating device is deactivated. The processing circuit is coupled to the image sensing unit, and is arranged for generating three-dimensional image information according to at least the first sensing signal and the second sensing signal. The three-dimensional image information includes depth information. The control circuit is arranged for control activation and deactivation of the infrared light generating device, sensing operations of the image sensing unit, and signal processing operations of the processing circuit.

Description

Sensing device and sensing method

The present invention relates to a sensing device, and more particularly to a sensing device that detects an image through an infrared light sensing mechanism to generate three-dimensional image information and related sensing methods.

The traditional image sensor captures two-dimensional (2D) images and cannot determine the image depth of the sensed object. Therefore, the captured image is monotonous and easily distorted.

In addition, traditional mobile devices (such as smart phones, tablet computers, or notebook computers) are equipped with image sensors and other types of sensors to detect object images or surrounding environments. News. For example, a user facing camera with an image sensor can be used to capture an image, wherein an ambient light sensor (ALS) and a proximity sensor are installed in the vicinity of the user's camera lens ( Proximity sensor (PS) and infrared emitter (IR emitter). The ambient light sensor adjusts the brightness of the display screen according to the intensity of the ambient light. In addition, the ambient light sensor can also be used to turn on the flash when the user's camera lens is enabled to capture the image. The proximity sensor and the infrared light emitter are used to detect whether the mobile device is close to the user's ear or to detect whether the mobile device is placed in the backpack. When the mobile device is detected to be attached to the user's ear or placed in the backpack, the proximity sensor causes the mobile device to turn off the backlight and the touch sensor to extend the battery life of the mobile device and reduce Mistaken situation. However, various sensors in the mobile device are respectively provided with different modules or integrated circuits (ICs), which causes cost and size. increase.

Therefore, there is a need for a sensing device that can capture more realistic image information (eg, three-dimensional (3D) image information) and integrate multiple sensors into a single module or a single integrated circuit.

In view of the above, one of the objects of the present invention is to provide a sensing device that detects an image through an infrared light sensing mechanism to generate three-dimensional image information and related sensing methods to solve the above problem.

Another object of the present invention is to provide a three-dimensional image sensing device that integrates multiple sensors in a single module or a single integrated circuit to integrate an optical-mechanical system and/or an optical mechanical system (optical). -mechanical-electrical system), which reduces costs and improves performance.

In accordance with an embodiment of the present invention, a sensing device is disclosed. The sensing device comprises an infrared light generating component, an image sensing unit, a processing circuit and a control circuit. The image sensing unit is configured to detect a first infrared light signal reflected from an object to generate a first sensing signal when the infrared light generating component is turned on, and to detect when the infrared light generating component is turned off. A second infrared light signal reflected from one of the objects is detected to generate a second sensing signal. The processing circuit is coupled to the image sensing unit for generating a three-dimensional image information of the object based on the first sensing signal and the second sensing signal, wherein the three-dimensional image information includes a depth information. The control circuit is coupled to the infrared light generating component, the image sensing unit, and the processing circuit for controlling the opening and closing of the infrared light generating component, the sensing operation of the image sensing unit, and the processing circuit Signal processing operation.

In accordance with an embodiment of the present invention, a sensing method is disclosed. The sensing method comprises the steps of: enabling an infrared light generating component to detect a first infrared light signal reflected from an object to generate a first sensing signal; and disabling the infrared light generating component to detect Generating a second sensing signal from the second infrared light signal of the object to generate a second sensing signal; and generating a three-dimensional image of the object based on at least a signal difference between the first sensing signal and the second sensing signal Information, wherein the 3D image information includes a depth information.

The sensing device and the sensing method provided by the invention can obtain the depth information/three-dimensional image information of the object to present the image more realistically. In addition, the sensing device provided by the present invention integrates image sensing, ambient light sensing (including ambient color sensing, ambient color temperature sensing), proximity sensing, infrared light emission, and gesture recognition into a single module/single unit. Among the integrated circuits, the cost can be greatly reduced and the system performance can be improved.

100‧‧‧Image Processing System

110‧‧‧ lens

120‧‧‧Sensing device

130‧‧‧Image Processing Function Block

132‧‧‧Digital Image Processor

134‧‧‧Image Compressor

136‧‧‧Transport interface

138‧‧‧ storage device

212‧‧‧Infrared light generating components

222, 722, 822‧‧‧ image sensing unit

224‧‧‧ Visible light detection unit

226‧‧‧ Dark Sensing Unit

228‧‧‧Infrared light detection unit

232‧‧‧Processing circuit

233‧‧‧Related double sampling circuit

234‧‧Amplifier

235‧‧‧Addition Circuit

236‧‧‧ analog digital conversion circuit

237‧‧‧Dark-order compensation circuit

238‧‧‧Digital processing circuit

239‧‧‧ Serial interface

242‧‧‧Control circuit

243‧‧‧Timing controller

244‧‧‧Infrared LED Driver

245‧‧‧Voltage regulator

246‧‧‧ Clock Generator

247‧‧‧Control register

248‧‧‧Power Control Circuit

249‧‧‧ interrupt circuit

522_VR, 722_VR‧‧‧ Visible light sensing components

522_IR, 722_IR‧‧‧ Infrared light sensing components

522_R, 722_R‧‧‧ red light sensing components

522_G, 722_G‧‧‧ Green light sensing components

522_B, 722_B‧‧‧Blu-ray sensing components

822_VI‧‧‧Infrared light passes through and visible light passes through the sensing element

822_R‧‧‧Infrared light passes through and red light passes through the sensing element

822_G‧‧‧Infrared light passes through and the green light passes through the sensing element

822_B‧‧‧Infrared light passes through and blue light passes through the sensing element

LED_A, LED_C, GND, RSTB, ADRSEL, D[9:0], PCLK, HSYNC, VSYNC, SCL, SDA, INTB, PWDN, MCLK, IR_LED, VDD‧‧‧ contacts

S_R1, S_R2‧‧‧ infrared light signal

S_VR‧‧‧ visible light reflection signal

S_C1, S_C2‧‧‧ control signals

DR1, DR2, DR3‧‧‧ sensing signals

R, G, B‧‧ subpixels

D‧‧‧dark pixels

I‧‧‧Infrared light detector

ST‧‧‧Substrate

DL‧‧‧ dielectric layer

D_R, D_G, D_B, D_IR, D_RI, D_GI, D_BI‧‧‧Photodetector

F_R‧‧‧ red filter

F_G‧‧‧Green Filter

F_B‧‧‧Blue filter

F_IRP‧‧‧Infrared light pass filter

F_IRC‧‧‧Infrared cut-off filter

F_RI‧‧‧Infrared light passes through and the red light passes through the filter

F_GI‧‧‧Infrared light passes through and the green light passes through the filter

F_BI‧‧‧Infrared light passes through and the blue light passes through the filter

T_R, T_G, T_B, T_IRC, T_TRP, T_IRB‧‧‧ penetration curves

T1, T2, T3, T4‧‧‧ time points

910, 912, 914, 916, 918, 920, 922, 924, 926, 928, 930, 932, 934, 936, 938, 1010, 1012, 1014, 1016, 1018, 1020, 1022, 1024, 1026, 1028, 1030, 1032, 1110, 1112, 1114, 1116, 1118, 1120, 1122, 1124, 1126, 1128, 1130, 1132, 1134, 1210, 1212, 1214, 1216, 1218, 1220, 1222, 1224, 1226, 1228, 1230, 1232, 1234‧‧ steps

1 is a block diagram showing an embodiment of an image processing system of the present invention.

Fig. 2 is a schematic view showing a practical example of the sensing device shown in Fig. 1.

Figure 3 is a signal timing diagram of the control signals of the infrared light generating element and the image sensing unit shown in Fig. 2.

FIG. 4 is a schematic diagram showing an example of an image sensing unit, a visible light detecting unit, a dark sensing unit, and an infrared light detecting unit shown in FIG. 2 .

Fig. 5 is a cross-sectional view showing the sensing element structure of the image sensing unit shown in Fig. 4.

Fig. 6 is a graph showing the relationship between the wavelength of incident light and the transmittance of the filter shown in Fig. 5.

Fig. 7 is a cross-sectional view showing another embodiment of the component structure of the image sensing unit shown in Fig. 4.

Fig. 8 is a cross-sectional view showing another embodiment of the component structure of the image sensing unit shown in Fig. 4.

Figure 9 is a flow chart of an embodiment of an image sensing operation of the present invention.

Figure 10 is a flow chart of one embodiment of an ambient light sensing operation of the present invention.

Figure 11 is a flow chart of one embodiment of a proximity sensing operation of the present invention.

Figure 12 is a flow chart of an embodiment of the gesture recognition operation of the present invention.

In order to present the image of the object more realistically, the present invention respectively reflects the infrared light reflected by the object under the state in which the infrared light generating element is in an open state (ie, emitting infrared light) and a closed state (ie, no infrared light is emitted). The signal is detected to obtain a corresponding sensing signal, and the obtained sensing signal is processed to remove/reduce the influence of the ambient light on the sensing signal, thereby obtaining accurate object depth information and/or three-dimensional image information. .

Please refer to FIG. 1 , which is a block diagram of an embodiment of an image processing system of the present invention. As can be seen from FIG. 1 , the image processing system 100 includes a lens 110 , a sensing device 120 , and an image processing function block 130 . Taking the image of the user's hand as an example, the lens 110 can collect and transmit the light reflected by the hand to the sensing device 120. The sensing device 120 can then generate image information according to the received optical signal to the image processing function block. 130. The image processing function block 130 includes a digital image processor 132, an image compressor 134, a transmission interface 136 (eg, a parallel interface or a serial interface), and a storage device 136. (For example, storing a complete image frame). Since the skilled artisan should be aware of the operational details of the image information generated by the sensing device 120 via the digital image processor 132, the image compressor 134, the transmission interface 136, and the storage device 136, further description of the image processing function block 130 is provided. I won't go into details here.

It should be noted that the sensing device 120 is an integrated sensing device, and more specifically, the sensing device 120 can perform image sensing, ambient light sensing (including ambient color sensing, ambient color temperature sensing), Functions such as proximity sensing, temperature sensing, object position detection, and/or gesture recognition are integrated into a single integrated circuit (or a single module). In addition, the sensing device 120 can capture a three-dimensional image of the user's hand to provide a more realistic output image. Further explanation is as follows.

Please refer to FIG. 2 , which is a schematic diagram of a practical example of the sensing device 120 shown in FIG. 1 . In this implementation example, the sensing device 120 can include, but is not limited to, an infrared light generating component 212 (eg, an infrared light-emitting diode (IR LED)), an image sensing unit. 222, a visible light detecting unit 224, a dark sensing unit 226, an infrared light detecting unit 228, a processing circuit 232, a control circuit 242, and a temperature sensor 252. In this implementation example, the pad VDD is coupled to a power source (not shown in FIG. 2), the contact GND is coupled to a ground voltage (not shown in FIG. 2), and infrared light is generated. The component 212 is coupled between the contacts LED_A and LED_C, the contact RSTB is used to receive a reset signal (not shown in FIG. 2), and the contact ADRSEL is used to receive the address selection signal (not shown in Figure 2). In addition, the visible light detecting unit 224 is disposed at the periphery of the image sensing unit 222 for performing at least one of ambient light sensing and color sensing; the dark sensing unit 226 is configured. The image sensing unit 222 is configured to generate a reference signal (not shown in FIG. 2) for dark/black level compensation; and the infrared light detecting unit 228 is disposed in the image sense. The periphery of the measuring unit 222 is configured to perform at least one of proximity sensing, object position detection and gesture detection. In this embodiment, the dark sensing unit 226 is disposed on the periphery of the visible light detecting unit 224, and the infrared detecting unit 228 is disposed on the periphery of the dark sensing unit 226. However, this is for illustrative purposes only, and is not used. It is intended to be a limitation of the present invention. For example, it is also feasible to arrange the infrared light detecting unit 228 between the visible light detecting unit 224 and the image sensing unit 222.

The control circuit 242 is coupled to the infrared light generating component 212 (via the contact IR_LED), the image sensing unit 222, the visible light detecting unit 224, the dark sensing unit 226, the infrared light detecting unit 228, and the processing circuit 232. The sensing unit 222 and the processing circuit 232 are also coupled to each other. In addition, the control circuit 242 can be used to control the infrared light generating component 212, the image sensing unit 222, the visible light detecting unit 224, the dark sensing unit 226, the infrared light detecting unit 228, and the processing circuit 232. Operation. When the infrared light generating element 212 is turned on (ie, emitting infrared light), the image sensing unit 222 can detect the first infrared light signal S_R1 reflected from an object (for example, the hand shown in FIG. 1), and A first sensing signal DR1 (eg, a photocurrent signal) is generated. The first infrared light signal S_R1 received by the image sensing unit 222 is mainly generated because the object reflects the infrared light emitted by the infrared light generating element 212. Therefore, the energy of the first infrared light signal S_R1 can be used to determine the The distance between the object and the image sensing unit 222, that is, the first sensing signal DR1 generated by the image sensing unit 222 includes the distance information between the object and the sensing device 120.

However, the first sensing signal DR1 may further include infrared light information from the environment (for example, the reflected signal generated by the infrared light in the object reflection environment), and therefore, the control circuit 242 may further turn off the infrared light generating element 212. (ie, the infrared light is not emitted), and the image sensing unit 222 detects the second infrared light signal S_R2 reflected from the object to generate a second sensing signal DR2, wherein the second sensing signal DR2 can be regarded as The sensing result obtained by sensing the reflected signal generated by the infrared light in the object reflection environment. Next, the processing circuit 232 can generate one of the three-dimensional image information of the object according to the first sensing signal DR1 and the second sensing signal DR2. For example, the 3D image information may include a depth information, wherein the depth information may indicate a distance between the object and a reference point (or a reference surface) (eg, a certain surface on the object) The distance between the point and the sensing device 120 is either a change in the image depth of the object (for example, a three-dimensional grayscale image of the object).

For example, the processing circuit 232 can generate the depth information of the object according to the signal difference between the first sensing signal DR1 and the second sensing signal DR2. More specifically, the processing circuit 232 can directly sense the first The signal signal DR1 is subtracted from the second sensing signal DR2 to remove/reduce the influence of the ambient light on the sensing result, thereby obtaining accurate object depth information. However, this is for illustrative purposes only and is not intended to be a limitation of the invention. In a design change, the processing circuit 232 can also participate The first sensing signal DR1 is adjusted according to the second sensing signal DR2, and then the first sensing signal DR1 is processed to generate the depth information.

The image sensing unit 222 can detect the visible light reflection signal S_VR reflected from the object to generate a third sensing signal DR3. Therefore, the third sensing signal DR3 includes the color information of the object. The processing circuit 232 can generate the depth information (for example, a three-dimensional gray-scale image) of the object according to the first sensing signal DR1 and the second sensing signal DR2. Therefore, the processing circuit 232 can be configured according to the first sensing signal DR1. The second sensing signal DR2 and the third sensing signal DR3 are used to generate three-dimensional image information (ie, color stereoscopic image information) of the object. The implementation of the image sensing unit that can simultaneously detect infrared light and visible light is described later.

In an implementation example, the enabling timing of the infrared light generating component 212 and the image sensing unit 222 can be controlled to obtain better signal quality. Please refer to Figure 3 together with Figure 2. FIG. 3 is a signal timing diagram of the control signals of the infrared light generating element 212 and the image sensing unit 222 shown in FIG. In this implementation example, the control circuit 242 can generate a plurality of control signals S_C1 and S_C2 to respectively control the on/off of the infrared light generating element 212 and the sensing operation of the image sensing unit 222. As can be seen from FIG. 3, after the infrared light generating element 212 is enabled, the control circuit 242 enables the image sensing unit 222 to receive the first infrared light signal S_R1 (eg, time point T1), and the infrared light generating element 212. After the image sensing unit 222 is enabled (eg, after integrating the first sensing signal DR1 for a predetermined time), the control circuit 242 can simultaneously disable the image sensing unit 222 and the infrared light generating component 212 (eg, time). Point T2). In a design change, the control circuit 242 can also enable the infrared light generating component 212 and the image sensing unit 222 at the same time. In short, when the image sensing unit 222 performs sensing, the infrared light generating element 212 is in an enabled state (ie, emits infrared light) to ensure that the received first infrared light signal S_R1 is mainly from infrared. The infrared light emitted by the light generating element 212.

When the infrared light generating element 212 is turned off, the control circuit 242 can enable the image sensing unit 222 to receive the second infrared light signal S_R2 (eg, time point T3) for a predetermined time (eg, the second sensing signal SR2) After the integration time), the image sensing unit 222 is disabled (for example, time point T4). In this implementation example (but the invention is not limited thereto), when the infrared light generating component 212 is turned off, the control circuit 242 can enable the image sensing unit 222 to detect the visible light reflecting signal S_VR reflected from the object. The third sensing signal DR3 is generated (for example, the time point T3 to the time point T4), so that the image sensing unit 222 can simultaneously perform image depth information and image color information detection.

In addition, when the control circuit 242 disables the image sensing unit 222 (ie, during the turned off period), the control circuit 242 can enable the visible light detecting unit 224 (ie, in an on state (turned on) And performing at least one of ambient light sensing or color sensing to optimize power consumption of the integrated sensing device 120. Similarly, when the control circuit 242 disables the image sensing unit 222 (ie, during the off state), the control circuit 242 can enable the infrared light detecting unit 228 (ie, in an on state) to perform proximity proximity. At least one of measurement, object position detection and gesture detection to optimize power consumption of the integrated sensing device 120.

In the implementation example shown in FIG. 2, the control circuit 242 can include, but is not limited to, a timing controller 243, an IR LED driver 244, a voltage regulator 245, and a clock. The generator 246, a control register 247, a power control circuit 248, and an interrupt circuit 249. The timing controller 243 can be used to generate the control signal S_C1 to control the infrared light emitting diode driver 244, and generate the control signal S_C2 to control the image sensing unit 222. The infrared light emitting diode driver 244 can enable/disable the infrared light generating element 212 according to the control signal S_C1. The clock generator 246 can receive an external clock (eg, a primary clock; not shown in FIG. 2) from the contact MCLK. The power control circuit 248 can receive a power source from the contact PWDN. Control signals (not shown in Figure 2) to control the power mode of operation. The interrupt circuit 250 can receive an interrupt signal from the contact INTB (not shown in FIG. 2). Since the skilled artisan should be able to understand the operational details of the circuit components included in the control circuit 242, further description will not be repeated here.

In addition, the processing circuit 232 can include, but is not limited to, a correlated double sampling circuit (CDS circuit) 233, an amplifier 234, an adding circuit 235, an analog-to-digital converter 236, and a dark-order compensation circuit. (dark/black level compensation circuit) 237, a digital processing circuit 238, and a serial interface (serial interface, serial I/F) 239 (for example, two wire inter-integrated circuit, two-wire I ² C)). The signals output by the image sensing unit 222 (for example, the first sensing signal DR1 and the second sensing signal DR2) can be connected to the correlated double sampling architecture of the programmable double sampling circuit 233 and the amplifier 234 with programmable gain setting. deal with. The summing circuit 235 can add the output of the amplifier 234 and the output of the dark-order compensation circuit 237 to an analog signal (that is, the output of the adding circuit 235), and the analog-to-digital converter 236 then converts the analog signal into one. The digital signal (i.e., the output of the analog to digital converter 236), wherein the output of the dark matrix compensation circuit 237 is generated based on the digital signal. The digital processing circuit 238 can perform subsequent processing on the digital signal (eg, threshold comparison, hysteresis detection, and other detection algorithms) by a plurality of contacts D[9:0], PCLK, HSYNC, and VSYNC. The data is transmitted to the image processing function block 130 shown in FIG. The serial interface 239 can be used for synchronous serial communication between chips, and is coupled to a contact SCL corresponding to a serial clock line (SCL) (not shown in FIG. 2) and a corresponding serial array. The contact SDA of the serial data line (SDA) (not shown in Figure 2). Since the skilled artisan should be able to understand the operational details of the various circuit components in the processing circuit 232, further description will not be repeated here.

Please refer to Figure 4 and Figure 5 together with Figure 2. 4 is an image sensing unit 222, a visible light detecting unit 224, a dark sensing unit 226, and an infrared light detecting unit shown in FIG. 228 is a schematic diagram of an example of implementation, and FIG. 5 is a cross-sectional view of the sensing element architecture of image sensing unit 222 shown in FIG. In this implementation example, the image sensing unit 222 can be implemented by an array of M columns and N rows of sensors (for example, an active pixel sensor array (APS array)) (eg, Figure 4), where M and N are positive integers. In addition, as can be seen from FIG. 5 , the image sensing unit 222 can include at least one infrared light sensing element 522_IR and at least one visible light sensing element 522_VR. The infrared light sensing component 522_IR is coupled to the processing circuit 232 for detecting the first infrared light signal S_R1 and the second infrared light signal S_R2 to respectively generate the first sensing signal DR1 and the second sensing signal DR2; The measuring component 522_VR is coupled to the processing circuit 232 for detecting the visible light reflecting signal S_VR to generate the third sensing signal DR3. In this implementation example, the third sensing signal DR3 may include a red light conversion signal, a green light conversion signal, and a blue light conversion signal, respectively, and the red light sensing element 522_R included in the visible light sensing element 522_VR, A green light sensing element 522_G and a blue light sensing element 522_B detect the visible light reflecting signal S_VR.

In practice, a plurality of photodetectors D_R, D_G, D_B and D_IR may be disposed on a substrate ST, and a dielectric layer DL is deposited on the plurality of photodetectors D_R, D_G, D_B and D_IR, and then A red filter F_R, a green filter F_G, a blue filter F_B, and an infrared light passing filter F_IRP are disposed/coated on the dielectric layer DL to respectively realize the red light sensing element 522_R, The green light sensing element 522_G, the blue light sensing element 522_B, and the infrared light sensing element 522_IR.

In this implementation example, each filter can be implemented by a thin film filter (but the invention is not limited thereto), and in addition, between the wavelength of the incident light and the transmittance of each filter. See Figure 6 for the relationship. It can be seen from FIG. 6 that the visible light can be filtered into three bands by the red filter F_R, the green filter F_G and the blue filter F_B, which respectively correspond to the penetration curves T_R, T_G and T_B, and the infrared light can be It is filtered by the infrared light through the filter F_IRP into a band corresponding to the penetration curve T_IRP. Therefore, when the image sensing unit 222 receives the visible light reflection signal S_VR, the light detector D_R can detect the visible light reflection signal S_VR through the red color filter F_R to generate the red light conversion signal (for example, a current signal), and the light The detector D_G can detect the visible light reflection signal S_VR through the green filter F_G to generate the green light conversion signal, and the light detector D_B can detect the visible light reflection signal S_VR through the blue filter F_B to generate the Blu-ray conversion signal. In addition, the photodetector D_IR can pass the infrared light through the filter F_IRP to detect the first infrared light signal S_R1 and the second infrared light signal S_R2 to respectively generate corresponding infrared light conversion signals (ie, first sensing) Signal DR1 and second sensing signal DR2). The processing circuit 232 can generate the color three-dimensional image information of the object according to the generated infrared light conversion signal, the red light conversion signal, the green light conversion signal, and the blue light conversion signal.

The component structure of the above image sensing unit is for illustrative purposes only and is not intended to be a limitation of the present invention. In a design change, the component structure shown in FIG. 5 may further include a yellow filter and its corresponding photodetector (not shown in FIG. 5) to increase the vividness of the color. In another design change, cyan, magenta, yellow, and black filters (ie, printed four primary colors) and their corresponding photodetectors may be used instead of the red, green, and blue filters described above. And its corresponding photodetector. In other words, as long as the image sensing unit can simultaneously detect the visible light signal and the infrared light signal to generate the three-dimensional image information of the object, it falls within the scope of the present invention in accordance with the inventive spirit of the present invention.

Please note that a plurality of sub-pixels (for example, red light sensing element 522_R, green light sensing element, blue light sensing element 522_B, and infrared light sensing element 522_IR) shown in FIG. 5 should be understood by those skilled in the art. The sensor array shown in Fig. 2 has a plurality of arrangements (for example, stripe arrangement, triangle arrangement or square arrangement), and therefore will not be described again here. In addition, the penetration curve of the filter shown in FIG. 6 is for illustrative purposes only. For example, the penetration curve corresponding to the infrared light passing through the filter F_IRP may also be a band pass penetration curve T_IRB.

Please refer to FIG. 7 , which is a cross-sectional view showing another embodiment of the component structure of the image sensing unit 222 shown in FIG. 4 . The component structure of the image sensing unit 722 shown in FIG. 7 is based on the component structure of the image sensing unit 222 shown in FIG. 5, and the main difference between the two is the sensing component architecture shown in FIG. Also included is an infrared light cut filter F_IRC. More specifically, the component structure of the infrared light sensing component 722_IR included in the image sensing unit 722 is substantially the same as the component structure of the infrared light sensing component 722_IR shown in FIG. 5, and the image sensing unit 722 includes The visible light sensing element 722_VR may be an infrared light cutoff and the visible light passes through the sensing element, and may include a red light sensing element 722_R, a green light sensing element 722_G, and a blue light sensing element 722_B. Compared with the red light sensing element 522_R/green light sensing element 522_G/blue light sensing element 522_B shown in FIG. 5, since the red light sensing element 722_R/green light sensing element 722_G/blue light sensing element The 722_B also has an infrared cut-off filter F_IRC, so that the signal in the infrared band can be filtered to enhance the conversion signal generated by the corresponding photodetector (for example, the red-light conversion signal, the green-light conversion signal, and the like). The quality of the blue light conversion signal, wherein the relationship between the wavelength of the incident light corresponding to the infrared cut filter F_IRC and the transmittance can be expressed by the penetration curve T_IRC shown in FIG. Since the skilled artisan can understand the operation details of the image sensing unit 722 and the design changes thereof (for example, filters containing other colors) after reading the related descriptions of FIGS. 1 to 6 , further explanation is given here. I won't go into details.

In a design change, the image sensing unit 222 shown in FIG. 2 may also include the component structure of the visible light sensing element 522_VR shown in FIG. 5 and the visible light sensing element 722_VR shown in FIG.

Please refer to Figure 8 together with Figure 2. Fig. 8 is a cross-sectional view showing another embodiment of the component structure of the image sensing unit 222 shown in Fig. 4. The component structure of the image sensing unit 822 shown in FIG. 8 is based on the component structure of the image sensing unit 222 shown in FIG. 5, and both The main difference is that the sensing element architecture shown in Figure 8 uses a dual band bandpass filter. More specifically, the image sensing unit 822 can include at least one infrared light passing through and the visible light passing through the sensing component 822_VI, which is coupled to the processing circuit 232 for detecting the first infrared light signal S_R1 and the second infrared light signal. The S_R2 generates the first sensing signal DR1 and the second sensing signal DR2, and the visible light reflecting signal S_VR to generate the third sensing signal DR3. In this implementation example, the third sensing signal DR3 may include a red light conversion signal, a green light conversion signal, and a blue light conversion signal, respectively, which are passed by the infrared light and the visible light passes through the sensing element 822_VI. The red light is generated by the sensing element 822_R, an infrared light passing through, and the green light passing through the sensing element 822_G and an infrared light and the blue light detecting the visible light reflecting signal S_VR through the sensing element 822_B. In addition, the infrared light passes through and the red light passes through the sensing element 822_R, the infrared light passes through, and the green light passes through the sensing element 822_G and the infrared light passes through and the blue light passes through at least one of the sensing element 822_B to detect the first infrared light signal S_R1 And the second infrared light signal S_R2 to generate the first sensing signal DR1 and the second sensing signal DR2, respectively.

In practice, a plurality of photodetectors D_RI, D_GI, and D_BI may be disposed on a substrate ST, and a dielectric layer DL is deposited on the plurality of photodetectors D_RI, D_GI, and D_BI, and then the dielectric layer DL is deposited. Setting/coating an infrared light through and red light passes through the filter F_RI, infrared light passes through and the green light passes through the filter F_GI and the infrared light passes through and the blue light passes through the filter F_BI to respectively make infrared light pass and red light The sensing element 822_R passes through the infrared light and the green light passes through the sensing element 822_G and the infrared light passes through and the blue light passes through the sensing element 822_B. In this example, the penetration curve of the infrared light passing through and the red light passing through the filter F_RI may be a superposition of the penetration curves T_R and T_IRB shown in FIG. 6, and the infrared light passes through and the green light passes through the filter. The penetration curve corresponding to the slice F_GI may be the superposition of the penetration curves T_G and T_IRB shown in FIG. 6, and the penetration curve corresponding to the infrared light passing through and the blue light passing filter F_BI may be shown in FIG. The superposition of the penetration curve T_B and T_IRB.

When the image sensing unit 822 receives the first infrared light signal S_R1, the second infrared light signal S_R2, and the visible light reflection signal S_VR shown in FIG. 2, the light detector D_RI can pass the infrared light and pass the red light through the filter. The F_RI detects the visible light reflection signal S_VR to generate the red light conversion signal (for example, a current signal); the light detector D_GI can pass the infrared light and the green light passes through the filter F_GI to detect the visible light reflection signal S_VR to generate The green light conversion signal; and the light detector D_BI can pass the infrared light and the blue light passes through the filter F_BI to detect the visible light reflection signal S_VR to generate the blue light conversion signal. In addition, each of the photodetectors can detect the first infrared light signal S_R1 and the second infrared light signal S_R2 through the corresponding dual band pass filters to respectively generate corresponding infrared light conversion signals (for example, the first The sensing signal DR1 and the second sensing signal DR2). The processing circuit 232 can generate the color three-dimensional image information of the object according to the generated infrared light conversion signal, the red light conversion signal, the green light conversion signal, and the blue light conversion signal.

Please refer to Figure 4 together with Figure 2. As can be seen from FIG. 4, the visible light detecting unit 224 can include a plurality of pixels (composed of the red sub-pixel R, the green sub-pixel G, and the blue sub-pixel B). In a practical example, the red sub-pixel R, the green sub-pixel G, and the blue sub-pixel B may respectively adopt the red light sensing element 522_R, the green light sensing element 522_G, and the blue light sensing element 522_B shown in FIG. 5 . The component architecture. In another implementation example, the red sub-pixel R, the green sub-pixel G, and the blue sub-pixel B may respectively adopt the red light sensing element 722_R, the green light sensing element 722_G, and the blue light sensing element shown in FIG. 7 . Component architecture of 722_B. In another implementation example, in order to determine the coefficient of illumination calculation, the visible light detecting unit 224 may include a first pixel (not shown in FIG. 4) having an infrared light cut filter and a visible light sensitivity having attenuation. a second pixel (not shown in FIG. 4), wherein the first pixel detects visible light to obtain a first visible light sensing signal, and the second pixel detects a primary infrared light spectrum to obtain a second Visible light sensing signal. The processing circuit 232 can determine the coefficient of the illuminance calculation according to the signal ratio between the first visible light sensing signal and the second visible light sensing signal. In addition, the processing circuit 232 can also adjust image information (eg, brightness) according to the sensing result of the visible light detecting unit 224.

The infrared light detecting unit 228 can include a plurality of IR detectors (labeled "I"). In the case where the infrared light detecting unit 228 is applied to the proximity sensing operation, when the control circuit 242 enables the infrared light generating element 212, the plurality of infrared light detectors are enabled, and the infrared light detecting unit 228 A first infrared light sensing signal is generated. When the control circuit 242 disables the infrared light generating component 212, the plurality of infrared light detectors are still in an on state. Therefore, the infrared light detecting unit 228 can generate a first Two infrared light sensing signals. The signal level difference between the first infrared light sensing signal and the second infrared light sensing signal can be output to the processing circuit 232 for proximity sensing. In the case where the infrared light detecting unit 228 is applied to the gesture recognition operation, when the control circuit 242 enables the infrared light generating element 212, the plurality of infrared light detectors can be enabled according to a specific order, wherein Only one infrared detector will be enabled during the time period. For example, in a first time period, the control circuit 242 can enable an infrared light detector, and sequentially disable and disable the infrared light generating component 212, so that the infrared light detector can respectively generate corresponding a first sensing signal and a second sensing signal are sent to the processing circuit 232, and the processing circuit 232 can obtain a signal level difference between the first sensing signal and the second sensing signal; The control circuit 242 can enable another infrared light detector, and the processing circuit 232 can obtain a corresponding signal level difference. By analogy, the processing circuit 232 can receive the sensing signals (signal level differences) of the plurality of infrared light detectors according to the specific order, and then recognize the gesture according to the relationship between the intensity of the sensing signals and the time. In addition, the processing circuit 232 can also recognize the gesture based on the relationship between the intensity of the sensing signal of the single infrared light detector and the time (ie, determining whether the object is far away or close).

Dark sensing unit 226 can include a plurality of dark pixels (labeled "D"). Since the sensing signals generated by the plurality of dark pixels are not generated by the illumination, the plurality of sensing signals obtained by the image sensing unit 222/visible light detecting unit 224/infrared light detecting unit 228 can be deducted from the plurality of sensing signals. The sensing signal generated by the dark pixel compensates the sensing signal obtained by the image sensing unit 222/visible light detecting unit 224/infrared light detecting unit 228.

In the embodiment shown in FIG. 4, the visible light detecting unit 224, the dark sensing unit 226, and the infrared light detecting unit 228 may each include a plurality of sensing elements (for example, a plurality of pixels (including red, green, and blue sub-pixels). a pixel, a plurality of dark pixels, and a plurality of infrared light detectors, wherein the plurality of sensing elements surround the image sensing unit 222 (ie, the sensor array) to more completely obtain the corresponding The image information of the visible range of the lens 110 shown in Fig. 1. In addition, since the visible light detecting unit 224/infrared light detecting unit 228 can have the same/similar sensing element structure as the image sensing unit 222, the same/similar process can be used to integrate multiple sensing functions. Sensing devices and reducing production costs.

In addition, the image sensing unit 222, the visible light detecting unit 224, and the infrared light detecting unit 228 can be independently enabled or disabled. The following briefly describes the flow of image sensing, ambient light sensing, proximity sensing, and gesture recognition. Please refer to Figure 9 together with Figure 2. FIG. 9 is a flow chart of an embodiment of an image sensing operation of the present invention, wherein the image sensing operation can be simply summarized as follows.

Step 910: Start.

Step 912: Select a sensing mode of the sensing device 120 (eg, image sensing, ambient light sensing, proximity sensing, gesture recognition, and temperature sensing mode).

Step 914: Set the integration time of the sensing signal.

Step 916: Enable the corresponding wafer (ie, the sensing device 120, or the wafer containing the sensing device 120).

Step 918: Set the sensing address of the image sensing unit 222 to the 0th column.

Step 920: The signal level of one of the image sensing units 222 is transmitted to the correlated double sampling circuit 233.

Step 922: Reset the sensing signal to transmit a reset level of the sensing signal to the correlated double sampling circuit 233.

Step 924: Output a level difference between the signal level and the reset level to the amplifier 234.

Step 926: The potential difference is amplified.

Step 928: Dark-level compensation circuit 237 is utilized to perform dark-order compensation.

Step 930: The analog digital converter 236 is used to convert an analog signal generated by the adding circuit 235 into a digital signal.

Step 932: The digital signal processing circuit 238 is used to process the digital signal, and output a digital data output.

Step 934: Increase the sensing address of the image sensing unit 222 by one column.

Step 936: Determine whether the sensing address of the image sensing unit 222 is the last column. If yes, go to step 938; otherwise, go to step 920.

Step 938: Read the next image frame.

In step 920, the integration time can be adjusted according to the sensitivity of different sensing elements to obtain a better sensing result. The above is an example of processing a single image frame. Since the skilled artisan will understand the operation details of each step shown in FIG. 9 after reading the related descriptions of FIG. 1 to FIG. 8, further description will not be repeated here.

Please refer to FIG. 10, which is a flowchart of an embodiment of an ambient light sensing operation (or color sensing operation) of the present invention, wherein the ambient light sensing process can be simply summarized as follows.

Step 1010: Start.

Step 1012: Select an ambient light sensing mode.

Step 1014: Set the integration time of the sensing signal.

Step 1016: Set the amplification gain.

Step 1018: Enable the corresponding wafer.

Step 1020: Pair a first pixel with a second pixel (for example, one of the red, green, and blue sub-pixels shown in FIG. 4) (for example, the red, green, and blue colors shown in FIG. 4 The other pixel of the color sub-pixel is detected to generate a first sensing signal and a second sensing signal, respectively.

Step 1022: Perform analog-to-digital conversion processing on the first sensing signal, and perform analog digital conversion processing on the second sensing signal.

Step 1024: Output the converted first sensing signal and the converted second sensing signal to a data buffer.

Step 1026: Read the data of the data register.

Step 1028: Determine whether to read the next data? If yes, go to step 1020; otherwise, go to step 1030.

Step 1030: Disable the corresponding wafer.

Step 1032: End.

In step 1026, the coefficient of illuminance calculation may be determined according to the ratio of the signal between the converted first sensing signal and the converted second sensing signal. In step 1028, if the ambient light sensing operation continues, then return to step 1020 to read the next data. Since the skilled artisan will understand the operation details of each step shown in FIG. 10 after reading the related descriptions of FIG. 1 to FIG. 9, further description will not be repeated here.

Please refer to FIG. 11 , which is a flowchart of an embodiment of the proximity sensing operation of the present invention, wherein the proximity sensing process can be simply summarized as follows.

Step 1110: Start.

Step 1112: Select the proximity sensing mode.

Step 1114: Set the integration time.

Step 1116: Set the amplification gain.

Step 1118: Enable the corresponding wafer.

Step 1120: When one infrared light emitting diode is turned on, detecting one of the first sensing signals of one pixel (for example, a pixel labeled "I" shown in FIG. 4).

Step 1122: When the infrared light emitting diode is turned off, detecting a second sensing signal of the pixel.

Step 1124: performing analog-to-digital conversion processing on the first sensing signal, and performing analog-to-digital conversion processing on the second sensing signal.

Step 1126: Output the converted first sensing signal and the converted second sensing signal to a data buffer.

Step 1128: Read the data of the data register.

Step 1130: Determine whether to read the next data? If yes, go to step 1120; otherwise, go to step 1130.

Step 1132: Disable the corresponding wafer.

Step 1134: End.

In step 1128, the distance between the object and the sensing device can be determined according to the signal level difference between the converted first sensing signal and the converted second sensing signal. Since the skilled artisan will understand the operation details of each step shown in FIG. 11 after reading the related descriptions of FIG. 1 to FIG. 10, further description will not be repeated here.

Please refer to FIG. 12, which is a flowchart of an embodiment of the gesture recognition operation of the present invention, wherein the gesture recognition process can be simply summarized as follows.

Step 1210: Start.

Step 1212: Select a gesture recognition mode.

Step 1214: Set the integration time.

Step 1216: Set the amplification gain.

Step 1218: Select an infrared light emitting diode.

Step 1220: When the infrared light emitting diode is turned on, detecting one of the first sensing signals of one pixel (for example, a pixel labeled "I" shown in FIG. 4).

Step 1222: When the infrared light emitting diode is turned off, detecting a second sensing signal of the pixel.

Step 1224: Perform analog-to-digital conversion processing on the first sensing signal, and perform analog digital conversion processing on the second sensing signal.

Step 1226: Output the converted first sensing signal and the converted second sensing signal to a data buffer.

Step 1228: Read the data of the data register.

Step 1230: Determine whether to read the next data? If yes, go to step 1218; otherwise, go to step 1232.

Step 1232: Disable the corresponding wafer.

Step 1234: End.

In step 1228, the gesture can be identified according to the relationship between the intensity of the sensing signal and time. In this embodiment, the relationship between the intensity of the sensing signal of a single pixel and the time is taken as an example. In a design change, the relationship between the intensity of the individual signal and the time of the plurality of pixels can also be referenced at the same time to identify the position of the object/the gesture of the user. Since the skilled artisan will understand the operation details of each step shown in FIG. 12 after reading the related descriptions of FIG. 1 to FIG. 11, further description will not be repeated here.

It should be noted that the image sensing unit 222 shown in FIG. 4 has an element structure of an infrared photodetector/proximity sensor (for example, the red image shown in FIG. 5) The external light sensing component 522_IR), therefore, the sensing device 120 shown in FIG. 2 can also determine the position of the object and/or the gesture corresponding to the object according to the first sensing signal DR1 and the second sensing signal DR2. . That is to say, the processing circuit 232 shown in FIG. 2 can further identify the gesture corresponding to the object according to the relationship between the obtained depth information and time, for example, if the obtained depth information indicates the object and the sensing. The distance between the devices 120 is reduced, and a proximity gesture can be applied to the sensing device 120 on behalf of the user. Furthermore, since the image sensing unit 222 shown in FIG. 4 can detect the object image, the sensing device 120 shown in FIG. 2 can also directly determine the position of the object based on the obtained three-dimensional image information and/or Or identify the gesture corresponding to the object. In a practical example, a plurality of proximity sensors can also be directly embedded into the red, green and blue image sensor array to implement a multifunctional integrated sensing device. In addition, the image sensing unit 222 shown in FIG. 2 can also obtain 3D image information (for example, grayscale 3D image) through only the proximity sensor, and can recognize the position of the object and the corresponding gesture, in other words, the 4th. The sensor array shown in the figure can also contain only the component architecture of the proximity sensor.

As can be seen from the above, the image processing system of the present invention integrates an image sensor, a proximity sensor and an ambient light sensor, and utilizes a cross-function sensor (for example, for detecting images and identifying A proximity sensor for gestures and a visible light detector for detecting ambient light and color to improve system performance.

120‧‧‧Sensing device

212‧‧‧Infrared light generating components

222‧‧‧Image sensing unit

224‧‧‧ Visible light detection unit

226‧‧‧ Dark Sensing Unit

228‧‧‧Infrared light detection unit

232‧‧‧Processing circuit

233‧‧‧Related double sampling circuit

234‧‧Amplifier

235‧‧‧Addition Circuit

236‧‧‧ analog digital conversion circuit

237‧‧‧Dark-order compensation circuit

238‧‧‧Digital processing circuit

239‧‧‧ Serial interface

242‧‧‧Control circuit

243‧‧‧Timing controller

244‧‧‧Infrared LED Driver

245‧‧‧Voltage regulator

246‧‧‧ Clock Generator

247‧‧‧Control register

248‧‧‧Power Control Circuit

249‧‧‧ interrupt circuit

LED_A, LED_C, GND, RSTB, ADRSEL, D[9:0], PCLK, HSYNC, VSYNC, SCL, SDA, INTB, PWDN, MCLK, IR_LED, VDD‧‧‧ contacts

S_R1, S_R2‧‧‧ infrared light signal

S_VR‧‧‧ visible light reflection signal

S_C1, S_C2‧‧‧ control signals

DR1, DR2, DR3‧‧‧ sensing signals

Claims (18)

A sensing device includes: an infrared light generating component; an image sensing unit configured to detect a first infrared light signal reflected from an object when the infrared light generating component is turned on to generate a first sensing a signal, and when the infrared light generating component is turned off, detecting a second infrared light signal reflected from the object to generate a second sensing signal; a processing circuit coupled to the image sensing unit for at least And generating, according to the first sensing signal and the second sensing signal, the three-dimensional image information of the object, wherein the three-dimensional image information includes a depth information; and a control circuit coupled to the infrared light generating component, the image The sensing unit and the processing circuit are configured to control the opening and closing of the infrared light generating component, the sensing operation of the image sensing unit, and the signal processing operation of the processing circuit. The sensing device of claim 1, wherein the processing circuit generates the depth information of the object according to a signal difference between the first sensing signal and the second sensing signal. The sensing device of claim 1, wherein the image sensing unit further detects a visible light reflection signal reflected from the object to generate a third sensing signal, and the processing circuit is based on the first The sensing signal, the second sensing signal, and the third sensing signal generate the three-dimensional image information of the object. The sensing device of claim 3, wherein the image sensing unit detects the visible light reflection signal reflected from the object when the infrared light generating element is turned off to generate the third sensing signal. The sensing device of claim 3, wherein the image sensing unit comprises: at least one infrared light sensing component coupled to the processing circuit for detecting the first infrared light signal and the first a second infrared light signal for respectively generating the first sensing signal and the second sensing signal; and at least one visible light sensing component coupled to the processing circuit for detecting the visible light reflecting signal to generate the third Sensing signal. The sensing device of claim 5, wherein the third sensing signal comprises a red light conversion signal, a green light conversion signal, and a blue light conversion signal, and the at least one visible light sensing element comprises: a red light sensing component coupled to the processing circuit for detecting the visible light reflecting signal to generate the red light converting signal; a green light sensing component coupled to the processing circuit for detecting the visible light reflection The signal is used to generate the green light conversion signal; and a blue light sensing component is coupled to the processing circuit for detecting the visible light reflection signal to generate the blue light conversion signal. The sensing device of claim 5, wherein the at least one visible light sensing element comprises at least one infrared light cutoff and visible light passes through the sensing element. The sensing device of claim 7, wherein the third sensing signal comprises a red light conversion signal, a green light conversion signal, and a blue light conversion signal, and the at least one infrared light is cut off and the visible light passes. The measuring component comprises: an infrared light cut-off and red light passing through the sensing component coupled to the processing circuit for detecting the visible light reflecting signal to generate the red light converting signal; an infrared light cutoff and the green light passing through the sensing component And coupled to the processing circuit for detecting the visible The light reflects the signal to generate the green light conversion signal; and an infrared light is turned off and the blue light passes through the sensing component and is coupled to the processing circuit for detecting the visible light reflecting signal to generate the blue light converting signal. The sensing device of claim 3, wherein the image sensing unit comprises: at least one infrared light passing through and visible light passing through the sensing component coupled to the processing circuit for detecting the first infrared light The signal and the second infrared light signal respectively generate the first sensing signal and the second sensing signal, and detect the visible light reflecting signal to generate the third sensing signal. The sensing device of claim 9, wherein the third sensing signal comprises a red light conversion signal, a green light conversion signal, and a blue light conversion signal, and the at least one infrared light passes through and the visible light passes The measuring component comprises: at least one infrared light passing through and the red light passing through the sensing component, coupled to the processing circuit, for detecting the visible light reflecting signal to generate the red light converting signal; at least one infrared light passing through and the green light passing The measuring component is coupled to the processing circuit for detecting the visible light reflecting signal to generate the green light converting signal; and the at least one infrared light passing through and the blue light passing through the sensing component is coupled to the processing circuit for detecting The visible light reflects the signal to generate the blue light conversion signal; wherein the at least one infrared light passes through and the red light passes through the sensing element, the at least one infrared light passes through, and the green light passes through the sensing element and the at least one infrared light passes through and the blue light passes through At least one of the measuring elements detects the first infrared light signal and the second infrared light signal to respectively generate the first sensing signal and the second sensing signal . The sensing device of claim 1, wherein the processing circuit further identifies a proximity gesture and a distant gesture according to the relationship between the depth information and time. The sensing device of claim 1, further comprising: an infrared light detecting unit disposed at a periphery of the image sensing unit and controlled by the control circuit for proximity sensing and object position detection At least one of the sensing and the gesture detection, wherein at least one of the proximity sensing, the object position detection, and the gesture detection is performed during a period in which the control circuit disables the image sensing unit. The sensing device of claim 1, further comprising: a visible light detecting unit disposed at a periphery of the image sensing unit and controlled by the control circuit for ambient light sensing and color sensing At least one of the following, wherein at least one of ambient light sensing and color sensing is performed during a period in which the control circuit disables the image sensing unit. The sensing device of claim 1, further comprising: a dark sensing unit disposed at a periphery of the image sensing unit and controlled by the control circuit to generate a reference signal for dark-order compensation ( Dark level compensation). A sensing method includes: enabling an infrared light generating component to detect a first infrared light signal reflected from an object to generate a first sensing signal; and disabling the infrared light generating component to detect a reflection Generating a second sensing signal from a second infrared light signal of the object; and generating a 3D image information of the object based on at least a signal difference between the first sensing signal and the second sensing signal The three-dimensional image information includes a depth information. The sensing method of claim 15, further comprising: detecting a visible light reflection signal reflected from the object to generate a third sensing signal; and at least according to the first sensing signal and the second Sensing the signal difference between the signals to generate the object The step of the three-dimensional image information includes: generating the three-dimensional image information of the object according to the signal difference and the third sensing signal. The sensing method of claim 18, wherein the step of detecting the visible light reflecting signal reflected from the object to generate the third sensing signal is performed during the period in which the infrared light generating element is disabled. The sensing method of claim 15, further comprising: identifying a proximity gesture and a distant gesture according to the relationship between the depth information and time.