US10659765B2 - Three-dimensional (3D) image system and electronic device - Google Patents

Three-dimensional (3D) image system and electronic device Download PDF

Info

Publication number
US10659765B2
US10659765B2 US15/959,272 US201815959272A US10659765B2 US 10659765 B2 US10659765 B2 US 10659765B2 US 201815959272 A US201815959272 A US 201815959272A US 10659765 B2 US10659765 B2 US 10659765B2
Authority
US
United States
Prior art keywords
light
transmission gate
coupled
emitting unit
output
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US15/959,272
Other languages
English (en)
Other versions
US20190052863A1 (en
Inventor
Meng-Ta Yang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Shenzhen Goodix Technology Co Ltd
Original Assignee
Shenzhen Goodix Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Shenzhen Goodix Technology Co Ltd filed Critical Shenzhen Goodix Technology Co Ltd
Assigned to Shenzhen GOODIX Technology Co., Ltd. reassignment Shenzhen GOODIX Technology Co., Ltd. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: YANG, MENG-TA
Publication of US20190052863A1 publication Critical patent/US20190052863A1/en
Priority to US16/815,027 priority Critical patent/US20200213577A1/en
Application granted granted Critical
Publication of US10659765B2 publication Critical patent/US10659765B2/en
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/254—Image signal generators using stereoscopic image cameras in combination with electromagnetic radiation sources for illuminating objects
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2509—Color coding
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2531—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object using several gratings, projected with variable angle of incidence on the object, and one detection device
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
    • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
    • G01B11/2536—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object using several gratings with variable grating pitch, projected on the object with the same angle of incidence
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/521—Depth or shape recovery from laser ranging, e.g. using interferometry; from the projection of structured light
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/14609—Pixel-elements with integrated switching, control, storage or amplification elements
    • H—ELECTRICITY
    • H01—ELECTRIC ELEMENTS
    • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
    • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
    • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
    • H01L27/144—Devices controlled by radiation
    • H01L27/146—Imager structures
    • H01L27/14601—Structural or functional details thereof
    • H01L27/14625—Optical elements or arrangements associated with the device
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/50—Control of the SSIS exposure
    • H04N25/57—Control of the dynamic range
    • H04N25/58—Control of the dynamic range involving two or more exposures
    • H04N25/587—Control of the dynamic range involving two or more exposures acquired sequentially, e.g. using the combination of odd and even image fields
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/71—Charge-coupled device [CCD] sensors; Charge-transfer registers specially adapted for CCD sensors
    • H04N25/75—Circuitry for providing, modifying or processing image signals from the pixel array
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/77—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components
    • H04N25/771—Pixel circuitry, e.g. memories, A/D converters, pixel amplifiers, shared circuits or shared components comprising storage means other than floating diffusion
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/76—Addressed sensors, e.g. MOS or CMOS sensors
    • H04N25/78—Readout circuits for addressed sensors, e.g. output amplifiers or A/D converters
    • H04N5/2256—
    • H04N5/3745—
    • H04N5/378—
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10052—Images from lightfield camera

Definitions

  • the present application relates to a three-dimensional (3D) image system and an electronic device, and more particularly, to a 3D image system and an electronic device capable of withstanding ambient light.
  • 3D three-dimensional
  • 3D structured light measuring method uses the projection of the structured light and its geometry relationship to obtain the 3D information of an object. Firstly, a projecting device is used to project a coded structural light pattern onto the object, and a camera is used to capture the projected image. Secondly, matching between the captured image and the structural light pattern is performed, and a matching point is obtained. Finally, the 3D information is solved according to the triangular relationship of the projecting point, the matching point and the object.
  • an embodiment of the present application provides a three-dimensional (3D) image system, comprising a structural light module, configured to emit a structural light
  • the structural light module comprises a first light-emitting unit, the first light-emitting unit receives a first pulse signal and emits a first light according to the first pulse signal, a duty cycle of the first pulse signal is less than a specific value, an emission power the first light-emitting unit is greater than a specific power, and the first light has a first wavelength; and a light-sensing pixel array, configured to receive a reflected light corresponding to the structural light.
  • the duty cycle of the first pulse signal is less than 1/50.
  • the emission power of the first light-emitting unit is greater than 4 watts.
  • the structural light module comprises a diffraction unit, and the diffraction unit forms a diffraction effect on the first light and generates the structural light.
  • the diffraction unit is a diffraction optical element.
  • the light-sensing pixel array comprises a plurality of light-sensing pixel circuits
  • a light-sensing pixel circuit of the plurality of light-sensing pixel circuits comprises a light-sensing component; a first photoelectric readout circuit, coupled to the light-sensing component, configured to output a first output signal; and a second photoelectric readout circuit, coupled to the light-sensing component, configured to output a second output signal; wherein a pixel value corresponding to the light-sensing pixel circuit is a subtraction result of the first output signal and the second output signal.
  • the first photoelectric readout circuit comprises a first transmission gate, coupled to the light-sensing component; a first output transistor, coupled to the first transmission gate; and a first read transistor, coupled to the first output transistor, configured to output the first output signal; and the second photoelectric readout circuit comprises a second transmission gate, coupled to the light-sensing component; a second output transistor, coupled to the second transmission gate; and a second read transistor, coupled to the second output transistor, configured to output the second output signal.
  • the first transmission gate is conducted when the first light-emitting unit emits light
  • the second transmission gate is conducted when the first light-emitting unit does not emit light
  • a conduction time interval of the first transmission gate is longer than an emitting time interval of the first light-emitting unit.
  • the light-sensing pixel circuit comprises a first reset transistor and a second reset transistor, the first reset transistor is coupled to the first transmission gate, and the second reset transistor is coupled to the second transmission gate.
  • the duty cycle of the first pulse signal is time variant.
  • the structural light module comprises at least a second light-emitting unit, the at least a second light-emitting unit receives at least a second pulse signal and emits at least a second light according to the at least a second pulse signal, a duty cycle of the at least a second pulse signal is less than the specific value, an emission power of the at least a second light-emitting unit is greater than the specific power, and the at least a second light has at least a second wavelength, respectively.
  • the light-sensing pixel array comprises a plurality of light-sensing pixel circuits
  • a light-sensing pixel circuit of the plurality of light-sensing pixel circuits comprises a light-sensing component; a first photoelectric readout circuit, coupled to the light-sensing component, configured to output a first output signal; at least a second photoelectric readout circuit, coupled to the light-sensing component, configured to output at least a second output signal; and a third photoelectric readout circuit, coupled to the light-sensing component, configured to output a third output signal; where a pixel value corresponding to the light-sensing pixel circuit is a sum of the first output signal and the at least a second output signal minus a product of the third output signal and a number of the first output signal and the at least a second output signal.
  • the first photoelectric readout circuit comprises a first transmission gate, coupled to the light-sensing component; a first output transistor, coupled to the first transmission gate; and a first read transistor, coupled to the first output transistor, configured to output the first output signal;
  • a second photoelectric readout circuit of the at least a second photoelectric readout circuit comprises a second transmission gate, coupled to the light-sensing component; a second output transistor, coupled to the second transmission gate; and a second read transistor, coupled to the second output transistor, configured to output the second output signal;
  • the third photoelectric readout circuit comprises a third transmission gate, coupled to the light-sensing component; a third output transistor, coupled to the third transmission gate; and a third read transistor, coupled to the third output transistor, configured to output the third output signal.
  • the first transmission gate is conducted when the first light-emitting unit emits light
  • the at least a second transmission gate of the at least a second photoelectric readout circuit is conducted when the at least a second light-emitting unit emits light
  • the third transmission gate is conducted when the first light-emitting unit and the at least a second light-emitting unit do not emit light
  • a conduction time interval of the first transmission gate is longer than an emitting time interval of the first light-emitting unit
  • an conduction time interval of the at least a second transmission gate is longer than an emitting time interval of the at least a second light-emitting unit.
  • the light-sensing pixel circuit comprises a first reset transistor, at least a second reset transistor and a third reset transistor, the first reset transistor is coupled to the first transmission gate, the at least a second reset transistor is coupled to the at least a second transmission gate of the at least a second photoelectric readout circuit, and the third reset transistor is coupled to the third transmission gate.
  • a time in which the first transmission gate is conducted and a time in which the second transmission gate is conducted are separated by a time blank.
  • the first wavelength and the at least a second wavelength are different.
  • the duty cycles of the first pulse signal and the at least a second pulse signal are time variant.
  • the light-emitting unit within the structural light module of the present application receives the pulse signal, which is pulse modulated, and emits instantaneous strong light, such that the emitted structural light has immunity against the ambient light, to improve over disadvantages of the prior art.
  • FIG. 1 is a schematic diagram of three-dimensional (3D) image system according to an embodiment of the present application.
  • FIG. 2 is a schematic diagram of a light-sensing pixel circuit according to an embodiment of the present application.
  • FIG. 3 is a timing diagram of the light-sensing pixel circuit in FIG. 2 .
  • FIG. 4 is a schematic diagram of a 3D image system according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of a light-sensing pixel circuit according to an embodiment of the present application.
  • FIG. 6 is a timing diagram of the light-sensing pixel circuit in FIG. 5 .
  • FIG. 7 is a schematic diagram of a structural light according to an embodiment of the present application.
  • FIG. 8 is a schematic diagram of a structural light according to an embodiment of the present application.
  • FIG. 9 is a schematic diagram of an electronic device according to an embodiment of the present application.
  • FIG. 10 is a schematic diagram of a structural light module according to an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a light-sensing pixel circuit
  • FIG. 12 illustrates waveforms of pulse signals and transmission gate signals of different electronic devices.
  • FIG. 1 is a schematic diagram of a three-dimensional (3D) image system 10 according to an embodiment of the present application.
  • the 3D image system 10 comprises a structural light module 12 _ t and a camera module 12 _ r .
  • the structural light module 12 _ t is configured to generate a structural light SL, and project the structural light SL onto an object, where the structural light SL has stripe pattern, as shown in FIG. 7 .
  • the camera module 12 _ r may capture an image of the structural light SL projected onto the object, and calculate depth information of the object (using the triangular measuring method) according to curveness of the structural light SL on the object, so as to obtain a 3D image related to the object.
  • the structural light module 12 _ t comprises a pulse signal generator 120 , a light-emitting unit 122 and a diffraction unit 124 .
  • the light-emitting unit 122 may be a light-emitting diode (LED) or a laser emitting unit.
  • the diffraction unit 124 may be a diffraction optical element (DOE).
  • the pulse signal generator 120 is configured to generate a pulse signal pm 1 .
  • the light-emitting unit 122 is coupled to the pulse signal generator 120 , to receive the pulse signal pm 1 and generate/emit a first light L 1 to the diffraction unit 124 according to the pulse signal pm 1 .
  • the pulse signal pm 1 is a pulse modulated signal, i.e., the pulse signal pm 1 may be regarded as square waves with small duty cycle.
  • the light-emitting unit 122 when the light-emitting unit 122 emits the first light L 1 , the light-emitting unit 122 would have a large emission power.
  • a duty cycle of the pulse signal pm 1 may be 1/1000 in general, and not limited herein. As long as the duty cycle of the pulse signal pm 1 is less than 1/50, requirements of the present application are satisfied.
  • an emission power of the light-emitting unit 122 is between 4 watts and 5 watts in general, and not limited herein. As long as the emission power of the light-emitting unit 122 is greater than 4 watts, requirements of the present application are satisfied.
  • the light-emitting unit 122 may be regarded as emitting a strong light instantaneously, which is similar to a photoflash of a general camera), so as to enhance a light signal strength related to the structural light SL received by the camera module 12 _ r , such that the light signal related to the structural light SL has immunity against the ambient light, to improve over disadvantages of the prior art.
  • the camera module 12 _ r includes a light-sensing pixel array 14 and a lens 18 .
  • the light-sensing pixel array 14 receiving a reflected light corresponding to the structural light SL, comprises a plurality of light-sensing pixel circuits 16 .
  • the output signals of the light-sensing pixel circuits 16 may correspond to pixel values of the image captured by the camera module 12 _ r .
  • the circuit structure and operational mechanism of the light-sensing pixel circuit 16 are not limited. For example, please refer to FIG. 2 and FIG. 3 .
  • FIG. 2 is an equivalent schematic circuit diagram of the light-sensing pixel circuit 16 according to an embodiment of the present application.
  • FIG. 2 is an equivalent schematic circuit diagram of the light-sensing pixel circuit 16 according to an embodiment of the present application.
  • the light-sensing pixel circuit 16 comprises a light-sensing component PD and photoelectric readout circuits 16 _ 1 , 16 _ 2 .
  • Both the photoelectric readout circuits 16 _ 1 and 16 _ 2 include transmission gates, reset transistors, output transistors and read transistors.
  • the transmission gates are coupled to the light-sensing component PD.
  • Gates of the reset transistors receive a reset signal Reset.
  • Gates of the read transistors receive a row select signal ROW.
  • the transmission gates of the photoelectric readout circuits 16 _ 1 and 16 _ 2 receive signals TX 1 and TX 2 , respectively.
  • the transmission gate, the reset transistor and the output transistor of the photoelectric readout circuit 16 _ 1 are connected to a node FD_ 1 .
  • the transmission gate, the reset transistor and the output transistor of the photoelectric readout circuit 16 _ 2 are connected to a node FD_ 2 .
  • the light-sensing pixel circuit 16 also includes an anti-blooming transistor, and a gate of the anti-blooming transistor receives a signal TX 4 .
  • Operational mechanism of the light-sensing pixel circuit 16 is described as follows.
  • the pulse signal pm 1 is high
  • the light-emitting unit 122 emits the first light L 1 .
  • the transmission gate of the photoelectric readout circuit 16 _ 1 is conducted.
  • a conduction time interval T 1 of the transmission gate within the photoelectric readout circuit 16 _ 1 is wider than a time interval T 4 of the pulse signal pm 1 being high, i.e., the conduction time interval of the transmission gate within the photoelectric readout circuit 16 _ 1 is longer than an emitting time interval of the light-emitting unit 122 .
  • the transmission gate of the photoelectric readout circuit 16 _ 1 When the transmission gate of the photoelectric readout circuit 16 _ 1 is conducted (i.e., the signal TX 1 is high), i.e., within the conduction time interval T 1 , the light-sensing component PD receives the first light L 1 and the ambient light, and the transmission gate of the photoelectric readout circuit 16 _ 1 may drain out the photocharge generated by the light-sensing component PD because of receiving the first light L 1 and also the ambient light and store the photocharge at the node FD_ 1 .
  • the transmission gate of the photoelectric readout circuit 16 _ 2 may be conducted in a short time (the signal TX 2 is high). At this time, the light-sensing component PD receives the ambient light only, and the transmission gate of the photoelectric readout circuit 16 _ 2 may drain out the photocharge generated by the light-sensing component PD because of receiving the ambient light and store the photocharge at the node FD_ 2 .
  • the read transistor the photoelectric readout circuit 16 _ 1 When the read transistors of the photoelectric readout circuits 16 _ 1 and 16 _ 2 are conducted, the read transistor the photoelectric readout circuit 16 _ 1 outputs an output signal Pout 1 (which is related to the first light L 1 and the ambient light), and the read transistor of the photoelectric readout circuit 16 _ 2 outputs an output signal Pout 2 (which is related to the ambient light only).
  • the pixel value corresponding to the light-sensing pixel circuit 16 is a subtraction result of the output signal Pout 1 and the output signal Pout 2 (e.g., Pout 1 -Pout 2 ). Therefore, an effect of the ambient light may be eliminated in the pixel value of the light-sensing pixel circuit 16 .
  • the anti-blooming transistor of the light-sensing pixel circuit 16 is conducted (the signal TX 4 is high).
  • the light-sensing pixel circuit 16 would drain out the photocharge of the light-sensing component PD caused by receiving the ambient light, to maintain normal operation.
  • the structural light module may comprise two light-emitting units.
  • the two light-emitting units may emit lights alternatively, so as to enhance the strength corresponding to the structural light received by the camera module 12 _ r .
  • the two light-emitting units may emit lights with different wavelengths. Since the different wavelengths have various refractions, the structural light generated by passing through the diffraction unit may have denser stripe pattern, and resolution of the 3D image is further enhanced.
  • FIG. 4 is a schematic diagram of a 3D image system 40 according to an embodiment of the present application.
  • the 3D image system 40 is similar to the 3D image system 10 , and thus, same components are denoted by the same symbols.
  • the structural light module 42 _ t of the 3D image system 40 comprises another light-emitting unit 422 , in addition to the light-emitting unit 122 .
  • the light-emitting unit 422 receives a pulse signal pm 2 to generate a second light L 2 .
  • the second light L 2 and the first light L 1 may have different wavelengths.
  • a duty cycle of the pulse signal pm 2 may be 1/1000 (or less than 1/50).
  • An emission power of the light-emitting unit 422 may be between 4 watts and 5 watts (or greater than 4 watts).
  • the first light L 1 and the second light L 2 pass though the diffraction unit 124 , in which the diffraction effect is formed, such that a structural light SL′ is generated. Since the first light L 1 and the second light L 2 have the different wavelengths, the stripe pattern of the structural light SL′ is denser. As shown in FIG. 8 , a strip light sp 1 represents the structural light corresponding to the first light L 1 , and a strip light sp 2 represents the structural light corresponding to the second light L 2 .
  • a camera module 42 _ r of the 3D image system 40 comprises a light-sensing pixel array 44 .
  • the light-sensing pixel array 44 comprises a plurality of light-sensing pixel circuits 46 .
  • the circuit structure and operational mechanism of the light-sensing pixel circuit 46 are not limited. For example, please refer to FIG. 5 and FIG. 6 .
  • FIG. 5 is an equivalent schematic circuit diagram of the light-sensing pixel circuit 46 according to an embodiment of the present application.
  • FIG. 6 is a timing diagram of the light-sensing pixel circuit 46 .
  • the light-sensing pixel circuit 46 is similar to the light-sensing pixel circuit 16 , and thus, same components are denoted by the same symbols. Different from the light-sensing pixel circuit 16 , the light-sensing pixel circuit 46 further comprises the photoelectric readout circuit 46 _ 3 .
  • the circuit structure of the photoelectric readout circuit 46 _ 3 is the same as which of the photoelectric readout circuits 16 _ 1 and 16 _ 2 , where a transmission gate of the photoelectric readout circuit 46 _ 3 receives a signal TX 3 .
  • the transmission gate of the photoelectric readout circuit 46 _ 3 is conducted.
  • the light-sensing component PD receives the second light L 2 and the ambient light, and the transmission gate of the photoelectric readout circuit 46 _ 3 may drain out the photocharge generated by the light-sensing component PD because of receiving the second light L 2 and the ambient light and store the photocharge a node FD_ 3 .
  • the transmission gate of the photoelectric readout circuit 16 _ 2 may be conducted in a short time (the signal TX 2 is high).
  • the light-sensing component PD receives the ambient light only, and the transmission gate of the photoelectric readout circuit 16 _ 2 may drain out the photocharge generated by the light-sensing component PD because of receiving the ambient light and store the photocharge at the node FD_ 2 .
  • the read transistor the photoelectric readout circuit 16 _ 1 When the read transistors of the photoelectric readout circuits 16 _ 1 , 16 _ 2 and 46 _ 3 are conducted, the read transistor the photoelectric readout circuit 16 _ 1 outputs an output signal Pout 1 (which is related to the first light L 1 and the ambient light), the read transistor of the photoelectric readout circuit 16 _ 2 outputs an output signal Pout 2 (which is related to the ambient light only), and the read transistor of the photoelectric readout circuit 46 _ 3 outputs an output signal Pout 3 (which is related to the second light L 2 and the ambient light).
  • Pout 1 which is related to the first light L 1 and the ambient light
  • the read transistor of the photoelectric readout circuit 16 _ 2 outputs an output signal Pout 2 (which is related to the ambient light only)
  • the read transistor of the photoelectric readout circuit 46 _ 3 outputs an output signal Pout 3 (which is related to the second light L 2 and the ambient light).
  • the pixel value corresponding to the light-sensing pixel circuit 46 is a sum of the output signal Pout 1 and the output signal Pout 3 minus twice of the output signal Pout 2 (ie.e, Pout 1 +Pout 3 ⁇ 2*Pout 2 ), such that an effect of the ambient light may be eliminated.
  • the conduction time intervals of the transmission gates within the photoelectric readout circuits 16 _ 1 , 16 _ 2 are wider than pulse widths of the pulse signals pm 1 and pm 2 , i.e., the conduction time intervals of the transmission gates within the photoelectric readout circuits 16 _ 1 , 16 _ 2 are longer than the emitting time intervals of the light-emitting units 122 , 422 .
  • the conduction time interval T 1 of the transmission gate within the photoelectric readout circuit 16 _ 1 and the conduction time interval T 8 of the transmission gate within the photoelectric readout circuit 46 _ 3 are separated by a time blank T 7 .
  • the rest operational mechanism is referred to the paragraph stated in the above, which is not narrated herein for brevity.
  • FIG. 9 is a schematic diagram of an electronic device 9 according to an embodiment of the present application.
  • the electronic device 9 comprises a 3D image system 90 , where the 3D image system 90 may be realized by the 3D image system 10 or the 3D image system 40 .
  • the duty cycles of the pulse signals pm 1 and pm 2 may be changed randomly (i.e., the duty cycles of the pulse signals pm 1 and pm 2 are time variant).
  • the pulse signal generator 120 may generate a subsequent pulse (N+n) period of time later, where N may be a large integer and n may be a random number. Therefore, the structural lights corresponding to different electronic devices are prevented from interfering each other. Please refer to FIG. 12 , FIG.
  • the pulse signal pmA is the pulse signal received by the light-emitting unit within the electronic device A.
  • the pulse signal pmB is the pulse signal received by the light-emitting unit within the electronic device B.
  • the signal TXA is the signal received by the transmission gate within the photoelectric readout circuit of the light-sensing pixel circuit within the electronic device A.
  • the signal TXB is the signal received by the transmission gate within the photoelectric readout circuit of the light-sensing pixel circuit within the electronic device B. As shown in FIG.
  • the light-emitting time corresponding to the light-emitting unit of the electronic device A and which of the electronic device B would be interleaved.
  • the conduction time corresponding to the transmission gate of the electronic device A and which of the electronic device B would be interleaved as well.
  • FIG. 10 is a schematic diagram of a structural light module c 2 _ t according to an embodiment of the present application.
  • FIG. 11 is a schematic diagram of a light-sensing pixel circuit d 6 corresponding to the structural light module c 2 _ t .
  • the structural light module c 2 _ t comprises the light-emitting units 122 and 122 _ 1 - 122 _K, which emit light at different time instant.
  • the light-sensing pixel circuit d 6 comprises photoelectric readout circuits d 6 _ 1 -d 6 _K′ and d 6 _B.
  • the circuit structure of the photoelectric readout circuits d 6 _ 1 -d 6 _K′ and d 6 _B may be the same as which of the photoelectric readout circuits 16 _ 1 , 16 _ 2 or 46 _ 3 stated in the above.
  • the conduction time intervals of the transmission gate within the photoelectric readout circuits d 6 _ 1 -d 6 _K′ (K′ may represent K+1) are corresponding to the emitting time intervals of the light-emitting units 122 , 122 _ 1 - 122 _K.
  • the photoelectric readout circuits d 6 _ 1 -d 6 _K′ output the output signals Pout 1 -Pout K′.
  • the output signals Pout 1 -Pout K′ are related to the lights emitted from the light-emitting units 122 , 122 _ 1 - 122 _K and the ambient light.
  • the transmission gate of the photoelectric readout circuit d 6 _B is conducted when all the light-emitting units 122 , 122 _ 1 - 122 _K do not emit light, which means that the photoelectric readout circuit d 6 _B only reads the photocurrent caused by the ambient light, i.e., the output signal PoutB outputted by the photoelectric readout circuit d 6 _B is only related to the ambient light.
  • the pixel value corresponding to the light-sensing pixel circuit d 6 may be Pout 1 + . . . +PoutK′ ⁇ K′*PoutB.
  • the light-emitting unit within the structural light module of the present application receives the pulse signal, which is pulse modulated, and emits instantaneous strong light, such that the emitted structural light has immunity against the ambient light, to improve over disadvantages of the prior art.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Electromagnetism (AREA)
  • Power Engineering (AREA)
  • Remote Sensing (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Optics & Photonics (AREA)
  • Theoretical Computer Science (AREA)
  • Condensed Matter Physics & Semiconductors (AREA)
  • Computer Hardware Design (AREA)
  • Microelectronics & Electronic Packaging (AREA)
  • Studio Devices (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Transforming Light Signals Into Electric Signals (AREA)
  • Optical Radar Systems And Details Thereof (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
US15/959,272 2017-08-14 2018-04-22 Three-dimensional (3D) image system and electronic device Expired - Fee Related US10659765B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/815,027 US20200213577A1 (en) 2017-08-14 2020-03-11 Three-Dimensional (3D) Image System and Electronic Device

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/097332 WO2019033232A1 (zh) 2017-08-14 2017-08-14 三维影像系统及电子装置

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/097332 Continuation WO2019033232A1 (zh) 2017-08-14 2017-08-14 三维影像系统及电子装置

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US16/815,027 Division US20200213577A1 (en) 2017-08-14 2020-03-11 Three-Dimensional (3D) Image System and Electronic Device

Publications (2)

Publication Number Publication Date
US20190052863A1 US20190052863A1 (en) 2019-02-14
US10659765B2 true US10659765B2 (en) 2020-05-19

Family

ID=61154840

Family Applications (2)

Application Number Title Priority Date Filing Date
US15/959,272 Expired - Fee Related US10659765B2 (en) 2017-08-14 2018-04-22 Three-dimensional (3D) image system and electronic device
US16/815,027 Abandoned US20200213577A1 (en) 2017-08-14 2020-03-11 Three-Dimensional (3D) Image System and Electronic Device

Family Applications After (1)

Application Number Title Priority Date Filing Date
US16/815,027 Abandoned US20200213577A1 (en) 2017-08-14 2020-03-11 Three-Dimensional (3D) Image System and Electronic Device

Country Status (6)

Country Link
US (2) US10659765B2 (ja)
EP (1) EP3470773A1 (ja)
JP (1) JP6908689B2 (ja)
KR (2) KR102061182B1 (ja)
CN (2) CN107690565A (ja)
WO (1) WO2019033232A1 (ja)

Families Citing this family (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110612429B (zh) * 2018-04-10 2021-03-26 深圳市汇顶科技股份有限公司 三维影像测距系统及方法
WO2019196049A1 (zh) * 2018-04-12 2019-10-17 深圳市汇顶科技股份有限公司 影像传感系统及电子装置
EP3605016B1 (en) 2018-04-16 2022-04-06 Shenzhen Goodix Technology Co., Ltd. Image sensing system and electronic device
CN109277015A (zh) * 2018-08-30 2019-01-29 东莞市闻誉实业有限公司 原料搅拌照明装置
CN109313264B (zh) * 2018-08-31 2023-09-12 深圳市汇顶科技股份有限公司 基于飞行时间的测距方法和测距系统
US10915003B2 (en) * 2018-09-27 2021-02-09 Himax Technologies Limited Projecting apparatus for 3D sensing system
WO2020093294A1 (zh) * 2018-11-08 2020-05-14 深圳市汇顶科技股份有限公司 垂直腔面发光激光器、结构光模块及光投射方法与终端
KR20220039230A (ko) * 2020-09-22 2022-03-29 삼성전자주식회사 생체 정보 측정 장치 및 이를 포함하는 전자 장치

Citations (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW399742U (en) 1999-12-22 2000-07-21 Inventec Corp Projector automatic light and illuminance adjusting device
US20050206777A1 (en) 2002-06-04 2005-09-22 Florent Selves Digital camera adapted for taking images with a flashlight and corresponding method
CN101356450A (zh) 2005-12-19 2009-01-28 国家光学研究所 物体检测发光系统和方法
CN101520326A (zh) 2009-03-24 2009-09-02 熊全宾 一种利用不同颜色顺序发光体非接触式定位装置
KR20090097416A (ko) 2008-03-11 2009-09-16 삼성전자주식회사 주변 광 제거를 위한 ｃｍｏｓ 센서의 픽셀 및 그 동작방법
CN101604111A (zh) 2009-06-15 2009-12-16 张一鸣 一种防逆光拍摄的方法和装置
CN101627280A (zh) 2006-11-21 2010-01-13 曼蒂斯影像有限公司 三维几何建模和三维视频内容创建
US20110085043A1 (en) * 2009-10-08 2011-04-14 Honda Motor Co., Ltd. Image capturing apparatus, image capturing system, and processing method
US20110090385A1 (en) * 2008-06-04 2011-04-21 Honda Motor Co., Ltd. Imaging device
JP2011169701A (ja) 2010-02-17 2011-09-01 Sanyo Electric Co Ltd 物体検出装置および情報取得装置
TW201132925A (en) 2010-03-16 2011-10-01 Test Research Inc Measuring system for a 3D profile of object
CN102547305A (zh) 2004-04-30 2012-07-04 精工爱普生株式会社 对投影系统中的发光器件进行定序的方法和设备
CN102595030A (zh) 2011-01-12 2012-07-18 英属开曼群岛商恒景科技股份有限公司 具环境光感测的数字摄像装置
EP2500687A2 (en) 2011-03-14 2012-09-19 Heliotis SA Structured light projector
US20130229491A1 (en) * 2012-03-02 2013-09-05 Samsung Electronics Co., Ltd. Method of operating a three-dimensional image sensor
JP2013213812A (ja) 2012-03-30 2013-10-17 Mitsubishi Electric Corp 構造化光パターンを生成及び取得するための装置および方法
CN103400366A (zh) 2013-07-03 2013-11-20 西安电子科技大学 基于条纹结构光的动态场景深度获取方法
CN103712576A (zh) 2013-12-20 2014-04-09 上海瑞立柯信息技术有限公司 一种可编程控制的光栅投影装置
CN104134426A (zh) 2014-07-07 2014-11-05 京东方科技集团股份有限公司 像素结构及其驱动方法、显示装置
CN104266607A (zh) 2014-09-22 2015-01-07 电子科技大学 镜面目标轮廓光学测量系统和方法
US20150116601A1 (en) 2013-10-31 2015-04-30 Htc Corporation Handheld electronic device and image projection method of the same
CN104717408A (zh) 2013-12-11 2015-06-17 廖彩姿 影像撷取机构的光控设备
US20150271476A1 (en) * 2011-04-26 2015-09-24 Semiconductor Components Industries, Llc Structured light imaging system
US9154708B1 (en) 2014-11-06 2015-10-06 Duelight Llc Image sensor apparatus and method for simultaneously capturing flash and ambient illuminated images
US20150341619A1 (en) 2013-01-01 2015-11-26 Inuitive Ltd. Method and system for light patterning and imaging
US20160109575A1 (en) * 2013-06-06 2016-04-21 Heptagon Micro Optics Pte. Ltd. Sensor System With Active Illumination
TW201629444A (zh) 2015-02-10 2016-08-16 聯詠科技股份有限公司 用於影像感測器之感光單元及其感光電路
CN106094398A (zh) 2016-08-08 2016-11-09 哈尔滨理工大学 彩色复合相移条纹结构光投影装置和方法
CN106289092A (zh) 2015-05-15 2017-01-04 高准精密工业股份有限公司 光学装置及其发光装置
CN107024850A (zh) 2017-05-26 2017-08-08 清华大学 高速结构光三维成像系统

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9296111B2 (en) * 2013-07-22 2016-03-29 LuxVue Technology Corporation Micro pick up array alignment encoder
CN105789202B (zh) * 2016-05-20 2018-09-14 京东方科技集团股份有限公司 有源像素传感器电路、驱动方法和图像传感器

Patent Citations (34)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TW399742U (en) 1999-12-22 2000-07-21 Inventec Corp Projector automatic light and illuminance adjusting device
US20050206777A1 (en) 2002-06-04 2005-09-22 Florent Selves Digital camera adapted for taking images with a flashlight and corresponding method
CN102547305A (zh) 2004-04-30 2012-07-04 精工爱普生株式会社 对投影系统中的发光器件进行定序的方法和设备
CN101356450A (zh) 2005-12-19 2009-01-28 国家光学研究所 物体检测发光系统和方法
US20100074532A1 (en) 2006-11-21 2010-03-25 Mantisvision Ltd. 3d geometric modeling and 3d video content creation
CN101627280A (zh) 2006-11-21 2010-01-13 曼蒂斯影像有限公司 三维几何建模和三维视频内容创建
KR20090097416A (ko) 2008-03-11 2009-09-16 삼성전자주식회사 주변 광 제거를 위한 ｃｍｏｓ 센서의 픽셀 및 그 동작방법
US20110090385A1 (en) * 2008-06-04 2011-04-21 Honda Motor Co., Ltd. Imaging device
CN101520326A (zh) 2009-03-24 2009-09-02 熊全宾 一种利用不同颜色顺序发光体非接触式定位装置
CN101604111A (zh) 2009-06-15 2009-12-16 张一鸣 一种防逆光拍摄的方法和装置
US20110085043A1 (en) * 2009-10-08 2011-04-14 Honda Motor Co., Ltd. Image capturing apparatus, image capturing system, and processing method
JP2011169701A (ja) 2010-02-17 2011-09-01 Sanyo Electric Co Ltd 物体検出装置および情報取得装置
TW201132925A (en) 2010-03-16 2011-10-01 Test Research Inc Measuring system for a 3D profile of object
CN102595030A (zh) 2011-01-12 2012-07-18 英属开曼群岛商恒景科技股份有限公司 具环境光感测的数字摄像装置
EP2500687A2 (en) 2011-03-14 2012-09-19 Heliotis SA Structured light projector
US20150271476A1 (en) * 2011-04-26 2015-09-24 Semiconductor Components Industries, Llc Structured light imaging system
US20130229491A1 (en) * 2012-03-02 2013-09-05 Samsung Electronics Co., Ltd. Method of operating a three-dimensional image sensor
JP2013213812A (ja) 2012-03-30 2013-10-17 Mitsubishi Electric Corp 構造化光パターンを生成及び取得するための装置および方法
US20150341619A1 (en) 2013-01-01 2015-11-26 Inuitive Ltd. Method and system for light patterning and imaging
CN105705962A (zh) 2013-06-06 2016-06-22 新加坡恒立私人有限公司 具有主动照明的传感器系统
US20160109575A1 (en) * 2013-06-06 2016-04-21 Heptagon Micro Optics Pte. Ltd. Sensor System With Active Illumination
JP2016524709A (ja) 2013-06-06 2016-08-18 ヘプタゴン・マイクロ・オプティクス・プライベート・リミテッドＨｅｐｔａｇｏｎ Ｍｉｃｒｏ Ｏｐｔｉｃｓ Ｐｔｅ． Ｌｔｄ． 能動照明を備えるセンサシステム
CN103400366A (zh) 2013-07-03 2013-11-20 西安电子科技大学 基于条纹结构光的动态场景深度获取方法
US20150116601A1 (en) 2013-10-31 2015-04-30 Htc Corporation Handheld electronic device and image projection method of the same
CN104597967A (zh) 2013-10-31 2015-05-06 宏达国际电子股份有限公司 手持式电子装置及其投影方法
CN104717408A (zh) 2013-12-11 2015-06-17 廖彩姿 影像撷取机构的光控设备
CN103712576A (zh) 2013-12-20 2014-04-09 上海瑞立柯信息技术有限公司 一种可编程控制的光栅投影装置
CN104134426A (zh) 2014-07-07 2014-11-05 京东方科技集团股份有限公司 像素结构及其驱动方法、显示装置
CN104266607A (zh) 2014-09-22 2015-01-07 电子科技大学 镜面目标轮廓光学测量系统和方法
US9154708B1 (en) 2014-11-06 2015-10-06 Duelight Llc Image sensor apparatus and method for simultaneously capturing flash and ambient illuminated images
TW201629444A (zh) 2015-02-10 2016-08-16 聯詠科技股份有限公司 用於影像感測器之感光單元及其感光電路
CN106289092A (zh) 2015-05-15 2017-01-04 高准精密工业股份有限公司 光学装置及其发光装置
CN106094398A (zh) 2016-08-08 2016-11-09 哈尔滨理工大学 彩色复合相移条纹结构光投影装置和方法
CN107024850A (zh) 2017-05-26 2017-08-08 清华大学 高速结构光三维成像系统

Also Published As

Publication number Publication date
KR102163643B1 (ko) 2020-10-12
JP6908689B2 (ja) 2021-07-28
CN110906864B (zh) 2022-04-29
EP3470773A4 (en) 2019-04-17
EP3470773A1 (en) 2019-04-17
WO2019033232A1 (zh) 2019-02-21
US20200213577A1 (en) 2020-07-02
JP2019533134A (ja) 2019-11-14
KR102061182B1 (ko) 2019-12-31
CN110906864A (zh) 2020-03-24
US20190052863A1 (en) 2019-02-14
KR20190031422A (ko) 2019-03-26
CN107690565A (zh) 2018-02-13
KR20190141800A (ko) 2019-12-24

Similar Documents

Publication Publication Date Title
US10659765B2 (en) Three-dimensional (3D) image system and electronic device
CN111123289B (zh) 一种深度测量装置及测量方法
WO2020247046A1 (en) Mixed active depth
EP3767335A1 (en) Distance measurement method and apparatus, and distance measurement sensor and distance measurement sensing array
CN109283508A (zh) 飞行时间计算方法
CN106791497B (zh) 一种脉冲增益调制式单像素三维成像系统及方法
US10931904B2 (en) Pixel circuit and image sensing system
US11796679B2 (en) Time of flight sensor and method
CN113805186A (zh) 飞行时间tof装置和电子设备
CN113805185A (zh) 飞行时间tof装置和电子设备
JP6887036B2 (ja) 三次元イメージングシステム及び電子デバイス
CN113805187A (zh) 飞行时间tof装置和电子设备
WO2022168500A1 (ja) 測距装置およびその制御方法、並びに、測距システム
TW202343020A (zh) 基於單光子崩潰二極體的遞色產生器及包括遞色產生器的飛行時間感測器
EP3015958B1 (en) Light sensor array device
Bantounos et al. Towards a solid‐state light detection and ranging system using holographic illumination and time‐of‐flight image sensing
CN213041995U (zh) 飞行时间tof装置和电子设备
CN213041994U (zh) 飞行时间tof装置和电子设备
US9804000B2 (en) Optical sensor array apparatus
US12123978B2 (en) Distance measurement method and apparatus, and distance measurement sensor and distance measurement sensing array
US11438486B2 (en) 3D active depth sensing with laser pulse train bursts and a gated sensor
WO2019196049A1 (zh) 影像传感系统及电子装置
JP2024100634A (ja) 距離測定のための装置
CN117880650A (zh) 图像传感器、电子设备及操作电子设备的方法

Legal Events

Date Code Title Description
AS Assignment

Owner name: SHENZHEN GOODIX TECHNOLOGY CO., LTD., CHINA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:YANG, MENG-TA;REEL/FRAME:045605/0713

Effective date: 20171117

FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

ZAAA Notice of allowance and fees due

Free format text: ORIGINAL CODE: NOA

ZAAB Notice of allowance mailed

Free format text: ORIGINAL CODE: MN/=.

ZAAA Notice of allowance and fees due

Free format text: ORIGINAL CODE: NOA

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20240519