KR102473740B1 - Concurrent rgbz sensor and system - Google Patents

Abstract

Two-dimensional (2D) color information and 3D depth information are obtained simultaneously from the 2D pixel array. The 2D pixel array is arranged in a plurality of first group rows. The second group of the array is operable to generate 2D color information and the pixels of the third group of the pixel array are operable to generate 3D depth information. The rows of the first group include a first number of rows, and the rows of the second group include a second number of rows less than or equal to the first number of rows. The third group of rows includes a third number of rows less than or equal to the second number of rows. Alternatively, 2D color information is received from a row selected from the second group of rows, and 3D depth information is received from a row selected from the third group of rows.

Description

CONCURRENT RGBZ SENSOR AND SYSTEM}

The present invention relates to image sensors, and more specifically, but not limited to, certain embodiments disclosed herein are laser point scan and complementary metal oxide semiconductor (CMOS) image sensors (also used for two-dimensional imaging of 3D objects). ) and a method for measuring depth for a 3D object.

Three-dimensional (3D) imaging systems are increasingly used in a variety of applications such as industrial production, video games, computer graphics, robotic surgery, consumer displays, surveillance video, 3D modeling, real estate sales, and more.

Existing three-dimensional imaging technologies may include time-of-flight (TOF) based range imaging, stereo vision systems and structured light (SL) methods, for example.

In the TOF method, the distance to a three-dimensional object is solved based on the known speed of light. That is, the distance to the 3D object is obtained by measuring the round trip time (RTT) required for the light signal to travel between the camera and the 3D object for each point of the image. A TOF camera may use a scannerless approach to capture the entire scene with each laser or light pulse. Some example applications of the TOF method may include advanced automotive applications such as active pedestrian safety or line collision detection based on real-time street images. Additionally, some example applications of the TOF method may include tracking human motion, such as interacting with a game on a video game console. Also, example applications may include helping robots find items, such as sorting objects in industrial machine vision, items on a conveyor belt - and the like.

In a stereoscopic or stereo vision system, two cameras spaced apart horizontally are used to obtain two different views of a scene or a three-dimensional object within a scene. By comparing these two images, relative depth information can be obtained for a 3D object. Stereo vision is very important in fields such as robotics to extract information about the relative position of 3D objects in the vicinity of an autonomous system/robot. Other applications for robotics may include object recognition where stereoscopic depth information enables a robotic system to separate closed image components. The robot cannot otherwise differentiate between two separate objects, such as one object in front of another that partially or completely hides the other object. Three-dimensional stereo displays are also used in entertainment and automation systems.

In the SL method, the three-dimensional shape of an object can be measured using a projection light pattern and an imaging camera. In the SL method, a known light pattern, i.e. an open grid, horizontal bar or parallel stripe pattern, is projected onto a scene or a three-dimensional object within the scene. The projected pattern may be deformed or displaced when hitting the surface of a three-dimensional object. Such transformations allow the SL vision system to calculate the depth and surface of an object. Thus, projecting a narrow band of light onto a three-dimensional surface can produce line light. Line light may appear more distorted from other projections than the projector's distortion. Projection can also be used for geometric reconstruction of the optical surface shape. SL-based 3D imaging has other applications, such as fingerprinting by police forces within a 3D scene, in-line inspection of parts in production, health care for live measurement of human body shape or human skin and similar microstructures. can be used for

A technical problem to be solved by the present invention is to provide a triangulation-based system and a depth measurement method for a 3D object.

An embodiment of the present invention receives an image of at least one object from an image sensor, the image sensor including a two-dimensional (2D) pixel array arranged in a plurality of rows of a first group, the array pixels in rows of a second group are operable to generate two-dimensional color information of the at least one object, and pixels in a third group of the array generate three-dimensional depth information of the at least one object wherein the first group of rows includes a first number of rows, the second group of rows includes a second number of rows less than or equal to the first number of rows, and The three groups of rows include a third number of rows less than or equal to the second number of rows. An embodiment of the present invention alternately receives 2D color information of at least one object from one row selected from the rows of the second group, and receives 2D color information of at least one object from one row selected from the rows of the third group. Receive 3D depth information.

An embodiment of the present invention provides an image sensor unit including a 2D pixel array and a controller. The two-dimensional pixel array is arranged in a first group of a plurality of rows. The pixels of the second group of rows of the array are operable to generate two-dimensional color information based on an image of at least one object received by the two-dimensional pixel array. A third group of pixels of the array is operable to generate 3-dimensional depth information of at least one object. The rows of the first group include a first number of rows, the rows of the second group include a second number of rows less than or equal to the first number of rows, and the rows of the third group include the second number of rows. and a third number of rows less than or equal to the number of rows. The controller alternately selects one row from the rows of the second group and outputs 2D color information generated based on the image of at least one object, selects one row from the rows of the third group, and outputs at least one row. Outputs the generated 3D depth information of the object of

An embodiment of the present invention provides a system including a two-dimensional pixel array, a controller, and a display. The two-dimensional pixel array has a plurality of rows of a first group in which the pixels of the rows of the second group of the array are operable to generate two-dimensional color information of at least one object received by the two-dimensional pixel array. are arranged A third group of pixels of the array is operable to generate 3-dimensional depth information of at least one object. The rows of the first group include a first number of rows, the rows of the second group include a second number of rows less than or equal to the first number of rows, and the rows of the third group include the second number of rows. and a third number of rows less than or equal to the number of rows. The controller alternately selects one row from the rows of the second group and outputs 2D color information generated based on the image of at least one object, and selects one row from the rows of the third group to output at least one row. Outputs the generated 3D depth information of the object of The display is connected to the 2D pixel array and the controller, displays a first image of at least one object based on the generated 2D color information and displays a second image of at least one object based on the generated 3D depth information. display

According to an embodiment of the present invention, depth measurement for a 3D object is implemented.

1 is a diagram showing a highly simplified configuration of a system according to an embodiment of the present invention.
2 illustrates an exemplary operational configuration of the system of FIG. 1 according to one embodiment of the present invention.
3 is an exemplary flowchart illustrating performance of 3D depth measurement according to an embodiment of the present invention.
4 is an exemplary diagram illustrating execution of a point scan for 3D depth measurement according to an embodiment of the present invention.
5 is a diagram showing exemplary timestamping of scanned light spots according to an embodiment of the present invention.
FIG. 6 is a diagram showing some exemplary detailed circuitry of the two-dimensional pixel array of the image sensor in FIGS. 1 and 2 and associated processing circuitry of the image processing unit in accordance with one embodiment of the present invention.
7A is an exemplary configuration diagram of an image sensor unit according to an embodiment of the present invention.
7B is an exemplary detailed configuration diagram of a CDS+ADC unit for 3D depth measurement according to an embodiment of the present invention.
8 illustrates exemplary timing of different signals within the system of FIGS. 1 and 2 to generate timestamp-based pixel-by-pixel outputs in a three-dimensional linear mode of operation in accordance with another embodiment of the present invention. It is also the timing.
9 exemplarily shows a lookup table (LUT) for indicating use of the LUT to determine 3D depth values according to another embodiment of the present invention.
10 is a timing diagram exemplarily showing the timing of different signals in the system of FIGS. 1 and 2 to generate a 2D RGB image using a 2D linear mode operation according to another embodiment of the present invention. .
11 illustrates exemplary timing of different signals within the system of FIGS. 1 and 2 to generate timestamp-based pixel-by-pixel outputs within a three-dimensional logarithmic (log) mode of operation in accordance with another embodiment of the present invention. This is the timing diagram shown by
FIG. 12 is a diagram showing an overall configuration within the system of FIGS. 1 and 2 according to an embodiment of the present invention.
13 is an exemplary flowchart showing a process for simultaneously generating and obtaining 2D color information and 3D depth information according to an embodiment of the present invention.
14 is a diagram exemplarily showing that a distance to a translucent object and a distance to an object behind the translucent object are performed for 3D depth measurement according to an embodiment of the present invention.
15 is a diagram exemplarily showing that depth imaging of a translucent medium is performed for 3D depth measurement according to an embodiment of the present invention.
16 is a diagram exemplarily illustrating that depth imaging of an object is performed for 3D depth measurement in the presence of multiple return paths according to an embodiment of the present invention.

In the detailed description that follows, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it should be understood that a person skilled in the art may implement the inventive concepts without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail in order not to unnecessarily obscure aspects of an embodiment. In addition, embodiments of the present invention may implement a 3D measuring device to perform low power in any imaging device or system including, for example, a smart phone, a user equipment (UE), a laptop computer, and the like.

Reference throughout this specification to “one embodiment” or “an embodiment” means that a particular feature, structure, or characteristic recited in connection with the embodiment is at least included in the one embodiment disclosed. Thus, the phrases “in one embodiment,” “within an embodiment,” or “according to an embodiment” (or other phrases having a similar introduction) are not necessarily all referring to the same embodiment. .

The term "exemplary" as used herein means "serving as an example or illustration." The meaning of “exemplarily” in any embodiment should not be construed as necessarily preferred or advantageous over other embodiments. Moreover, particular features, structures, or characteristics may be combined in any suitable manner within one or more embodiments. Also, depending on the context of this disclosure, singular terms may include a corresponding plural form and plural terms may include a corresponding singular form. Similarly, hyphenated terms such as "two-dimensional", "pre-set", or "pixel-by-pixel" may be used with corresponding non-corresponding terms such as, for example, "two-dimensional", "pre-set", or "specific pixel". The hyphenated versions can sometimes be used interchangeably, and capitalized items (e.g., "Counter Clock", "row select", "PIXOUT", etc.) interchangeably correspond to non-capitalized versions (e.g. , "counter clock", "row selection", "pixout", etc.). These occasional interchangeable uses are not to be regarded as inconsistent.

The terms “connected,” “operably connected,” “connected,” “connected,” “electrically connected,” and the like referred to herein generally refer to being electrically/electronically connected in an operative manner herein. can be used interchangeably to indicate

Similarly, if a first entity electrically transmits and/or receives information signals (including address, data, or control information) to/from a second entity, regardless of signal type (analog or digital) ( If receiving (via wire or wireless), the first entity is considered to be “communicating” with the second entity (or entities). It should be noted that the various figures (including component figures) shown and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are shown for illustrative purposes only.

The terms "first", "second", etc., as used herein, are used as labels for preceding nouns, except as explicitly defined, in any form of order (e.g., spatial , temporal, logical, etc.) Also, the same reference number may be used across two or more drawings to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. This use is for ease of description and brevity, and does not imply that configurations or structural details of such components or units are the same across all embodiments, and that such commonly referenced parts/modules do not serve as guidelines for a particular embodiment. It does not just mean that it is a way to implement

It is observed here that the already mentioned 3D technology has many drawbacks. For example, a TOF-based 3D imaging system may require high power to drive optical or electronic shutters. These systems typically operate over ranges of a few meters to tens of meters, but the resolution of these systems is reduced for measurements over short distances, making 3D imaging within distances of the order of one meter impractical. . Therefore, a TOF system is not required for cell phone-based camera applications where most pictures are taken at close range. A TOF sensor may also require specific pixels with large pixel sizes, usually 7 microns (μm) or more. These pixels may also be susceptible to ambient light.

The stereoscopic imaging scheme only works for textured surfaces. It has high-computational complexity because it requires matching features and finding a counterpart between images of a stereo pair of objects. This is an undesirable feature in applications that require high system power and need to save power, such as in smartphones. In addition, stereo imaging requires high-bit resolution sensors with two regular, two lenses, making the entire assembly unsuitable for applications in portable devices such as cell phones and tablets, where devices are often valued as property. make

The SL scheme involves distance ambiguity and also requires high system power. For 3D depth measurement, the SL method may require multiple images with multiple patterns which increases computational complexity and power consumption. Also, SL imaging may require regular image sensors with high bit resolution. Therefore, structured light-based systems may not be suitable for low-cost, low-power, small image sensors in smartphones.

In contrast to the 3D technology described above, certain embodiments of the present invention provide a technology for implementing a low-power, 3D imaging system in portable electronic devices such as smart phones, tablets, and terminals. A 2D image sensor as a particular embodiment disclosed herein can capture both 2D RGB (red, green, blue) images and 3D depth measurement with visible light laser scanning. Meanwhile, ambient light may be rejected during 3D depth measurement. Although, in the description below, visible light lasers will often be referred to as light sources for point scan and 2D RGB sensors as image/light capture devices, it should be understood that such references are intended for consistency of illustration and description only. Applications of low-power, consumer mobile electronic devices having cameras in smartphones, tablets, terminals (UEs), and the like can be seen in the visible laser and RGB sensor-based examples described below. It will be appreciated that the invention disclosed herein is not limited to the visible laser RGB sensor based examples discussed below. Rather, according to embodiments of the invention disclosed herein, a point scan based 3D depth measurement and ambient light rejection method can be performed using various combinations of 2D sensors and laser light sources (for point scan), The laser light sources include, but are not limited to, (i) a 2D color (RGB) sensor having a visible light laser source being a laser source generating an R, G, B light laser or a combination of these lights; (ii) a visible light laser with a 2D RGB color sensor with an infrared (IR) cut filter; (iii) a Near Infrared (NIR) laser with a 2D IR sensor; (iv) an NIR laser with a 2D NIR sensor; (v) an NIR laser with a 2D RGB sensor (without an IR cut filter); (vi) an NIR laser with a 2D RGB sensor (without an NIR cut filter); (vii) a 2D RGB-IR sensor with visible or NIR laser; (viii) a 2D RGBW (red, green, blue, white) sensor with either visible or NIR laser; and so forth.

During 3D depth measurement, the entire sensor can operate as a binary sensor in cooperation with the laser scan to reconstruct the 3D content. In certain embodiments, the sensor's pixel size may be as small as 1 micron. Also, due to the low bit resolution, analog-to-digital converter (ADC) units within an image sensor according to certain embodiments disclosed herein require significantly less processing power compared to the power required for higher bit resolution sensors within a typical 3D imaging system. may require Because, since the 3D imaging module according to the embodiment of the present invention requires low system power, the need for less processing power can be well suited for application in low-power devices such as smart phones.

In a particular embodiment, the invention disclosed herein uses a triangular and point scan with a laser light source for 3D depth measurement with a group of line sensors. The laser scanning plane and imaging plane are oriented to use epipolar geometry. According to one embodiment of the present invention, an image sensor may use timestamps to remove ambiguity within a triangulation approach, thereby reducing the amount of depth calculations and system power. Each pixel within the same image sensor or image sensor can be used in 3D laser scan mode as well as normal 2D (RGB color or non-RGB) imaging mode. In laser scan mode, however, the resolution of the ADCs in the image sensor is reduced to a binary output (only 1-bit resolution), so that within the chip incorporating the image sensor and related processing units, the read speed is improved and, for example, within the ADC units Power consumption due to switching is reduced. Moreover, since the point scan scheme allows the system to take all measurements within one pass, latency for depth measurement is reduced and motion blur is reduced.

As noted above, in particular embodiments, the full image sensor may be used for routine 2D RGB color imaging using ambient light as well as 3D depth imaging using visible laser scans, for example. Such dual use of the same camera unit can save space and cost for mobile devices. Moreover, in certain applications, visible lasers for 3D applications may be better for user eye stability compared to Near Infrared (NIR) lasers. The sensor can have a higher quantization coefficient in the visible spectrum than in the NIR spectrum, resulting in lower power consumption of the light source. In one embodiment, the dual-use image sensor may properly operate in a linear mode for 2D imaging as a regular 2D sensor. However, for 3D imaging, the sensor can operate in linear mode under normal light conditions and in logarithmic mode under strong ambient light to facilitate continued use of visible laser sources through rejection of strong ambient light. In addition, ambient light rejection may also be required for NIR lasers, unless the bandwidth of the passband of the IR cut filter applied with the RGB sensor is very narrow.

1 shows a highly simplified configuration of system 15 according to one embodiment of the present invention. As shown, system 15 may include an imaging module 17 that is coupled to and communicates with a processor or host 19 . System 15 may include a memory module 20 coupled to processor 19 . The memory module 20 stores information content such as image data received from the imaging module 17 . In a particular embodiment, the entire system 15 may be encapsulated within a single IC or chip. Alternatively, the modules 17, 19, and 20 may be implemented on separate chips. Furthermore, memory module 20 may include more than one memory chip, and processor module 19 may include multiple processing chips as well. In any case, details regarding the packaging of the modules in Figure 1 and how the modules are fabricated and implemented (either on a single chip or using multiple discrete chips) are not described in an embodiment of the present invention and such details are not provided herein. don't

System 15 may be a low power electronic device configured for 2D and 3D camera applications according to an embodiment of the present invention. System 15 may be portable or non-portable. Although one embodiment of the portable system 15 is not limited. Mobile devices, cell phones, smartphones, user equipment (UE), tablets, digital cameras, laptop or desktop computers, electronic smartwatches, machine-to-machine (M2M) communication units, virtual reality (VR) devices or popular consumer electronic tools such as modules, robots, and the like. On the other hand, one embodiment of the non-portable system 15 is a game console in a video arcade, an interactive video terminal, a vehicle, a machine system, an industrial robot, a VR device, a driver-side mounted camera in a vehicle (eg, whether the driver is drowsy driving) monitoring), etc. 3D imaging functionality provided in accordance with an embodiment of the present invention is not limited to, but is not limited to, virtual reality applications on a virtual reality device, online chatting/gaming, 3D texting, products (e.g., calories in a piece of food item). contents) for various applications such as searching online or local (device-based) catalogs/databases, robotics and machine vision applications, automotive applications, car driving applications, etc. can be used

In certain embodiments described herein, imaging module 17 may include light source 22 and image sensor unit 24 . As described in more detail with reference to FIG. 2 below, in one embodiment, light source 22 may be a visible laser. In another embodiment, the light source may be a NIR laser. The image sensor unit 24 may include a pixel array and auxiliary processing circuitry as shown in FIG. 2, which will also be described below.

In one embodiment, processor 19 may be a CPU, which may be a general purpose microprocessor. In the description of the present invention, the terms "processor" and "CPU" may be used interchangeably for convenience of description. However, instead of or in addition to a CPU, processor 19 may be any other processor, such as, but not limited to, a microcontroller, digital signal processor (DSP), graphics processing unit (GPU), dedicated application integrated circuit (ASIC) processor, or the like. It will be appreciated that it may include a tangible processor. Additionally, in one embodiment, processor/host 19 may include one or more CPUs capable of operating in a distributed processing environment. Processor 19 is a MIPS (Microprocessor without Interlocked Pipeline Stages) that relies on, but is not limited to, x86 instruction set architecture (32-bit or 64-bit versions), Power PC® ISA, or Reduced Instruction Set Computer (RISC) ISA. ) to execute instructions and process data according to a particular instruction set architecture, such as an instruction set architecture. In one embodiment, the processor 19 may be a system on chip (SOC) having additional functions in addition to the functions of a CPU.

In a particular embodiment, the memory module 20 is a DRAM-based three-dimensional stack, such as but not limited to synchronous DRAM (SDRAM), or a DRAM-based memory module, such as but not limited to a High Bandwidth Memory (HBM) module or a Hybrid Memory Cube (HMC) memory module. (3DS) may be a dynamic random access memory (DRAM) such as a memory module. In other embodiments, memory module 20 may be a Solid-State Drive (SSD), non-3DS DRAM module, or other semiconductor-based storage system, including but not limited to Static Random Access Memory (SRAM), Phase-Change Random Access Memory (PRAM or PCRAM), Resistive Random Access Memory (RRAM or ReRAM), Conductive-Bridging RAM (CBRAM), Magnetic RAM (MRAM), Spin-Transfer Torque MRAM (STT-MRAM) ), and the like.

FIG. 2 illustrates an exemplary operational configuration of system 15 of FIG. 1 according to one embodiment of the present invention. System 15 may be used to obtain depth information (along the Z axis) for a 3D object, such as 3D object 26, which may be an individual object or an object within a scene. In one embodiment, depth information may be calculated by the processor 19 based on scan data received from the image sensor unit 24 . In another embodiment, the depth information may be calculated by the image sensor unit 24 itself, as in the case of the image sensor unit in the embodiment of FIG. 7A. In particular embodiments, the depth information is 3D to allow a user of the system 15 to interact with the 3D image of the object or to use the 3D image of the object as part of a game or other application running on the system 15. It can be used by processor 19 as part of a user interface. According to an embodiment of the present invention, 3D imaging can be used for other purposes or applications, and can be applied to practically any scene or 3D objects.

2 the X axis is taken to be horizontal along the front of the device 15, the Y axis is the vertical direction (outside the page in the figure), and the Z axis is the general orientation of the object 26 within which the device is imaged ( 15) and extended away from it. For depth measurements, the optical axes of modules 22 and 24 may be parallel to the Z axis. Other optical arrangements may be suitably used to implement the principles disclosed herein, and alternative arrangements thereof should be construed as falling within the scope of the technical spirit of the present invention.

The light source module 22 may illuminate the 3D object 26 as exemplarily shown by arrows 28 and 29 associated with corresponding dotted lines 30 and 31 . The dotted lines represent optical radiation that can be used to point scan a 3D object 26 in an optical field scene or an irradiation path of a light beam. The line-by-line point scan of the surface of an object uses an optical radiation source, which in an embodiment can be a laser light source 33 driven and controlled by a laser controller 34. can be performed by A light beam from the laser source 33 may be point scanned, in the X-Y direction, across the surface of the 3D object 26 via the projection optics 35 under the control of the laser controller 34 . The point scan may project light spots on the surface of the 3D object along a scan line, as described in more detail below with reference to FIGS. 4 and 5 . The projection optics may be focusing lenses, a glass/plastic surface, or other cylindrical optical element that focuses the laser beam of laser 33 as a point or spot onto that surface of object 26 . In the embodiment of FIG. 2 , the convex structure is shown as a focusing lens 35 . However, other suitable lens designs may be selected for the projection optics 35 in addition to this. The object 26 may be placed at a convergence position when irradiation light from the light source 33 is focused by the projection optics 35 as a light spot. Thus, within a point scan, a point or narrow area/spot on the surface of the 3D object 26 can be sequentially illuminated by the focused light beams from the projection optics 35 .

In a particular embodiment, the light source (33: or irradiation source) is a diode laser, a light emitting diode (LED) emitting visible light, a NIR laser, a point light source, or a monochromatic irradiation source within the visible light spectrum (eg, a white lamp and a monochromator). combinations, etc.), or other types of laser light sources.

Laser 33 may be fixed in one position within the housing of device 15, but may rotate in the X-Y direction. Laser 33 may be X-Y addressed (eg, by laser controller 34) to perform a point scan of 3D object 26. In one embodiment, visible light may be substantially green light. Visible light illumination from the laser light source 33 can be projected onto a 3D surface (not shown) using a mirror (not shown), or the point scan can be done entirely without a mirror. In a particular embodiment, the light source module 22 may include added or subtracted components compared to those shown in the exemplary embodiment of FIG. 2 .

In the embodiment of FIG. 2 , light reflected from a point scan of object 26 may travel along a collection path indicated by arrows 36 and 37 and dotted lines 38 and 39 . When light is received from the laser light source 33 , the light condensing path may transfer photons reflected or scattered from the surface of the object 26 . It should be understood that the depiction of the various propagation paths using solid arrows and dotted lines in FIG. 2 (which applies to FIGS. 4 and 5 as well) is for illustrative purposes only. Depictions should not be construed as depicting actual optical signal propagation paths. In practice, the illumination and collection signal paths may differ from those shown in FIG. 2 . As shown in FIG. 2, it may not be clearly defined.

Light received from the irradiated object 26 may be focused on one or more pixels of the 2D pixel (pixel) array 42 via collection optics 35 in the image sensor unit 24 . Like projection optics 35 , collection optics 44 may include focusing lenses, glass/plastic surfaces, or other cylindrical objects that focus the projected light received from object 26 onto one or more pixels within 2D array 42 . It may be an optical element of. In the embodiment of Figure 2. The convex structure is shown as a focusing lens 44 . However, any other suitable lens design may be selected for collection optics 44 . Also, for ease of explanation, only a 3 x 3 pixel array is shown in FIG. 2 (also in FIG. 6). However, it will be appreciated that such state-of-the-art pixel arrays contain thousands or even millions of pixels. The pixel array 42 may be an RGB pixel array in which different pixels may collect light signals of different colors. As noted above, in certain embodiments, pixel array 42 may include any 2D RGB sensor, such as a 2D RGB sensor with an IR cut filter, a 2D IR sensor, a 2D NIR sensor, a 2D RGBW sensor, a 2D RGB-IR sensor, and the like. may be a sensor. As will be described in more detail below, system 15 uses the same pixel array for a 2D RGB color image of object 26 (or a scene containing the object) as well as a 3D image of object 26 (including depth measurements). (42) can be used. Additional structural details of pixel array 42 are discussed below with reference to FIG.

The pixel array 42 converts the received photons into corresponding electrical signals, which are processed by an associated image processing unit 46 to determine a 3D depth image of the object 26 . In one embodiment, image processing unit 46 may use triangulation for depth measurement. The triangulation method will be described later with reference to FIG. 4 . The image processing unit 46 may include related circuits for controlling the operation of the pixel array 42 . An example of image processing and control circuits is illustrated in FIGS. 7A and 7B , and described further below.

The processor 19 may control operations of the light source module 22 and the image sensor unit 24 . For example, system 15 may have a mode switch (not shown) controllable by a user to switch from a 2D image mode to a 3D image mode. When the user selects the 2D image mode using the mode switch, the processor 19 can activate the image sensor unit 24 but disable the light source module 22 since 2D imaging can use ambient light. have. On the other hand, if the user selects a 3D image using the mode switch, processor 19 may activate both modules 22 and 24, and ambient light may be too strong to be rejected as linear mode (described below). As in the case of an unavailable number, it may also trigger a change in the level of the reset (RST) signal in the image processing unit 46 to switch the image from linear mode to logarithmic mode. Processed image data received from image processing unit 46 may be stored in memory 20 by processor 19 . Processor 19 may also display a 2D or 3D image selected by the user on a display screen (not shown) of device 15. Processor 19 may be programmed with software or firmware to perform various processing tasks described herein. Alternatively or additionally, processor 19 may include programmable hardware logic circuits to perform some or all functions of processor 19 . In a particular embodiment, memory 20 may include program code, look-up tables (as shown in FIG. 9, and will be described below), and/or memory 20 to enable processor 19 to perform functions of processor 19. Intermediate calculation results can be stored for

3 is an exemplary flow chart 50 showing how a three-dimensional depth measurement is performed according to one embodiment of the present invention. The various operations shown in FIG. 3 may be performed by one module, a combination of modules, or system components within system 15. In the description disclosed herein, only through embodiments, specific tasks are described as being performed by special modules or system components. Other modules or system components will be suitably configured to perform those tasks well.

In FIG. 3, at block 52, system 15 (more specifically processor 19) performs a one-dimensional (1D) point scan of a 3D object, such as object 26 in FIG. It can be performed along the scanning line using a light source such as the light source module 22 of As part of a point scan, light source module 22 may be configured, for example by processor 19, to project a series of light spots onto the surface of 3D object 26 in a line-by-line manner. At block 54, pixel processing unit 46 within system 15 may select a row of pixels within an image sensor, such as 2D pixel array 42 in FIG. The image sensor 42 has a plurality of pixels arranged in a 2D array forming an image plane, with selected rows of pixels forming an epipolar line of a scanning line (at block 52) on the image plane. A brief description of the epipolar geometry is provided below with reference to FIG. 4 . At block 56, pixel processing unit 46 may be operatively configured by processor 19 to detect each light spot using a corresponding pixel within the row of pixels. It is observed here that light reflected from an irradiated light spot is detected by a single pixel or more than one pixel, such that light reflected from an irradiated light spot is focused by collection optics 44 onto two or more adjacent pixels. . On the other hand, light reflected from two or more light spots may be collected at a single pixel in the 2D array 42 . The timestamp-based method discussed below avoids depth calculation-related ambiguities resulting from images of two different spots by the same pixel or an image of a single spot by two different pixels. ) is removed. At block 58, as suitably configured by processor 19, image processing unit 46 performs pixel-by-pixel detection (block 56) of a corresponding light spot in a series of light spots (within a point scan at block 52). ) to generate pixel-by-pixel output. Consequently, at block 60, the image processing unit 46 projects at least the pixel-by-pixel output (at block 58) and the corresponding light spot (at block 52) for the corresponding light spot on the surface of the 3D object. The 3D distance (or depth) can be determined based on the scan angle used by the light source for the image. Depth measurement will be described in more detail with reference to FIG. 4 .

4 is an exemplary diagram showing how a point scan is performed for 3D depth measurement according to an embodiment of the present invention. In Fig. 4, the XY rotation capability of the laser source 33 is indicated by the arrows 62 and 64 representing the angular motions of the laser in the X direction (with angle "β") and in the Y direction (with angle "α"). is shown using In one embodiment, laser controller 34 may control the XY rotation of laser source 33 based on scanning commands/input received from processor 19 . For example, if the user selects a 3D imaging mode, processor 19 may instruct laser controller 34 to initiate a 3D depth measurement of the object surface facing projection optics 35 . Laser controller 34 may, in response, initiate a 1D XY point scan of the object surface through XY movement of laser light source 33 . As shown in FIG. 4, the laser 33 projects light spots along 1D horizontal scanning lines (S _R 66 and S _R+1 68 are distinguished by dotted lines in FIG. 4). By doing so, the surface of the object 26 can be point-scanned. Because of the curvature of the surface of object 26, light spots 70-73 may form a scanning line S _R 66 in FIG. For convenience and clarity of explanation, light spots constituting the scan line S _R+1 (68) are not identified using reference numerals. Laser 33 may scan object 26 along rows R, R+1, etc. (e.g., one spot at a time in a left-to-right direction). . The values of rows R, R+1, etc. may be associated with rows of pixels within the 2D pixel array 42, so these values are known. For example, within the 2D pixel array 42 in FIG. 4, pixel row R is identified using reference numeral 75 and pixel row R+1 is identified using reference numeral 76. It will be appreciated that rows R and R+1 are selected from a plurality of rows of pixels for illustrative purposes only.

A plane containing the rows of pixels in the 2D pixel array 42 may be referred to as an image plane, whereas a plane containing scanning lines such as lines S _R and S _R+1 may be referred to as a scanning plane. can In the embodiment of FIG. 4 , the image plane and the scanning plane are such that each row of pixels R, R+1 and so forth has a corresponding scanning line S _R and S _R+1 within the 2D pixel array 42. ) and others are directed to use epipolar geometry. A row of pixels R is relative to the corresponding scanning line SR if the projection of the illuminated spot (within the scanning line) into the image plane forms a distinct spot along the line being the row R itself. can be considered to have the same polarity. For example, in FIG. 4 , arrow 78 represents the illumination of light spot 71 by laser 33 , while arrow 80 indicates that light spot 71 is illuminated by condensing lens 44 . indicates being imaged or projected along row R 75 by . Although not shown in FIG. 4, it is observed that all of the light spots 70-73 can be imaged by corresponding pixels within row R. Thus, in one embodiment, the physical arrangements, such as the position and orientation of lasers 33 and pixel array 42, correspond to corresponding rows within pixel array 42 (rows of pixels form the epipolar lines of the scanning line). Light spots irradiated within the scanning line on the surface of the object 26 may be set to be captured or detected by the pixels within.

It will be appreciated that within the 2D pixel array 42 the pixels may be arranged in rows and columns. The irradiated light spots can be referenced by corresponding rows and columns in the pixel array 42 . For example, in FIG. 4, light spot 71 in scanning line SR is X _R to indicate that spot 71 has been imaged by row R and column i (Ci) in pixel array 42. _{, can be specified as i} . Column Ci is indicated by a dotted line 82. Other irradiated spots can be similarly identified. As noted above, light reflected from two or more light spots may be received by a single pixel within a row, or light reflected from a single light spot may be received by one or more pixels within a row of pixels. A timestamp-based approach, which will now be described, can disambiguate depth calculations arising from such multiple or overlapping projections.

In FIG. 4 , an arrow with reference numeral 84 indicates the depth or distance Z (along the Z axis) of the light spot 71 from the X axis, such as the X axis shown in FIG. 2 along the front of the device 15 . In FIG. 4 , the dotted line with reference numeral 86 represents an axis that can be visualized as being contained within a vertical plane containing the projection optics 35 and the collection optics 44 . For convenience of explanation of the triangulation method, the laser source 33 is shown as being on the X-axis 86 instead of the projection optics 35 in FIG. 4 . In the triangulation method, the value of Z can be determined using the equation

The parameters in equation (1) above are also shown in FIG. 4 . Based on the physical configuration of the device 15, the values for the parameters on the right side of equation (1) can be determined in advance. Parameter h in equation (1) is the distance (along the Z-axis) between the collection optics 44 and the image sensor 42 (assumed to lie in the vertical plane behind the collection optics 44); parameter d is the offset distance between the light source 33 and the collection optics 44 associated with the image sensor 24; Parameter q is the offset between the collection optics 44 and the pixel detecting the corresponding light spot (in the example of Fig. 4, the detection/imaging pixel i is represented by the column Ci associated with the light spot X _R,i (7)1). is the distance; the parameter θ is the scan angle or beam angle of the light source with respect to the light spot under consideration (light spot 71 in the example of Fig. 4). Alternatively, the parameter q may be regarded as an offset of a light spot within the scene field of pixel array 42 .

It is shown through equation (1) that the parameters θ and q vary for a given point scan, while the other parameters h and d are essentially fixed due to the physical geometry of the device 15. Since row R 75 is an epipolar of the scanning line S _R , the depth difference or depth profile of object 26, as indicated by the values of parameter q for difference light spots imaged in the horizontal direction, is equal to It can be reflected as much as the shift. As described below, according to a particular embodiment disclosed herein, a timestamp-based method may be used to find a correspondence between the pixel position of the captured light spot and the corresponding scan angle of the laser source 33 . On the other hand, the timestamp may represent the relationship between the values of the parameters θ and q. From the known value of the scan angle θ and the corresponding position of the imaged light spot (as indicated by parameter q), the distance Z to the light spot can be determined using trigonometric equation (1).

The use of triangulation for distance measurement is disclosed in related literature including, for example, US Patent Application Publication No. 2011/0102763 A1 (Brown et al). The description in Brown's above publication relating to triangulation-based distance measurement is incorporated herein by reference in its entirety.

5 shows exemplary timestamping for light spots scanned according to an embodiment of the present invention. Additional details of the generation of individual timestamps will be discussed later with reference to the description of FIG. 8 . Compared to FIG. 4 , in the embodiment of FIG. 5 , collection optics 44 and laser 33 are shown in an offset arrangement to reflect the actual physical geometry of these components, as shown in the embodiment of FIG. 2 . Illustratively, scanning line 66 is shown in FIG. 5 along with corresponding light spots 70-73. As mentioned above, the corresponding light spots can be projected based on a point scan from left to right of the object surface by the sparse laser point source 33 . Therefore, as shown, the first light spot 70 can be projected at time t ₁ , the second light spot 71 can be projected at time t ₂ , and other light spots are the same. These light spots can be detected/imaged by each of the pixels 90-93 within pixel row R 75, which is the epipolar line of the scanning line S _R as described above. In one embodiment, the charge collected by each pixel is output to the image processing unit 46 for determining the depth per pixel as described below when light spot detection is performed in the form of an analog voltage. can Analog pixel outputs (pixouts) are indicated centrally by arrow 95 in FIG. 5 .

As shown in FIG. 5 , each detection pixel 90-93 within row R may have an associated column number (here, columns C ₁ to C ₄ ). Furthermore, it is seen from Fig. 4 that each pixel column C _i (i = 1, 2, 3, ...) has an associated value for the parameter q in equation (1). Therefore, as will be described later, if a per-pixel timestamp (t ₁ -t ₄ ) is generated for the detection pixels 90-93, the timestamp may indicate the number of columns of pixels, which is the per-pixel value of the parameter q. becomes In an additional embodiment, spot-by-spot detection using pixels within pixel array 42 causes image processing unit 46 to link each timestamp to a corresponding irradiated spot. After all, since the laser 33 is properly controlled to irradiate each spot within the required sequence with preset values, it allows linking to the scan angle θ for each spot. Accordingly, the timestamps are each in the form of values for the parameters (θ and q) in equation (1), for the pixel location of the captured laser spot and for each pixel-by-pixel signal received from the pixel array 42. Provides a correspondence between scan angles. As described above, the values of the scan angle and the corresponding position of the detected spot in the pixel array 42, as reflected through the value of the parameter q in equation (1), can allow the depth determination for the light spot. have. In this way, a 3D depth map of the surface of object 26 within the field scene of pixel array 42 may be created.

6 shows exemplary detailed circuitry of two-dimensional pixel array 42 and some of the associated processing circuitry within image processing unit 46 of image sensor 24 in FIGS. 1 and 2, in accordance with one embodiment of the present invention. show As noted above, while pixel array 42 is shown as having nine pixels 100-108 arranged as a 3x3 array for ease of illustration only, in reality the pixel array may consist of tens to thousands of pixels in multiple rows and columns. It may contain millions of pixels. In one embodiment, each pixel 100 - 108 may have the same configuration as shown in FIG. 6 . In the embodiment of FIG. 6 , the 2D pixel array 42 includes a Complementary Metal Oxide Semiconductor (CMOS) array in which each pixel is a Four Transistor Pinned Photo-diode (4T PPD) pixel. For convenience of illustration, constituent circuit elements of only the pixel 108 are denoted by reference numerals. The description below of the operation of pixel 108 applies equally to the other pixels 101-107, and the operation of each individual pixel is not described here.

The 4T PPD pixel 108 (other pixels 101-107 are similar) is connected as shown in the figure, with a pinned photo-diode (PPD) 110 and four N-channel Metal Oxide Semiconductor Field Effect Transistors (NMOS) 111-114. includes Transistor 111 can operate as a transfer gate (TG), floating diffusion (FD) transistor, and broadly, 4T PPD pixel 108 can operate as follows. First, the PPD 110 converts incident photons into electrons so that an optical input signal is converted into an electrical signal in a charge domain. The transfer gate 111 can then be “closed” to transfer all photon-generated electrons from the PPD 110 to the floating diffusion. Thus, signals in the charge domain are converted to the voltage domain for convenience of subsequent processing and measurement. The voltage at the floating diffusion can later be transmitted as a PIXOUT signal to the ADC using transistor 114 and converted into an appropriate digital signal for subsequent processing. Further details of PIXOUT generation and processing will be described below with reference to FIGS. 8, 10, and 11.

6, row decoder/driver 116 in image processing unit 46 provides three different signals to control the operation of pixels in pixel array 42, resulting in a per-column PIXOUT signal. s 117-119 are shown. In the embodiment of FIG. 5, output 95 may centrally represent those PIXOUT signals 117-119. A row select (RSEL) signal may be provided to select the appropriate row of pixels. In one embodiment, the row to be selected is the epipolar line of the current scanning line (of light spots) projected by the laser source 33 . Row decoder/driver 116 may receive address or control information for a row to be selected via row address/control inputs 126, for example, from processor 19. For purposes of this discussion, it is assumed that row decoder/driver 116 selects the row of pixels that contains pixel 108. A transistor, such as transistor 114, may be connected to each RSEL line 122-124 as shown, within each row of pixels in pixel array 42. A reset (RST) signal may be applied to the pixels in the selected row to reset the pixels to a preset high voltage level. RST signals 128-130 for each row are shown in FIG. 6 and will be described in more detail with reference to waveforms in FIGS. 8, 10, and 11. A transistor, such as transistor 112, within each pixel, may receive each RST signal as shown. A transmit (TX) signal may be provided to initiate transmission of the pixel-by-pixel output voltage (PIXOUT) for subsequent processing. TX lines 132-134 for each row are shown in FIG. A transfer gate transistor such as transistor 111 may receive each TX signal as shown in FIG. 6 .

As noted above, in particular embodiments disclosed herein, the remaining components within 2D array 42 and image sensor unit 24 may be used for 2D RGB (or non-RGB) as well as 3D depth measurement. As a result, as shown in FIG. 6, the image sensor unit 24 includes circuits for Correlated Double Sampling (CDS) as well as column-specific ADCs, one per column of pixels, to be used during 2D and 3D imaging. It may include a pixel column unit 138 having. The pixel column unit 138 may receive and process the PIXOUT signals 117-119 to generate a digital data output (Dout) signal 140 from which a 2D image may be created or a 3D depth measurement may be obtained. Pixel column unit 138 may also receive reference input 142 and ramp input 143 during processing of PIXOUT signals 117-119. More detailed operation of the pixel column unit 138 will be described later. In the embodiment of FIG. 6 , column decoder unit 145 is coupled to pixel column unit 138 . The column decoder 145 may receive, for example, a column address/control input 147 from the processor 19 for a column to be selected together with a given row select (RSEL) signal. Column selection can be sequential, enabling sequential reception of pixel outputs from each pixel selected by the corresponding RSEL signal. The processor 19 may recognize the currently projected scanning line of light spots and provide appropriate row address inputs to select a row of pixels forming an epipolar line of the current scanning line, and may also provide an individual row address within the selected row. Appropriate column address inputs may be provided to enable the pixel column unit 138 to receive outputs from the pixels.

Although the discussion herein is focused primarily on the 4T PPD pixel design shown in FIG. 6 for 2D and 3D imaging in accordance with the spirit of the present invention, other types of pixels may be used within pixel array 42 in other embodiments. will be able to use For example, in one embodiment, each pixel in the pixel array 42 may be a 3T pixel omitting a transfer gate transistor, such as transistor 111 in the 4T PPD design in FIG. 6 . In other embodiments, 1T pixels or 2T pixels may also be used. In another embodiment, each pixel in pixel array 42 may have a shared transistor pixel configuration in which transistors and readout circuits may be shared between two or more neighboring pixels. In a shared transistor pixel configuration, each pixel may have at least one photodiode and one transfer gate transistor, and the remainder of the transistors may be shared between two or more pixels. One example of such a shared transistor pixel is a 2 shared (1x2) 2.5T pixel where 5 transistors (T) are used for 2 pixels resulting in a 2.5T/pixel configuration. Another example of a shared transistor pixel that can be used in pixel array 42 is a 1x4 4 shared pixel in which four pixels share a readout circuit, but each pixel has at least one photodiode and one TX transistor. Other pixel configurations than those enumerated herein may be implemented as appropriate for 2D and 3D imaging in accordance with the spirit of the disclosed subject matter.

7A is an exemplary configuration diagram of an image sensor unit such as image sensor unit 24 in FIG. 6 according to an embodiment of the present invention. For the avoidance of complexity, a brief description of the configuration within FIG. 7A is provided herein, further details of relevant operation will be described below with reference to FIGS. 8, 10, and 11. As shown, the image sensor unit 24 in FIG. 7A may include a row decoder unit 149 and a row driver unit 150, both of which correspond to the row decoder/driver 116 in FIG. Can be included collectively. Although not shown in FIG. 7A, row decoder unit 149 receives a row address input (such as input 126 in FIG. 6) from processor 19, for example, and transmits the appropriate RSEL, RST, and TX signals to the row decoder. The input may be decoded to enable the row driver unit 150 to provide to the row selected/decoded by (149). Row driver unit 150 may also send control signals (not shown) to, for example, processor 19 to allow row driver unit 150 to apply appropriate voltage levels for the RSEL, RST, and TX signals. can be received from In image sensor unit 24 in FIG. 7A , column ADC unit 153 may represent pixel column unit 138 in FIG. 6 . For convenience of illustration, in FIG. 7A , various driver signals for each row, such as RSEL, RST, and TX signals from the row driver 150, will be collectively referenced using a single reference numeral 155. Similarly, column-by-column pixel outputs (PIXOUTS), such as PIXOUT signals 117-119 in FIG. 6, will all be collectively referenced using a single reference number 157. The column ADC unit 153 receives the PIXOUT signals 157, a reference input 142 (from the reference signal generator 159), and a ramp signal 143 into a pixel-by-pixel output by a corresponding column-by-column ADC for a column of pixels. You can receive to create. 2D imaging will be described later in more detail with reference to FIG. 10 . In one embodiment, the ADC unit 153, as in the case of the pixel column unit 138 in FIG. 6, generates a CDS output (not shown) that is the difference between the reset level of the pixel and the level of the received signal. may include circuits for In a particular embodiment, 3D depth values may be combined with a 2D image to create a 3D image of an object.

The column ADC unit 153 may include a separate ADC per pixel column within the 2D array 42 . The ADC for each column may receive each ramp input 143 from the ramp signal generator 163 together with the PIXOUT signals 157 . In one embodiment, the ramp signal generator 163 may generate the ramp input 143 based on a reference voltage level received from the reference signal generator 159 . Each column-specific ADC in the ADC unit 153 may process the received inputs to generate a corresponding digital data output (Dout) signal 140 . The ADC unit 153 can receive information from the column decoder 145 about which column ADC outputs are read and sent to the Dout bus 140, the column to select for a given row to receive the appropriate pixel output. information about can be received. Although not shown in FIG. 7A, the column decoder unit 145 may receive a column address input (such as input 147 in FIG. 6) from the processor 19, and select an appropriate pixel column (a column ADC unit). 153) to decode the input. In the embodiment of FIG. 7A , the decoded column address signals may be collectively identified using reference numeral 165 .

The digital data output 140 from the ADC units may be processed by a digital processing block 167. In one embodiment for the 2D RGB imaging mode, the data output 140 for each ADC may be a multi-bit digital value substantially corresponding to the actual photon charge collected by each pixel. Meanwhile, in the 3D depth measurement mode, the data output 140 for each ADC may be a timestamp value indicating a time when each pixel detects a corresponding light spot. This time stamping approach according to an embodiment of the present invention will be described in more detail in the following description. The digital processing block 167 includes circuits that provide timing generation, image signal processing (ISP), such as processing of data outputs 140 for 2D imaging mode and processing for depth calculation for 3D imaging mode, and others. etc. may be included. In this regard, the digital processing block 167 inputs the processor 19 to perform, for example, a 2D RGB/non-RGB image on a display screen (not shown) of the device 15, or a 3D depth image on a 3D object. To enable, it may be coupled to interface unit 168 to provide processed data, such as output 170 . Interface unit 168 may include a phase locked loop (PLL) unit for generation of clock signals supporting timing generation functionality within digital processing block 167 . In addition, interface unit 168 is a mobile industry processor interface (MIPI) providing industry standard hardware and software interfaces to other components or circuit elements in device 15 for data generated by digital block 167. can include MIPI specifications support a wide range of mobile products and provide specifications for a mobile device's camera, display screen, power management, battery interface, and more. MIPI standard interfaces can produce improved operability between a mobile device's peripherals, such as a smartphone's camera or display screen, and the mobile device's application processor(s). Here, the vendor(s) providing the neighborhoods may not be the same vendor.

In the embodiment of FIG. 7A , timestamp measurement unit 171 is coupled to column ADC unit 153 to generate measurement signals 172 appropriate for the individual column-by-column ADCs, thus measuring the pixel-by-pixel timestamp in 3D measurement mode. Produces output representing values. This time stamping approach will be described in more detail with reference to FIG. 8 .

7B shows an exemplary detailed configuration of a CDS+ADC unit 175 for 3D depth measurement according to an embodiment of the present invention. For convenience of explanation, the unit 175 will be referred to as an 'ADC unit' hereinafter, but it will be appreciated that the unit 175 includes a CDS function in addition to the ADC function. A simplified version of the CDS unit is shown using capacitor 176 in FIG. 7B. In one embodiment, each column in the 2D pixel array 42 may have a single slope ADC unit per column, similar to the ADC unit 175. In the embodiment of FIG. 6, taking one ADC per column as an example, three ADC units may exist in the pixel column unit 138. As shown, in the embodiment of FIG. 7B , the ADC 175 may include two Operational Transconductance Amplifiers (OTA) 177 and 179 serially connected to the binary counter 181 and the line memory unit 183. For ease of illustration, inverting (−) and non-inverting (+) voltage inputs to OTAs 177 and 179 are shown in FIG. 7B, and biasing inputs and power supply connections are not shown. It will be appreciated that an OTA is an amplifier in which a different input voltage produces an output current. Thus, the OTA can function as a voltage controlled current source. The biasing inputs can be used to provide current or voltage to control the transconductance of the amplifier. The first OTA 177 uses the column number received from the column decoder 145 from the CDS unit 176 to obtain a CDS version of the PIXOUT voltage from a pixel such as pixel 108 in FIG. 6 selected within the active row. can receive The CDS version of the fix out signal will be referred to as 'PIX_CDS'. OTA 177 may also receive Vramp voltage 143 from ramp signal generator 163 (FIG. 7A). The OTA 177 may generate an output current when the fixout voltage 157 drops below the Vramp voltage 143 as will be described later with reference to FIG. 8 . The output of the OTA 177 may be filtered by the second OTA 179 before being applied to the binary counter 181 . In one embodiment, the binary counter 181 receives the clock (Clk) input 185 and generates an output current by the first OTA 177 based on the clock cycles counted while the set time is triggered. It can be a 10-bit ripple counter that produces timestamp value 186. In the context of the embodiment of FIG. 7A , the clock input 185 may be a system wide clock, a clock for each image sensor generated by the PLL unit 168 or another clock generator (not shown) in the device 15 . The timestamp value 186 for each pixel may be stored in the line memory 183 for the number of columns (column #) of the pixel, and may be continuously output to the digital processing block 167 as the Dout signal 140 . Column number input 165 may be received from column decoder unit 145 as shown in FIG. 7A.

In a particular embodiment, the RGB color model may be used for sensing, reproduction, and display of images on mobile devices, such as apparatus 15 in FIGS. 1 and 2 . In the RGB color model, light signals having three primary colors, such as R, G and B, can be added together in a variety of ways to create a wide array of colors in the final image. The CDS method can be used in 2D RGB imaging to measure electronic values, such as pixel/sensor output voltages, in a way that allows the removal of undesirable offsets. For example, a CDS unit such as CDS unit 176 may be applied within an ADC unit for each column such as ADC unit 175 to perform correlated double sampling. In CDS, the output of a pixel can be measured twice, once with known conditions and once with unknown conditions. A value measured under known conditions can be subtracted from a value measured under unknown conditions to produce a value that has a known correlation to the physical quantity being measured. That is, the photoelectronic charge represents the pixel-by-pixel portion of the image signal. With CDS, noise can be reduced by removing the pixel's reference voltage (such as the pixel's voltage after reset) from the pixel's signal voltage at the end of each integration period. Thus, in CDS, the reset value is sampled before the pixel's charge is transferred to the output. The reference value is "subtracted" from the value after the pixel's charge is transferred.

In a particular embodiment, it is observed that the ADC unit 175 is used for both 2D imaging and 3D depth measurement. All inputs for such a shared configuration are not shown in FIG. 7B. The Vramp signal corresponding to the case of shared use may also be different for 2D imaging.

8 is a timing diagram illustrating the timing of different signals within the system 15 of FIGS. 1 and 2 to generate timestamp-based pixel-by-pixel outputs in a three-dimensional linear mode of operation in accordance with another embodiment of the present invention. Fig. 190. As noted above, in certain embodiments, all pixels within the same image sensor 24 may be used for 2D as well as 3D imaging. 3D depth measurement can be performed using 3D linear mode or 3D logarithmic mode depending on the level of ambient light. As described below with reference to FIG. 11 , the 3D logarithmic mode can be used for depth measurement when ambient light rejection is required. The description of FIG. 8, however, relates to timing waveforms related to the 3D linear mode.

Briefly, as described above with reference to FIGS. 4 and 5 , 3D object 26 may be point scanned by laser light source 33 along row R 75 of pixel array 42 one spot at a time. . Here, R is based on the corresponding epipolar relationship with respect to the scanning line S _R (66). After scanning one row, the scanning operation repeats for the other rows. When the laser projects the next spot, the previous projected light spot can be imaged by the corresponding pixel in row R. The pixel-by-pixel outputs from all pixels in row R can be read into the depth processing circuit/module in digital processing block 167 of FIG. 7A.

To generate per-pixel output, the corresponding row may have to be initially selected using the RSEL signal. In the context of FIG. 8, row decoder/driver 116 in FIG. 6 converts a row of pixels, including pixels 106-108, by providing RSEL signal 122 at a high level as shown in FIG. Suppose you choose Thus, all pixels 106-`108 are selected together. For convenience of explanation, the same reference numbers are used in FIG. 8 for signals, inputs, or outputs as shown in FIGS. 6 and 7 . Initially, all pixels 106-108 in the selected row may be reset to a high voltage using RST line 128. The reset level of the pixel may indicate the absence of pixel-by-pixel detection of the corresponding light spot. In 3D linear mode according to an embodiment of the present invention, RST signal 128 promotes integration of photoelectrons received by pixels 106-108 and results in corresponding pixel output (PIXOUT) signals 117-119. It can be released from the high level for a preset time to obtain Facilitating the integration of photoelectrons and obtaining a corresponding pixel output signal is shown in Fig. 8 and will be described below. The PIXOUT1 signal 119 represents the output supplied by the pixel 108 to the corresponding ADC unit, which is represented using a dashed line with the pattern "------". The PIXOUT2 signal 118 represents the output supplied by the pixel 107 to the corresponding ADC unit, which is represented using a dotted line with the pattern "……". On the other hand, in 3D logarithmic mode according to an embodiment of the present invention, the RST signal may remain high for a selected low during generation of the pixel output as described below. In one embodiment, other RST lines, such as lines 129-130 in FIG. 6, can be kept high or “on” for unselected rows to prevent blooming. To be precise, the PIXOUT signals 118 and 119 in FIG. 8 (similar pixout signals in FIGS. 10 and 11) are OTA in FIG. 7B within each column-specific ADC unit, such as ADC unit 175 in FIG. Before the PIX_CDS signals are applied for the first OTA, such as 177, they may be slightly modified by a CDS unit, such as CDS unit 176 in FIG. 7B. For simplicity of illustration and ease of explanation, the PIXOUT signals in FIGS. 8, 10, and 11 are treated as representative of respective PIX_CDS signals (not shown) and are considered to have direct input to respective OTAs 177.

After resetting, if a photodiode in a pixel receives incident luminance, such as photoelectrons in light reflected from a light spot projected onto the surface of the 3D object 26, the photodiode will generate a corresponding photocurrent. can Detection of incident light by a pixel may be referred to as an "on event", whereas decreasing the density of incident light may produce an "off event". The photocurrent generated in response to the on event may decrease the pixel output voltage PIXOUT from an initial reset level. The pixels function as transducers that convert received luminance/light signals into corresponding electrical (analog) voltages as designated as PIXOUT signals in FIGS. 6, 8, 10, and 11. Each pixel may be individually read in a sequence in which corresponding light spots are projected by a laser source in an embodiment. The analog fixout signal may be converted into a digital value by a corresponding column ADC. In 2D imaging mode, the ADC can function as an ADC and generate multibit outputs. As described below, in the 3D depth measurement mode, the ADC functions as a time-to-digital converter and can generate a timestamp value representing the time when a light spot is detected by a pixel.

Referring back to Figure 8, after a pixel reset (RST 128 is high) is performed, the column ADCs associated with pixels 106-108 may of course be reset before RST is released. The transmit (TX) signal 132, however, may remain high throughout. ADCs can be reset using either a common ADC reset signal or an individual ADC-specific reset signal. In the embodiment of Figure 8. The common ADC_RST signal 192 is simply shown appearing in the column ADC unit 153 (FIG. 7A) to reset (to a high level) column-specific ADCs, such as ADC 175. In one embodiment, the ADCs can be reset after the pixel is reset to a set binary value, such as a binary "0" or other known number. 8, these reset values for the ADCs associated with pixels 108 and 107 are "fields" (194) within signals ADCOUT1 (or ADC output "A") and ADCOUT2 (or ADC output "B"), respectively. and 195), respectively. It should be noted that the term “field” is used for convenience only when describing the ADC outputs shown in FIGS. 8, 10, and 11. It will be appreciated that the ADC output may not actually contain all of these "fields" at the same time, but may be a particular digital value depending on the current stage of signal processing for the ADC. That is, if the ADC is reset, the output can be a binary “0”, and if the ADC is triggered to count clock pulses, the output can be a count value as in the case of 3D depth measurements in FIGS. 8 and 11. . If the ADC is used for 2D color imaging as in the case of FIG. 10, the output may be a multi-bit value representing an image signal. Accordingly, the ADC output signals of FIGS. 8, 10 and 11 are shown as such "fields" merely to represent different digital values, and the ADC may produce a final output in the forward direction. In FIG. 8, reference numeral "197" is used to refer to the ADCOUT1 signal representing the output of the ADC related to the pixel 108, and reference numeral "198" is used to refer to the ADCOUT2 signal representing the output of the ADC related to the pixel 107. do. Each of the outputs 197 and 198 is selected by the column decoder when each ADC reads the memory. Dout signal 140 (Figs. 6 and 7). Before being reset, ADC outputs 197 and 198 may have an unknown value, as indicated by the symbol "X" in fields 199 and 200.

After the ADC is reset, the predetermined threshold can be enabled by reapplying the ramp input (Vramp: 143) to a predefined voltage level after the pixel reset signal 128 and the ADC reset signal 192 are released. have. In the embodiment of Figure 8, the RAMP input 143 is common to all column individual ADCs, thus providing the same Vramp voltage to each ADC. However, in another embodiment, different Vramp values may be separately applied to two or more ADCs as ramp inputs for each ADC. Also, in certain embodiments, the Vramp threshold may be a programmable parameter that can be changed as desired. After the threshold value (RAMP signal) is activated, the ADCs per pixel can wait for the “ON event” of the corresponding pixel before operating the binary counter, like the counter 181 in FIG. 7B.

In 3D depth measurement mode, each ADC can produce a single bit output representing a binary “0” or “1” as opposed to multi-bit output in the case of 2D imaging mode (described below). Thus, in the case of an RGB sensor, all color information received by pixels in the RGB pixel array 42 can be effectively ignored. In the absence of incident light sensed by a pixel, the corresponding ADCOUT signal may hold a binary "0" value. Thus, without any on event the columns can hold the digital value "0" (or other known number) for their respective ADCOUT signals. When a pixel is struck by incident light as described above, however, the pixel's PIXOUT line may begin to drop at the reset level, as indicated by the downward slopes of the PIXOUT1 and PIXOUT2 signals in FIG. If pixel charge is to be read starting with the pixel receiving charge first, such as a reading starting with the rightmost pixel within a row and ending with the leftmost pixel, e.g., as in FIG. 5, then t1 is the shortest time instant; t4 is the longest time instant. Thus, in the embodiment of FIG. 8 , the output of pixel 108 (PIXOUT1) may be read before the output of pixel 107 PIXOUT2. As soon as the progressively falling PIXOUT1 reaches the Vramp threshold 143, the single bit ADCOUT1 can be flipped from binary "0" to "1". Instead of bit output "1", however, when that ADC flips the bit from "1" to "0". time can be recorded. That is, the ADC associated with pixel 108 may start a binary counter within the ADC to act as a time-to-digital converter, as indicated by the "count up" field 202 in ADCOUT1. During the “up count” period, a counter in the ADC may also count the clock pulses of the CLK signal 185 that may be applied to each ADC, for example, as shown in FIG. 7B. The counted clock pulse appears as counter clock-1 signal 204 in FIG. The counter value in the "up count" field may be provided as a per-pixel output for pixel 108. A similar counting is shown by the counter clock-2 signal 205 in FIG. 8. For the charge collected by the pixel 107 may occur at the ADC associated with the pixel 107. A per-pixel count value ("up count" field 207) may be provided by each ADC as a per-pixel output for pixel 107. After scanning all the pixels in one row, the pixel-by-pixel charge collection operation can be repeated for another row while the outputs from the previous scan row are read into the depth calculation unit in digital block 167.

Each ADC output can effectively represent a respective timestamp value providing a temporal indication of detection by a pixel of a light spot on the surface of an object illuminated by laser light source 33.

A timestamp can be considered to capture the time of arrival of light for a pixel. In one embodiment, the timestamp value is calculated by the digital processing block 167 from a count value (counted clock pulses) received from the ADC unit. It may be generated for the detected light spot. For example, digital block 167 may generate a time stamp by relating the count value to an internal system time or other reference time. Since the time stamp is generated at the receiving end, it may not necessarily indicate the exact time when the light spot is projected by the light source. The time stamp value, however, may allow digital block 167 to establish a time correlation between light spots timestamped in a particular temporal order by time correlation. That is, time correlation indicates that the distance to the initially illuminated light spot is determined first and so on until the distance to the last illuminated (illuminated) light spot is determined. In one embodiment, the time stamp approach may also facilitate resolution of ambiguities that may result from multiple light spots being imaged on the same pixel, as described below.

All ADC based counters may stop simultaneously when ramp signal 143 is provided again after a predetermined time period has elapsed. In FIG. 8 , the transition of ramp signal 143 marking the result of the predetermined time period for pixel charge integration is indicated by dashed line 210 . The RSEL 122 and RST 128 signals have changes in the level of the ramp signal (line 210, 143) and can transition to their states substantially simultaneously. In one embodiment, all ADC based counters may be reset on line 210. In another embodiment, all ADC-based counters can be reset any time prior to selection of the next row of pixels to read the pixel charge. Despite the reset of the ADC counter at the end of scanning a row of pixels, the time stamp value for each pixel in pixel array 42 may remain distinct. This is because the establishment of a relationship of timestamp values due to internal system time or other sources of time reference may leave global and continuously-running.

In the example of FIG. 8 , a post-scan pixel, e.g., pixel 107, may have an ADC output that is less than the output of a previously scanned pixel, such as pixel 108. Therefore, as shown, ADCOUT2 may have a smaller count value (or smaller counted number of clock pulses) than ADCOUT1. Or, in another embodiment, when the counter for each ADC counts when an on event is detected, such as when the pixel's fixout signal falls below a given threshold value (Vramp), the post-scan pixel is the same as the previous scan pixel. It can have an ADC output larger than a pixel.

The circuit and waveform diagrams shown in Figures 6, 8, 10, and 11 are based on single slope ADCs per column up counters. It should be understood that the timestamping scheme may be implemented as an up or down counter depending on design choices. Also, single slope ADCs with global counters can be used as well.

For example, in one embodiment, instead of using a separate column-based counter, a global counter (not shown) can be shared by all column ADCs. In this case, the ADC enters the line memory of FIG. 7B when the column-based comparator unit detects an on event, such as when the column-based comparator unit first detects the respective fixout signal drooping below the ramp threshold 143. A column memory such as (183) can be configured to latch the output of the global counter to generate the appropriate ADC individual output within each ADC.

Although not shown in FIG. 8 , the dark current offset can be removed by reducing the Vramp threshold at the same rate as the reduction of the dark current. Dark current can be a relatively small electrical current flowing through a photosensitive device, even when photons do not enter the photosensitive device, such as a photodiode. Dark current in the image sensor can cause noise or unwanted artifacts in the collected charge. Dark current may be generated by defects in pixels and may have the same result as photocurrent. Thus, due to the dark current, the pixel output can still decrease without the presence of light (or the absence of light received by the pixel). As shown in the context of row 75 in FIG. 5 and described with reference to FIG. 8 , therefore, during charge collection, pixels in a row are scanned from right to left, for example, pixels on the left are on the right. It is possible to integrate more dark current than pixels of . Thus, to prevent registration of any false events due to the dark current, a predetermined ramp threshold (Vramp) is set at the rate at which the dark current increases along a row of pixels to compensate for the reduced level of pixel output due to the dark current: can be reduced or adjusted. In one embodiment, the adjusted threshold may be used for a pixel to compare the level of the pixel's PIXOUT signal. Accordingly, the value of the threshold voltage (Vramp) can be varied and individually programmed for each ADC. In one embodiment, all pixels associated with a specific ADC may have the same Vramp value. In another embodiment, each pixel may have a programmable Vramp value for each pixel in a corresponding ADC.

It is observed herein that when a row of light spots scans along the surface of an object, two or more different spots from the scanned object can be imaged on the same pixel. Spots can be within the same scanning line or on adjacent scanning lines. When multiple spots are scanned across the surface of an object, such overlapping images can negatively affect the correlation of those spots and pixel on events, and can lead to depth measurement ambiguity. It can be seen, for example, from the aforementioned equation (1) that the depth measurement is related to the scan angle θ and the pixel location of the imaged light spot, as given by the parameter q in equation (1). Depth measurements may not be accurate if the scan angle is not precisely known for a given light spot. Similarly, depth calculation can also be ambiguous if two or more light spots have the same q value. A timestamp-based approach according to certain embodiments disclosed herein may be used to maintain an accurate correlation between the pixel position of a captured light spot and the corresponding scan angle of the laser source. That is, the time stamp may indicate a relationship between values of parameters q and θ. If both spots are on the same pixel or column from the data output point of the scene, the time-to-digital conversion within the timestamping scheme determines which light spot was first received by the imaging system, i.e., the digital processing block 167 of FIG. 7B. It is allowed to establish a temporal correlation between these two spots to establish . Such correlation may not be readily possible in systems that do not use timestamping, such as stereo vision systems as described above or systems using structured light schemes. As a result, such systems need to perform a lot of data retrieval and pixel matching to solve the corresponding problems.

In one embodiment, if multiple light spots are imaged by the same pixel, the time stamps of these light spots can be compared to identify the previously received light spot, and the distance is the distance between all light spots successively received at the same pixel. Neglecting, it can be computed only for that light spot. In this embodiment, the timestamp of the first received light spot can be treated as a pixel-by-pixel output for the corresponding pixel. Or, in another embodiment, the distance may be calculated for the last received light spot, ignoring all other light spots imaged by the same pixel. In some cases, any light spot received between the first or last light spot may be disregarded for depth calculation. Mathematically, the scan time of the light spots projected from the light source can be given as t(1),..., t(n) where t (i+1)　-　t(i)　=　d(t) (constant) . Pixel/column outputs can be given as a(0), a(1),...,a(n) timestamps for on events. a(i) is always after t(i) but before a(i+1). If a(i) and a(k) (i　　k) are associated with the same pixel/column then only one of them can be salvaged as discussed before to remove any ambiguity in the depth computation. Based on the temporal relationship between the scan time and the output time (indicated by the timestamp), a processing unit, eg digital block 167, may construct an output in which the output point(s) are missing. The processing unit may not be able to recover the missing position, but depth calculations from possible output points may be sufficient to provide an acceptable 3D depth profile of the object. It should be noted that, in one embodiment, it may be possible for two different pixels to image each part of the same light spot. In an embodiment, based on the proximity of the output timestamp values from the two pixels, the processing unit may infer that one light spot is imaged by two different pixels. To resolve the ambiguity, the processing unit can use the timestamp to find the average of each position value q, and use the average value of q in equation (1) to calculate the 3D depth for the shared light spot. .

9 exemplarily shows a look-up table (LUT:215) for illustrating the use of a look-up table (LUT) to determine 3-dimensional depth values according to another embodiment of the present invention. ) can be used instead of the triangulation-based depth calculation described above. LUT 215 lists the parameters θ, q and Z for scan line S _R . The relationship between these parameters is given by equation (1) above. The LUT 215 may pre-store the values of these parameters for multiple scan lines. Only one of the scan lines S _R is shown in FIG. 9 . The preconfigured LUT 215 may be stored within internal memory (not shown) of processor 19, system memory 20 (FIGS. 1 and 2) or stored within 167 in digital processing block (FIG. 7A). . First, to construct the LUT 215, a light spot along the scan line S _R can be projected at a reference distance Z _i using, for example, 1 meter and a specific scan angle θi. The predetermined values Z _i and θ _i may be used to obtain the corresponding value qi. The value qi may indicate the column/pixel in which the imaged spot should appear for scan line S _R . The difference values between Z _i and θ _i can be used to obtain the corresponding value qi. If there is a difference ΔZ between the actual and the preset value Z _i for the light spot within the scanning line S _R , then the corresponding column/pixel must be moved by Δq. The value of LUT 215 can be adjusted as needed. In this way, for each scanning line S _R , LUT 215 can be pre-configured with depth values Z _i as a function of θ _i and q _i using triangulation equation (1). As mentioned previously, pre-configured LUTs may be stored within the device 15. During operation, the actual values θ _i and q _i for each light spot within the scan line of the light spots projected onto the user-selected 3D object are input to a LUT such as LUT 215 to look up the corresponding value Z _i can be used as Processor 19 or digital block 167 may be configured to perform a lookup, for example. Thus, in certain embodiments, a 3D profile of an object may be created by interpolating into a LUT calculated using triangulation.

It is observed from the foregoing description that timestamp-based measurement of 3D depth using triangulation in accordance with certain embodiments disclosed herein allows the ADC to operate as a binary comparator with only a single bit low resolution. Thus, significantly less switching power is consumed within the ADC, conserving system power. Also, high bit resolution in typical 3D sensors, on the other hand, may require more processing power. Additionally, timestamp-based ambiguity resolution can save system power compared to typical image processing methods that require significant processing power to find and match pixel data to resolve ambiguity. Latency is also reduced because all depth measurements can be performed at once, imaging/detection of all point scan light spots within a single imaging step. In certain embodiments, each pixel in a pixel array can be a single storage pixel, making it as small as 1 micrometer (μm) in size. In a single storage pixel design, there can be only one photodiode and one junction capacitor per pixel (such as transistor 111 in FIG. 6) to integrate and store photoelectrons. On the other hand, a pixel with one photodiode and multiple capacitors storing photoelectrons coming from different times cannot be reduced to such a small size. Accordingly, a low-power 3D imaging system having a small sensor according to specific embodiments described herein may be easily implemented in mobile applications such as, but not limited to, cameras in smartphones or tablets.

As mentioned above, the same image sensor, for example image sensor unit 24 in FIGS. 1 and 2, can be used for both 2D (dimensional) imaging and 3D depth measurement according to an embodiment disclosed herein. Such a dual mode image sensor may be part of a camera system on a mobile phone, smartphone, laptop computer, or tablet, for example, or part of a camera system in an industrial robot or VR equipment. In certain embodiments, a mode switch may be present on the device to allow the user to select between a typical 2D camera mode or a 3D imaging mode using depth measurement as previously discussed. In typical 2D camera mode, in certain embodiments, the user can capture color (RGB) images or snapshots of a scene or a specific 3D object within a scene. In 3D mode, however, the user can create a three-dimensional image of the object based on the camera system performing point scan based depth measurement in the manner described above. In both modes, the same image sensor can be used throughout to perform the desired imaging. That is, each pixel of the image sensor can be used for two applications such as 2D or 3D imaging.

10 is a timing diagram 230 illustrating the timing of different signals within the system 15 of FIGS. 1 and 2 to generate a two-dimensional image using a two-dimensional linear mode of operation in accordance with a particular embodiment of the present invention. to be. It should be noted that under ambient light illumination, which may include occasional use of a camera flash or other similar component (not shown), a 2D image may become an RGB image of a scene or a 3D object within a scene. In contrast to the 3D imaging-related embodiments in FIGS. 8 and 11 , however, in the case of a 2D image in the embodiment of FIG. 10 , there may be no illumination by the laser light source 33 ( FIG. 2 ). Many of the signals shown in FIG. 10 are also shown in FIG. 8 . Only the salient aspects of FIG. 10 in view of the previous detailed description in FIG. 8 are described herein. It should be understood that the control signals RSEL, RST, TX, RAMP, and ADC_RST as in FIG. 10 are for a row of pixels including pixels 106-108 of FIG. 6, so for ease of explanation, these Signals are identified using the same reference numerals as those used in FIG. 8 despite differences in the timing of signals and waveforms in FIGS. 8 and 10 . Also, the illustration in FIG. 10 is for a single pixel, ie pixel 108 in FIG. 6 . Accordingly, in FIG. 10 the PIXOUT signal 119, the counter clock signal 204, and the ADCOUT signal 197 are shown using the same reference numerals as the corresponding signals PIXOUT1, counter clock 1 and ADCOUT1 in FIG. 8. Pixel output 119 is produced by linearly integrating the photoelectrons collected by pixel 108 over a period of time. As before, the description of FIG. 10 in the context of pixel 108 also applies to corresponding signals associated with other pixels in pixel array 42 .

As described above, in a specific embodiment, the ADC for each column, such as the ADC unit 175 in FIG. 7B, may be a single slope ADC. In the case of FIG. 8 the pixels within the same row may be selected and reset together, as shown by RSEL signal 122 and RST signal 128 in FIG. 10 . Column ADCs can be reset using a common ADC_RST signal 192. The reset state of the ADC associated with pixel 108 in FIG. 10 may be indicated by field 234 in ADCOUT signal 197. After the pixel 108 and corresponding ADC are reset, the threshold or reference voltage level may be enabled as indicated by the voltage level 236 for the Vramp signal 143. The ramp is ramped down from this voltage level 236 to digitize the ADC unit's comparator offset as given by field 238 of the ADCOUT signal 197. In one embodiment, clock pulses in counter clock 204, such as offset 238, may be used to generate a count value. Clock pulses may be counted from the time the ramp signal 143 reaches the threshold level 236 until the level of the ramp signal drops to the reset level of the pixel output PIXOUT. Then, each transmit (TX) line 132 can be pulsed to initiate the transfer of the charge (charge) stored in the photodiode 110 to the floating diffusion 111 for readout. While a TX pulse is provided, the Vramp signal 143 rises to the threshold level 236, and a counter in the per-pixel ADC, such as counter 181 in FIG. 7B, inverts as indicated by field 240. It can be initialized with an offset value. Inverted offset value 240 can represent the negative of offset value 238 . After the provision of the TX pulses 132 ceases, the ADC unit for the pixel 108 continues to digitize the received pixel signal PIXOUT until the Vramp threshold 143 drops to the level of the PIXOUT signal 119. can start This operation is illustrated by the "up count" field 242 in the ADCOUT signal 197. The count value 242 can be based on the clock pulses of the counter clock 204, as shown using reference numeral 243, and the offset count (in field 238) of the image signal for pixel 108. It can represent a combined value including a per-pixel part. The comparator 243 in the ADC unit may compare the digitized comparator offset value in field 238 to the up count value 242 . Thus, in one embodiment, the RGB image signal 244 can be obtained by adding the ADC values in fields 240 and 242, thus the combined value in the "up count" field 242 (offset + The operation of removing the offset value 238 from the signal) is effectively achieved.

The operation shown in FIG. 10 may be performed for each pixel of the pixel array 42 . Each column ADC may produce a corresponding RGB image signal in the form of a multi-bit output from an ADC-based counter such as counter 181 in FIG. 7B. A multi-bit output, such as the output at 244 in FIG. 10, may be needed to effectively represent the color content of an image signal. The RGB image signal outputs from the ADCs of the column ADC unit 153 form a Dout signal 140 (FIG. 7A) which can be processed by the digital block 167 for representing a 2D color image in a scene via the MIPI interface 168. and 7b).

Additional details of the 2D imaging and related waveforms in FIG. 10 can be obtained from US Pat. No. 7,990,304 B2 issued Aug. 2, 2011 to Lim et al. The 2D imaging-related descriptions in the above patents in relation to the technical spirit of the present invention disclosed herein are incorporated herein by reference in their entirety.

11 shows the timing of different signals within the system 15 of FIGS. 1 and 2 to generate timestamp-based pixel specific outputs within a 3-dimensional logarithmic (log) mode of operation in accordance with a particular embodiment of the present invention. It is a timing diagram 250 showing by way of example. As mentioned above, 3D depth measurement can be performed using either a 3D linear mode or a 3D logarithmic mode depending on the level of ambient light. Also, during 3D depth measurement, a 3D object, such as 3D object 26 of FIG. 2, may be illuminated by visible light (or other light such as NIR light) as well as ambient light from a laser scan. Thus, the 3D logarithmic mode can be used for depth measurement when the ambient light is too strong to be rejected by the 3D linear mode. In terms of CDS-based imaging to remove the offset or noise from the final image signal, a logarithmic mode may not be needed for the 2D imaging related waveforms shown in FIG. 10 . In the case of 3D depth measurement according to certain embodiments disclosed herein, however, strong ambient light can interfere with light from the laser light source during point scans. In the three-dimensional linear mode of operation, such interference may overwhelm or suppress the visible/NIR light reflected from the point-scanned light spot, resulting in inaccurate detection of light received from the light spot. Thus, in particular embodiments, rejecting pixel charge due to ambient light may be desirable when the intensity of ambient light is sensed above a certain luminance level (or intensity threshold), such as 10,000 (10K) lux. . Such ambient light cancellation can be achieved using the 3D logarithmic mode shown in FIG. 11 .

As described above, the same reference numerals are used in FIGS. 8, 10 and 11 for similarly named signals (or signals having similar functions) and for convenience of description. It will be appreciated that the signals shown in Figures 8, 10, and 11 relate to a particular mode of imaging, however. For example, the timing diagram 230 shown in FIG. 10 shows the special relationship between the signals shown therein when the user selects the operation of the 2D imaging mode. Signals with similar names in Figures 8 and 11, however, relate to the operation of the 3D imaging mode, and thus may have different timing relationships. Also, some signals between FIGS. 8 and 11 may have different waveforms because FIG. 8 relates to a 3D linear mode operation while FIG. 11 relates to a 3D logarithmic mode operation. In view of the previous detailed description of FIG. 8 , only the salient aspects of FIG. 11 will be described herein. As with FIG. 8, timing diagram 250 in FIG. 11 is also shown relative to pixels 107 and 108 in FIG. The description of FIG. 11 may, however, be applied to all other pixels in pixel array 42 .

In a three-dimensional (3D) linear mode, the per-pixel output can be generated by linearly collecting the photoelectrons collected by the pixels over a period of time. As such, in linear mode, the output voltage of a pixel is proportional to the total number of photons collected/integrated during a given period of time. In the 3D logarithmic mode, however, the output per pixel may be proportional to the natural logarithm of an instantaneous light current generated by a pixel during a predetermined time interval upon detection of laser light reflected from a 3D object. Mathematically, the photocurrent is generated by a photodiode such as the PPD 110 of FIG. 6 and can be expressed as the following relationship. in other words,

where I _ph is the photocurrent of the diode, V _ph is the voltage across the diode, and V _T is the thermal voltage. Thus, V _ph and each pixel output (PIXOUT) can be made proportional to the natural logarithm of the instantaneous diode current I _ph , just as ambient light rejection is required. As mentioned above, heavy ambient light can limit photon collection once linear integration is complete. Thus, in this situation, sensing of the instantaneous photocurrent using the 3D logarithmic mode may be more desirable.

In certain embodiments, device 15 may include an ambient light sensor (not shown). The processor 19 or digital block 167 may be configured to sense the intensity of the ambient light as soon as the 3D imaging mode is selected by the user deciding whether to use the 3D linear mode or the 3D logarithmic mode. In one embodiment, the ambient light level may be sensed substantially concurrently with the provision of an RSL signal that may indicate initiation of imaging of light reflected from the point scan light spots. In another embodiment, the ambient light level may be sensed by the laser source substantially simultaneously with the initiation of the visible light point scan. Based on the level of ambient light, the processor 19 or digital block 167 can select either a 3D linear mode or a 3D logarithmic mode of depth measurement. In another embodiment, the ambient light level may be periodically and continuously sensed during the 3D depth measurement. In this case, the 3D mode of operation can be switched from linear mode to logarithmic mode and vice versa at any time before or during the progressive imaging operation. Other approaches for sensing the ambient light level may be considered as appropriate.

Referring to the embodiment of FIG. 11 , in the 3D logarithmic mode, the row specific RST signal 128 can be asserted (or turned on “high”) (asserted) and selected during the entire period of pixel output generation. Can be held/asserted high for low. In contrast, in the three-dimensional linear mode of FIG. 8, the RST signal 128 is turned off (or de-asserted) later during the linear integration of photoelectrons, but resets the pixels in the row to a preset voltage level. can be initially provided (turned on "high") for TX signal 132 can, however, be held high as in the case of the 3D linear mode of FIG. 8 . Thus, in certain embodiments, an appropriate level of the RST signal may be used to select logarithmic mode versus linear mode. In logarithmic mode, in one embodiment, after the ADCs associated with pixels 107 and 108 are reset using RST signal 192, the ADCs output pixel output signals PIXOUT when the signals are received. The ambient level is initially sampled to enable the ADCs to perform proper consideration of the signal levels of . After the ADC resets, the RAMP threshold 143 may be activated, and the ADC counters may enter a "wait state" to wait for an "ON event" to occur at each pixel. When a pixel receives incident light (reflection from a projection light spot), the pixel's PIXOUT signal can start to fall. In contrast to the linear fall in FIG. 8 , the PIXOUT signals 118 and 119 in FIG. 11 may represent short instantaneous dips 252 and 253 , respectively, which fall in the visible light reflected by each pixel. It detects and reflects the instantaneous photocurrent generated. When the PIXOUT signals 118 and 119 reach the predetermined Vramp threshold 143, the ADC counters can start counting. All counters are given by the transition of RAMP signal 143 to a "high" state after the predetermined time for charge integration has expired, and can stop simultaneously, as indicated by dotted line 255. The count values are represented by data field 257 of ADCOUT1 and data field 259 of ADCOUT2 of pixels 108 and 107, respectively. Count values in logarithmic mode can be different from those in linear mode, so difference reference symbols are used for the "up count up" field in the ADCOUT signal in FIGS. 8 and 11 . As in the case of FIG. 8 , subsequent pixel scans may have smaller count values for the ADC output than count values of previous scans.

As mentioned earlier with reference to FIG. 8 . Instead of column up counters, down counters may be used in the ADC units in FIGS. 10 and 11 . Similarly, a global counter-based scheme can be implemented instead of individual ADC-specific counters.

As mentioned above, the same image sensor (and all pixels in the corresponding pixel array) can be used according to the technical principles disclosed herein for routine 2D imaging as well as 3D depth measurement. In 2D mode, the sensor can operate in linear mode like a normal 2D sensor. During 3D depth measurement, however, the sensor can operate in linear mode under moderate ambient light but can switch to logarithmic mode signal detection under strong ambient light to allow use of visible (or NIR) light sources. Therefore, since the same 4T PPD pixels can be used for both 2D and 3D images, the imaging (imaging) method described herein can be compatible with existing 2D sensor designs. This allows the sensor design to be smaller in size (with smaller pixels), more versatile, and capable of low power operation. These properties in turn save space and cost for mobile devices that include such an image sensor. Moreover, in consumer mobile devices and some other applications, the use of a visible light laser (in addition to ambient light) for 3D depth measurements may be better for eye safety than conventional NIR sensors. In the visible spectrum, the sensor can have a higher quantization efficiency compared to the NIR spectrum resulting in lower power consumption for the light source, ie saving power in the mobile device.

One embodiment disclosed herein includes a 2D pixel array in which 2D color image information and 3D depth information are acquired simultaneously to provide fully synchronized frame rate, color phase, depth and scene points. In one embodiment, color image information and 3D depth information are output from rows of a 2D pixel array in an interleaved or alternating type of method. That is, color image information is output from the first selected rows following depth information output from the same row, color image information is output from the next selected row following depth information output from the same next row, and so on. Alternatively, depth information is output from the first selected row following color image information output from the same row, and depth information is output from the next selected row followed by color image information output from the same next row.

In accordance with the inventive concept disclosed herein, one embodiment shown in FIGS. 1, 2, 6, 7A, and 7B is a 2D color image to provide fully synchronized frame rate, color phase, depth, and scene points. It includes a 2D pixel array from which information and 3D depth information are acquired simultaneously. In one embodiment, image sensor unit 24 includes a 2D pixel array arranged in a plurality of rows where each pixel of the array is substantially the same as the other pixels in the array. In other embodiments, one or more pixels of the array may be not substantially identical to other pixels of the array. In one embodiment, the rows of the array may be operated to generate 2D color information of an object being imaged as disclosed herein, and may be operated to generate 3D depth information of an object as disclosed herein. In another embodiment, one or more rows of the array can be operated to generate both 2D color information and 3D depth information and other rows of the array can be operated to generate either 2D color information or 3D depth information but not both. can be operated. In another embodiment, specific rows scanned for 2D color information and/or 3D depth information may be smaller than the total number of rows in the 2D pixel array. In one embodiment, simultaneous generation of 2D color information and 3D depth information does not require a frame buffer because output information of digital signal processing is not required, and the information is output shortly after being obtained.

13 is an exemplary flowchart showing a process for simultaneously generating and obtaining 2D color information and 3D depth information according to an embodiment of the present invention. The various operations shown in FIG. 13 may be performed by a single module or a combination of modules or system components within system 15, for example. In the description herein, by way of example only, certain tasks are described as being performed by particular modules or system components. Also, other modules or system components may be suitably configured to perform such tasks.

The process starts at the start block. At block 302, system 15, in particular processor 19, may perform 2D color image capture of an object, such as object 26 of FIG. 2 along a first row of 2D pixel array 42 of FIG. In one embodiment, the first row may be the first physical row (eg, corresponding to the first row or the last row as shown in FIG. 6) of the 2D pixel array 42 . In another embodiment, the first row may be different from the first physical row of the 2D pixel array 42 (eg corresponding to the first row or the last row as shown in FIG. 6 ). In one embodiment, color image information is read from the 2D pixel array 42, as described with respect to FIG.

At block 303, system 15 may perform a 1D point scan of a 3D object, such as object 26 in FIG. 2, along the scanning line using a light source, such as light source module 22 in FIG. In one embodiment, the selected scanning line corresponds to a second row corresponding to the same row as the row scanned for color image information in block 302 . In another embodiment, the selected scanning line corresponds to a second row that is not the same row as the row that was scanned for color image information in block 302 . 3D depth information is read from the 2D pixel array 42, as described in connection with FIG. In one embodiment, the order of blocks 302 and 303 may be reversed.

At block 304, system 15 determines whether all rows of the 2D pixel array have been scanned for both color image information and 3D depth information. If not, the process moves to block 305. If at block 305 the indices corresponding to the rows of the color image information and the rows of the 3D depth information scan are incremented (or decreased in some cases), then the process returns to block 302. In an embodiment of the present invention in which 2D color information and 3D depth information are obtained from the same row, the indexes may be the same index. In an embodiment in which 2D color information and 3D depth information are obtained from different rows, the indexes may be different. If it is determined at block 304 that all rows of the 2D pixel array have been scanned for both color image information and 3D depth information, the process ends.

In an embodiment in which the number of rows outputting 2D color information is greater than the number of rows outputting 3D depth information, the number of selected rows of color image information is interleaved or alternating type for each row of 3D depth information can be output for

In one embodiment, the same row R (or column C) is illuminated multiple times to adaptively adjust the timing and/or density (intensity) of a point light output from a laser light source (eg, adjusting the timing of laser pulses). It can be scanned so that the response time of each particular pixel within a row and the mechanical properties of the projection optics 35 of FIG. 2 are better matched. Such techniques may be used to measure the imaging module 17 of FIG. 1 .

FIG. 14 illustratively shows that a distance to a translucent object 401 such as glass and a distance to an object 402 behind the translucent object 401 are performed for 3D depth measurement according to an embodiment of the present invention. show In FIG. 14 , an X-Y addressable light source such as a laser light source point-scans an object 402 through a translucent object 401 . Reflection 404 from translucent object 401 and reflection 405 from object 402 pass through lens 406 and are captured by pixels 407 and 408 respectively within row R of the 2D pixel array. is detected The two detected reflections have substantially the same timestamps and the output depth of the two reflections can be determined as disclosed herein.

15 illustratively shows that depth imaging of a translucent medium 501 such as fog, rain, etc. is performed for 3D depth measurement according to an embodiment of the present invention. An X-Y addressable light source 503, such as a laser light source, point scans the translucent medium 501, the reflection 504 will pass through the lens 506, and the pixels 507 within row R of the 2D pixel array. ) range will be activated. And reflections will have substantially the same timestamps. The thickness 501 of the medium may be determined based on the timestamps disclosed herein.

16 exemplarily shows that depth imaging of an object 601 is performed for 3D depth measurement in the presence of multiple return paths according to an embodiment of the present invention. An X-Y addressable light source 602, such as a laser light source, point scans a shiny object 601, and stray reflections 603 may bounce back from other objects. In this situation the reflected stray light 603 may not be visible within the contour plane of the low R being scanned and thus may not be detected as a reflection 605 from the object 601 where the point scan is directed.

12 shows an overall configuration for system 15 in FIGS. 1 and 2 according to one embodiment disclosed herein. Accordingly, for convenience of reference and description, the same reference numbers are used in Figures 1,2, and 12 for common system components/units.

As described above, the imaging module 17 uses the hardware shown in FIGS. 2, 6, 7a, 7b and 13 of exemplary embodiments to achieve 2D imaging and 3D depth measurement according to the spirit of the present invention. can include The processor 19 may be configured to interface with a plurality of external elements. In one embodiment, imaging module 17 may function as an input device. An input device provides data input. The data input is provided to processor 19 for further processing in the form of pixel event data, such as processed data output 170 above in FIG. 7a. Processor 19 may also receive inputs from other input devices (not shown). Other input devices may be part of system 15 . Some examples of such input devices may include a computer keyboard, touch pad, touch screen, joystick, physical or virtual clickable buttons, and/or a computer mouse/pointing device. In FIG. 12 , processor 19 is coupled to system memory 20 , peripheral storage unit 265 , one or more output devices 267 , and network interface unit 268 . In FIG. 12 the display unit is shown as an output device 267 . In some embodiments, output device 267 may include a touchscreen display. In some embodiments, system 15 may include more devices than one example of devices shown. Some embodiments of system 15 include a computer system (desktop or laptop), tablet computer, mobile device, cell phone, video game console or console, machine-to-machine (M2M) communication unit, robot, vehicle, virtual reality devices, stateless "thin" client systems, vehicle dash-cam or rear-view camera systems, or any other form of computing or processing device. In various embodiments all of the components in FIG. 12 may be housed within a single housing. Accordingly, system 15 may be configured as a standalone system or configured in any other suitable form factor. In some embodiments, system 15 may be configured as a client system rather than a server system.

In particular embodiments, system 15 may include one or more processors (eg, in a distributed processing configuration). If system 15 is a multiprocessor system, there may be more processors than one example of processor 19 . Also, there may be multiple processors coupled to processor 19 via respective interfaces (not shown). Processor 19 may be a system on a chip (SoC) and/or may include one or more CPUs.

As noted above, system memory 20 may be any semiconductor based storage system. Although not limited to, the semiconductor based storage system may be DRAM, SRAM, PRAM, RRAM, CBRAM, MRAM, STT-MRAM, and others. In some embodiments, the memory unit 20 may include at least one 3DS memory module along with one or more non-3DS memory modules. Non-3DS memory may include double data rate, or double data rate 2, 3, or 4 synchronous dynamic random access memory (DDR/DDR2/DDR3/DDR4 SDRAM). In addition, non-3DS memory may include Rambus® DRAM, flash memory, various forms of lido only memory (ROM), and the like. Also, in some embodiments, the system memory 20 may include a plurality of different types of semiconductor memories as opposed to a single type of memory. In other embodiments, system memory 20 may be a non-transitory data storage medium.

Peripheral storage unit 265 may include a magnetic, optical, magneto-optical, or solid state storage medium in various embodiments. Such media may be hard drives, optical drives (eg CDs or DVDs), non-volatile random access memory (RAM) devices, and the like. In some embodiments, peripheral storage unit 265 may include more complex storage devices/systems. Such storage devices/systems may be disk arrays (which may exist in a suitable Redundant Array of Independent Disks (RAID) configuration) or storage area networks (SANs). Peripheral storage unit 265 may be connected to processor 19 through a standard peripheral interface. Standard peripheral interfaces include Small Computer System Interface (SCSI) interfaces, Fiber Channel interfaces, Firewire® (IEEE 1394) interfaces, Peripheral Component Interface Express (PCI Express™) standards-based interfaces, Universal Serial Bus (USB) protocol-based interfaces, or other It can be any suitable interface. A variety of such storage devices may be non-transitory data storage media.

The display unit 267 may be an example of an output device. Other examples of output devices include graphics/display devices, computer screens, alarm systems, CAD/CAM (Computer Aided Design/Computer Aided Machining) systems, video game stations, smartphone display screens, or any other type of data output device. do. In some embodiments, input device(s) such as imaging module 17 and output device(s) such as display unit 267 may be connected to processor 19 via I/O or peripheral interface(s). .

In one embodiment, network interface 268 may communicate with processor 19 to enable system 15 to connect to a network (not shown). In another embodiment, network interface 268 may include media and/or protocol content (whether wired or wireless) for coupling system 15 to any suitable devices and networks. In various embodiments, the networks may include Local Area Networks (LANs), Wide Area Networks (WANs), wired or wireless Ethernet, telecommunications networks, or other suitable types of networks.

System 15 may include an on board power supply unit 270 for providing electrical power to the various system components shown in FIG. 12 . The power supply unit 270 may receive batteries or be connected to an AC electrical power outlet. In one embodiment, the power supply unit 270 may convert solar energy into electric power.

In one embodiment, imaging module 17 may be integrated with a high-speed interface. Although not limited to, the high-speed interface can be a Universal Serial Bus 2.0 or 3.0 (USB 2.0 or 3.0) interface or higher. The high-speed interface is built into any personal computer (PC) or laptop. A non-transitory, computer-readable data storage medium such as, but not limited to, system memory 20 or a peripheral data storage unit such as a CD/DVD may store program code or software. have. Within imaging module 17, processor 19 and/or digital processing block 167 in FIG. 7A may be configured to execute program code. Device 15 may be operated to perform 2D imaging and 3D depth measurement as described above, such as operations previously described with reference to FIGS. 1-11 and 13-16. The program code or software may be proprietary software or open source software. When executed by an appropriate processing entity, such as processor 19 and/or digital block 167, the software may enable the processing entity. This is to capture and process pixel events using precise timing. Software renders pixel events in various formats and displays them in 2D and/or 3D formats. As noted above, in some embodiments, digital processing block 167 within imaging module 17 may perform some processing of the pixel event signals, which processing may send the pixel output data to processor 19 for further processing and display. performed prior to transmission. In another embodiment, the processor 19 may also perform the functions of the digital block 167 when the digital block 167 is not part of the imaging module 17 .

For purposes of explanation and not limitation in the previous description, specific details are set forth in order to provide a thorough understanding of the disclosed technology. Such details are particular architectures, waveforms, interfaces, techniques, etc. However, it will be apparent to one skilled in the art that such techniques may be implemented in other embodiments that depart from these specific details. That is, a person skilled in the art will be able to create various arrangements. Even if not explicitly described or shown herein, the disclosed technical principles may be embodied. Some examples, well-known devices, circuits, and methods in detailed descriptions are omitted so as not to obscure the description of the disclosed technology with unnecessary details. All statements herein reciting principles, aspects, and embodiments of the disclosed technology, as well as specific examples thereof, are intended to encompass both structural and functional equivalents. Additionally, such equivalents are intended to include both currently known equivalents as well as equivalents developed in the future. Equivalents can be any elements developed that perform the same function, regardless of structure.

Thus, for example, it will be appreciated by those skilled in the art that the block diagrams herein ( FIGS. 1 and 2 ) may represent a conceptual view of an illustrative circuit or other functional units embodying a technical principle. can be understood by Similarly, it will be appreciated that the flowchart of FIG. 3 represents various processes. The various processes may be substantially performed by a processor (eg processor 19 in FIG. 12 and/or digital block 167 in FIG. 7A). Such processors include, by way of example, general purpose processors, dedicated processors, general processors, digital signal processors (DSPs), a plurality of microprocessors, one or more processors in conjunction with a DSP core, controllers, microcontrollers, ASICs, FPGAs, other types of ICs, and /or may include state machines. Some or all of the functions described above within the context of FIGS. 1-11 and 13-16 may be provided within hardware and/or software, and also by such a processor.

If certain aspects of the subject matter disclosed herein require software-based processing, such software or program code may reside within a computer-readable data storage medium. As noted above, such data storage media may be part of peripheral storage 265 . Alternatively, the data storage medium may be the system memory 20 or an internal memory (not shown) of the processor 19 . In one embodiment, processor 19 or digital block 167 may execute instructions stored on such media to perform software-based processing. A computer readable data storage medium may be a non-transitory data storage medium containing a computer program, software, firmware, or microcode for execution by a general-purpose computer or processor as described above. Examples of computer-readable data storage media include ROM, RAM, digital registers, cache memory, semiconductor memory devices, and magnetic media, such as internal hard disks, and also magnetic tapes and erasers, such as CD-ROM disks and DVDs. It includes capable disks, magneto-optical media, and optical media.

In accordance with the spirit of the present invention, alternative embodiments of the imaging module 17 or the system 15 that includes the imaging module may include additional components that provide additional functionality. The additional functions include any functions identified above and/or functions necessary to support solutions according to the disclosed spirit of the present invention. Although features and components are described in particular combinations, each feature or component may be used alone without other features and components, or may be used in various combinations with or without other features. . As mentioned above, the various 2D and 3D imaging functions disclosed herein may be implemented through the use of hardware (circuit hardware) and/or hardware capable of executing software/firmware, wherein hardware capable of executing software/firmware may be coded. It executes software/firmware in the form of written instructions or microcode. Coded instructions or microcode are stored on a computer readable data storage medium (mentioned above). Accordingly, it will be understood that such functions and illustrated functional blocks may be hardware implemented and/or computer implemented, resulting in a machine implementation.

In the above, the system and method have been disclosed. The systems and methods are systems and methods in which the same image sensor, i.e. all pixels within the image sensor, are used to capture both a 2D image of a 3D object and a 3D depth measurement for the object. The image sensor may be, but is not limited to, part of a camera in a mobile device such as a smartphone or the like. Laser Light A laser source can be used to point scan the surface of an object with light spots. The light spots are detected by a pixel array in the image sensor. Detection is performed to create a 3D depth profile of the object using triangulation. In 3D mode, the laser can project a series of light spots on the surface of an object along a scan line. Reflected light spots can be detected using a row of pixels in a pixel array. One row forms the epipolar line of the scan line. Detected light spots can be timestamped to remove any ambiguity within the triangulation. Timestamping reduces depth computation and system power. The timestamp also provides a correspondence between the pixel position of the captured laser spot and each scan angle of the laser light source to determine the depth using triangulation. Image signals in the 2D mode may appear as multi-bit outputs from the ADC unit in the image sensor. However, the ADC unit may only generate binary outputs to generate timestamp values for 3D depth measurement. To reject strong ambient light, the image sensor can be operated in a 3D logarithmic mode as opposed to a 3D linear mode.

As will be appreciated by those skilled in the art, the innovative concepts disclosed herein can be modified and varied across a wide range of applications. Accordingly, the scope of the technical idea of the patented invention should not be limited by any specific guidelines as described above, but should be defined by the following claims.

Claims (20)

Receive an image of at least one object at an image sensor, the image sensor comprising a two-dimensional (2D) pixel array arranged in a plurality of rows of a first group, the pixels of rows of a second group of the array comprising: is operable to generate 2D color information of the at least one object, pixels of a third group of the array are operable to generate 3D depth information of the at least one object, and rows of the first group are The second group of rows includes a first number of rows, the second group of rows includes a second number of rows less than or equal to the first number of rows, and the third group of rows includes a third number of rows less than or equal to the second number of rows. contains a number of rows; and
receiving 2D color information of the at least one object from one row selected from the rows of the second group in an alternating manner, and 3D depth information of the at least one object from one row selected from the rows of the third group Including receiving,
wherein the 2D pixel array generates the 2D color information using a linear mode, and generates the 3D depth information using the linear mode or logarithmic mode. 2. The method of claim 1, wherein the row selected from the second group of rows is the same as the row selected from the third group of rows. 2. The method of claim 1, wherein the row selected from the second group of rows is different from the row selected from the third group of rows. The method of claim 1, wherein the 3D depth information includes triangulation information corresponding to spots of scanned rows of at least one object. A two-dimensional (2D) pixel array arranged in a plurality of rows of a first group, the pixels of the rows of a second group of the array receiving 2D color information based on the image of at least one object received by the 2D pixel array , wherein a third group of pixels in the array is operable to generate 3D depth information of the at least one object, wherein the first group of rows includes a first number of rows; the second group of rows includes a second number of rows less than or equal to the first number of rows, and the third group of rows includes a third number of rows less than or equal to the second number of rows; and
Selecting one row from the second group of rows in an alternating manner to output 2D color information generated based on the image of the at least one object, and the generated 3D depth information of the at least one object a controller coupled to the 2D pixel array for selecting a row from the third group of rows to output
wherein the 2D pixel array generates the 2D color information using a linear mode, and generates the 3D depth information using the linear mode or logarithmic mode. 6. The image sensor unit according to claim 5, wherein the row selected from the second group of rows is the same as the row selected from the third group of rows. 6. The image sensor unit according to claim 5, wherein the row selected from the second group of rows is different from the row selected from the third group of rows. 6 . The image sensor unit of claim 5 , wherein the 3D depth information includes triangulation information corresponding to spots of a scanned row of at least one object. 9. The image sensor unit according to claim 8, wherein the 3D depth information is based on linear integration of photoelectrons generated by pixels in a row selected from the third group of rows. 9. The image sensor unit according to claim 8, wherein the 3D depth information is based on logarithmic integration of photoelectrons generated by pixels in a row selected from the third group of rows. 9. The image sensor unit of claim 8, wherein the triangulation information includes timestamp information for spots of the scanned row. The image sensor unit of claim 8 , further comprising a laser light source for irradiating spots of the scanned row. 13. The image sensor unit of claim 12, wherein the laser light source comprises one of a visible laser light source, a near infrared laser light source, a point light source, a monochromatic illumination source, an XY addressable laser light source, and a laser scanner. A two-dimensional (2D) pixel array arranged in a plurality of rows of a first group, the pixels of the rows of a second group of the array receiving 2D color information based on the image of at least one object received by the 2D pixel array , wherein a third group of pixels in the array is operable to generate 3D depth information of the at least one object, wherein the first group of rows includes a first number of rows; the second group of rows includes a second number of rows less than or equal to the first number of rows, and the third group of rows includes a third number of rows less than or equal to the second number of rows;
Selecting one row from the second group of rows in an alternating manner to output 2D color information generated based on the image of the at least one object, and the generated 3D depth information of the at least one object a controller coupled to the 2D pixel array for selecting a row from the third group of rows to output ; and
operate to display a first image of the at least one object based on the generated 2D color information and display a second image of the at least one object based on the generated 3D depth information, and operate with the controller A display connected to the 2D pixel array,
wherein the 2D pixel array generates the 2D color information using a linear mode, and generates the 3D depth information using the linear mode or logarithmic mode. 15. The method of claim 14, wherein the 3D depth information includes triangulation information corresponding to spots of a row scanned of the at least one object,
The system of claim 1, wherein the triangulation information includes timestamp information for spots in the scanned row. 16. The system of claim 15, wherein the 3D depth information is based on linear integration of photoelectrons generated by pixels in a row selected from the third group of rows. 16. The system of claim 15, wherein the 3D depth information is based on logarithmic integration of photoelectrons generated by pixels in a row selected from the third group of rows. The method of claim 15, further comprising a laser light source for irradiating spots of the scanned row,
The system of claim 1 , wherein the laser light source comprises one of a visible laser light source, a near infrared laser light source, a point light source, a monochromatic illumination source, an XY addressable laser light source, and a laser scanner. 16. The system of claim 15, wherein the display comprises a touchscreen display. 20. The system of claim 19, wherein the system comprises a portion of a mobile communication device.

Images