KR20080057345A - Imaging system with adjustable optics - Google Patents

Abstract

This invention provides a solution for determining a non-exposure time during which imaging optics can be adjusted without affecting the image being captured. In the solution an image sequence comprising at least two images is acquired, at least one of which said at least two images is used as measurement image and at least one other of which said at least two images is used as final image. Exposure times are determined for the measurement image and the final image. By means of the exposure times for the measurement image and the final image, the non-exposure time can be determined. As a result of this, the imaging optics can be adjusted during non-exposure time.

Description

Imaging system with adjustable optics

The present invention relates to the field of imaging. More specifically, it relates to imaging in imaging systems with adaptive optics.

Over the years, digital imaging systems such as digital cameras have played a remarkable role in the field of imaging technology. Digital cameras store images in digital format as characterized by one or more equipped processors. In terms of electrical characteristics, a digital camera (or a digital camera module) has a technology already integrated in another electric device, and a typical example is a portable telephone (mobile phone) equipped with a digital camera. Depending on the master device (eg, the device on which the camera module is mounted), the camera module can communicate with many other components or systems within the device. For example, in the case of a camera phone, the camera module operates in communication with one or more processors, and in the case of a digital camera, it may include other types of dedicated signal processing components.

In the field of digital imaging systems, adaptive optics can be used to electrically control image focusing to adjust the characteristics of the captured image, such as auto-focusing or optical zoom. These functions play a more important role in the imaging device. Auto-focusing or zooming was performed with conventional lens optics with movable lens components, and recently with optical systems using lenses with adaptive shapes or other adaptive means to affect refractive power. do.

The imaging system includes a lens system that collects light to produce a scene image. Light is focused on a semiconductor device that records light electrically. Such a semiconductor device may generally be a device such as a complementary metal oxide semiconductor (CMOS) or a charge-coupled device (CCD) sensor.

The sensor consists of a set of light-sensitive pixels, which convert light into electric charges, which in turn are converted into digital image data. A technique called binning can be applied to the sensor. Binning is a technique that combines adjacent pixels to increase the effective sensitivity of the imaging system and reduce the amount of pixels in the image.

The imaging system also includes shutter means. The main shutter types are a global shutter and a rolling shutter. Recently, rolling shutters are used with CMOS sensors, and global shutters are used with CCD sensors. In this case, however, the shutter may be used in other ways. Shutter means are used to limit the exposure of the image sensor. The shutter operation at least includes operations such as reset, exposure and read operations, and also operations such as open and close occur. Both global and rolling shutter means can be implemented electrically or mechanically, and in mechanical implementation various apertures or Neutral Density (ND) filters can be used. Rolling shutters have a characteristic of exposing images substantially in a line unit. The global shutter exposes all the pixels in the image at one time.

The imaging system also includes a focus detector which primarily measures current focusing values from one or more regions within an image and utilizes the measured results in a control unit included in the imaging system. The focus is measured based on the contrast between adjacent areas in the image, and according to the result, the controller finds an optimal focus that can chleog the contrast in the image.

The imaging system also includes an exposure detector which measures the exposure level of the current light in the image pixel and the resulting value is also used in the controller. The controller compares the target exposure level with the current exposure level. According to the result of comparing the exposure time, the analog gain value, digital gain value, aperture, and ND filter are controlled. The control also uses the information being received via the user interface. For example, when a user wants to apply a zoom function to a specific image, the controller changes the position of the lens. The optical driver is used when the lens system is moved, and is mainly controlled by using an inter-intergrated cirguit (I2C) command or a pulse width modulation (PWM) signal.

The imaging system includes or is connected to an input device (eg control button for zoom function, image selection, shutter controller). Flash is also often used in imaging systems. All image processing and substantive image processing, including focus detectors, exposure detectors, and controllers, may be performed in one or more combinations of camera modules, camera processors, application engines, and baseband engines. Such processing may also be performed via software or hardware processing. In image processing, at least the detector or controller must operate in real time.

In this specification, the image may represent a still image, a video image, or a viewfinder image. The still image generates visual information based on being stationary. Still images are captured and stored in memory immediately. The video image produces a visual video that changes over time. In a video image, a series of visual images are taken to continuously provide an image. The viewfinder image provides the image on the viewfinder display. In a digital imaging system, the viewfinder is an integrated color display for previewing the scene you are capturing. The viewfinder image shown on the display is mainly taken by the image sensor and displayed on the viewfinder display after the sensor or processor has lowered the resolution than the original resolution. In general, there is no need to save the viewfinder image. The viewfinder image is immediately updated on the viewfinder display so that the user can immediately view the image on the screen without delay.

In the image, the focus may be automatic (auto-focusing) or manually according to the user's motion. In addition, the AF (auto-focus) function may be implemented using single-shot auto-focusing or continuous auto-focusing. Single-shot auto-focusing is mainly used when shooting still images, and continuous auto-focusing is used for video image shooting.

Single-shot auto-focusing is mainly implemented in such a way that the lens moves with a fixed increment within that range, and records the value detected by the focus detector. After scanning, the lens moves to the point where the contrast is maximum. Single-shot auto-focusing is activated, for example, when the shooting button is pressed halfway down. When the photographing button is fully pressed, the image optics are already properly adjusted, and images are captured immediately to provide the user with satisfaction of photographing. The performance of the focus system is determined by the time it takes to find the optimal focus and the accuracy of the focused image.

In continuous auto-focusing, the value detected by the focus detector is determined from the image taken substantially continuously, and whenever the result of the focus detector indicates that the image optic needs to be adjusted, the focus can be adjusted by adjusting the image optic. Is improved. In particular, the images taken from the video image are displayed on the viewfinder display in real time. The advantage of continuous auto-focusing is that the viewfinder image is always in focus because the optics can stay focused. This is not only essential for video shooting, but also very useful for shooting stills. A single still image can be shot with no delay or minimal delay by fine tuning through basic single focusing and quick single shot focusing.

From the above, it is clear that focusing requirements of different conditions are required in still images, video images, and viewfinder displays. Depending on whether a rolling shutter type or global shutter type exposure control is used, the lens operation used for auto-focusing (or zooming) during the exposure may create several types of artifacts. In particular, when a rolling shutter is used, since the blank time between images is mainly short (when the image sensor has not collected any optical image information), there is insufficient time for the lens to move, resulting in artifacts in the image. . In addition, when modern high-resolution sensors are used, the image of the viewfinder is mainly subsampled, binned, or downscaled due to the bandwidth limitations of the interface between the camera module and the successive series of images. As a result, since the resolution of the viewfinder image is limited, the quality of auto-focusing detection is limited to the image at the second half during subsequent image processing.

In most major applications where rolling shutters are used, auto-focusing detection information can be calculated immediately with dedicated hardware or software when image information for the detection area is used in the focus detector. That is, auto-focusing can be performed on selected portions of the image using the full resolution of the entire image, without having to base the sub-sampled, binned or downscaled image.

Primarily, the focus detection zone is located in the middle of the image area. In this case a decision may be made about the lens operation in the next frame before all lines of the current image are fully exposed and transmitted. However, in this case, the problem is that if the lenses move immediately after the auto-focusing process is performed in the center of the image, the last lines of the current image are exposed by the moving lens and the artifacts caused by this are captured or in view. Will be visible in the image.

As a result, a similar kind of artifact may occur if the exposure of the next image frame begins before the lens stops operating. This situation can occur with both focus lens and optical zoom lens operation. In this situation, the first line of the image is broken when using the rolling shutter, and the whole image is damaged when using the global shutter. The zoom lens can be moved only if, for example, the image sensor is not exposed between image frames. It is very important when ordering a lens operation. Also, if dedicated hardware is used, there may be a delay before the lens actually moves.

If you're looking for image focusing, especially single shot auto-focusing, you'll need a lot of time, so you'll miss the scene you're already shooting by the time the camera focuses on the image. This situation is frequently seen in the case of sports or other activity footage, for example, when an object moves fast or the situation changes rapidly.

Examples of using a rolling shutter using adaptive optics can be found in the related art. For example, a command may be given to move the lens immediately after moving the lens without considering the effect on the image being captured. In this case, the last line of the image is generally corrupted. In another example, instructions for lens operation are made immediately after the entire image is captured. In this case, the lens operation is somewhat delayed until the entire image is captured. Depending on the blanking length and the exposure time, the lens moves the blank section. However, while the interval is short, the lens operation continues for a relatively long time, mainly breaking the first line of the next image.

In the past, auto-focusing detection was performed by measuring the auto-focusing detection value in units of frames. This type of detection method requires that the entire image frame, or the entire sub-sampled image frame, be converted to AD when focus detection is performed. However, some frames frequently skip this process due to lack of time in performing proper image viewing or proper focusing detection. As a result, the focusing time becomes longer. In the case of a video image, the frames are rarely skipped, but artifacts due to exposure occur, allowing the lens to be seen on the recorded video sequence.

As described above, when the lens needs to move for the purpose of focusing or zooming, a solution for solving the existing problem is required to properly expose the image without damaging the captured image.

In the present invention, the state of the image sensor in the process of adjusting the image optics for focusing or zooming is not fully described. The present invention aims to maximize the controllable time of the optics and at the same time minimize the artifacts that occur in the captured image. It also aims to minimize response time in order to provide improved usage to the user.

The present invention aims to propose a solution to image exposure by using the optical adjustment operation properly together and to minimize the artifacts occurring in the captured image at the same time.

It is another object of the present invention to provide various methods for minimizing response time in an image focusing process.

An imaging method, an imaging device, an imaging module, and a computer readable medium for obtaining an image sequence have been disclosed. This is a step of acquiring an image sequence including at least two images, wherein at least one image is used as the measurement image and the other at least one image is used as the final image; Determining a measurement image exposure time and a final image exposure time; Determining a non-exposure time between the measured image exposure time and the final image exposure time; And allowing imaging optic adjustment during the non-exposure time.

The above object can be attained through a method, a module and a computer readable medium for determining the non-exposure time disclosed by the features as claimed in claims 32, 33, 34, 36, 37 and 38.

The first embodiment of the present invention is called a timing method for optical adjustment. In this embodiment, an appropriate timing is determined by the auto-focus detection method. The timing tells how the auto focus and / or zoom optics should be adjusted.

Since the first embodiment of the present invention defines the exact time when the focus or zoom optic can be adjusted, image artifacts may not occur. If the frame blank time is small in a given situation, the present invention will allocate more time to the margin for optical control beyond the blank time.

The first embodiment also minimizes latency in the control loop and improves real time functionality. The reason is that since the autofocus statistical calculation has already been performed in the previous frame, the optical adjustment can be applied before the next frame.

The settling time in the auto-focus / zoom hardware may be, for example, a range such as the total time, blank time required to start moving it for the optical position and finally stop. Therefore, it is important to provide a settling time long enough before exposing the pixel of interest. Long settling times require a small starting current in the auto-focus / zoom activation controller and are therefore particularly useful in portable devices with limited power capacity with batteries. When the total non-exposure time is used for the optics adjustment, the activator need not be too fast. That means the optics can be adjusted with less power.

A second embodiment of the present invention is to detect auto-focus from a plurality of optical positions in one frame. This is done via optics adjustment when the detection area pixels are not exposed, but still during the exposure of the entire image frame. The detection area is the area of interest in the image, which is used for focus detection.

The second embodiment of the present invention provides a method for faster focus detection. In addition, since continuous focus is not always necessary, the advantage of lower power consumption can be obtained. This improves usability.

The third embodiment of the present invention controls the blank period or exposure and lens operating time.

This embodiment provides an undamaged video or viewfinder image, which may be provided at the maximum repetition frequency under light and control conditions of the image. However, if you do not require the maximum image frequency, you can adjust the optics at a fixed image frequency and, if the optics are adjusted (a lot, little, no), get the maximum exposure time. Similarly, when the exposure time is short, the zoom is accelerated or the temporary peak effect is reduced, the optics are adjusted more slowly.

The third embodiment also allows for flexible switching of the automatic night / day mode so that it is not larger than the exposure time but can slow down the image frequency accordingly.

Hereinafter, preferred embodiments of the present invention will be described with reference to the accompanying drawings. It should be noted that the same elements among the drawings are denoted by the same reference numerals and symbols as much as possible even though they are shown in different drawings.

1 shows an example of an image sequence.

2 shows an example of a timing scheme for optical adjustment.

3 shows an example of an image frame including an auto-focus window.

4 shows an example of an auto-focus system.

5 shows an example of one optical position and adjustment during a frame period.

6 shows an example of a method of measuring N optical positions in one frame.

7 shows an example of focus measurement as a function of optical position.

8 shows an example of a full focus scan during one frame

9 shows an example of a focus scan in a global shutter;

10 shows an example of the static blank interval scheme.

11 shows an example of a dynamic blank interval scheme.

12 shows an example of an apparatus according to the invention.

The present invention relates to an imaging system using adaptive optics. The imaging system may be a still image camera, a digital video camera, a mobile terminal capable of driving still images or video images, or both, or another electrical device capable of driving images. Imaging systems include movable adaptive optics (eg, auto-focus lenses or optical zoom lenses) and sensors (eg, CCD sensors or CMOS sensors).

The system further includes image processing means associated with the image sensor and may be mounted in a camera module, separate processing circuitry, an application engine of the mobile terminal or a mixed form thereof. The processing circuit includes at least the function of forming an image, improving the image and improving the real time control, for example, lightening (EC), white balance (WB) and sharpness (F). Real-time processing can be implemented automatically and the user does not need to take any action.

The imaging system may also include or be coupled to an input device. Camera operation can be controlled through the input device. Examples of such operations include zoom control, object selection, mode selection and image capture or video image activation. The lens described below includes, for example, an existing lens or a liquid lens or the like. Thus, those skilled in the art will understand the motion and motion in terms of "lens motion" or "lens motion" as the actual motion of an existing lens. However, when a liquid lens is used, for example, the motion and the motion are interpreted as somewhat different adjustment motions. That is, in this case, the light is projected to the image sensor through the movement and the image is outlined.

As described in the background of the present invention, the imaging system also includes shutter means such as a global shutter or a rolling shutter. This term is also used below for clarity. This term should not be interpreted to unnecessarily limit the technical scope of the present invention, but rather to understand the main technical concepts or features of the present invention.

1 shows an example of an image sequence comprising at least two frames F1, F2.

One of the at least two frames is the measured image F1 and the other is the final image F2. The final image is a stored image, and the measured image is an image that can be used, for example, to measure focus or exposure time.

Measuring area M1 is defined in measuring image F2 and used for the measurement. The final image may be a raw image being acquired at the sensor. Thus, digital image processing or other algorithms can be applied to the final image before it is actually stored.

Various measurement images may exist in one image frame. The measured image is usually smaller than the final image and is not stored. However, when the image sequence is a video image sequence, the measured images are determined as the final image and stored.

"Blank time" refers to the time when a sensor is unable to record meaningful image data due to frames / lines, pixel resets, other sensor structures, or user-defined controls. "Blank time" does not necessarily mean only the time at which the image is not exposed, but may also mean the time when pixels are not forwarded from the sensor.

1 shows the blank time between two frames F1, F2. Light is continuously received at the rolling shutter, but just before the reading is actually done, the pixels and lines are reset before the exposure time. The time that a pixel is not exposed at the sensor is within the vertical blank period. The sensor may still be exposed during each blank period (eg, at least the next line is exposed during the line blank period). Blank time may occur between image frames, between lines, and between pixels.

As shown in Fig. 1, each image has a unique exposure time. The exposure of the final image F2 partially overlaps with the blank time. However, the exposure of the measured image F1 responds completely differently from the final image F2. The exposure of the measurement image F1 starts before reading the measurement area M1. It should be noted in FIG. 1 that the exposure of the final image F2 does not last until the blank time after the final image F2.

The "non-exposure time" between F1 and F2 is the time from the start of the blank time to the section in which no exposure is performed on at least the next pixel, line or image and the section in which the lens can operate. In FIG. 1, the non-exposure time starts at the point where the exposure of the measured image F1 ends. And the non-exposure time can be extended depending on the time required for the operation of the lens. In the case of video, the non-exposure time cannot be extended by the viewfinder or measured images.

A way to extend the non-exposure time is to increase the channel rate to read the image faster. Implemented in this way, the image is read faster, so the blank takes longer, and this time can be used for lens operation. The prior art describes disadvantages of how to increase the channel rate. This is because EMC-noise can occur when the cost and channel rate are increased. However, increasing the channel rate has become possible through lesser processes or via differential signal interfaces.

In the case of a viewfinder image, focus and measurement are performed on fewer images. Thus, the image can be sub-sampled, binned or down-scaled to provide a smaller sized image. Less images can be read faster. Thus, the non-exposure time is longer. In the case of still images, the measured images are smaller than the actual final image, so the non-exposure time remains automatically.

The basic method may comprise acquiring a series of image frames or a sequence of images forming a series of image sections, eg, lines. At least a portion of the image sequence is used to control the lens.

Blank times are defined within an image sequence between images, lines or pixels, and non-exposure times are also defined. The lens moves at the end during the non-exposure time, which constitutes part of the blank time. The reason for this is to ensure that the lens does not operate (auto-focus, zoom) based on exposure. Thus, the time period in which the lens moves and the period in which exposure is not performed (eg, non-exposure time) are defined. Non-exposure time can basically be defined as the gap between successive images and their exposure.

The imaging system includes means for performing the operation during the non-exposure time. Such actions include, for example, defining exposure times, defining locations of information within an image, and the like. In addition, the system needs to know the operation of the pixel and the operation already performed during the non-exposure time. In addition, all delays that occur when performing each operation should be identified and operated. For example, it is necessary to know how much delay occurs depending on the degree of difference in lens operation. In addition, the above means should be able to know whether the focus is lost or wrong.

Hereinafter, the present invention will be described through various embodiments.

1) Timing method for lens operation

2 illustrates a timing solution for lens operation. In the sensor, pixel data is exposed (1) and transmitted. Processor 110 receives image data of the image being measured, and also includes auto-focusing logic 112 where, for example, auto-focusing detection is made.

The detection unit 112 calculates the auto-focus statistics, and then transmits the statistics calculated at the imaging sensor to the control CPU 113 (eg, transmitted in an image frame or transmitted through I2C). After the calculation is performed, the auto-focus statistic is sent to the control CPU 113 section (3). The control CPU 113 reads the auto-focus statistics (4) and determines how much non-exposure time is needed. For example, auto-focus and / or zoom lenses determine how to change.

If it is determined that lens operation is required, the control CPU 113 can also use the information received from the user interface. A line counter can be used in receiver logic 111 to receive image data from sensor 100. The control CPU 113 reads the record of the line counter at the receiver after the auto-focus statistics have been calculated (5). By determining the number of pixels in the image sensor, the transmission clock frequency, and the delays that may occur with the zoom hardware, it is possible to determine whether the lens can be operated.

In one embodiment, an image frame is received from a 1600 * 1200 sensor at a rate of 20 frames per second (f / s), the transmission time is 40 μs per line, and there are five blanking lines per frame. If the lens adjustment decision has already been determined when line number 1020 is received, there are still 180 lines at the end of the image (1200-1020) and 7.2 milliseconds is required to complete the image transfer. Thus, if the delay in the zoom hardware is 1 millisecond, then a delay of as many as 25 lines (1000 μs / 40 μs) is required, and the zoom hardware 114, 115 after line number 1175 1200-25 has been received. You can command (6, 7) and then the zoom optic can start moving after the current image is complete.

The command 6 for the lens driver 114 can be made after it is assured that the operation of the lens does not disturb the current image exposure. The control CPU 113 also controls the exposure value and the blank section so that the next image frame is not corrupted by the lens operation.

If no current or next image frame is needed, the control CPU 113 must determine if the next focus detection area has not been compromised by lens operation. For example, single shot focus should be adjusted as soon as possible. Therefore, when a global shutter is used, the timing of the global shutter must be known in order to operate the lens as soon as possible.

3 shows the auto-focus position in the measured image frame. The auto-focus position can be considered the measurement area and is indicated by reference numerals 101-108 in the sensor array 100. One exposure (for rolling shutter) embodiment associated with the first five lines in the (CMOS) sensor using three line exposure times is shown in Table 1.

Table 1: Example of Exposure for Timing Method

To illustrate this embodiment, take Step 4 of Table 4 as an example. In Step 4, after the first line (line 1) is read, two rows (row 2, 3) are exposed. The fourth line (line 4) is reset. The next steps then proceed until the fifth line (line 5) is read in Step 8. The previous steps Step 1-3 initiate the exposure operation, which is usually done during the vertical blank period. If you trace the operation of line 1 in Step 1-4, you can see that the line is read first, then exposed for two lines, and finally read. On the other hand, if you trace the function of the reset, starting from line 1 in step 1 and moving forward line by line until step 5 is reached during step 5. In this embodiment, subsequent lines and blank times are not shown. And assume that the blank time is greater than the exposure time, line 1 need not be reset for the next image until the last line of the current image is read.

In the present embodiment, the line counter has been described in the context related to the receiver logic 111 to determine the state of the image sensor. In addition, a line counter may also be used for time measurement. In this embodiment, the time measurement is performed in a situation where the exposure time is very short and the lens operation is very short. Short exposure time means that the exposure time is shorter than the vertical blank section, which is the time the lens moves within the blank time. Primarily, a short exposure time means that it can be shorter than the following example (e.g.). e.g. 50 / (20 * (1200 + 50)) s = 1 / 500s = 2 ms, where 50 is the amount of blank lines (= maximum exposure time in this embodiment), 20 is the amount of frames read per second ( 2Mpix video includes 1200 lines). Therefore, since the exposure time is shorter than this, there is sufficient time for the lens to operate within the blank period. This is because otherwise, the first line of the next image is exposed before the last line of the current image is read.

Note also that the blank time is not double, for example when the amount of blank lines is double. This is because (100 / (20 * (1200 + 100)) s = 1 / 260s ~ = 3.85ms. Also note that the reading of the sensor pixels must be accelerated simultaneously, otherwise the sensor will have 20 frames per second. In general, at least 15 frames per second are required for the rolling shutter to not cause image distortion, and in fact, image reading can be performed within a shorter time, which reduces image distortion, resulting in a static blank time. It is also possible to increase as large as possible, and it is also possible to keep the blank time small to capture twice the amount of frames from the sensor, but in practical operation, since the lens operation affects the image, the maximum exposure time is also reduced. In another embodiment, described later (3. Lens Movement and Image Exposure Using Blank Time), the blank time is static (3). .1) or dynamically (3.2) when used.

Table 1 shows one embodiment of the first five lines of the sensor with 1200 visible image lines and 50 blank lines. The reading of the first line starts in step 4, and steps 1-3 occur during the blank time. The exposure time is for three lines, and the exposure time is 120 μs when incorporating the exposure time into an embodiment with 1200 lines and 50 blank lines and a line reading within 40 μs. Thus, 47 lines will remain during lens operation (= non-exposure time), which means 47 * 40 = 1.88ms. Thus, in this example, the total blank time of the sensor is 50 * 40μs = 2ms.

2) fast focus  Detection method ( fast focus solution )

4 shows an example of an auto-focus system. The auto focus system includes at least a sensor 300, a focus detector 312, an auto-focus controller 314, an optic driver 315, and an optic 316.

The basic approach is to operate the lens within that range, record the contrast value and then move the lens to the position with the best contrast value. Embodiments of the present invention allow for faster focus finding than when using related techniques.

The main idea of the present invention is to measure focus from one or multiple lens positions in one or multiple frames. This makes the focus search faster. This embodiment can be implemented in the following manner.

In this method, the position for the lenses does not have to have a fixed increment, but the auto-focus control scheme must select the position of the new lens to perform the measurement.

2.1 Measuring one lens position within one frame

5 shows one frame read time T _f One embodiment in which one lens position is measured is shown. Contrast is detected in the measurement area M in the image frame. The measured value for the lens position is obtained by collecting high frequency (and / or bandpass) content in the lower region of the measurement region M. It is also possible that only a set of measured subregions is used in the evaluation step. The lens is P _n and P _{n +1} during the lens operation time between the point at which the last line of the measuring area M is read out and the point at which the first line of the measuring area M in the next frame N + 1 (not shown in FIG. 5) starts. Move the position. If the lens operates out of this time frame, the lines in the measurement area acquire mixed data and the measurement area no longer corresponds to only one lens position. The P _{n + 1} position is measured within the next frame N + 1.

The time allotted for lens operation is, for example, the non-exposure time as follows.

At this time, T _exp is the exposure time.

2.2 Measuring N Lens Positions in One Frame

6 shows that two lens positions are measured within one exposure frame. Contrast is detected in the M1 and M2 regions within image frame I. The measurements are captured by collecting high frequency (and / or band pass) content from the sub-regions. Only the set of subregions measured in the evaluation step may be used.

Exposure of the first line in the M1 area starts with the L _lead line reading. This means that the first line in the M1 area is exposed during the L _lead line read. The lens _shifts the P _n and P _{n +1} positions for the time T _{LensM1 - M2} between the point at which the last line of the M1 region in the image frame N is read out and the point at which exposure of the first line of the M2 region begins.

This lens is the T _{LensM2 - M1} between the last line reading point of the M2 region in image frame N and the start of exposure of the first line in M1 region in next frame N + 1. Time (not shown) Moves P _{n +1} and P _{n +2} positions.

When the lens operates outside this time frame, the lines in the measurement area acquire mixed data, and the measurement area does not correspond to only one lens position. The P _{n +2} position and the P _{n +3} position are measured in the next frame. The exposure time Texp is generally constant within one frame, but one skilled in the art will appreciate that it may vary within one frame.

7 shows the result after scanning the lens operating range. FIG. 7 shows a curve related to FIG. 6 and associated with two separate measurement areas. One of the two separate measurement areas contains more information than the other area. The peak focus position can be calculated by combining the curves.

The exposure time Texp is used to define how many measurements can be performed in one frame. In addition, lens characteristic values, such as, for example, Modulation Transfer Function (MTF / PSF), Point Spread Fuction, and Point Spread Function, may be used to evaluate the focus.

Example 2.2 describes two measurement areas by way of example, but one of ordinary skill in the art should be aware that they are not limited to only two areas. In addition, although the lens position has been continuously described in the 2.2 embodiment, the position may also be different. The distance between lens positions does not always need to be the same. The size or position of the measurement area may vary depending on the exposure time and the time required for the lens operation.

2.3. Consecutive moments within one frame (or two frames)

This embodiment is similar to the embodiment 2.1 (see FIG. 8) except that the lens stops at a specific lens position. The lower regions of the M region contain data from the subsections of the entire lens operating range. Therefore, this must be taken into account when interpreting the focus value. In addition, the exposure time should be made in consideration of this calculation. This embodiment is useful when the image is a flat object such as a document or a business card.

In this embodiment, the lens moves at a fixed speed, but can move at various speeds or paths, for example, over a wide range of cycles during the frame. As yet another embodiment, the lenses may operate from minimum to maximum between the first frame and the lenses may restore operation from maximum to minimum between the second frame. In this way, two curves can be generated, and the influence of different contrast regions within other regions in the image can be reduced. In addition, lens characteristic values (eg, MTF / PSF) may be used when calculating focus.

2.4 Fast with Global Shutter Focusing

As described above, global shutters are generally used with CCD sensors. However, DMOS sensors can also include global resets and global shutters. 9 shows an example where the drawn image 810 can be used for auto-focus metering and the full image 800 can be used as a viewfinder image.

The first timing chart 801 includes a general operation mode in which the frame rate and the focusing speed are limited to the analog to digital conversion (ADC) speed. Naturally, exposure time can also be a limiting factor if the exposure time is very long. The second timing chart 802 shows a system in which the focusing speed is maximized and the viewfinder image is not captured at all.

In this case, the focusing speed is limited by the lens operation, reset and exposure time. The third timing chart 803 shows an example where fast focusing can be achieved, but the preview image still looks at the proper frame rate. If clipping is performed, the charges outside the clipping region are ignored or not converted to AD.

Fast focus detection has a significant advantage in the time needed to find focus. For example, if " X " corresponds to the required measurement amount, the time can be reduced by the 2.1 embodiment (by limiting the exposure time), and by the 2.2 embodiment the time can be reduced to X / N. In the 2.3 embodiment, the time can be reduced by increasing the frame rate during the measurement. The 2.3 embodiment also reduces power consumption since it does not require constant focus.

3) Lens shift and image exposure using blank time

This embodiment describes a blank section being controlled (a dynamically varying blank section) or an exposure and lens operating time controlled (a static blank section). When a dynamically varying blank interval is used, the maximum image frame rate can be obtained within a known exposure value and lens operating time that does not compromise the image information. It also enables automatic day-night scene changes within the maximum image frame rate even with or without lens motion. The frame rate is not constant when a dynamically varying blank period is used. However, the frame rate may be constant in the static blank period.

Blank time scenarios can be used with both rolling shutters and global shutters. When using a global shutter, it should be noted that the lens operation starts immediately after the shutter lens is closed, even if the entire image is not transmitted from the sensor. It should also be noted that the blank solution in both shutters can be implemented without changing the system clock in the imaging sensor. This is a great advantage when the PLL (Phase Locking Loop_) is set, such as when there is no need to send poor quality images or skip image frames.

3.1 static static Blank section

In one implementation according to this embodiment (Fig. 10), the desired image frequency is used on the bus after the maximum limit of exposure time in a manner in which the lens can operate within a defined constraint of the blank time depending on how the lens is moved. Obtained by using the maximum blank time based on the rate at which it is present. Exposure time can be limited in such a way that shorter exposure times are replaced by analog or digital gain values.

Reference numerals 96a and 96b denote exposure times of the fully visualized image frames, and reference numerals 97a and 97b denote non-exposure times of the AF-statistic blocks. 10, frame blanks 92a, 92b. 92c, line blanks 91a, 91b, inserted / incident data 94a, 94b, 94c, 94d are shown. The line read is indicated by 98 sine. The exposure time is represented by 95 and the visible data is represented by 99.

In such a system, the image frequency remains constant by setting the amount of the entire line to a constant. However, within the limits of the bus rate and the read rate, the time for exposure limitation and lens operation are still required, without compromising the image in any way other than increasing the gain to reduce the dynamic range. There is plenty of time limit.

When static blank intervals are used, the exposure and lens operating time are limited, so that the image can be captured without generating any artifacts in the image. This means that the blank time should be as long as possible to be able to operate by the interface within the required frame rate. Also, if long exposures or long lens operating times are required, the other or two shutters must be limited. This means that the exposure time must be compensated for example using an analog gain value and the zoom speed is reduced.

3.2 Dynamic Blank Segment

In another embodiment of the present invention, the rolling shutter uses frame blank times 102a, 102b, 102c in each image. This is necessary for lens operation and the first line exposure of the next image. Thus, the frame blank times 102a, 102b, 102c must be greater than the corresponding exposure times 105a, 105b in order to change according to the image and obtain sufficient non-exposure time for the complete image.

Lens movement may begin immediately after the last visible line of the previous frame N-1 is exposed. Lens motion control thus requires a delay time for the lens to begin operation at the exposure of the last visible pixel (line) in the direction in which the lens is supposed to operate (zoom control or auto-focusing control). In addition, immediately after the lens operation occurs, exposure of the first pixel (line) can be started by resetting the pixel on the line. As long as you know which operation (zoom or autofocusing control) is being performed and what the lens operation amount and direction are, the lens operation and time required for processing the next image (N + 1 frame) 106a, 107a, 106b and 107b may be known in advance before the current image (N frame) is exposed. In the case of a viewfinder picture, focusing is important.

Thus, in this case, the lens operating times 107a and 107b differ from the lens operating times 106a and 106b in the case of still or video images. Zoom control is done by the user, and there is inevitably significant hysteresis in performing continuous auto-focusing control so that the lens does not move back and forth frequently frequently. In addition, since the exposure 105b of the next image (N + 1 frame) is already known, the amount of blank line required within the current image (N frame) frame blank area 102b is easily calculated.

In this embodiment, the read rate of the sensor does not change, so there is no collision with the sensor's system clock, but the amount of visible and blank lines must be transmitted and in the image. Blank pixels in line blank areas 101a and 101b are pixels that are not included in the image being displayed, such as visible images 109a and 109b. There may also be other invisible image areas such as vertical blanks, auxiliary / inserted data, fake pixels, dark pixels, black pixels, manufacturer specific data. Changing the amount of visible lines corresponds to image cropping, which is usually done when the image is digitally zoomed.

In FIG. 11, a blank line is inserted to change a frame blank, and a sensor of a standard mobile imaging architecture (SMIA) specification is used. It is also possible to change the line blank by inserting pixels at the ends of the lines 101a and 101b.

SMIA sensors function to control a constant image frequency without changing the original system clock. These sensors are also designed to achieve long exposure times without the need for read rates or continuous exposure of the image to image limits. Also, lens motion is not visible on either side of the image. Thus, such a system exposes the image in such a way that the lens can be moved when desired without compromising the image to provide the largest possible image frequency while obtaining the desired exposure time.

Note also that the blank area can be increased even slightly more dynamically than absolutely necessary to obtain a more suitable viewfinder frame update. For example, a blank line is added so that from the start of the current frame to the start of the next frame it is increased such that, for example, 1, 2, ..., n / 60s.

3.3. General ( General )

10 and 11 illustrate the static and operational blank interval schemes, respectively. 10 and 11, the lens operating time, exposure time and focusing data area are shown.

The biggest difference in FIGS. 10 and 11 is that the blank time is the same in FIG. 10, and the lens operation 96a, 97a, 96b, 97b is changed based on the exposure time 95a, 95b (or vice versa) required. In FIG. 11, the lens operation 106a, 107a, 106b, 107b time and exposure time 105a, 105b are known and the blank section is changed based on this.

The previous 3.1 and 3.2 processes also apply to global shutters in the following manner.

If the lens operation can be started, then the exposure of the image is closed by the global shutter. The lens operation is started regardless of how long the corresponding image reading from the sensor lasts. Similarly, global reset of the pixel (or opening of the global shutter) and thus exposure of the new image will begin immediately after the lens has moved to the correct position and the previous visible image has been read by the sensor, and the global shutter is opened after the abnormal reading. .

Note that in this case, the non-exposure time is the time during which the sensor is read after the shutter is closed and the blank time before the first line used in the next image is reset (generally a global reset). During the non-exposure time, the sensor does not receive (or discard) light in the visible pixels in the final image or in the pixels used for measurement.

It should be noted that if a process targeting the maximum image frequency (3.2) is not used, more time is used for lens operation and shorter time is used for exposure. Because of this, optical zoom / auto-focusing lenses operate faster or with smaller temporary capacity in the same path than normal.

In some cases, no attempt is made to prevent the viewfinder image from being temporarily damaged. Because we do not store such images for further use. Thus, the fastest auto-focusing function possible can be implemented in both processes with long exposure times (3.1, 3.2). This implementation is obtained with lens movement during the exposure period of the pixel / pixel line that does not belong to the focusing control. This movement is visible in the viewfinder image but not in the last saved still image. In addition, for video images, it is best to implement the first auto-focusing control as soon as possible. As a result, the image can be corrupted unless the area used for statistics is damaged.

Note that the control lens is rarely needed because the exposure time can always be controlled at the read rate depending on the limit value of the control lens. Thus, in implementations that target the maximum image frequency 3.2, the blank time can often be set to zero (or sensor limit). In addition, implementations targeting the maximum image frequency work well in day / night mode automatic control (regardless of lens operation). In this case, the blank increases with light conditions, but the viewfinder image is not slowed down more than necessary.

In this case, the desired result or undamaged image with the maximum image frequency under light and control situations is achieved. In addition, when the maximum image frequency is not the target lens operation with a fixed image frequency is possible, and the maximum exposure item may be guaranteed according to the way the lens moves. Similarly, if the exposure time is short, it is also possible to speed up zooming or reduce the transient peak effect when the lens movement slows down.

avatar

The above embodiments may be implemented on a control CPU of an imaging system that is part of a mobile device, digital camera, web camera or similar electronic device. Dedicated hardware may be required at the imaging sensor or receiver for faster and more accurate timing.

12 illustrates an example implementation of an electrical device. The apparatus 1200 of FIG. 12 comprises communication means 1220 with or connected to a transmitter 1221 and a receiver 1222. It may also include other communication means 1280 also having a transmitter 1281 and a receiver 1282. The first communication means 1220 is for telecommunications, and the other communication means 1280 may be used for short-range communication means, such as a Bluetooth system, a Wireless Local Area Network (WLAN), or another system for use locally with other devices. Can be.

The apparatus 1200 according to the embodiment of FIG. 12 also includes a display 1240 for displaying visual information and imaging data. The device 1200 also includes interaction means such as a keypad 1250 for data entry. If the display is a touch-screen display, it may include a stylus instead of or in addition to the keypad 1250.

Device 1200 includes audio means 1260, such as earphone 1261 and microphone 1262, and optionally includes a codec for coding (decoding, if necessary) audio information. Device 1200 includes or is coupled to imaging system 1210.

The controller 1230 may be integrated into the device 1200 to control functionality or to execute an application within the device 1200. The controller 1230 may include one or more processors (CPU, DSP). The apparatus also includes a memory 1270 for storing data, applications, and computer program code, for example. Those skilled in the art will appreciate that an imaging system may be further integrated with various functions to improve the performance of the system.

The best embodiments have been disclosed in the drawings and specification above. Although specific terms have been used herein, they are used only for the purpose of describing the present invention and are not used to limit the scope of the present invention as defined in the meaning or claims.

Therefore, those skilled in the art will understand that various modifications and equivalent other embodiments are possible from this. Therefore, the true technical protection scope of the present invention will be defined by the technical spirit of the appended claims.

Claims (38)

Obtaining an image sequence comprising at least two images, wherein at least one image is used as the measurement image and the other at least one image is used as the final image; Determining a measurement image exposure time and a final image exposure time; Determining a non-exposure time between the measured image exposure time and the final image exposure time; And Allowing imaging optics adjustments during the non-exposure time; Imaging method comprising a. The method of claim 1, And a smaller image is used as the measurement image and a larger image is used as the final image in the image sequence. The method of claim 1, And the measured image is used as the final image in the case of a video image or a viewfinder image. The method of claim 1, And the final image is stored. The method of claim 1, wherein the exposure time Controlling the size of the measured image, sub-sampling the measured image, changing the channel rate used for reading the image sequence, controlling the measured image exposure time or the final image exposure time or both, Imaging method characterized in that it is extended by the above method or a combination of the above methods. The method of claim 1, And the measurement image is one of a measurement image frame and a measurement region in the image frame. The method of claim 1, Auto focusing statistics are calculated from the measured image. The method of claim 7, wherein And the non-exposure time is determined by information including the auto-focus statistics and the amount of pixels in the image sensor and the transmission clock frequency delayable time in zoom hardware. The method of claim 1, At least one measurement area is defined at least in the measurement image, in which case the auto-focus is measured in the at least one measurement area at at least one lens position. The method of claim 9, And a focus measurement is obtained by collecting high frequency content or band-pass frequency content from a lower area of said at least one measurement area. The method of claim 10, wherein the optics Between the first line exposure start point of the second measurement area in the same image as the last line reading point of the first measurement area, between the first measurement area exposure point in the first image and the second measurement area exposure point in the second image, or Imaging method characterized in that it is continuously adjusted during imaging. The method of claim 1, Blank time is used when the non-exposure time is determined. The method of claim 12, Maximum blank time is used to define the non-exposure time for imaging optic adjustment. The method of claim 12, And the blank time is controlled in accordance with the exposure time to define the non-exposure time. The method of claim 1, And the image sequence is formed of a still image, a video image or a viewfinder image or a combination of such images. Obtaining an image sequence comprising at least one measured image and at least one final image; Calculating auto-focus statistics from the measured image; Determining the non-exposure time using the auto-focus statistic, wherein the non-exposure time is a determination step used for the final time. . Obtaining an image sequence comprising at least one measured image and at least one final image; Defining at least one measurement area in at least the measurement image; Measuring auto-focus from at least one lens position within the measurement area; Defining the non-exposure time between a reading point of the last line of the measurement area and an exposure start point of the first line of the next measurement area; And determining a non-exposure time used for imaging optic adjustment. 18. The method of claim 17, wherein the optics are Between the last line reading point of the measurement area and the first line exposure start point of the measurement area in a continuous image, between one measurement area exposure point in one image and the next measurement area exposure point in the image, or A method of determining the non-exposure time used for imaging optic adjustment, characterized in that it is continuously adjusted during imaging. Obtaining an image sequence comprising at least one measured image and at least one final image; Defining at least one blank time occurring within the image sequence; Controlling the image optics adjustment in the blank time or controlling the blank time to define the non-exposure time. An imaging device comprising at least an image sensor that collects light provided to a processor as an adaptive imaging optic, an image sequence, comprising: The image sequence includes at least two images and an exposure control for controlling exposure of the images, at least one of the at least two images being a measurement image and the other being a final image, in which case the imaging device Determining a measurement image exposure time and a final image exposure time; Determining a non-exposure time between the measured image exposure time and the final image exposure time; And Allowing adjustment of the imaging optics during the non-exposure time. The method of claim 20, And using a smaller image as the measurement image and using a larger image as the final image in the image sequence. The method of claim 20, wherein the measurement image is And the final image in a video image or a viewfinder image. The method of claim 20, And storing the final image. The method of claim 20, And calculating auto-focus statistics on the measured image. The method of claim 24, And determining the non-exposure time by the auto-focus statistic, information including the amount of pixels in the image sensor, and transmit clock frequency delayable time in zoom hardware. The method of claim 20, Defining at least one measurement area in at least the measurement image; And And measuring auto-focus in the at least one measurement area at at least one lens position. The method of claim 20, And using the maximum blank time to define the non-exposure time. The method of claim 20, And controlling the blank time in accordance with the exposure time to define the non-exposure time. The method of claim 20, And the image sequence is formed of a still image, a video image, a viewfinder image, or a combination of the above. The method of claim 20, And the shutter means is a rolling shutter or a global shutter. 16. An imaging module, comprising determining a non-exposure time at which a method according to any one of claims 1 to 15 can be implemented. Obtaining an image sequence comprising at least one measurement image and at least one final image; Calculating auto-focus statistics from the measured image; Determining non-exposure time using the auto-focus statistic, wherein the non-exposure time may be used in the final image to determine a non-exposure time. Obtaining an image sequence comprising at least one measurement image and at least one final image; Defining at least one measurement area in at least the measurement image; Measuring auto-focus in the at least one measurement area at at least one lens position; And And defining the non-exposure time between the reading point of the last line of the measurement area and the exposure start point of the first line of the next measurement area. Obtaining an image sequence comprising at least one measurement image and at least one final image; Defining at least one blank time occurring in the image sequence; And controlling the image optics adjustment within the blank time or controlling the blank time to define the non-exposure time. 16. A computer readable medium for imaging comprising code stored on a readable medium for implementing on a computer the method of any of claims 1-15. A computer readable medium for imaging comprising code stored on a readable medium for implementing on a computer the method of claim 16. A computer readable medium for imaging comprising code stored on a readable medium for implementing on a computer the method of at least one of claims 17 or 18. A computer readable medium for imaging comprising code stored on a readable medium for implementing on a computer the method of claim 18.

Images