JP5086270B2 - Imaging system with adjustable optics - Google Patents

Description

  The present invention relates generally to the field of imaging, and in particular, to imaging using an imaging system having an adjustable optical system.

  Over the past few years, digital imaging systems such as digital cameras have played a notable role in imaging technology. A digital camera is characterized by one or more built-in processors and records images in digital form. Due to the electronic nature of digital cameras, digital cameras (or digital camera modules) can be easily integrated into other electronic devices, and mobile communication devices (mobile terminals) among other electronic devices are today Then it is a general example. Depending on the master device (ie, the device in which the camera module is integrated), the camera module can communicate with multiple other elements and systems of the master device. For example, in a camera phone, the camera module typically communicates with one or more processors, and in the case of a digital camera, the master device can include other types of dedicated signal processing elements.

  Adjustable optics associated with digital imaging systems relate to the ability to use electronically controlled image focusing functions such as autofocusing and optical zoom functions to adjust the characteristics of the image to be acquired. These operations are becoming increasingly important in imaging devices. Autofocusing and zooming can be performed by conventional lens optics with moving lens components, and nowadays optics based on lenses with adjustable shape or other adjustable means affecting refractive power Can be used.

  The imaging system includes a lens system that focuses light to form an image of a scene. The light is focused on a semiconductor device that electrically records the light. This semiconductor device can typically be, for example, a CMOS (complementary metal oxide semiconductor) sensor or a CCD (charge coupled device) sensor. A CMOS sensor or CCD sensor is mainly composed of a set of photosensitive pixels that convert light into electric charges, and the electric charges are further converted into digital image data. A technique called binning can be used in a CMOS sensor or a CCD sensor. Binning combines the charge of neighboring pixels to increase the effective sensitivity of the imaging system and reduce the number of pixels in the image.

  The imaging system also includes shutter means. The main types of shutters are global shutters and rolling shutters. Currently, rolling shutters are used with CMOS sensors and global shutters are used with CCD sensors, but these global shutters and rolling shutters can be used in different ways. The shutter means is used to limit the exposure of the image sensor. The shutter operation includes at least operations such as reset, exposure, and reading, but operations such as opening and closing can also occur. Both global shutter and rolling shutter shutter means can be realized electronically or mechanically, but variable aperture or neutral density (ND) filters can also be used in the mechanical realization. A rolling shutter is known for its feature of exposing an image substantially line by line, and a global shutter is intended to expose all pixels in the image substantially simultaneously.

  The imaging system also typically includes a focus detector that measures the current focus value from one or more regions of the image, and the result is used in a control function included in the imaging system. The focus measurement is typically based on the contrast between adjacent areas of the image, so the control function attempts to find the optimal focus of the image by maximizing the contrast of the image.

  The imaging system also includes an exposure detector that measures the current level of light exposure of the pixel, and the result is also used in the control function. The control function utilizes the current level of exposure and compares the current level of exposure with the target exposure level. Based on this comparison, the exposure time, analog gain, digital gain, aperture and ND filter are controlled. The control function also uses information received from the user interface. For example, if the user desires to zoom the image, the control function begins to change the lens position. An optical system driver is used when moving the lens system, and the optical system driver is typically controlled by an I2C (Inter-Integrated Circuit) command or by using a pulse width modulation (PWM) signal.

  The imaging system can include or be connected to an input device (eg, control buttons for zooming, scene selection and shutter control). A flash is also typically used in an imaging system. All image processing, including focus detectors, exposure detectors, control functions and actual image processing can be performed in a camera module, camera processor, application engine, baseband engine, or any combination thereof. Processing can also be realized by using software processing blocks or hardware processing blocks. At least the detector and control functions of image processing need to operate in real time.

  In this specification, imaging refers to still imaging, video imaging, or finder imaging. Still imaging produces visual information characterized by no movement. The still image is stored in the memory immediately after shooting. Video imaging produces a visual display that moves with time. In video imaging, a series of visual displays are taken to give an impression of movement when shown continuously. Viewfinder imaging provides an image on the viewfinder display. The viewfinder of a digital imaging system is typically an integrated color display that provides a preview of the scene that the user acquires. The viewfinder image seen on the display is typically retrieved from the image sensor after being scaled down by the image sensor, or obtained from the original resolution displayed on the viewfinder display in the processor. Typically, there is no need to store a finder image. Preferably, the finder image should be immediately updated on the finder display with minimal delay to give the user a good real-time feel and response.

  Focusing during imaging can be performed automatically (autofocus) or manually with user interaction. Furthermore, an autofocus (AF) function can be realized by using single shot autofocus or continuous autofocus. Typically, single shot autofocus is applied when acquiring a still image, and continuous autofocus is applied in video imaging.

  Typically, single shot autofocus is realized and the focus detection value is recorded so that the lens moves through the range of the lens using a fixed increment. When the scan is finished, the lens is moved to the position where the contrast is found to be maximum. Single shot autofocus can be performed, for example, by pressing the shooting button halfway. Therefore, when the shooting button is kept pressed, the imaging optical system is already properly adjusted, so that an image can be acquired immediately after giving a good user feeling. The performance of the focusing system can be characterized by the time required to find the optimal focus and the accuracy of the focused image.

  In continuous autofocusing, focus detection values are determined from images acquired substantially continuously, and focusing is improved by adjusting the imaging optics whenever the focus detection values indicate the need to adjust the imaging optics To do. Typically, the acquired image is also displayed in real time on the viewfinder display, particularly in video imaging. The advantage of continuous autofocus is that the optical system continues to focus continuously, so the viewfinder image remains in focus throughout that time. In video recording this is clearly necessary, but it is also very useful when recording still images, and a single still image can be converted into a basic using a fast single shot focusing procedure. It can be obtained without delay or after a short delay by fine tuning the continuous focusing.

  As explained so far, focusing on still images, focusing on video imaging and focusing on viewfinder displays have somewhat different requirements. Depending on whether rolling shutter type exposure control or global shutter type exposure control is used, different types of artifacts are caused by lens movement during autofocus (or zooming) during exposure. In particular, when using a rolling shutter, the blanking time between images (when no optical image information is collected by the image sensor) is usually very short, and therefore during the blanking time without artifacts in the image. There is not enough time available to move the lens. Even when using modern high-resolution sensors, the viewfinder image usually needs to be subsampled, binned and downscaled due to bandwidth limitations of the interface between the camera module and the next part of the image processing chain. Thus, the quality of autofocus detection is limited in the rear parts of the image processing chain due to the resolution of the viewfinder image.

  In most time critical applications, when a rolling shutter is used, the autofocus detection information can be calculated by dedicated hardware or software as soon as image information from the detection area is available at the focus detector. . In other words, autofocusing need not be based on a subsampled, binned or downscaled viewfinder image, but autofocusing is performed on the selected portion of the image using the full resolution of the selected portion. May be. Typically, the focus detection region is located in the middle of the image region. In this case, the lens movement decision for the next frame can be utilized before all lines of the current image are fully exposed and transmitted. In that case, if the lens is moved immediately after the autofocus process has been performed on the central part of the image, the last line of the current image is exposed by the moving lens and the resulting artifacts are acquired. Or it may be easily seen in the observed image.

  The same kind of artifacts can occur if exposure of the next image frame is started before the lens movement is finished. This situation can be caused by both the movement of the focusing lens and the movement of the optical zoom lens. In this case, the first line of the image is damaged by the rolling shutter, and the entire image is damaged by the global shutter. The zoom lens can only be moved when the image sensor is not exposed, that is, between image frames. Command timing is very important. Even when dedicated hardware is used, there may be a delay before the lens actually moves.

  In particular, image focusing using single shot autofocus takes a very long time and in a rapidly changing situation, the scene to be acquired when the camera system is finally ready to focus the image May no longer be available. Such situations are typical, for example, when imaging a sport or other activity, where the scene includes rapidly moving objects and rapidly changing situations.

  Realization of the use of a rolling shutter with adjustable optics can be found from the related art. For example, the lens movement command can be given immediately after the lens movement is determined without considering the influence of the acquired image. In these cases, the last line of the image typically begins to break. In another example, the lens movement command is given only after the entire image is acquired. In this case, the start of lens movement is sufficiently delayed until the entire image is acquired, and the lens is moved during the blanking period, depending on the blanking length and exposure time. However, because the blanking period is often short, the first line of the next image begins to break. The reason is that the movement of the lens continues for a very long time.

  Autofocus detection is conventionally performed by measuring an autofocus detection value for each frame. This type of detection requires that the entire image frame or subsampled image frame must be AD converted when performing focus detection. Often some frames begin to be skipped due to lack of proper focus detection or proper image viewing time. This further increases the convergence time. With video images, frames are usually not skipped, but artifacts caused by exposure and lens movement may be visible from the recorded video sequence.

  To overcome the shortcomings of the prior art, there is a need to find a solution that properly exposes the image without damaging the acquired image when the lens needs to be moved for focusing or zooming purposes.

  According to the applicant's understanding, knowledge of the state of the image sensor is not yet fully utilized when timing the adjustment of the imaging optical system during focusing or zooming. It is an object of the present invention to provide a solution that maximizes the time available to adjust the optical system and minimizes artifacts that occur in acquired images. It is also an object of the present invention to provide an improved user experience with minimal response time.

  It is an object of the present invention to provide a solution that properly exposes an image while minimizing artifacts that occur in the acquired image and adjusting the optical system.

  Another object of the present invention is to provide various methods for minimizing response time in the image focusing process.

  These objectives are to obtain an image sequence comprising at least two images, at least one of which is used as a measurement image and at least one of which is used as a final image, to determine a measurement image exposure time and a final image exposure time, It can be achieved by an imaging method, an imaging device, an imaging module and a computer program product that determines a non-exposure time between an exposure time and a final image exposure time and allows adjustment of an imaging optical system during the non-exposure time .

  These objects can also be achieved by a method, module and computer program product for determining the non-exposure time as described in the characterizing part of claims 32, 33, 34, 36, 37, 38.

  The first example of the present invention is a so-called timing solution for adjusting the optical system. In this example, an appropriate timing is determined by the autofocus detection value. This timing represents how much the focus and / or zoom optical system needs to be adjusted.

  Since the first example of the present invention defines an exact point in time when the focus or zoom optical system can be adjusted, image artifacts can be avoided. If the frame blanking time for a given situation is small, the present invention provides a timing margin for optical system control that exceeds the blanking time.

  The first example minimizes control loop latency and improves real-time performance. The reason is that the autofocus statistic calculation is completed in the previous frame and it is guaranteed that the optical adjustment is easily applied before the next frame.

  The set time for the autofocus / zoom hardware, that is, the total time required to finally fix the position of the optical system after starting the movement of the optical system is in the same range as the blanking time. Therefore, it is important that the set time can be sufficiently long before exposing the pixel of interest. By increasing the set time, the starting current of the autofocus / zoom actuator controller can be reduced, which is particularly advantageous in portable applications where only limited capacity batteries are available. When the entire non-exposure time can be used for adjusting the optical system, the actuator does not have to be very fast, which means that less power can be used for adjusting the optical system.

  The second example of the present invention detects autofocus from the positions of a plurality of optical systems within one frame. This can be done by adjusting the optical system while exposing the entire image frame, even when the pixels in the detection area are not exposed. The detection region is a region of interest in the image and is used for focus detection.

  With this second example, the time for finding the focal point can be further shortened. Furthermore, since a continuous focus is not always required, the power consumption can be further reduced. This example can also improve the usefulness.

  The third example of the present invention controls blanking time or exposure time and lens moving time.

  This example also provides an unbroken video image and viewfinder image that can be provided at the maximum repetition frequency in the light and control conditions. However, if the maximum image frequency is not desired, the optical system can be adjusted by a fixed image frequency, ensuring maximum exposure time regardless of the optical system adjustment (more, less, or no). Similarly, if the exposure time is short, zooming can be accelerated, or the temporary peak effect can be reduced when adjusting the optical system at low speed. The third example also allows for a flexible automatic night / day mode, whereby the image frequency can be reduced in a range that follows the exposure time but does not exceed the exposure time.

  The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate examples relating to the present invention, and together with the description, explain the objects, advantages and principles of the invention.

  The present invention relates to an imaging system having an adjustable optical system. The imaging system can be a digital still camera, a digital video camera, a mobile terminal capable of taking still pictures, taking video, or both, or any other electronic device capable of taking pictures. The imaging system includes an adjustable optical system (eg, autofocus lens or optical zoom lens) that can be moved and a sensor (eg, CCD sensor or CMOS sensor). The imaging system further comprises image processing means associated with the image sensor and which can be located in a camera module, a separate processing circuit, a mobile device application engine, or a combination thereof. The processing operation includes at least image formation, image processing, and real-time control such as illumination (EC), white balance (WB), and sharpness (F). Real-time processing can be realized automatically, and no user action is required. The imaging system also includes or is connected to an input device, which can control the operation of the camera. These operations can be, for example, a launcher that performs zoom control, object selection, mode selection, and image acquisition or video shooting. When a lens is referred to in the following description, it means an optical system including, for example, a normal lens, a liquid lens, or the like. Thus, when “lens movement” or “lens movement” is described in the specification, one skilled in the art will understand that movement is the actual operation of a normal lens, but for example when using a liquid lens Movement is another adjustment operation, whereby light can be projected onto the image sensor and the image can be outlined.

  Furthermore, as described in the background of the invention, the imaging system also includes shutter means such as a global shutter and a rolling shutter. In the following description, certain terms are used, but such terms are intended to be used for clarity. These terms are not intended to unnecessarily define or limit the scope of the invention, but are intended to form features or a better concept for the invention.

  FIG. 1 shows an example of an “image sequence” comprising at least two frames F1, F2. One of the frames is a measurement image F1, and the other is a final image F2. The measurement image can be used, for example, to measure convergence time or exposure time. A measurement area M1 can be defined for the measurement image F1, and the measurement area M1 is used for measurement. The final image can be a raw image acquired from a sensor. Thus, digital image processing or other algorithms can be performed on such final images prior to actual storage. There may be a plurality of measurement images in one image frame. The measurement image is typically smaller than the final image and no measurement image is stored. However, when the image sequence is a video image sequence, typically the measured image is also referred to as the final image and stored.

  “Blanking time” refers to the reason that a sensor cannot record meaningful image data due to a frame / line or pixel reset or any other sensor architecture or user defined control. The “blanking time” does not necessarily correspond to the time when the image is not exposed, and corresponds to the time when the pixel is not sent forward from the sensor. The blanking time is shown between the two frames F1 and F2 in FIG. The rolling shutter continuously receives light, but before actual reading, the pixels and lines are reset before the exposure time. The time during which the sensor pixels are not exposed is within the vertical blanking period. Each sensor can be exposed during the blanking period (eg, at least the next line is exposed during line blanking). Blanking time occurs between image frames, lines, and pixels.

  Both images have their own exposure time as shown in FIG. The exposure of the final image F2 partially overlaps the blanking time. However, the measurement image F1 can be exposed completely different from the final image F2. Exposure of the measurement image F1 starts before the measurement area M1 is read. As can be seen from FIG. 1, the exposure of the final image F2 does not last until the blanking time following the final image F2.

  The “non-exposure time” between frame F1 and frame F2 defines the length of time from the start of the blanking time, during which no exposure is performed on at least the next pixel, line or image and the lens Can be moved. In FIG. 1, the non-exposure time starts when the exposure of the measurement image F1 ends. The non-exposure time can be extended depending on the time required for lens movement. In the case of video, the non-exposure time cannot be extended to the same extent as the viewfinder image or measurement image.

  Extending the non-exposure time can be realized by increasing the channel rate, which makes the image readout even faster. In such an implementation, since the image is read out more quickly, the blanking time that can be used for lens movement is further increased. Note that the prior art teaches against the increase in channel rate. This is because of cost, and increasing the channel rate results in EMC noise. However, this type of increase is possible by using fewer processes and a differential serial interface.

  In the case of a viewfinder image, convergence and measurement are targeted at smaller images, so the image is subsampled, binned or downloaded, reducing the size of the image. The readout can be faster as the image gets smaller, which further increases the remaining non-exposure time. In the case of a still image, the measured image is smaller than the actual final image, thus automatically leaving unexposed time.

  The basic method comprises obtaining an image sequence composed of a series of image frames or series of image sections, eg lines. At least a portion of the image sequence is used to control the lens. The blanking time is defined in an image sequence between images, lines or pixels. Furthermore, the non-exposure time is defined. The lens finishes moving during the non-exposure time, and the non-exposure time can comprise at least a portion of the blanking time. The purpose of the solution is not to move the lens (autofocus, zoom) during exposure. Accordingly, a period of lens movement where exposure is not performed, that is, a non-exposure time is defined. Non-exposure time can basically be defined as the time interval between successive images and between exposures.

  The imaging system comprises means for enabling operation during non-exposure times. These operations include, for example, defining the exposure time and defining the position of the image from which information is retrieved. In addition to the non-exposure time, the imaging system needs to know the operation of the pixel cell and the previously performed operation. Further, the above means are arranged so as to know all delays for each operation, for example, how much delay is obtained by various lens movement amounts. Further, the above means can know whether the convergence is performed or the convergence is performed erroneously.

  The following description discloses various examples of the present invention.

1) Lens Movement Timing Solution FIG. 2 shows a solution for achieving proper timing of lens movement. Pixel data is exposed and transmitted from the sensor 100 (1). The processor 110 receives image data that is a measurement image, and includes, for example, autofocus logic 112 that performs autofocus detection. The detection block 112 calculates autofocus statistics, or the autofocus statistics are already calculated at the imaging sensor and are sent to the CPU 113 (eg, through I2C or in an image frame). After the calculation, information of autofocus statistics is provided to the control CPU 113 (3). The control CPU 113 reads the autofocus statistics and makes the necessary determination as to the required non-exposure time, i.e. how much the autofocus and / or zoom lens needs to be changed. When the control CPU 113 determines that lens movement is necessary, it can also use information received from the user interface. In this example, a line counter of receiver logic 111 that receives image data from sensor 100 can be used. The control CPU 113 reads the line counter register of the receiver 111 after calculating the autofocus statistics (5) and moves the lens by knowing the number of pixels of the image sensor, the transmission clock frequency and the possible delay of the zoom hardware. You can decide whether you can. In one example, image frames from a 1600 × 1200 sensor are received at a rate of 20 frames / second (f / s). In the case of 50 blanking lines / frame, the transmission time is 40 microseconds / line. If the lens adjustment is ready when 1020 lines are received, 180 lines (1200-1020) remain at the end of the image and 7.2 microseconds to complete the image transmission. Cost. This means that the delay of the zoom hardware is 1 millisecond, whereby the delay takes 25 lines (1000 microseconds / 40 microseconds) and the zoom hardware (114, 115) Before giving (6,7) a command to wait until a 1175 line number (1200-25) is received, so that the zoom optics only starts moving after the current image is complete. .

  The command (6) for the lens driver 114 is not given until it can be assured that the lens movement does not affect the exposure of the current image. The control CPU 113 also controls the exposure value and the blanking period. As a result, the next image frame is not damaged by the movement of the lens. When the current or next image frame is not needed, the control CPU 113 takes care that the next focus detection area is not damaged by lens movement, for example when trying to perform single shot focus as quickly as possible. When using the global shutter, it is necessary to know the timing of the global shutter in order to start the lens movement as quickly as possible.

  FIG. 3 shows the autofocus location within the measurement image frame. The autofocus location can be considered as a measurement region, and is denoted by reference numerals 101 to 108 in the sensor array 100. One exposure example (using a rolling shutter) associated with the first 5 lines of a (CMOS) sensor using a 3-line exposure time is shown in Table 1 below.

  To illustrate this example, consider step 4 in the table. In step 4, the first line (line 1) is read, the next two rows (rows 2 and 3) are exposed, and the fourth line (line 4) is reset. This is followed by the next step until the fifth line (line 5) is read in step 8. The previous steps 1-3 start the exposure operation, which typically occurs during the vertical blanking period. When tracking the operation of line 1 in steps 1-4, it can be seen that the line is first reset, then exposed during two lines, and finally read. In contrast, when tracking a reset, it can be seen that the reset proceeds one line forward in step 1 from line 1 until a reset is present on line 5 during step 5. In this example, the next line or blanking time is not shown, and it is assumed that the blanking time is longer than the exposure time, so that line 1 is relative to the next image until the last line of the current image is read out. There is no need to reset.

  In this example, the line counter is described in relation to receiver logic 111 that evaluates the state of the imaging sensor. Furthermore, a line counter can be used for time measurement. This example works in situations where the exposure time is very short and the lens movement is very quick. A short exposure time means that the exposure time is shorter than the vertical blanking time, so that the lens movement time is within the blanking time. Typically, a short exposure time can be less than, for example, 50 / (20 * (1200 + 50)) seconds = 1/500 seconds = 2 milliseconds, where 50 is the amount of blanking lines (books In the example, the maximum exposure time) and 20 is the amount of frames read out per second (a 2 Mpix image comprises 1200 lines). Therefore, if the exposure time is shorter than this, there is sufficient time to move the lens within the blanking period. The reason is that if there is not enough time, the first line of the next image is exposed before the last line of the current image is read. If the amount of blanking lines is doubled, for example, the blanking time is not exactly doubled. The reason is that (100 / (20 * (1200 + 100)) seconds = 1/260 seconds≈3.85 seconds, and at the same time, the readout of the sensor pixels also increases, because the speed is increased. Otherwise it may not have 20 frames / second from the sensor, typically at least 15 frames / second is required to allow the image distortion caused by the rolling shutter to be tolerated. The static blanking time can also be increased as much as possible because the actual readout of the image takes place in a short time, thus reducing image distortion, and exposure and lens. More time is allowed for the move, while keeping the blanking time short and twice as many frames can be acquired from the sensor, in practice, In other situations, the example described later (3. Lens movement and image exposure by using blanking time) should be used. Yes, in this case, the blanking time increases either static (3.1) or dynamic (3.2).

  Table 1 shows an example of the first five lines of a sensor having, for example, 1200 visible image lines and 50 blanking lines. Reading the first line starts at step 4 and steps 1-3 are performed during the blanking time. The exposure time is 3 lines, and the exposure time is 120 microseconds by associating the exposure time with the example where the sensor has 1200 lines and 50 blanking lines and where the line readout takes place at 40 microseconds. . Therefore, 47 lines remain for lens movement (= non-exposure time), which means 47 * 40 = 1.88 milliseconds. The overall blanking time of the sensor in this example is 50 * 40 microseconds = 2 milliseconds.

2) Fast focus detection solution FIG. 4 shows an example of an autofocus system. The autofocus system includes at least a sensor 300, a focus detection module 312, an autofocus control module 314, an optical system driver 315, and an optical system 316. The basic method is to move the lens through the lens range, record the contrast value, and move the lens to the optimal contrast position. This example of the present invention allows a focus to be found quickly using related techniques. The concept of the present invention is to measure the focus from one or more lens positions within one or more frames, thus further reducing the focus search. This example will be described by the following method. In these methods, there is no need to increase the lens position by a certain amount, and autofocus control selects a new lens position for measurement.

2.1 Measurement of one lens position in one frame FIG. 5 shows an example in which one lens position is measured within one frame readout time _Tf . Contrast is detected from the measurement area M of the image frame. The measurement value of the lens position is obtained by collecting high-frequency components (and / or band-pass components) from the sub-region of the measurement region M. Only a set of measured sub-regions can be used in the evaluation stage. Reading of the last line of the measurement area M and the (not shown in FIG. 5) on the lens moving time in T _LENS between the start of exposure of the first line of the measurement area M in the next frame N + 1, the position of the lens P _n And the position P _{n + 1} . If the lens is moved outside these time windows, the measurement area lines acquire mixed data and the measurement area no longer corresponds to only one lens position. The position P _{n + 1} is measured in the next frame N + 1.

  The time allocated for lens movement, i.e. the non-exposure time, is

となる。この場合、Ｔ_expは露出時間を表す。

It becomes. In this case, T _exp represents the exposure time.

2.2 Measurement of N lens positions in one frame In FIG. 6, two lens positions are measured in one exposed frame. Contrast is detected from the regions M1 and M2 of the image frame I. Measurements are obtained by collecting high frequency components (and / or bandpass components) from the sub-regions. Only a set of measured sub-regions can be used in the evaluation stage. The exposure of the first line in the region M1 is started when the line L _read is read. This means that the first line of the region M1 starts reading exposure of the line L _read . During the time T _LensM1-M2 between the reading of the last line of region M1 of image frame N and the start of exposure of the first line of region M2, the lens is _moved between position P _n and position P _{n + 1.} Move. During the time T _LensM2-M1 (not shown in the image) between the reading of the last line of region M2 of image frame N and the start of exposure of the first line of region M1 of next frame N + 1. Move between position P _{n + 1} and position P _{n + 2} . If the lens is moved outside these time windows, the measurement area lines acquire mixed data and the measurement area no longer corresponds to only one lens position. The positions P _{n + 1} and P _{n + 2} are measured in the next frame. The exposure time T _exp is typically constant in one frame, but those skilled in the art understand that the exposure time T _exp can vary in one frame.

  FIG. 7 shows the result after scanning the lens movement range. FIG. 7 relates to FIG. 6 and shows curves associated with two separate measurement areas, where one measurement area has more information than the other measurement area. The peak focus position can be estimated by combining the curves.

The exposure time T _exp is used to define the number of times a measurement can be performed within one frame. Lens characteristic values, such as MTF / PSF (modulation transfer function / point spread function), can be utilized when evaluating focus determination.

  Although Example 2.2 illustrates two measurement regions, those skilled in the art will understand that it is not limited to two numbers. Similarly, in the above example, the lens position is continuous, but the lens position may be different. The distance between the lenses need not always be the same. The size and position of the region can vary depending on the exposure time and the time required for lens movement.

2.3 Continuous Instantaneous within 1 Frame (or 2 Frames) This example (see FIG. 8) is similar to Example 2.1, but in this example the lens stops at a specific lens position. The sub-region of the region M has data from a partial section of the entire lens movement range. This must be taken into account when interpreting the focus value. The exposure time must also be considered in these calculations. This example is useful when extracting an image of a flat object such as a document or a business card.

  In this example, the lens can be moved at a constant speed, but the lens can be moved at variable speeds and miracles, eg, in the range of multiple cycles in the frame. In other implementations, the lens can be moved from minimum to maximum between first frames and the lens can be moved from maximum to minimum between second frames. In this way, two curves can be formed and as a result the influence of different contrast areas in different parts of the image can be reduced. Lens characteristic values (e.g., MTF / PSF) can also be utilized when evaluating focus determination.

2.4 High-speed focusing using a global shutter As described above, a global shutter is conventionally used together with a CCD sensor. However, the CMOS sensor can also have a global reset and a global shutter. FIG. 9 shows an example in which the crop image 810 is used for autofocus measurement and the full image 800 can be used as a finder image. The first timing chart 801 has a normal operation mode in which the frame rate and the convergence speed are normally limited by an ADC (analog-digital conversion) speed. Of course, if the exposure time is very long, the exposure time can be a limiting factor. The second timing chart 802 shows a system in which the convergence speed is maximized but no finder image is acquired. In this case, the convergence speed is limited by lens movement, reset and exposure time. A third timing chart 803 shows an example in which high-speed focusing can be achieved but a preview image can be shown at an ideal frame rate. When cropping is performed, all changes outside the crop area window can be ignored and no AD conversion is performed.

  A fast focus detection solution has the important advantage of being the time required to find a focus. For example, if X corresponds to the required measurement, the time can be reduced to X frames (by limiting the exposure time) according to Example 2.1, and the time can be reduced to X / N can be reduced to N, time can be reduced by one frame according to Example 2.3, and time can be reduced according to Example 2.4 by increasing the frame rate for measurement. Example 2.3 also reduces power consumption. The reason is that continuous convergence is not required.

3) Lens movement and image exposure by using blanking time In this example, the blanking period is controlled (dynamically changing blanking period) or the exposure time and lens moving time are controlled (static). A typical blanking period) method will be described. When using a dynamically varying blanking period, the maximum image frame rate can be achieved within a known exposure time and lens travel time without compromising image information. Automatic day and night light scene variations within the maximum image frame rate can be used with or without lens movement. When using a dynamically changing blanking period, the frame rate is not constant. However, a static blanking period allows a constant frame rate.

  Rolling shutters and global shutters can be used in both blanking time scenarios. By using the global shutter, the lens movement can be started immediately after the shutter is closed even if the entire image is not transmitted from the sensor. It is also important to remember that both blanking solutions can be implemented without changing the image sensor system clock. This is a great advantage. The reason is that it is not necessary to skip image frames or transmit bad quality image frames when setting up a phase locked loop (PLL) in such situations.

3.1 Static Blanking Period In one implementation of this example (FIG. 10), the desired image frequency is achieved by a maximum blanking time based on the rate using the bus, after which the exposure time is The lens is maximally limited to allow movement of the lens within a blanking time limit defined according to the method of movement. The exposure time can be limited to replace the shortened exposure time with analog or digital gain. Reference numerals 96a and 96b represent the non-exposure time of the complete visible image frame, while reference numerals 97a and 97b represent the non-exposure time of the AF statistics block. FIG. 10 shows frame blanking 92a, 92b, 92c, line blanking 91a, 91b, and embedded / auxiliary data 94a, 94b, 94c, 94d. The readout of the line is referenced by reference numeral 98, where the exposure time is referenced by reference numeral 95 and the visible data is referenced by reference numeral 99.

  This system assumes that the image frequency is kept constant by setting the total number of lines constant. However, it is necessary for sufficient time and lens movement for limited exposure, within the bus rate limit and readout speed limit, without increasing the gain and thus compromising the image other than reducing the dynamic area. There is enough time limit.

  When a static blanking period is used, the exposure time and lens movement time are limited so that an image can be acquired without artifacts in the image. This means that the blanking period needs to be as long as possible, which is made possible by the interface within the required frame rate. Furthermore, if long exposure times and long lens travel times are required, one or both must be limited. This means that the exposure time needs to be compensated by analog gain, for example, and the zooming speed is reduced.

3.2 Dynamic Blanking Period In another implementation of the current example for a rolling shutter, the frame blanking times 102a, 102b, 102c required for lens movement and exposure of the first line of the next image are for each image. The exposure amount is defined by the exposure times 105a and 105b. Thus, the frame blanking times 102a, 102b, 102c vary according to the image and need to be greater than the corresponding exposure times 105a, 105b in order to have sufficient non-exposure time for a complete image. An example of this is shown in FIG.

  The lens movement can be started immediately after the last visible line of the previous image (frame N-1) is exposed. Therefore, the control of lens movement is controlled in which direction the lens is supposed to move (zoom control or autofocus control) and the delay required to start the lens movement from the exposure of the last visible pixel (line). You can start by knowing when you are left. Furthermore, after placing the lens in place, the exposure of the first pixel (line) can be initiated by resetting the pixels in that line. The lens movement to be performed on the next image (frame N + 1) and the time required (106a, 107a, 106b, 107b) indicate which operation-zooming control or autofocus control-is performed, and the amount of lens movement and If the direction is known, it is already known before exposing the current image (frame N). In the case of a finder image, only focusing is important, so the lens movement time (107a, 107b) is different from the lens movement time (106a, 106b) of a still image or video image. Zoom control is by the user, and there must be sufficient hysteresis in continuous autofocus control to prevent the lens from moving back and forth very frequently. Furthermore, the exposure (105b) of the next image (frame N + 1) is already known, thereby calculating the amount of blanking lines required in the blanking region (102b) of the current image (frame N). Becomes easier.

  In this example, the sensor readout rate does not change, so it does not need to affect the sensor's system clock and only affects the amount of visible and blanking lines that are transmitted and present in the image. . Blanking pixels in the line blanking region (101a, 101b) are pixels that are not included in the displayed image, that is, the visible images 109a, 109b. There may be other invisible image areas such as vertical blanking, auxiliary / embedded data, dummy pixels, dark pixels, black pixels and manufacturer specific data. The change in the amount of visible line corresponds to the image cropping normally performed when the image is digitally zoomed.

  FIG. 11 shows the change of frame blanking by inserting a blanking line. At least the sensor of the standard mobile imaging architecture (SMIA) specification changes the line blanking by inserting a pixel at the end of the line. (101a, 101b). The SMIA sensor is originally intended to control the image frequency constant without changing the system clock. Similarly, these sensors are designed to achieve long exposure times without having to read the image rate or continuous exposure over the image limit. By using the control structure of this example, as large an image rate as possible can be achieved. Furthermore, lens movement is not shown at every stage of the image. Thus, this system provides as high an image frequency as possible with the desired exposure time by exposing the image so that the lens can be moved when desired without damaging the image. Note also that the blanking area can be increased dynamically without requiring any appropriate finder frame updates. For example, from the start of the current frame to the start of the next frame is 1, 2,. . . , N / 60 seconds to add a blanking line.

3.3 Overview FIGS. 10 and 11 illustrate a static blanking period solution and a dynamic blanking period solution, respectively. In both FIGS. 10 and 11, the lens movement time, exposure time and focusing data area are described. The largest difference between FIG. 10 and FIG. 11 is based on the exposure time (95a, 95b) in FIG. 10 where the blanking period is equal and the lens movement time (96a, 97a, 96b, 97b) is required. Fluctuates (and vice versa). In FIG. 11, the lens movement time (106a, 107a, 106b, 107b) and the exposure time (105a, 105b) are known, and the blanking period varies based on these times.

  The above processes (3.1, 3.2) are performed as follows using a global shutter. Image exposure is closed by the global shutter, after which lens movement can begin. The lens movement is initiated regardless of how long the image reading from the sensor continues. Similarly, a global reset of the pixel (ie, opening of the global shutter), and hence the exposure of a new image, moves the lens to the correct position and reads the previous visible image from the sensor and immediately after opening the global shutter To begin. In this case, the non-exposure time is the time when the sensor is read after closing the shutter, and the blanking time before the line used at the beginning of the next image is reset (typically a global reset). During the non-exposure time, the sensor does not receive light for pixels that are visible in the last image or used for measurement.

  Regardless of whether or not the process (3.2) targeting the maximum image frequency is used, the exposure time decreases as the time available for lens movement increases. For this reason, the optical zoom / autofocus lens moves on the same path more quickly than usual or with a temporary capacity less than usual.

  In certain situations, it is not desirable to prevent temporary damage to the viewfinder image. The reason is that such images are not stored for later use. Thus, autofocus with the fastest and longest exposure time possible can be achieved for both processes (3.1, 3.2) by lens movement during exposure of pixels / pixel lines not belonging to focus control. This movement is visible in the viewfinder image, but not in the final image and the stored still image. Even for video images, it is preferable to implement the initial autofocus control as quickly as possible, so that the image is not corrupted unless the area used for statistics is compromised.

  Since the exposure time can be controlled almost every time at the readout speed according to the limit value, there is almost no need to control the lens. Therefore, the blanking time can often be set to zero (or sensor limit) in the run (3.2) for the maximum image frequency. Furthermore, the realization for the maximum image frequency is well performed in the automatic control of the night / day mode (with or without lens movement), so that blanking depends on the lighting conditions. Although it increases, the viewfinder image does not become slower than necessary.

  Such an implementation provides an undamaged image and, if desired, an image with a maximum image frequency in the illumination and control states. Further, when the maximum image frequency is not targeted, the lens can be moved at a fixed image frequency, and the maximum exposure item can be secured according to the lens moving method. Similarly, if the exposure time is short, zooming can be accelerated or the temporary peak effect can be reduced when lens movement is slow.

Implementation The above example can be implemented on the control CPU of an imaging system that is part of an electronic device such as a mobile device, digital camera, web camera or the like. In order to have faster and more accurate timing, dedicated hardware implementations may be required in the image sensor or receiver block. FIG. 12 shows a possible form of the electronic device. The apparatus 1200 of FIG. 12 comprises or is connected to a communication means 1220 having a transmitter 1221 and a receiver 1222. There may be other communication means 1280 having a transmitter 1281 and a receiver 1282. The first communication means 1220 can be adapted for remote communication, and the other communication means 1280 can be adapted for Bluetooth system, WLAN system (wireless local area network) or other devices and communicating with other devices. It can be a narrow-band communication means of the kind like other systems that perform. The apparatus 1200 according to the example of FIG. 12 also includes a display 1240 that displays visual information and image data. Further, the device 1200 can comprise interaction means such as a keypad 1250 for entering data. Further, regardless of whether the display is a touch screen display, the device 1200 can have a stylus pen instead of the keypad 1250. The apparatus 1200 includes audio means 1260 such as an earphone 1261 and a microphone 1262, and selectively includes a codec that encodes (decodes if necessary) audio information. The apparatus 1200 includes or is connected to the imaging system 1210. A control unit 1230 can be incorporated into the device 1200 to control functions and execute applications in the device 1200. The control unit 1230 can include one or more processors (CPU, DSP). Further, the apparatus 1200 comprises a memory 1270 that stores data, applications and computer program code, for example. Those skilled in the art will appreciate that the imaging system can also incorporate any number of capabilities and functions that appropriately improve the efficiency of the system.

  The foregoing detailed description has been given for clarity of understanding only and is not necessarily intended to limit the scope of the claims.

An example of an image sequence is shown. An example of a timing solution for optical system adjustment is shown. An example of an image frame provided with an autofocus window is shown. An example of an autofocus system is shown. An example of the position and adjustment of one optical system during a frame period is shown. An example of measuring the positions of N optical systems within one frame is shown. An example of focus measurement as a function of optical system position is shown. An example of omnifocal scanning in one frame is shown. An example of focus scanning using a global shutter will be described. An example of a static blanking period solution is shown. An example of a dynamic blanking period solution is shown. 2 shows an example of a device according to the invention.

Claims (29)

In an imaging method for obtaining an image sequence comprising at least two images, wherein at least one is used as a measurement image and at least one is used as a final image ,
Determining a measurement image exposure time and a final image exposure time and a non-exposure time between the measurement image exposure time and the final image exposure time;
Allowing adjustment of the imaging optical system during the non-exposure time, comprising:
The non-exposure time, control of the size of the measurement image, sub-sampling of the measurement image,
An imaging method , wherein adjustment is performed by one or a combination of control of one or both of the measurement image exposure time and the final image exposure time, or a change in a channel rate used for reading the image sequence .   The imaging method according to claim 1, wherein a small image of the image sequence is used as the measurement image, and a large image is used as the final image.   The imaging method according to claim 1, wherein the measurement image is used as the final image in the case of a video image or a finder image.   The imaging method according to claim 1, wherein at least the final image is stored.   The imaging method according to claim 1, wherein the measurement image is one of a measurement image frame and a measurement area of the image frame.   The imaging method according to claim 1, wherein autofocus statistics are calculated from the measurement image. The imaging method according to claim 6 , wherein the non-exposure time is determined by the autofocus statistics, information including a pixel amount of an image sensor, and a possible delay of a transmission clock frequency in zoom hardware.   The imaging method according to claim 1, wherein at least one measurement region is defined in at least a measurement image, and autofocus is measured from at least one lens position in the at least one measurement region. The imaging method according to claim 8 , wherein the focus measurement value is acquired by collecting a high-frequency component and a passband frequency component from a sub-region of the at least one measurement region. The imaging optical system may be configured such that the exposure of the first measurement area of the first image and the first measurement area between the readout of the last line of the first measurement area and the exposure of the first line of the second measurement area in the same image. The imaging method according to claim 9 , wherein the adjustment is performed during imaging or always during exposure of the second measurement region of the two images.   The imaging method according to claim 1, wherein blanking time is used when determining the non-exposure time. The imaging method according to claim 11 , wherein a maximum blanking time is used to define the non-exposure time when adjusting the imaging optics. The imaging method according to claim 11 , wherein the blanking time is controlled according to an exposure time in order to define the non-exposure time. The imaging method according to claim 1 , wherein the image sequence is formed by a still image, a video image, a viewfinder image, or a combination thereof. In a method for obtaining an image sequence comprising at least one measured image and at least one final image, in order to determine the non-exposure time used for adjusting the imaging optics
a) an option to calculate autofocus statistics from the measured image and to determine the non-exposure time used for the final image by the autofocus statistics;
b) defining at least two measurement areas in at least the measurement image, measuring autofocus from at least one lens position in the measurement area, reading out the last line of the measurement area and the first line of the next measurement area An option to define a non-exposure time between the start of exposure and
c) defining at least one blanking time occurring during the image sequence, and controlling the adjustment of the imaging optical system within the blanking time or defining the non-exposure time by controlling the blanking time A method with at least one of the following options :
The non-exposure time, control of the size of the measurement image, sub-sampling of the measurement image,
Adjusting the channel rate used to read the image sequence by adjusting one or a combination of one or both of the measurement image exposure time and / or the final image exposure time control . For the option b), the imaging optics is one in a continuous image between the readout of the last line of the measurement area of one image and the start of exposure of the first line of the measurement image. 16. The imaging method according to claim 15 , wherein the adjustment is performed during the time between the exposure of one measurement area of the image and the exposure of the next measurement area of the same image, or always during imaging. Adjustable imaging optics and at least configured to collect light supplied to the processor as an image sequence comprising at least two images, at least one of which is a measurement image and at least one of which is a final image One image sensor,
In an imaging apparatus comprising exposure control means configured to control exposure of an image ,
Determining a measurement image exposure time and a final image exposure time and a non-exposure time between the measurement image exposure time and the final image exposure time;
An imaging device configured to allow adjustment of an imaging optical system during the non-exposure time,
The non-exposure time, control of the size of the measurement image, sub-sampling of the measurement image,
An imaging apparatus that adjusts by one or a combination of a change in a channel rate used for reading out the image sequence, a control of one or both of the measurement image exposure time and the final image exposure time . The imaging device according to claim 17 , wherein a small image of the image sequence can be used as the measurement image, and a large image can be used as the final image. The imaging apparatus according to claim 17 , wherein the measurement image is used as the final image in the case of a video image or a finder image. The imaging device according to claim 17 , wherein the final image can be stored. The imaging apparatus according to claim 17 , wherein autofocus statistics can be calculated from the measurement image. The imaging apparatus according to claim 21 , wherein the non-exposure time can be determined by the autofocus statistics, information including a pixel amount of an image sensor, and a possible delay of a transmission clock frequency in zoom hardware. 18. The imaging device according to claim 17 , wherein at least one measurement region is defined in at least a measurement image, and autofocus can be further measured from at least one lens position in the at least one measurement region. The imaging device according to claim 17 , wherein a maximum blanking time can be used to define the non-exposure time. The imaging device according to claim 17 , wherein the blanking time can be controlled according to the exposure time in order to define the non-exposure time. The imaging apparatus according to claim 17 , wherein the image sequence is formed by a still image, a video image, a finder image, or a combination thereof. The imaging apparatus according to claim 17 , wherein the shutter unit is a rolling shutter or a global shutter. An imaging module capable of performing the method according to any one of claims 1 to 16 to determine the non-exposure time. A computer program executed by a computer to carry out the method according to one of claims 1 to 16 .

Images