CN101518054A - A hand jitter reduction system for cameras - Google Patents
A hand jitter reduction system for cameras Download PDFInfo
- Publication number
- CN101518054A CN101518054A CNA2007800354470A CN200780035447A CN101518054A CN 101518054 A CN101518054 A CN 101518054A CN A2007800354470 A CNA2007800354470 A CN A2007800354470A CN 200780035447 A CN200780035447 A CN 200780035447A CN 101518054 A CN101518054 A CN 101518054A
- Authority
- CN
- China
- Prior art keywords
- gain
- time
- exposure
- automatic exposure
- lighting level
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/684—Vibration or motion blur correction performed by controlling the image sensor readout, e.g. by controlling the integration time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/72—Combination of two or more compensation controls
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/76—Circuitry for compensating brightness variation in the scene by influencing the image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N2101/00—Still video cameras
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Studio Devices (AREA)
- Exposure Control For Cameras (AREA)
Abstract
A camera system in normal mode and hand jitter reduction (hjr) mode may comprise generating a first exposure time-gain product by multiplying the normal mode exposure time with the normal mode gain. It may further comprise modifying the normal mode exposure time and gain and multiplying these modified parameters to generate a second exposure time-gain product for a hjr mode that reduces the difference between the first exposure time-gain product and the second exposure time-gain product. To reduce the difference the normal mode frame rate may also be modified. Operation of a camera in normal mode may be in response to a sensed light level being above a threshold. The hjr mode may be selected by the user while the camera is operating. The hjr mode may be used in response to a sensed light level being lower than the threshold.
Description
The crosscorrelation application case
The application's case is advocated the rights and interests of the 60/760th, No. 768 U.S. Provisional Application case that is entitled as " hand jitter reduction system design (HAND JITTERREDUCTION SYSTEM DESIGN) " of application on January 19th, 2006.The present invention with all with the application's case in the 11/534th of being entitled as of co-applications on the 25th September in 2006 " hand that compensation rotatablely moves is shaken minimizing (HANDJITTER REDUCTION COMPENSATING FOR ROTATIONAL MOTION) ", No. 935 (attorney docket 060268) common patent application cases co-pending are relevant with the 11/534th, No. 808 (attorney docket 060193) the common patent application case co-pending that is entitled as " the hand shake that is used for the compensated linear displacement reduces (HAND JITTER REDUCTION FOR COMPENSATING FOR LINEAR DISPLACEMENT) ".
Technical field
The present invention relates to digital picture and handle, and more particularly, relate to a kind of hand jitter reduction system that is used for camera.
Background technology
To the demand of multimedia application in the mobile communication with surprising speed increment.Nowadays, the user can send from the internet and receive still image and download images and video, for watching on mobile unit or mobile phone.The integrated growth trend that further help in mobile communication multimedia function of digital camera in the mobile unit.
The given limited amount resource as battery capacity, processing power and transmission speed that is associated with mobile unit needs effective digital imagery treatment technology to support multimedia function.This requires to reduce the exploitation that the computational complexity that is used for multimedia application is kept the more senior hardware and software of picture quality simultaneously.The exploitation of this type of hardware and software causes the lower power consumption and the longer stand-by time of mobile unit.
An aspect of digital imagery process relate to remove from picture fuzzy.May cause fuzzy by the hand shake.Mobile the cause hand of user's hand when taking digital picture with camera shaken.Even the user does not recognize mobile, hand also may constantly move.Described mobile phase is to less, if but mobile phase is bigger for the time for exposure, and digital picture may blur so.Object or person in the picture can look and move.Also may by object/person when the pictures taken move cause fuzzy.Also may cause fuzzy by the restriction of the optical system that is used to capture picture.
According under the condition, for example used digital camera takes a long time the registration picture in the mobile unit at low light.Slight the moving that the longer time for exposure has increased that hand produces can cause the possibility of bluring.Similarly, the longer time for exposure has increased the probability that moving of object/person can be bigger with respect to the time for exposure.
The current technology that is used to compensate mobile camera moving relates to uses miniature gyroscope or other mechanical device.As if described technology all do not have acceptable manner and come especially to shine under the condition at low light with digital form compensation mobile camera moving.Need be to be suitable for moving the fuzzy quantity in effective processing a resource shrinkage digital picture of using under all conditions.
Summary of the invention
The details of one or more configurations of statement in the accompanying drawings and the description below content.From describe content, graphic and claims, will understand other features, objects and advantages.
A kind of camera system that comprises that hand shake reduces (hjr) minimizing pattern and normal mode can be operated by time for exposure and gain being multiply by mutually generation exposure time-gain product.Can there be normal mode exposure time-gain sum of products hjr exposure mode time-gain product.The exposure time-gain product of normal exposure time and normal gain in the normal mode can be stored in the table when needed or be used.Exposure time-gain product in the hjr pattern also can be stored in the table.Can by revise in (add, deduct, multiply by or divided by) normal mode the time for exposure-clauses and subclauses in the product table (or clauses and subclauses) produce the exposure time-gain product in the hjr pattern.So, can not necessarily need independent table, but the equivalent exposure time-gain product in the hjr pattern can compare with the exposure time-gain product in the normal mode.When in the hjr pattern, operating, change parameter to reduce poor between exposure time-gain product in the hjr pattern and the exposure time-gain product in the normal mode.As long as the imageing sensor in the camera system can be higher than minimum average light level, described difference just can reduce.In the hjr pattern, can not satisfy in any district of lighting level of minimum average amount, may can not in the hjr pattern, keep and exposure time-gain product identical in normal mode at imageing sensor.The operation of camera in normal mode can be higher than threshold value in response to institute's sensed light level.The operation of camera in the hjr pattern can be selected by the user.Can be lower than described threshold value and use the hjr pattern in response to institute's sensed light level.
Description of drawings
By example rather than in the mode that limits various configurations are described in the accompanying drawing.
Fig. 1 is the block diagram of a configuration of explanation digital imagery process.
Fig. 2 is functional block diagram of a configuration of a front end image processing module in the explanation digital image processing system.
Fig. 3 is functional block diagram of another configuration of the front end image processing module in the explanation digital image processing system.
Fig. 4 A is that explanation is at the curve chart of normal mode time for exposure to lighting level.
Fig. 4 B is that explanation is at the curve chart of normal mode gain to lighting level.
Fig. 5 A is that shake reduces the curve chart of exposure mode time to lighting level at hand in explanation.
Fig. 5 B is that shake reduces the curve chart of modal gain to lighting level at hand in explanation.
Fig. 6 is the flow chart that explanation produces the method through revising the automatic exposure parameter.
Fig. 7 illustrates flow chart how to operate camera system in normal mode and hand shake minimizing pattern.
Fig. 8 is the automatic exposure parameter in the camera system is revised in explanation in hand shake minimizing pattern a flow chart.
Embodiment
This paper uses word " exemplary " to represent " as example, example or explanation ".It is preferred or favourable that this paper is described as that any configuration, scheme, design of " exemplary " or calibration all may not be interpreted as comparing with other configuration, scheme, design or calibration.In general, this paper describes and reduces in the digital picture because the technology of bluring that hand shake and/or the illumination condition lower than normal illumination condition cause.The technology of calibrating the camera that can operate in normal mode and hand shake minimizing pattern is also described.
In conventional camera apparatus, when the user took snapshot (current finish by pressing button), only a frame was used to produce picture usually.The method of using an above frame to produce picture gets nowhere usually, because it produces poor outcome.Utilize conventional camera apparatus, picture may move moving of producing owing to user's oneself hand and blur, and these hands move and are called the hand shake, and conventional camera apparatus also is exposed the required time quantum challenge of picture.According under the condition, increase the time for exposure at low light usually.It is observable because low light shines the noisiness that condition causes that the increase time for exposure has increased the user, and increase the possibility that the hand shake will produce blurred picture.Current, camera apparatus can contain the hand shake that miniature gyroscope produces with the compensation user.Yet, have many challenges of when being placed on gyroscope on the mobile unit, facing.Even when overcoming these whens challenge, digital hand jitter reduce technology can with have gyrostatic device and be used in combination.Current camera apparatus also can be at low light according to scalar gain under the condition.Regrettably, increase gain simply and amplified the noise that exists owing to low light levels.The result normally has the picture of poor quality.Similarly, the digital compensation for the hand shake does not always provide suitable result.Yet,, existing may reduce the hand shake and reduce than the noise under the light conditions by the technology that discloses among the present invention.
Fig. 1 is the block diagram that explanation is suitable for being integrated into the digital imagery process of the camera apparatus in the mobile unit.Mobile unit can be radio telephone, PDA(Personal Digital Assistant), laptop computer, or any other portable radio.Lens (not shown) can be used for focusing an image on the imageing sensor 102 in the image sensor module 104.In a kind of configuration, image sensor module 104 can have the memory that is used for storage gain and exposure parameter.Image sensor module 104 also can have and is used for revising gain and automatic exposure parameter control driver.In another configuration, image sensor module 104 can be coupled to integrated circuit (for example, travelling carriage modulator-demodulator (MSM
TM)) or have be used to store and revise the gain and the memory of automatic exposure parameter and/or other module of Control Driver.Imageing sensor 102 can be charge coupled device (CCD), complementary metal oxide semiconductors (CMOS) (CMOS) imageing sensor, or any other suitable imageing sensor.In at least a configuration of imageing sensor 102, semiconductor array is used in the different pixels place capture light of image.The colorful optical filter array (CFA) (not shown) that is positioned at imageing sensor 102 the place aheads can be used for single color (that is, red, green or blue) is delivered to each semiconductor.Modal CFA is RGB and CMYG pattern.Image sensor module 104 can drive or control chart image-position sensor 102 gains and or the time for exposure to revise.
With before taking snapshot and producing digital picture, preview mode can be captured the series of frames that is produced by imageing sensor 102 at user's pressing button.The subdivision of entire frame or frame is called as image or (being called interchangeably) picture.For illustration purposes, discussing image processing expediently is series of frames.But it should be understood that does not need to handle entire frame when using front end image processing module 106.In addition, frame sequence also is called as crossfire.Described crossfire can be provided to front end image processing module 106, locate it at this and removed mosaic so that obtain full RGB resolution as input to still image and video compressor 108.When crossfire passes through front end image processing module 106, in preview mode, can collect the statistics that produces the frame of digital picture about auxiliary.These statisticss can be (but being not limited to) exposure metrics, white balance metrics and focus metrics.
Front end image processing module 106 can will help the various signal feedback of control chart image-position sensor 102 in image sensor module 104.Still image and video compressor 108 can use the JPEG compression, or any other suitable compression algorithm.Automatic exposure control module 110 can receive and the proportional value of being handled by front end image processing module 106 of lighting level, and itself and the optical target of being stored are compared, so that at least one in the function of auxiliary front end image processing module 106.Image by the resume module in the front end image processing module 106 is the part of digital frame.Also crossfire can be sent to the view finder that can be arranged in display module 112.In preview mode, can be used for controlling automatic exposure from the preview decision of display module 112.
Have that preview mode in the mobile unit of digital camera can be used for normal mode or the hand shake reduces (hjr) pattern.The user can be via menu or is manually selected hjr pattern (being shown as hjr among Fig. 1 selects) by the user interface.Can determine for example automatic exposure parameters such as gain, automatic exposure time, frame rate and frame number to be processed to take snapshot and to produce digital picture in moment afterwards at user's pressing button.Collected statistics can be used for determining the automatic exposure parameter used during snapshot in normal mode and hjr pattern.Therefore, after user's pressing button, the hjr pattern can be different with image processing between the normal mode.Before user's pressing button, even selected the hjr pattern, preview mode is also handled image as it in normal mode.
Fig. 2 is functional block diagram of the configuration of a front end image processing module 106a in the explanation digital image processing system.Front end image processing module 106a can be used for compensating human visual system's response and the sensor signal that produces by imageing sensor 102 between difference.Can use various treatment technologies to proofread and correct these difference, described treatment technology comprises (by way of example) black correction and lens slippage 202, goes mosaic module 204, white balance and colour correction 206, gamma regulate 208, and color conversion 210.These processes are expressed as independent processing module in Fig. 2, but or can use shared hardware or software platform to carry out.In addition, these modules can comprise a plurality of image processing modules of carrying out identical function, allow whereby different images is carried out described function concurrently.
After the color conversion module processes frame, image three-colo(u)r component (Y, Cb and Cr) can be sent shake control module 212 in one's hands.Can will present in the shake control module 212 in one's hands from the various parameters of automatic exposure control module.Hand shake control module 212 can be used for a plurality of purposes.Hand shake control module 212 can be determined the image processing that snapshot takes place afterwards.Hand shake control module 212 can detect the value of hjr selection and determine whether that needing to carry out the hand shake reduces (hjr).Although the user has selected the hjr pattern, hand shake control module 212 can determine to take place the image processing as finishing in the normal mode.Hand shake control module 212 can determine to take place the image processing in the hjr pattern.Producing the digital picture image processing in the hjr pattern can comprise and capture single frame or a plurality of frame.If hand shake control module 212 is determined to capture a plurality of frames, by after the hjr control module, the parameter how many frames frame can handle together with indication noise minimizing/frame registration module 214 can be sent to noise minimizing/frame registration module 214 so.If will handle single frame, can the noise minimizing take place by using noise to reduce module 215 so on single frame.It can be Baeyer (bayer) filter that noise reduces module, or other similar filter.If will handle a plurality of frames, noise minimizing/frame registration module 214 available buffers are shaken the frame number numf of control module 212 appointments by hand so, and it is carried out the frame registration.According to how many frames and lighting level, the purpose of a plurality of frame registrations can be used for the purposes that noise reduces and/or blur minimizing.A plurality of frame registrations can be finished by frame registration module 216.
If hand shake control module 212 is determined image processing and as taking place in the normal mode, although the user selects the hjr pattern so, also can not used noise minimizing/frame registration module 214, and can use the output from (for example) color correction module 210.Determine what image processing (image processing in image processing in the normal mode or the hjr pattern) according to hand shake control module 212, can use signal (sel) to select 217 outputs of which multiplexer to send to post-processing module 218.The output of post-processing module 218 can be sent to static and image/video compressor reducer 108 and/or display module 112.
Except output selects signal (sel) and frame number to be used for the noise minimizing/frame registration, also exportable other parameter of hand shake control module 212: new automatic exposure frame rate (ae fr_new), new automatic exposure gain (ae gain_new), new automatic exposure time (ae time_new) and pending frame number (numf).These parameters can be sent to image sensor module 104 with control chart image-position sensor 102.Hand is shaken the also exportable digital gain of control module 212 and can be applied to image sensor module 104 any module place afterwards.As an example, can during white-balance/color correction module 206, apply digital gain.
One of ordinary skill in the art will recognize that though describe pixel usually, sub-pixel or a plurality of pixel also can be used as the input that enters among the front end image processing module 106a.In addition, the subclass of these picture contents or other form: also the pixel through RGB and space-frequency conversion can be sent shake control module in one's hands, for example hand shake control module 212.So, functional another configuration of capturing the front end image processing module in the digital image processing system is described among Fig. 3.
In Fig. 3, mobile hand shake control module 212, noise minimizing/frame registration module 214 and multiplexer 217 also is inserted in black correction and lens slippage module 202 and goes between the mosaic module 204.This configuration displaying hand shake control module 212 that is embedded among the front end image processing module 106b can be operated R, G and B picture content, and not only to Y, Cb and Cr operation.In general, hand shake control module 212 can operation in any subsequent module and before any display module 112 or still image and video compressor module 108 after imageing sensor 102 capture light.
Can calibrate normal mode and hand shake minimizing pattern by creating at least one automatic exposure time-gain table.Automatic exposure time-gain table can have time for exposure row and gain column.Clauses and subclauses in the time for exposure row can multiply by mutually with the clauses and subclauses in the gain column and produce the exposure time-gain product.Light level value can be represented in each the row clauses and subclauses or the index that enter in automatic exposure time-gain table, each lighting level can be mapped to the automatic exposure index that enters in automatic exposure time-gain table.Automatic exposure time-gain table can have various operating spaces, for example those districts that point out in Fig. 4 A, Fig. 4 B, Fig. 5 A and Fig. 5 B.In all four figure (Fig. 4 A, Fig. 4 B, Fig. 5 A and Fig. 5 B), exist in four possible lighting level districts that illustrate on the axis of abscissas, R1, R2, R3 and R4.The separation in district can be by different boundary (border _ a, border _ b and border _ c) control.The lighting level that the leftmost side of axis of abscissas is showed begins higher and then is reduced to lowly, shows as the rightmost side of axis of abscissas.Exposure time-gain product (that is (time for exposure) * (gain)) should be intended to not reduce in each district.So, the reduction of time for exposure or gain can cause gaining or the increase of time for exposure.Can calibrate automatic exposure time-gain table according to the setting and/or the characteristic of sensor type, camera.Also can use an above frame rate to calibrate described table.In addition, described table can be stored in the memory that is positioned on the integrated circuit (for example, front end image processing module 106 and still image and video compressor module 108) that carries out main image processing.Also described table can be stored in the memory that is not positioned on the integrated circuit that carries out main image processing.In both cases, whether no matter be positioned on the integrated circuit identical with main image processing, described table all can be coupled to image sensor module 104.
Fig. 4 A be explanation at the curve chart of normal mode time for exposure with (detect) lighting level, and Fig. 4 B is explanation at the curve chart of normal mode gain to (detect) lighting level.Because lighting level be height among the R1 in the district, so the time for exposure is shorter, i.e. low value.Similarly, in district R1, gain is in nominal value, and is promptly lower with respect to other district.For the object that slowly moves or when user's hand move very little (that is, about 5ms) time, can have the minimal advantage of the fuzzy and/or noise minimizing among the correction zone R1.Even will carry out fuzzy and/or the noise minimizing, its eyes for the user also are sightless.Therefore, border _ a is positioned at fuzzy and/or noise reduces for around the visible point of user, promptly allows the user to see fuzzy visibly and/or the time for exposure of noise minimizing and the combination of gain.Disengagement zone R1 can change according to different users and/or different camera with border _ a of district R2.Also border _ a can be considered as image processing in the normal mode and the threshold value between the image processing in the hjr pattern.
In district R1 and R2, camera can be in (for example, the fi1) operation down of a frame rate.Shown in Fig. 4 A, along with lighting level reduces, in district R1, the time for exposure increases.Time for exposure increases in district R1.Time for exposure can continue to increase till the time for exposure reaches maximum for frame rate 1 (fr1) in district R2.Under this lighting level (wherein the time for exposure at first is in maximum for frame rate 1 (fr1)), gain (shown in Fig. 4 B) can begin to increase.Gain can continue to increase till the predetermined value that is provided with between alignment epoch is in the point that has the acceptable noise level.If be increased to maximum, noise level may be unacceptable for the described given time for exposure so.For normal mode, border _ b can be expressed as time for exposure under the frame rate 1 (fr1) and be in the point that maximum and gain are in the predetermined value that wherein has the acceptable noise level.Disengagement zone R2 can change according to different cameras with border _ b of district R3.
Need to carry out as far as possible continuously being converted to another district from a district.Therefore, the exposure time-gain product of the leftmost side or identical with it among the exposure time-gain product accessible area R3 of the rightmost side among the district R2.Because the time for exposure is in maximum (in district R2 shown in the rightmost side), thus frame rate can be on the border _ reduction of b place.The reduction of frame rate (that is, making frame rate be changed to fr2 from fr1) allows the time for exposure to increase.Time for exposure can reach maximum (as distinguishing as shown in the R3 among Fig. 4 B) for new frame rate (frame rate 2 fr2).In order to keep distinguishing the continuity of exposure time-gain product between R2 and the district R3, the gain among the reduction district R3 is so that the increase of time for exposure skew.Gain can increase till it reaches maximum.Although can apply digital gain, the gain of using in the normal mode is generally simulation.In normal mode, border _ c is that analog gain reaches peaked point.On the border _ and the c place, in normal mode, the lighting level of correspondence can be stored as the optical target of checking in hand shake minimizing pattern.
Can calibrate hand shake minimizing (hjr) pattern by creating another automatic exposure time-gain table.Can carry out subtraction, addition, division or multiplication to produce required " equivalence " automatic exposure time-gain table in the hand shake minimizing pattern to normal mode automatic exposure time-gain table clauses and subclauses.Fig. 5 A and Fig. 5 B illustrate the characteristic in the row (time for exposure and gain) of another automatic exposure time-gain table or " equivalence " automatic exposure time-gain table.When taking snapshot in the hjr pattern, the automatic exposure time-gain table by equivalence is a normal mode when being in preview mode to the change of time for exposure and gain.Preview mode uses the characteristic and the automatic exposure parameter of normal mode.
To be explanation reduce the curve chart of exposure mode time to (detect) lighting level at hand shake to Fig. 5 A.To be explanation reduce the curve chart of modal gain to (detect) lighting level at hand shake to Fig. 5 B.Reduce in (hjr) pattern in hand shake, target be intended to keep with normal mode in identical exposure time-gain product.So, [time for exposure in the hjr pattern] * [gain in the hjr pattern] should be intended to identical with [time for exposure in the normal mode] * [gain in the normal mode].As previously mentioned, there is the minimal advantage of attempting to reduce the hand shake among the district R1, because it may be invisible for human eye.In district R1, the gain in the hjr pattern and time for exposure can with normal mode in identical.Therefore, the image processing that takes place in district R1 in normal mode also can take place in district R1 when being in the hjr pattern.Yet in district R2, R3 and R4, image processing takes place in the hjr pattern.The image processing that reduces the hand shake is taking place through revising the automatic exposure parameter, and the image processing that does not need to reduce the hand shake takes place with unmodified automatic exposure parameter.
Slight the moving that longer time for exposure increases that hand produces can cause the possibility of bluring.Therefore, in case eyes can detect fuzzy in district R2 and/or noise reduces, the time for exposure just should reduce.Therefore, the time for exposure (shown in Fig. 5 A) that reduces among the district R2 will cause the possibility of bluring to reduce the hand shake.In order to offset the minimizing of time for exposure, increase the gain (shown in Fig. 5 B) among the district R2.Note, in the hjr pattern, keep with normal mode in identical time for exposure and the shape of gain in distinguishing R2.One of difference in district R2 between normal mode and the hjr pattern is from the time for exposure of normal mode and gain and hjr pattern skew is arranged.In the hjr pattern, the time for exposure is in lower skew than in normal mode, and the ratio of gains is in higher skew in normal mode.Yet the exposure time-gain product is equivalent between normal mode and hjr pattern in the district.
As previously mentioned, in normal mode or hjr pattern, but the preview series of frames.Preview mode uses characteristic and (unmodified) automatic exposure parameter of normal mode.In the hjr pattern, the automatic exposure time-gain table of equivalence is the normal mode when being in preview mode.Therefore, when increasing or reduce time for exposure, gain or frame rate, it is with respect to the value in the preview mode.In the hjr pattern, can be by checking when the exposure time-gain product reaches the value identical with exposure time-gain product in the normal mode and determine border _ b.In the hjr pattern, if lighting level the district R3 in, frame rate can increase the amount greater than frame rate fr2 so.Why this can be because in preview mode, in district R3, frame rate is fr2 (being lower than the speed of fr1).For instance, if frame rate fr1 is 30 frame per seconds (fps), and frame rate fr2 is 15fps, and the frame rate of 15fps can increase a certain amount L in the hjr pattern so, up to frame rate fr1.In this example, L is 15fps.Therefore, in the hjr pattern, frame rate (for example) is increased to 30fps from 15fps allows the time for exposure among district R3, to be maximized at its value in district R2 of normal mode.Time for exposure is can cause gaining the reduction in district R3 of the increase among the R3 in the district.If in district R1, R2 and R3, use analog gain, so because the cause of the gain that in the hjr pattern, increases skew (in the leftmost side of district R2), analog gain among the district R3 can be saturated before border _ c, promptly reaches maximum (as the rightmost side finding of district R3).Operate in order to exceed maximum analog gain, can add digital gain to analog gain.Be in district R3 in the bottom of Fig. 5 B and distinguish between the R4 digital gain is described.
The lighting level at border in normal mode _ c place can be stored as predetermined optical target, in the hjr pattern, to check.In the hjr pattern,, may there be enough light to produce the minimum average light level that is used for imageing sensor 102 so for imageing sensor 102 if lighting level is lower than the optical target of the optical target of storing.Can be by light value (brightness and/or colourity) the phase adduction at each pixel place be determined minimum average light level divided by the sum of all pixels order.The another way that calculating is used for the minimum average light level of imageing sensor 102 is to abandon all pixels that are lower than a certain threshold value in calculating.For instance, do not use the pixel that is lower than value 10, and use residual pixel (being higher than a certain threshold value) to calculate minimum average light level.Usually, use the brightness light value, but also can use chromatic value.For illustration purposes, discuss brightness value.In the hjr pattern, if brightness value is lower than predetermined luminance (light) target, light (brightness) level will be in district R4 so.
If determined that lighting level is lower than luma target in the hjr pattern, need not so to change and in preview mode, be the frame rate of fr2.Yet, can reduce owing to the amount of light and regulate the time for exposure, and can increase the time for exposure.Can be increased to the maximum that is allowed for frame rate fr2 the time for exposure.In order to offset the increase of time for exposure, digital gain on the border _ the c place reduces, and can constantly increase in district R4.
In district R1, R2 and R3, normal mode is intended to identical with exposure time-gain product between the hjr pattern.That is, should reduce difference so that difference between the two approaching as far as possible zero.In district R4, imageing sensor 102 may not satisfy the lighting level of minimum average amount.Therefore, can apply digital gain to increase the gain stage among the district R4.
Fig. 6 is the flow chart that explanation produces the method through revising the automatic exposure parameter.As illustrated in the flow chart, can be in capture frame 600 after the snapshot.The last frame of handling in the preview mode can be stored in the memory.Capture and mean from the memory search frame and can be used for producing digital picture.Under the present frame speed and memory storage capabilities on the mobile unit, can exist in the memory after the snapshot and be less than ten frames.But be not each frame that when producing digital picture, may need in the memory.Usually, three institute's capture frame can be used for producing digital picture at the most.In order to produce after snapshot through revising the automatic exposure parameter, camera can be in the hjr pattern.Therefore, in another embodiment, entering hjr pattern 602 can be prior to capture frame after snapshot.But in Fig. 6, enter hjr pattern 602 and be illustrated as after capture frame after snapshot.In case be in the hjr pattern, the lighting level that is detected just with from the unmodified automatic exposure parameter correlation of normal mode connection 604.Can be under the lighting level that is detected from the gain and time for exposure 606 of memory search unmodified.Can finish the calculating 608 of the exposure time-gain product in the normal mode by multiplying each other with gain from the time for exposure of normal mode.Can calculate exposure time-gain product in the hjr pattern by revising from the automatic exposure parameter of normal mode, these automatic exposure parameters can be gain, time for exposure and/or frame rate.Can revise the automatic exposure parameter to reduce differing from 610 between exposure time-gain product in the hjr pattern and the exposure time-gain product in the normal mode.Therefore, exist corresponding to the exposure time-gain product in the hjr pattern of the exposure time-gain product in the normal mode.Ideally, described correspondence should be identical.Yet, because potential the lighting level that detects is slightly different between potential computational accuracy difference when multiplying each other and hjr pattern and the normal mode, so described correspondence can be inequality.Difference between the exposure time-gain product in hjr pattern and the normal mode reduces just enough as much as possible.In addition, in the district in the hjr pattern (for example district 4 above), imageing sensor 102 may not satisfy the lighting level of minimum average amount.Therefore, may be able to not in the hjr pattern, keep with normal mode in identical exposure time-gain product.
Be that flow chart how to operate camera system in normal mode and hand shake minimizing pattern is described among Fig. 7.In order to produce image, first action is a capture frame 700 after snapshot.In case capture, just carry out and check to find out the hjr pattern of selecting whether.If still non-selected hjr pattern can be transmitted unmodified automatic exposure parameter so with control chart image-position sensor 704.These parameters can be automatic exposure frame rate (ae fr), automatic exposure time (ae time), automatic exposure gain (aegain), measured brightness, and luma target.If selected the hjr pattern, the modification 706 of automatic exposure parameter takes place so when needed.In next action, carry out to check to find out that lighting level whether in district R1 708, promptly exists in the district of identical (or much at one) parameter in normal mode.If described lighting level can be transmitted unmodified automatic exposure parameter with control chart image-position sensor 704 so greater than the lighting level (that is, lighting level is in district R1) at border _ a place.Although selected the hjr pattern, the image processing in the normal mode can take place.In subsequent action, after transmission unmodified automatic exposure parameter is with the control chart image-position sensor, can continue the processing 710 of digital picture.If described lighting level is lower than the lighting level (performed as the inspection in the decision block 708) at border _ a place, can uses so through revising automatic exposure parameter and frame number (numf) and digital gain and come control chart image-position sensor 711.Through revising the automatic exposure parameter can be new automatic exposure frame rate (ae fr_new), new automatic exposure time (ae time_new), new automatic exposure gain (ae gain_new), measured brightness, and luma target.Can check the number numf 712 that will handle with the frame that produces digital picture.If numf is greater than 1, but a plurality of frames 714 of registration so.But the motion between the frame registration compensated frame (level, vertical or angled).It can increase the frame registration intensity of gained image afterwards.If numf is greater than 2, but also aid in noise minimizing of frame registration so.If numf is 1 and has selected hjr, the minimizing 716 of noise so also can take place.A kind of possible mode that reduces the noise in the frame is to use Baeyer to picture content to filter.After the noise minimizing takes place, can continue the processing 710 of digital picture.It should be noted that and to shake the frame 702,704,706 and 711 that finds in the control module 212 among Fig. 7 at hand.
Fig. 8 is the automatic exposure parameter in the camera system is revised in explanation in hand shake minimizing pattern a flow chart.Although the selection of hjr pattern may take place, but still can use the image processing in the normal mode.Carry out by decision block 800 and to check, if lighting level greater than the lighting level at border _ a place, so pending frame number numf with the generation digital picture is set to 1802, and uses normal image processing.In the case, the automatic exposure parameter is not made amendment.If lighting level less than the lighting level at border _ a place, can be used the image processing in the hjr pattern so.Pending frame number numf with the generation digital picture is set to 1804.As illustrated among Fig. 5 A and Fig. 5 B, can be reduced a certain amount M 806 time for exposure.Exist M can wider range, yet, be in or about 50% value has been showed good result.Gain also can increase a certain amount K 808.Because the exposure time-gain product in the hjr pattern is intended to keep the exposure time-gain product in the normal mode, thus K be in or about 200% value exhibition reduce the hand shake.
If the gain that increases surpasses the maximum analog gain of imageing sensor, can use the selection 810 of the minimum value of the gain that has increased K between gaining with maximum (simulation) so.Also can in as the district of R2, use it in the configuration of use digital gain therein.The ratio 812 that has increased the gain of K and new analog gain (ae_gain_new) can be compared with constant C 1.C1 is relevant with the maximum analog gain of transducer, and typical value can be 1.Select C1 and increased the gain of K and the ratio of new analog gain between minimum value 814, and can apply digital gain at described minimum value place.
Some different configurations and technology have been described.Described technology can be improved and remove fuzzy from image with the longer time for exposure.Described technology and configuration also can be the auxiliary hand that reduces of in fact any digital device of pictures taken and shake.Described technology and configuration may be implemented in hardware, software, firmware or its any combination.If be implemented in the software, so described technology and configuration can be directed to the computer-readable media that comprises computer readable program code (also can be described as computer code), and described computer readable program code the time is carried out one or more in the method mentioned above when execution in the device in pictures taken.
Described computer readable program code can computer-readable instruction form be stored on the memory.Under described situation, for example the processor of DSP can be carried out and be stored in the instruction in the memory so that carry out one or more in the technology described herein.In some cases, described technology can be carried out by DSP, and described DSP calls the various nextport hardware component NextPorts that multiply each other of for example time for exposure and gain to produce the exposure time-gain product.The exposure time-gain product that is disclosed may be implemented in one or more microprocessors, one or more application-specific integrated circuit (ASIC)s (ASIC) and one or more field programmable gate arrays (FPGA) or a certain other hardware-combination of software.These technology and configuration are within the scope of the appended claims.
Claims (28)
1. device, it comprises:
Be used for discerning the device of the lighting level of at least one institute's capture images;
Be used to detect the device that hand is shaken the selection of minimizing pattern;
Be used for dynamically selecting the hand jitter mode through revise the automatic exposure parameter, described at least one institute's capture images is carried out the device of image processing, the unmodified automatic exposure parameter of described hand shake minimizing pattern has the exposure time-gain product corresponding to the exposure time-gain product of normal mode through revising the automatic exposure parameter.
2. device according to claim 1, gain and the time for exposure wherein said comprising through modification automatic exposure parameter.
3. device according to claim 1 wherein saidly further comprises frame rate through revising the automatic exposure parameter.
4. the computer-readable media of a store sets of instructions, wherein said instruction set comprises when being carried out by one or more processors:
Be used for discerning the device of the lighting level of at least one institute's capture images;
Be used to detect the device that hand is shaken the selection of minimizing pattern;
Be used for dynamically selecting the hand jitter mode through revise the automatic exposure parameter, described at least one institute's capture images is carried out the device of image processing, the unmodified automatic exposure parameter of described hand shake minimizing pattern has the exposure time-gain product corresponding to the exposure time-gain product of normal mode through revising the automatic exposure parameter.
5. computer-readable media according to claim 4, gain and the time for exposure wherein said comprising through modification automatic exposure parameter.
6. computer-readable media according to claim 4 wherein saidly further comprises frame rate through revising the automatic exposure parameter.
7. device that comprises the carries out image processing of hand shake minimizing pattern, its have the device that is used to detect lighting level and in response to the lighting level of described detection to reduce the device of hand shake, single frames noise minimizing when the shake of the described hand of described minimizing comprises when described lighting level is in first scope in and the multiframe noise minimizing in described lighting level is in second scope time.
8. device according to claim 7, wherein said minimizing hand shake comprises modified automatic exposure parameter.
9. device according to claim 8, wherein said modified automatic exposure parameter is gain, time for exposure and frame rate.
10. device according to claim 7, wherein said multiframe noise minimizing alleviates fuzzy by the multiframe registration.
11. it is that Baeyer filters that device according to claim 7, wherein said single frames noise reduce.
12. device according to claim 7 wherein when the lighting level of described detection is higher than the lighting level of employed minimum when removing fuzzy in the digital picture, carries out the image processing in the hand shake minimizing pattern.
13. device according to claim 11, the lighting level of wherein said detection is mapped to the automatic exposure index.
14. device according to claim 7, wherein said first scope, first frame rate reaches maximum, and in response, described first frame rate reduces and increases the time for exposure to produce second frame rate.
15. device according to claim 14, wherein gain response is in time for exposure of described increase and reduce.
16. device according to claim 7, wherein said second scope comprises two scopes, the 3rd scope and the 4th scope.
17. camera apparatus according to claim 16, second border between wherein said the 3rd scope and described the 4th scope is to be determined by the described lighting level that does not satisfy luma target.
18. device according to claim 17 wherein applies digital gain in described the 3rd scope and described the 4th scope.
19. device according to claim 7, wherein said first scope is between border a and border b.
20. device according to claim 7, wherein said second scope is beyond the boundary _ b.
21. an integrated circuit, it is suitable for being coupled to imageing sensor, with in response to the modified automatic exposure parameter that is produced by described integrated circuit, shakes and shake the hand that reduces the digital picture of capturing in the minimizing pattern at hand, and described integrated circuit comprises:
Be used to discern the device of at least one institute's capture images;
Be used to detect the device that hand is shaken the selection of minimizing pattern;
Be used to discern the device of the lighting level that is associated with described at least one institute's capture images;
Be used for described lighting level is mapped to corresponding to the gain of normal mode and the device of time for exposure;
Be used for the device that calculates the first exposure time-gain product from the gain of described correspondence and time for exposure; And
Be used for producing modified automatic exposure parameter in response to the described exposure time-gain product of described calculating, to reduce the device of hand shake.
22. device according to claim 21, wherein said being used for produces through revising the device of automatic exposure parameter in response to the described exposure time-gain product of described calculating, comprises and revising described gain and time for exposure.
23. further comprising, device according to claim 22, the wherein said device that is used to produce modified automatic exposure parameter calculate the second exposure time-gain product.
24. device according to claim 21, wherein said integrated circuit is MSM
TM
25. a method that produces modified automatic exposure parameter, it comprises:
After snapshot, capture at least one frame;
Enter the hjr pattern;
Make the lighting level and the unmodified automatic exposure parameter correlation connection that are detected;
From memory search time for exposure and gain;
Time for exposure and the gain calculating first exposure time-gain product from described retrieval; And
Produce modified automatic exposure parameter based on the described first exposure time-gain product.
26. method according to claim 25 wherein saidly produces modified automatic exposure parameter based on the described first exposure time-gain product and comprises: multiply by, deduct, add and divided by the time for exposure and the gain of described retrieval.
27. method according to claim 25, gain and the time for exposure wherein said comprising through modification automatic exposure parameter.
28. method according to claim 27 wherein saidly further comprises frame rate through revising the automatic exposure parameter.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US11/534,993 US8120658B2 (en) | 2006-01-19 | 2006-09-25 | Hand jitter reduction system for cameras |
| US11/534,993 | 2006-09-25 | ||
| PCT/US2007/060809 WO2008039551A1 (en) | 2006-09-25 | 2007-01-20 | A hand jitter reduction system for cameras |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101518054A true CN101518054A (en) | 2009-08-26 |
| CN101518054B CN101518054B (en) | 2012-11-21 |
Family
ID=39230515
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2007800354470A Active CN101518054B (en) | 2006-09-25 | 2007-01-20 | A hand jitter reduction system for cameras |
Country Status (5)
| Country | Link |
|---|---|
| EP (1) | EP2070312A1 (en) |
| JP (2) | JP5507251B2 (en) |
| KR (1) | KR101120966B1 (en) |
| CN (1) | CN101518054B (en) |
| WO (1) | WO2008039551A1 (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103685935A (en) * | 2012-09-21 | 2014-03-26 | 黑莓有限公司 | Methods and devices for controlling camera image capture |
| CN104219448A (en) * | 2014-08-28 | 2014-12-17 | 小米科技有限责任公司 | Shooting method and shooting device |
| JP2015534765A (en) * | 2012-09-25 | 2015-12-03 | サムスン エレクトロニクス カンパニー リミテッド | Photographed image generation method and apparatus |
| CN105472263A (en) * | 2014-09-12 | 2016-04-06 | 聚晶半导体股份有限公司 | Image capture method and image capture device with use of method |
| CN105516583A (en) * | 2014-10-10 | 2016-04-20 | 鹦鹉股份有限公司 | Mobile appratus, in particular rotary-wing drone, provided with video camera delivering sequences of images dynamically corrected for the wobble effect |
| CN105849746A (en) * | 2013-12-27 | 2016-08-10 | 三菱电机株式会社 | Two-dimensional code reading device |
Families Citing this family (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP2347402B1 (en) | 2008-09-22 | 2016-07-13 | Freedom Scientific Inc. | Systems and methods for imaging objects |
| CN101510960B (en) * | 2009-03-26 | 2014-06-25 | 北京中星微电子有限公司 | Mobile phone camera shooting method and apparatus |
| US8659670B2 (en) * | 2009-04-20 | 2014-02-25 | Qualcomm Incorporated | Motion information assisted 3A techniques |
| US9641759B2 (en) | 2014-07-10 | 2017-05-02 | Lg Electronics Inc. | Mobile terminal and controlling method thereof |
| KR102176230B1 (en) * | 2017-07-14 | 2020-11-09 | 주식회사 엘지화학 | Method for analyzing polymer layer |
Family Cites Families (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5745808A (en) * | 1995-08-21 | 1998-04-28 | Eastman Kodak Company | Camera exposure control system using variable-length exposure tables |
| JPH1141515A (en) * | 1997-07-24 | 1999-02-12 | Minolta Co Ltd | Still camera |
| JPH11215432A (en) * | 1997-11-19 | 1999-08-06 | Ricoh Co Ltd | Digital camera |
| JPH11215431A (en) * | 1998-01-27 | 1999-08-06 | Hitachi Denshi Ltd | Television camera |
| JP2000047297A (en) * | 1998-07-28 | 2000-02-18 | Minolta Co Ltd | Digital still camera |
| JP2000224470A (en) * | 1999-02-02 | 2000-08-11 | Minolta Co Ltd | Camera system |
| JP2001045361A (en) * | 1999-07-30 | 2001-02-16 | Hitachi Ltd | Image pickup device |
| JP2003259199A (en) * | 2002-03-05 | 2003-09-12 | Sony Corp | Camera signal processor and exposure range expanding method thereof |
| JP2003322906A (en) * | 2002-04-30 | 2003-11-14 | Olympus Optical Co Ltd | Camera |
| JP3934506B2 (en) * | 2002-08-06 | 2007-06-20 | オリンパス株式会社 | Imaging system and image processing program |
| US7295232B2 (en) * | 2003-01-15 | 2007-11-13 | Canon Kabushiki Kaisha | Camera and program |
| JP2004221992A (en) * | 2003-01-15 | 2004-08-05 | Canon Inc | Imaging device and program |
| JP2004304252A (en) * | 2003-03-28 | 2004-10-28 | Minolta Co Ltd | Imaging systems |
| JP2004357202A (en) * | 2003-05-30 | 2004-12-16 | Canon Inc | Photographing apparatus |
| JP2005286790A (en) * | 2004-03-30 | 2005-10-13 | Fuji Photo Film Co Ltd | Mobile terminal device |
| JP4677748B2 (en) * | 2004-09-15 | 2011-04-27 | 株式会社ニコン | Imaging device |
| CN101095078B (en) * | 2004-12-29 | 2010-04-28 | 诺基亚公司 | Exposure of digital imaging |
-
2007
- 2007-01-20 KR KR1020097008384A patent/KR101120966B1/en active IP Right Grant
- 2007-01-20 EP EP07710242A patent/EP2070312A1/en not_active Ceased
- 2007-01-20 WO PCT/US2007/060809 patent/WO2008039551A1/en active Application Filing
- 2007-01-20 CN CN2007800354470A patent/CN101518054B/en active Active
- 2007-01-20 JP JP2009530477A patent/JP5507251B2/en active Active
-
2012
- 2012-04-18 JP JP2012094648A patent/JP5612017B2/en active Active
Cited By (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103685935A (en) * | 2012-09-21 | 2014-03-26 | 黑莓有限公司 | Methods and devices for controlling camera image capture |
| JP2015534765A (en) * | 2012-09-25 | 2015-12-03 | サムスン エレクトロニクス カンパニー リミテッド | Photographed image generation method and apparatus |
| CN105849746A (en) * | 2013-12-27 | 2016-08-10 | 三菱电机株式会社 | Two-dimensional code reading device |
| CN104219448A (en) * | 2014-08-28 | 2014-12-17 | 小米科技有限责任公司 | Shooting method and shooting device |
| CN104219448B (en) * | 2014-08-28 | 2015-12-02 | 小米科技有限责任公司 | Image pickup method and device |
| US9661223B2 (en) | 2014-08-28 | 2017-05-23 | Xiaomi Inc. | Method and device for photographing including camera shake |
| CN105472263A (en) * | 2014-09-12 | 2016-04-06 | 聚晶半导体股份有限公司 | Image capture method and image capture device with use of method |
| CN105472263B (en) * | 2014-09-12 | 2018-07-13 | 聚晶半导体股份有限公司 | Image acquisition method and the image capture equipment for using the method |
| CN105516583A (en) * | 2014-10-10 | 2016-04-20 | 鹦鹉股份有限公司 | Mobile appratus, in particular rotary-wing drone, provided with video camera delivering sequences of images dynamically corrected for the wobble effect |
Also Published As
| Publication number | Publication date |
|---|---|
| KR20090057461A (en) | 2009-06-05 |
| JP2012182806A (en) | 2012-09-20 |
| WO2008039551A1 (en) | 2008-04-03 |
| JP2010504718A (en) | 2010-02-12 |
| JP5507251B2 (en) | 2014-05-28 |
| JP5612017B2 (en) | 2014-10-22 |
| CN101518054B (en) | 2012-11-21 |
| KR101120966B1 (en) | 2012-03-05 |
| EP2070312A1 (en) | 2009-06-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101518054B (en) | A hand jitter reduction system for cameras | |
| US10602069B2 (en) | Digital camera and display method of digital camera | |
| CN101212571B (en) | Image capturing apparatus and focusing method | |
| US9237319B2 (en) | Imaging device and automatic focus adjustment method | |
| US7606476B2 (en) | Imaging device and imaging method | |
| US8120658B2 (en) | Hand jitter reduction system for cameras | |
| JPWO2014087807A1 (en) | Imaging apparatus, abnormal oblique incident light detection method and program, and recording medium | |
| JP2008078945A (en) | Imaging apparatus with blurring correction function, blurring correction method, and blurring correction processing program | |
| US11496666B2 (en) | Imaging apparatus with phase difference detecting element | |
| JP2009010616A (en) | Imaging device and image output control method | |
| US8860876B2 (en) | Image display using flash reach information | |
| US8218028B2 (en) | Imaging apparatus, image processing apparatus, and program | |
| JP2007052061A (en) | Imaging device, focusing operation control method, and electronic information apparatus | |
| JP2013031010A (en) | Imaging apparatus, imaging method, and program | |
| KR20110072213A (en) | Image correction method | |
| US9392181B2 (en) | Image capture apparatus and method of controlling the same | |
| EP2632146B1 (en) | Image processing apparatus | |
| US11057560B2 (en) | Image pickup apparatus, image pickup method, and recording medium | |
| TWI394439B (en) | A digital still camera with previewing function by picture in picture and method thereof | |
| JPWO2015156187A1 (en) | Imaging apparatus and control method thereof | |
| JP4876642B2 (en) | Imaging apparatus and program | |
| JP2021064836A (en) | Imaging apparatus | |
| JP2008311812A (en) | Imaging apparatus and display method of imaging apparatus | |
| JPWO2019193889A1 (en) | Image alignment aids, methods and programs and imaging devices | |
| JP2011030107A (en) | Defective pixel threshold value specifying device and defective pixel threshold value specifying program |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant |