EP3343899B1 - Electronic device for performing autofocus - Google Patents
Electronic device for performing autofocus Download PDFInfo
- Publication number
- EP3343899B1 EP3343899B1 EP17209056.5A EP17209056A EP3343899B1 EP 3343899 B1 EP3343899 B1 EP 3343899B1 EP 17209056 A EP17209056 A EP 17209056A EP 3343899 B1 EP3343899 B1 EP 3343899B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- lens set
- camera module
- processor
- reliability
- value
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 229920006395 saturated elastomer Polymers 0.000 claims description 10
- 238000001514 detection method Methods 0.000 claims description 6
- 230000006870 function Effects 0.000 description 18
- 230000008859 change Effects 0.000 description 16
- 238000000034 method Methods 0.000 description 14
- 230000003287 optical effect Effects 0.000 description 11
- 230000000737 periodic effect Effects 0.000 description 7
- 230000014509 gene expression Effects 0.000 description 5
- 238000012545 processing Methods 0.000 description 4
- 238000012937 correction Methods 0.000 description 3
- 230000003247 decreasing effect Effects 0.000 description 3
- 230000004044 response Effects 0.000 description 3
- 230000035945 sensitivity Effects 0.000 description 3
- 239000003086 colorant Substances 0.000 description 2
- 238000002591 computed tomography Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 2
- WQZGKKKJIJFFOK-GASJEMHNSA-N Glucose Natural products OC[C@H]1OC(O)[C@H](O)[C@@H](O)[C@@H]1O WQZGKKKJIJFFOK-GASJEMHNSA-N 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 238000002583 angiography Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000003705 background correction Methods 0.000 description 1
- 239000008280 blood Substances 0.000 description 1
- 210000004369 blood Anatomy 0.000 description 1
- 230000036772 blood pressure Effects 0.000 description 1
- 230000036760 body temperature Effects 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000005611 electricity Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 239000004744 fabric Substances 0.000 description 1
- 239000011521 glass Substances 0.000 description 1
- 239000008103 glucose Substances 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 238000002595 magnetic resonance imaging Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 230000007246 mechanism Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012806 monitoring device Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000011867 re-evaluation Methods 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000006641 stabilisation Effects 0.000 description 1
- 238000011105 stabilization Methods 0.000 description 1
- 238000005406 washing Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
- G03B13/32—Means for focusing
- G03B13/34—Power focusing
- G03B13/36—Autofocus systems
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B43/00—Testing correct operation of photographic apparatus or parts thereof
-
- G—PHYSICS
- G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
- G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
- G03B5/00—Adjustment of optical system relative to image or object surface other than for focusing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/67—Focus control based on electronic image sensor signals
- H04N23/673—Focus control based on electronic image sensor signals based on contrast or high frequency components of image signals, e.g. hill climbing method
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
- H04N23/682—Vibration or motion blur correction
- H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
Definitions
- the present disclosure relates to an auto focus (AF) method of a camera and an electronic device for performing the same.
- AF auto focus
- a camera has evolved from a device that exposes a film reel to a digital memory-based camera. Recently, cameras have been included as part of a portable electronic device such as smartphones.
- a camera may condense light incident from the outside using its lens and may convert the condensed light into an electric signal.
- the electric signal may configure image data, and may be stored in a memory or may be output on a display.
- Focusing in the mentioned camera may be important to obtain a high-quality image, for example, enhance sharpness of a subject.
- An automatic focus (AF) function may be to move a lens included in a camera back and forth to focus on a subject (i.e., allow a subject to be on focus).
- Each of various electronic devices including the camera may include a display and may output an image received from the camera on the display in real time (a preview or a live view).
- a field of view (FoV) of an image output on the display may be frequently wobbled (a so-called wobbling phenomenon).
- an aspect of the present disclosure is to provide an electronic device for providing a live view finder function through a main camera module and may perform AF through a sub-camera module. Accordingly, another aspect of the present disclosure is to provide an electronic device for obtaining data associated with AF from a sub-camera module and using the obtained data for an AF operation of a main camera module.
- an electronic device in accordance with the appended independent device claim.
- the processor can thereby suspend moving the second lens set and prevent the wobbling phenomenon in the displayed image, until the auto-focus function in relation to the first lens set achieves and surpasses a certainty of a correct auto focus direction of movement for the first lens, while not having achieved the fully completed auto focus yet.
- An electronic device may remove an image wobble in a main camera module which supports a live view and may enhance a speed of an AF function a user feels.
- a variety of effects directly or indirectly ascertained through the present disclosure may be provided.
- the expressions "have”, “may have”, “include” and “comprise”, or “may include” and “may comprise” used herein indicate existence of corresponding features (e.g., elements such as numeric values, functions, operations, or components) but do not exclude presence of additional features.
- the expressions "A or B”, “at least one of A or/and B”, or “one or more of A or/and B”, and the like may include any and all combinations of one or more of the associated listed items.
- the term “A or B”, “at least one of A and B”, or “at least one of A or B” may refer to all of the case (1) where at least one A is included, the case (2) where at least one B is included, or the case (3) where both of at least one A and at least one B are included.
- first, second, and the like used in this disclosure may be used to refer to various elements regardless of the order and/or the priority and to distinguish the relevant elements from other elements, but do not limit the elements.
- a first user device and "a second user device” indicate different user devices regardless of the order or priority.
- a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.
- the expression “configured to” used in this disclosure may be used as, for example, the expression “suitable for”, “having the capacity to”, “designed to”, “adapted to”, “made to”, or “capable of”.
- the term “configured to” must not mean only “specifically designed to” in hardware. Instead, the expression “a device configured to” may mean that the device is “capable of” operating together with another device or other components.
- a "processor configured to (or set to) perform A, B, and C” may mean a dedicated processor (e.g., an embedded processor) for performing a corresponding operation or a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor) which performs corresponding operations by executing one or more software programs which are stored in a memory device.
- a dedicated processor e.g., an embedded processor
- a generic-purpose processor e.g., a central processing unit (CPU) or an application processor
- An electronic device may include at least one of, for example, smartphones, tablet personal computers (PCs), mobile phones, video telephones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), Motion Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, mobile medical devices, cameras, or wearable devices.
- PCs tablet personal computers
- PDAs personal digital assistants
- PMPs Portable multimedia players
- MPEG-1 or MPEG-2 Motion Picture Experts Group Audio Layer 3
- MP3 Motion Picture Experts Group Audio Layer 3
- the wearable device may include at least one of an accessory type (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lens, or head-mounted-devices (HMDs), a fabric or garment-integrated type (e.g., an electronic apparel), a body-attached type (e.g., a skin pad or tattoos), or a bio-implantable type (e.g., an implantable circuit).
- an accessory type e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lens, or head-mounted-devices (HMDs)
- a fabric or garment-integrated type e.g., an electronic apparel
- a body-attached type e.g., a skin pad or tattoos
- a bio-implantable type e.g., an implantable circuit
- the electronic device may be a home appliance.
- the home appliances may include at least one of, for example, televisions (TVs), digital versatile disc (DVD) players, audios, refrigerators, air conditioners, cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, home automation control panels, security control panels, TV boxes (e.g., Samsung HomeSyncTM, Apple TVTM, or Google TVTM), game consoles (e.g., XboxTM or PlayStationTM), electronic dictionaries, electronic keys, camcorders, electronic picture frames, and the like.
- TVs televisions
- DVD digital versatile disc
- an electronic device may include at least one of various medical devices (e.g., various portable medical measurement devices (e.g., a blood glucose monitoring device, a heartbeat measuring device, a blood pressure measuring device, a body temperature measuring device, and the like), a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI), a computed tomography (CT), scanners, and ultrasonic devices), navigation devices, Global Navigation Satellite System (GNSS), event data recorders (EDRs), flight data recorders (FDRs), vehicle infotainment devices, electronic equipment for vessels (e.g., navigation systems and gyrocompasses), avionics, security devices, head units for vehicles, industrial or home robots, automatic teller's machines (ATMs), points of sales (POSs) of stores, or internet of things (e.g., light bulbs, various sensors, electric or gas meters, sprinkler devices, fire alarms, thermostats, street lamps, toasters, exercise equipment, hot water tanks, heaters, boilers,
- the electronic device may include at least one of parts of furniture or buildings/structures, electronic boards, electronic signature receiving devices, projectors, or various measuring instruments (e.g., water meters, electricity meters, gas meters, or wave meters, and the like).
- the electronic device may be one of the above-described devices or a combination thereof.
- An electronic device according to an embodiment may be a flexible electronic device.
- an electronic device according to an embodiment of this disclosure may not be limited to the above-described electronic devices and may include other electronic devices and new electronic devices according to the development of technologies.
- the term "user” may refer to a person who uses an electronic device or may refer to a device (e.g., an artificial intelligence electronic device) that uses the electronic device.
- FIG. 1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure.
- an electronic device 101 may include a first camera module 110, a second camera module 120, an image sensor interface (I/F) 130, an image signal processor (ISP) 140, a display 150, a memory 160, and a processor 170.
- the electronic device 101 may be implemented not to include some of the elements or may be implemented to include an element which is not shown.
- the first camera module 110 may include a first lens set 111, a first actuator 112, aperture walls 113 defining an aperture, and a first image sensor 114.
- the first camera module 110 may omit some of the elements or may be implemented to include additional elements (e.g., an optical image stabilization (OIS) module, a flash, or the like) not shown.
- OIS optical image stabilization
- the first camera module 110 may be referred to as a first optical system.
- the first lens set 111 may condense light reflected from the outside to the first camera 110.
- the condensed light may arrive at a first image sensor 114 via the aperture formed from aperture walls 113.
- the first lens set 111 may allow light reflected or generated from a subject to reach a pixel array of the first image sensor 114.
- the first actuator 112 may drive the first lens set 111 depending on control of the processor 170.
- the first actuator 112 may allow a subject 10 to be on focus by moving the first lens set 111.
- the first actuator 112 may include a servo motor or an ultrasonic motor. According to various embodiments, the first actuator 112 may be referred to as a lens actuator, an AF actuator, or the like.
- the aperture walls 113 may adjust an amount of light which arrives at (or incident to) the first image sensor 114 by narrowing or widening the aperture formed therebetween. In general, as the aperture walls 113 are more widely opened, an amount of light which arrives at the first image sensor 114 increases (an effective aperture may be more increased). As the aperture walls 113 are more narrowly closed, an amount of light which arrives at the first image sensor 114 decreases (an effective aperture may be more decreased). According to an embodiment, the aperture walls 113 may defining an aperture having fixed width. In other words, the effective aperture may have a specified value by the fixed aperture.
- the first image sensor 114 may include a pixel array in which a plurality of pixels are arranged as a two-dimensional (2D) grating shape. In certain embodiments, millions of pixels, even tens of millions of pixels, may be included in the pixel array. One of a plurality of reference colors may be assigned to each one of the plurality of pixels. For example, the plurality of reference colors may include "red, green, and blue (RGB)".
- the first image sensor 114 may be implemented using, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS).
- CCD charge-coupled device
- CMOS complementary metal-oxide-semiconductor
- the first image sensor 114 may generate an electric signal based on light received from the outside and may generate a digital image (e.g., a Bayer image) based on the electric signal.
- the image generated by the first image sensor 114 may be transmitted to the image sensor I/F 130.
- a first shutter 119 may adjust, for example, a time when the first image sensor 114 is exposed to light, under control of the processor 170. For example, if the first shutter 119 is operated slowly, more light may be incident to the first image sensor 114. If the first shutter 119 is operated faster, less light may be incident to the first image sensor 114. A time when the shutter 119 is operated may be determined according to a shutter speed set automatically or manually.
- the first shutter 119 may include a mechanical shutter or an electronic shutter of a manner of controlling an image sensor.
- the second camera module 120 may include the same or similar configuration to the first camera module 110.
- the second camera module 120 may include a second lens set 121, a second actuator 122 for driving the second lens set 121, a second aperture 123, a second image sensor 124, and a second shutter 129.
- a field of view (FoV) of the first camera 110 and an FoV of the second camera module 120 may be substantially the same as each other or may be within a constant range.
- the first camera 110 and the second camera module 120 may be exposed to the outside through a common surface of the electronic device 101 (e.g., a surface opposite to a surface where the display 150 is located).
- the second camera module 120 operates similar to the first camera module 110.
- a specification of each element included in the second camera module 120 may different from a specification of each element included in the first camera module 110.
- focal distances of the first lens set 111 and the second lens set 121, the number of pixels of the first image sensor 114 and the number of pixels of the second image sensor 124, each pixel pitch of the first image sensor 114 and each pixel pitch of the second image sensor 124, and the like may have different values.
- the image sensor I/F 130 may serve as a medium for transmitting and receiving data between image sensor and another element (e.g., the ISP 140).
- the image sensor I/F 130 may transmit image data generated by the first image sensor 114 or the second image sensor 124 to the ISP 140.
- the ISP 140 may include a plurality of image processing (IP) functions.
- the ISP 140 may correct an image obtained from the first camera module 110 or the second camera module 120, through the plurality of IP functions.
- the plurality of IP functions may include various IP functions, such as an IP block for color interpolation, an IP block for lens shading correction, an IP block for auto white balance, an IP block for lateral chromatic aberration correction, an IP block for optical inverse correction, an IP block for noise reduction, an IP block for edge enhancement, an IP block for gamma correction, or an IP block for out-of-focus (or shallow depth of field (DoF)).
- the plurality of IP blocks may be referred to as an image processing filter, an image processing module, or the like. Further, according to various embodiments, the ISP 140 may be included in the processor 170.
- the display 150 may visually output a variety of content (e.g., text, an image, a video, an icon, a widget, a symbol, or the like) to the user.
- the display 150 may be configured to output an image received from the first camera module 110. If an image from the first camera module 110 is output on the display 150 in real time, an output of the image may be referred to as a so-called live view.
- the display 150 may include a touch pad.
- the display 150 may detect a touch, a gesture, proximity, or a hovering input using an electronic pen or part of a user's body.
- the memory 160 may include a volatile and/or non-volatile memory.
- the memory 160 may store various instructions or data associated with the elements 110 to 150 and 170 included in the electronic device 101.
- the memory 160 may store instructions to perform various operations described in the present disclosure.
- the instructions may be implemented as software, for example, an application program, an operating system (OS), or firmware, may be stored in the memory 160, and may be embedded in hardware.
- OS operating system
- the processor 170 may be electrically connected with the elements 110 to 160 included in the electronic device 101 and may perform an arithmetic operation about control and/or communication of the elements 110 to 160 included in the electronic device 101.
- the processor 170 may control the first camera module 110 to perform AF on the subject 10 in response to a specified event.
- an image obtained through the second camera module 120 (or the second image sensor 124 of the second camera module 120) may be output on the display 150 in real time.
- the specified event may be an event for triggering AF and may include, for example, a selection (e.g., a touch) of a user for an object of the subject 10 displayed on the display 150, a change in scene, or the like.
- the change in scene may be determined from, for example, a change in brightness of a specified level or more of an image received from the first camera module 110 or the second camera module 120, a movement/change of the recognized subject 10, a change in sensitivity of a specified level or more, or the like.
- the processor 170 may move the first lens set 111 to focus on a subject, using AF value obtained by the first camera module 110.
- the processor 170 may move the first lens set 111 by controlling the first actuator 112.
- the processor 170 may use a contrast AF scheme or a phase difference AF scheme.
- the AF value may correspond to a contrast value or a phase difference value obtained from the first camera module 111.
- the processor 170 may evaluate reliability of the AF value of an image obtained by the first camera module 110.
- the processor 170 may determine whether to move the second lens set 121 included in the second camera module 120, based on the reliability.
- the reliability of the AF value may be determined by various elements. According to various embodiments, the processor 170 may evaluate reliability based on a level of a noise included in an AF value, brightness of an image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, a degree of a flicker phenomenon of the AF value, or two or more combinations thereof.
- the processor 170 may evaluate whether reliability is high or low, depending on a level of a noise included in an AF value.
- a noise may be included in an AF value. If an AF value is small, an error may occur upon focus detection. For example, if an AF/noise value is higher than a specified threshold, the processor 170 may determine that reliability is high. If the AF/noise value is lower than the specified threshold, the processor 170 may determine that the reliability is low.
- the processor 170 may evaluate whether reliability is high or low, depending on brightness of an obtained image. In general, since there is a high probability that a focus detection error will occur due to an increase in noise in a low illumination situation (a dark area), reliability may be low. Thus, for example, if brightness is higher than a specified threshold, the processor 170 may determine that reliability is high. If the brightness is lower than the specified threshold, the processor 170 may determine that the reliability is low.
- the processor 170 may evaluate whether reliability is high or low, based on a contrast ratio of an image. In general, since there is a high probability that a focus detection error will occur if a contrast ratio between the subject 10 and a background is low, reliability may be low. For example, if a contrast ratio is higher than a specified threshold, the processor 170 may determine that reliability is high. If the contrast ratio is lower than the specified threshold, the processor 170 may determine that the reliability is low.
- the processor 170 may evaluate whether reliability is high or low, based on a blur level (or a defocus amount, a bokeh size, or the like) of an obtained image.
- a blur level or a defocus amount, a bokeh size, or the like
- the blur level may be calculated by a ratio of a high-frequency filter component of an AF value to a low-frequency filter component of the AF value.
- the processor 170 may evaluate whether reliability is high or low, depending on the number of saturated pixels included in an obtained image. For example, if a bright subject such as a point light source is captured, since there is a high probability that a focus detection error will occur, reliability may be low.
- the number of the saturated pixels included in the image may be ascertained through, for example, saturated pixel information which may be obtained by the ISP 140. If the number of saturated pixels is lower than a specified number, the processor 170 may determine that reliability is high. If the number of the saturated pixels is higher than the specified number, the processor 170 may determine that the reliability is low.
- the processor 170 may evaluate whether reliability is high or low, depending on a degree of a flick phenomenon of an AF value.
- the flicker phenomenon may be a phenomenon in which an AF value wobbles. Since there is a higher probability that a focus detection error will occur as the flicker phenomenon is more intensified, reliability may be low.
- the degree of the flicker phenomenon may be determined through a wobble of an AF value or a period when the AF value wobbles. If the degree of the flicker phenomenon is lower than a specified level, the processor 170 may determine that reliability is high. If the degree of the flicker phenomenon is higher than the specified level, the processor 170 may determine that the reliability is low.
- the processor 170 may evaluate reliability in complex consideration of two or more elements.
- the evaluation of the reliability can be the result of the sum of weighted values of the above factors.
- the evaluating of reliability of an AF value may be performed in various methods other than the above-mentioned examples.
- the processor 170 may evaluate reliability based on a repeated pattern of a high-frequency component of an image.
- the processor 170 may calculate a movement parameter of the second lens set 121, corresponding to a movement parameter of the first lens set 111, by using the movement parameter of the first lens set 111 which is performing AF as a result of evaluating the above-mentioned reliability of the AF value. According to various embodiments, if there is a specification difference between an optical system of the first camera module 110 and an optical system of the second camera module 120, the processor 170 may convert a movement parameter of the second lens set 121, corresponding to a movement parameter of the first lens set 111, in consideration of a design specification difference or calibration data.
- the term "movement parameter" may include just the movement direction to a target location where the first and second lens sets 111 and 121 should be moved, such that the subject 10 is on focus, may include the movement direction to the target location and an estimated target range, or may include the movement direction to the target location as well as an amount of movement to the target location.
- the movement direction of the first and second lens sets 111 and 121 may be any one of a front direction (a direction approached to the subject 10) or a rear direction (a direction distant from the subject 10).
- the amount of movement may be represented as a discrete step of a guide mechanism configured to move the first and second lens sets 111 and 121 to the target location.
- the target range may be a discrete step range estimated that the target location belongs.
- the processor 170 may calculate a direction facing a target location to cause the first lens to focus on the subject and an amount of movement to the target location. Further, the processor 170 may calculate a movement direction of the second lens set 121 and an amount of movement of the second lens set 121, respectively corresponding to a movement direction of the first lens set 111 and an amount of movement of the first lens set 111.
- the processor 170 may relatively accurately calculate a direction and an amount of movement, in which the first lens set 111 is moved to a target location. Thus, a movement direction and an amount of movement for allowing the second lens set 121 to be on focus may be relatively accurately calculated.
- the processor 170 may move the second lens set 121 based on the calculated movement direction and the calculated amount of movement. Thus, a location of the second lens set 121 may be adjusted such that the subject 10 is on focus in the second camera module 120.
- the processor 170 may calculate a direction facing a target location where the first lens set 111 focuses on the subject and a target range estimated that the target location belongs. Further, the processor 170 may calculate a movement direction and a target range of the second lens set 121, corresponding to a movement direction and a target range of the first lens set 111.
- the processor 170 may relatively excellently calculate a direction in which the first lens set 111 of the first camera module 110 which is performing AF is moved. However, a target location where the first lens set 111 purportedly focuses on the subject may be not be accurate with a high degree of confidence. Instead, the processor 170 may set a target range estimated that the target belongs. The processor 170 may calculate a movement direction and a target range of the second lens set 121, corresponding to a movement direction and a target range of the first lens set 111.
- the processor 170 may move the second lens set 121 in the calculated movement direction, that is, a direction close to the target range.
- the processor 170 may evaluate reliability of an AF value again using the first camera module 110 on a periodic or aperiodic basis while the second lens set 121 is moved in a direction close to the target range.
- the processor 170 may evaluate reliability of an AF value again using the first camera module 110 if the second lens set 121 enters a target range corresponding to a target range of the first lens set 111, that is, the calculated target range.
- the reliability of an AF value may increases to be higher than the first threshold upon re-evaluation.
- a location of the second lens set 121 may be adjusted to focus on the subject 10 in the second camera module 120.
- the processor 170 may calculate only a direction facing a target location where the first lens set 111 is on focus.
- the processor 170 may move the first lens set 111 in the calculated direction and may evaluate reliability of an AF value again using the first camera module 110 on a periodic or aperiodic basis while moving the first lens set 111.
- the second lens set 121 of the second camera module 120 may maintain a previous location until reliability of an AF value from the first camera module 110 increases to the second threshold or more.
- reliability of an AF value may be evaluated to be very low.
- the processor 170 since it is impossible for the processor 170 to even calculate a direction facing a target location where the first lens set 111 focuses on the subject, it may evaluate reliability of an AF value on a periodic or aperiodic basis while moving the first lens set 111 in any direction.
- the second lens set 121 of the second camera module 120 may maintain a previous location until reliability of an AF value from the first camera module 110 increases to the second threshold or more.
- the processor 170 may move the second set 121. This may be represented as a formula using Equation 1. P L 2 _ current ⁇ P L 2 _ calculated > Th
- the threshold range Th may be set in various manners based on, for example, an acceptable depth of field (DoF) of the second camera 120 including the second lens set 121.
- DoF depth of field
- the acceptable DoF may be determined based on a pixel pitch of the second image sensor 124 included in the second camera module 120, an F-number of the second camera module 120, lens sensitivity of the second lens set 121, and/or the like.
- the subject 10 which is on focus may be stopped, but only light around the subject 10 may be bright. Since the subject 10 is stopped, the second lens set 121 (used to actually capture an image) may fail to need to be moved.
- the processor 170 may recognize a change in brightness of the light around the subject 10 as a change in scene. In this case, the processor 170 may move the first lens set 111 such that the subject 10 is on focus and may calculate a movement parameter of the second lens set 121, corresponding to a movement parameter of the first lens set 111.
- Equation 1 if a difference between a location indicated by a movement parameter of the second lens set 121 and a current location of the second lens set 121 is within a specified threshold range, the second lens set 121 may fail to need to be moved.
- a live view is implemented using the second camera module 120 including the second lens set 121, although light around the subject 10 which is previously on focus is rapidly changed, that is, although a change in scene is recognized, a wobbling phenomenon may fail to be generated.
- the subject 10 which is on focus may be moved left and right with respect to the direction of an optical axis of the first camera module 110.
- another subject may temporarily pass between the subject 10 and the electronic device 101.
- the second lens set 121 may fail to need to be moved.
- the processor 170 of the electronic device 101 may recognize the above case as a change in scene.
- the processor 170 may move the first lens set 111 such that the subject 10 is on focus and may calculate a movement parameter of the second lens 121, corresponding to a movement parameter of the first lens set 111.
- Equation 1 if a difference between a location indicated by a movement parameter of the second lens set 121 and a current location of the second lens set 121 is within a specified threshold range, the second lens set 121 may fail to need to be moved.
- the subject 10 which is previously on focus is moved left and right with respect to a direction of an optical axis or although another subject is temporarily brushed, the wobbling phenomenon may fail to be generated.
- FIG. 2 is a graph of the AF in a first camera according to an embodiment of the present disclosure.
- graph 201 indicating the reliability of the AF value as a function of time
- graph 202 indicating the location of the first lens set 111 included in a first camera module 110 (a sub camera module) as a function of time are shown.
- An AF value may vary according to an AF scheme.
- the AF value may correspond to a contrast value of an image obtained by the first camera module 110.
- the AF value may correspond to a phase difference value obtained by the first camera module 110.
- reliability of the AF value may increase and on the lens set may focus on the subject 10 with a greater degree of confidence.
- Reliability of an AF value of an image obtained from the first camera module 110 at an initial time 0 may be evaluated to be very low (see graph 201).
- a processor 170 of FIG. 1 since it is impossible for a processor 170 of FIG. 1 to even calculate a direction facing a target location d p where the first lens set 111 is on focus, as shown in graph 202, it may evaluate reliability of an AF value again on a periodic or aperiodic basis while moving the first lens set 111 in any direction.
- the processor 170 may move the first lens set 111, which is in an initial location d 0 during a time between 0 to t1, in a first direction (e.g., a direction approached to the subject 10) and may move the first lens set 111 in a second direction (e.g., a direction distant from the subject 10) during a time between t1 and t2 by controlling a first actuator 112 of the first camera module 110.
- a first direction e.g., a direction approached to the subject
- a second direction e.g., a direction distant from the subject
- the processor 170 may determine a direction for moving the first lens set 111 to the target location d p as the first direction by evaluating reliability of an AF value again on a periodic or aperiodic basis during the time between 0 to t2. According to another embodiment, the processor 170 may verify an increase and decrease in the AF value and may move the first lens set 111 in a direction in which an AF value is increased. For example, since an AF value is increased during the time between 0 and t1 and is decreased during the time between t1 and t2, the processor 170 may determine a direction for moving to the target location d p where the first lens set 111 is on focus as the first direction.
- the processor 170 may move the first lens set 111 in the determined first direction during a time between t2 and t3 by controlling a first actuator 112 of the first camera module 110.
- the processor 170 may evaluate reliability of an AF value on a periodic or aperiodic basis during the time between t2 and t3.
- the processor 170 may evaluate reliability of an AF value at a time between t3 and t4 to be higher than a second threshold Th2 and be lower than a first threshold Th1.
- the processor 170 may calculate a direction (e.g., the first direction) facing the target location d p where the first lens set 111 is on focus and a target range (e.g., a range between d 1 and d 2 shown in graph 202) estimated that the target location d p belongs.
- the processor 170 may calculate a movement direction (e.g., the first direction) and a target range (e.g., a range between D 1 and D 2 , see graph 301 of FIG. 3 ) of the second lens set 121, corresponding to a movement direction (e.g., the first direction) and the target range (e.g., the range between d 1 and d 2 shown in graph 202) of the first lens set 111.
- a movement direction e.g., the first direction
- a target range
- the processor 170 may evaluate reliability of an AF value at a time between t4 and t5 to be higher than the first threshold Th1.
- the processor 170 may calculate an amount of movement for moving to the target location d p as well as a direction (e.g., the first direction) facing the target location d p where the first lens set 111 focuses on the subject.
- the processor 170 may move the first lens set 111 based on the calculated movement direction and the calculated amount of movement.
- FIG. 3 is a drawing illustrating AF in a second camera according to an embodiment.
- graph 301 indicating a change in a location of a second lens set 121 included in a second camera module 120 (a main camera module) of FIG. 1 is shown, as a function of time.
- the second lens set 121 which is in an initial location D 0 may maintain a previous location during a time between 0 and t3. As shown in graph 201 of FIG. 2 , this is because the reliability of the AF value of an image obtained from a first camera module 110 of FIG. 1 during a time between 0 and t3 is evaluated to be less than or equal to a second threshold value Th2.
- the second lens set 121 may face a target range (a range between D 1 and D 2 ) during a time between t3 and t4 to move in a first direction. As shown in graph 201 of FIG. 2 , this is because the reliability of the AF value of an image obtained from the first camera module 110 during a time between t3 and t4 is evaluated to be higher than a second threshold Th2 and be lower than a first threshold Th1. Since the processor 170 calculates a target range of the second lens set 121 as a range between D 1 and D 2 during the time between t3 and t4, the processor may fail to determine a target location D p while moving towards the first direction during the time between t3 and t4.
- a target range a range between D 1 and D 2
- the second lens set 121 may continue moving to the target location D p in the first direction during a time between t4 and t5. As shown in graph 201 of FIG. 2 , this is because the reliability of the AF value of an image obtained from the first camera module 110 during a time between t4 and t5 is evaluated to be higher than the first threshold Th1 and because the target location D p of the second lens set 121, corresponding to a target location d p of a first lens set 111, is calculated.
- the second lens set 121 of the second camera module 120 may fail to move in an interval between t1 and t2 when reliability of an AF value of the first camera module 110 falls or an interval between 0 and t3 when the reliability of the AF value of the first camera 110 is lower than the second threshold Th2.
- the second lens set 121 may fail to move although the first lens set 111 of the first camera module 110 moves in the interval between t1 and t2 or between 0 and t3.
- the second lens set 121 may fail to move until at least a movement direction of the second lens set 121 is determined.
- the reliability of the AF value rises to be higher than the second threshold Th2 (when a movement direction of the second lens set 121 is determined)
- movement may be started.
- a time when the second lens set 121 is started to move is not limited to the above-mentioned example.
- the processor 170 may start to move the second lens set 121.
- the first camera module 110 may be used for AF (i.e., as a sub-camera), and the second camera module 120 may be used for image capture or a live view (i.e., as a main camera).
- a role each of the first camera module 110 and the second camera module 120 plays may change.
- the role each of the first camera module 110 and the second camera module 120 plays may be dynamically determined, and may even swap.
- the processor 170 may respectively obtain a first image and a second image from the first camera module 110 and the second camera 120 and may determine a role each of the first camera module 110 and the second camera module 120 plays, based on a blur level (or a defocus amount, a bokeh size, or the like) or a blur level of the second image.
- the processor 170 may select a camera module which may capture an image, a DoF of which is deep, between the first and second camera modules 110 and 120, as a sub-camera for AF.
- the processor 170 may select a camera module which may capture an image, a DoF of which is shallow, between the first and second camera modules 110 and 120, as a sub-camera for AF.
- the selected camera may be specified as a sub-camera for AF.
- a camera which is not selected may be specified as a main camera used for image capture or a live view.
- a camera module which may obtain an image with a deep DoF may be a camera module with a high F-number
- a camera module which may obtain an image with a shallow DoF may be a camera module with a low F-number.
- FIG. 4 is a graph illustrating a selection of a camera module according to an embodiment.
- graph 401 indicating an AF value according to a lens location, of a camera module which may obtain an image with a deep DoF
- graph 402 indicating an AF value according to a lens location, of a camera module which may obtain an image with a shallow DoF
- the AF value may vary according to an AF scheme. For example, if AF is performed in a contrast AF scheme, the AF value may correspond to a contrast value. For another example, if AF is performed in a phase difference AF scheme, the AF value may correspond to a phase difference value. In general, as the AF value is higher, a subject may be more accurately on focus.
- a lens location of each of the first and second camera modules 110 and 120 is l 1
- an AF value of each of the first and second camera modules 110 and 120 may detected to be low, and reliability of a lens location (a so-called peak location) to focus on the subject may also be low. Since the lens location l 1 is distant from a lens location l 0 (the so-called peak location) when an AF value indicates a maximum value (when on focus), a blur level may be detected to be high, and an AF value and reliability of the AF value may be detected to be low.
- a lens location of each of the first and second camera modules 110 and 120 is l 2
- an AF value of each of the first and second camera modules 110 and 120 may detected to be relatively high, and reliability of a lens location (the so-called peak location) to focus on the subject may also be high. Since the lens location l 2 is close to the lens location l 0 (the so-called peak location) when an AF value indicates a maximum value (when on focus), a blur level may be detected to be low, and an AF value and reliability of the AF value may be detected to be high.
- a change range of an AF value according to a lens location is higher in graph 402.
- a direction of lens movement for AF may be more accurately determined if using a camera module which may obtain an image with a shallow DoF.
- a camera module which may obtain an image with a shallow DoF, is used as a sub-camera module.
- an acceptable DoF (a focus allowable error) of the camera module (corresponding to graph 402) which may obtain the image with the shallow DoF may be less than an acceptable DoF of the camera module (corresponding to graph 401) which may obtain the image with the deep DoF.
- a result value about lens movement in graph 402 may be used in lens movement in graph 401, whereas it is difficult to use a result value about lens movement in graph 401 in lens movement in graph 402. This is because blurring which departs from an error range occurs since an AF acceptable DoF (or an allowable error) is exceeded, if the result value about lens movement in graph 401 is used in lens movement in graph 402.
- FIG. 5 is a flowchart illustrating an AF method according to an embodiment.
- the AF method may include operations 501 to 507.
- Operations 501 to 507 may be performed by, for example, an electronic device 101 shown in FIG. 1 .
- Each of operations 501 to 507 may be implemented with, for example, instructions performable (or executable) by a processor 170 of the electronic device 101.
- the instructions may be stored in, for example, a computer storage medium or a memory 160 of the electronic device 101 shown in FIG. 1 .
- reference numerals of FIG. 1 may be used for describing operations 501 to 507. While operations 501 to 507 are performed, an image obtained from a second camera module 120 may be output on a display 150 in real time.
- the processor 170 of the electronic device 101 may move a first lens set 111 to focus on a subject 10, using an AF value obtained by the first camera module 110. Operation 501 may be performed in response to, for example, occurrence of a specified event for triggering an AF operation.
- the specified event may include, for example, a selection of a user for an object of the subject 10 displayed on the display 150, a change in scene, or the like.
- the processor 170 may evaluate reliability of the AF value of the image obtained from the first camera module 110.
- the AF value may vary according to an AF scheme in operation 501. For example, if AF is performed in a contrast AF scheme in operation 501, the AF value may correspond to a contrast value of an image obtained by the first camera module 110. For another example, if AF is performed in a phase difference AF scheme in operation 501, the AF value may correspond to a phase difference value obtained by the first camera module 110.
- the reliability of the AF value may be determined by various elements.
- the processor 170 may evaluate reliability based on a level of a noise included in an AF value, brightness of an image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, a degree of a flicker phenomenon of the AF value, or two or more combinations thereof.
- the processor 170 may calculate a movement parameter of a second lens set 121, corresponding to a movement parameter of the first lens set 111, by using the movement parameter of the first lens set 111 included in the first camera module 110 which is performing AF in operation 501.
- the movement parameter may vary as a result of evaluating the reliability in operation 503 (a description will be given in detail with reference to FIG. 6 ).
- the processor 170 may move the second lens set 121 based on the movement parameter of the second lens set 121, calculated in operation 505.
- FIG. 6 is a flowchart illustrating an AF method according to another embodiment.
- the AF method may include operations 601 to 613.
- Reference numerals of FIG. 1 may be used for describing operations 601 to 613.
- a partial description duplicated by a description with reference to FIG. 5 may be omitted.
- an image obtained from a second camera module 120 while operations 601 to 613 are performed may be output on a display 150 like FIG. 5 .
- a processor 170 of an electronic device 101 may move a first lens set 111 to focus on a subject 10 using an AF value obtained by the first camera module 110. Operation 601 may be performed in response to, for example, occurrence of a specified event for triggering an AF operation (e.g., a selection of an object for the subject 10, a change in scene, or the like).
- a specified event for triggering an AF operation e.g., a selection of an object for the subject 10, a change in scene, or the like.
- the processor 170 may evaluate reliability of the AV value of the image obtained from the first camera module 110.
- the processor 170 may return to operation 601, may proceed to operation 605, or may proceed to operation 609, based on the result of evaluating the reliability of the AF value.
- a movement parameter of a first lens set 111 included in the first camera module 110 which is performing an AF operation in operation 601 may include a movement direction to a target location where the first lens set 111 is on focus or any direction. If the reliability of the AF value is less than or equal to the second threshold, it may difficult for the processor 170 to calculate a movement parameter of a second lens set 121 included in a second camera module 120 using just a movement direction of the first lens set 111. Thus, the processor 170 may return to operation 601 to perform an AF operation according to operation 601 again and may evaluate reliability of an AF value again in operation 603.
- a movement parameter of the first lens set 111 may include a movement direction to a target location where the first lens set 111 is on focus and a target range including the target location.
- the processor 170 may proceed to operation 605.
- a movement parameter of the first lens set 111 may include a direction to a target location where the first lens set 111 is on focus and an amount of movement to the target location.
- the processor 170 may proceed to operation 609.
- the processor 170 may calculate a movement parameter of the second lens set 121, corresponding to the movement parameter of the first lens set 111.
- the calculated movement parameter of the second lens set 121 may include a movement direction of the second lens set 121, corresponding to the movement direction of the first lens set 111, and a target range of the second lens set 121, corresponding to the target range of the first lens set 111.
- the processor 170 may move the second lens set 121 towards the movement direction of the second lens set 121, calculated in operation 605, that is, the target range of the second lens set 121, calculated in operation 605.
- the processor 170 may return to operation 601 to perform an AF operation in the first camera module 110 according to operation 601 and may evaluate reliability of an AF value again on a periodic or aperiodic basis in operation 603. For example, if the second lens set 121 enters the target range of the second lens set 121, calculated in operation 605, the processor 170 may evaluate reliability of the AF value again. The evaluating again of the reliability of the AF value may be repeated until the reliability is greater than the first threshold.
- the processor 170 since the processor 170 evaluates that the first threshold of the reliability of the AF value is greater than the first threshold, it may calculate a movement parameter of the second lens set 121, corresponding to the movement parameter of the first lens set 111.
- the calculated movement parameter of the second lens set 121 may include a movement direction of the second lens set 121, corresponding to the movement direction of the first lens set 111 and an amount of movement to a target location of the second lens set 121, corresponding to the target location of the first lens set 111.
- the processor 170 may determine whether a difference between a current location of the second lens set 121 and the target location of the second lens set 121 is within a specified threshold range.
- the threshold range may be set based on an acceptable DoF of the second camera module 120. If the difference is within the specified threshold range, the processor 170 may end the process. If the difference is out of the specified threshold range, the processor 170 may proceed to operation 613.
- the processor 170 may move the second lens set 121 to the target location based on the movement direction of the second lens set 121 and the amount of movement of the second lens set 121, calculated in operation 609.
- FIG. 7 is a flowchart illustrating a selection and AF method of a camera to perform AF according to an embodiment.
- the selection and AF method of the camera to perform the AF may include operations 701 to 709.
- Reference numerals of FIG. 1 may be used for describing operations 701 to 709.
- a partial description duplicated by a description of FIG. 5 may be omitted.
- a processor 170 of an electronic device 101 may select any one of a first camera module 110 or a second camera module 120 based on a blur level of an image obtained from the first camera module 110 and/or the second camera module 120.
- the processor 170 may select a camera module which may capture an image with a deeper DoF, for example, a camera module with a higher F-number, between the first and second camera modules 110 and 120.
- the processor 170 may select a camera module which may capture an image with a shallower DoF, for example, a camera module with a lower F-number, between the first and second camera modules 110 and 120.
- the selected camera module may be selected as a sub-camera module used for an AF operation.
- a camera module which is not selected may be a main camera module which provides a live view on a display 160 while operations 701 to 709 are performed.
- the selected camera module may function as the first camera module 110 described with reference to FIG. 6
- the camera module which is not selected may function as the second camera module 120.
- the processor 170 may perform the same operations as operations 501 to 507 shown in FIG. 5 .
- Operations 703 to 709 are outlined hereafter.
- the processor 170 may move a lens set of the camera module (hereinafter referred to as "sub-camera module") selected in operation 701 such that the lens set focuses on the subject, using an AF value obtained by the selected camera module.
- sub-camera module a lens set of the camera module
- the processor 170 may evaluate reliability of the AF value of the image obtained from the sub-camera module.
- the processor 170 may calculate a movement parameter of a lens set included in the camera module which is not selected (hereinafter referred to as "main camera module"), corresponding to a movement parameter of a lens set included in the sub-camera module, by using the movement parameter of the lens set as a result of evaluating the reliability of the AF value.
- main camera module a movement parameter of a lens set included in the camera module which is not selected
- the processor 170 may move the lens set of the main camera module depending on the movement parameter calculated in operation 707.
- a camera module which is most suitable for AF may be selected as a sub-camera module, and the camera module may be used for an AF operation. Further, since movement of a lens set included in the main camera module is determined based on reliability of an AF value in the sub-camera module, wobbling in the main camera module may be removed and an AF speed a user feels may be enhanced.
- an electronic device may include a first camera module including a first lens set, a second camera module including a second lens set, and a processor connected to the first camera module and a second camera module.
- the processor is configured to move the first lens focus on a subject, resulting in an auto focus (AF) value and an move the first lens set to focus on a subject, using an auto focus (AF) value obtained by the first camera module, evaluate reliability of the AF value, based on the reliability of the AF value, calculate a movement parameter of the second lens set, corresponding to a movement parameter of the first lens set, and move the second lens set depending on the calculated movement parameter.
- the movement parameter of the first lens set comprises a movement direction to a target location that causes the first lens to focus on the subject and an amount of movement to the target location.
- Move the second lens set depending on the calculated movement parameter comprises move the second lens set by another movement direction and another amount of movement respectively corresponding to the movement direction of the first lens set and the amount of movement of the first lens set.
- the movement parameter of the first lens set consists of a movement direction to a target location that causes the first lens to focus on the subject and a target range including the target location, and wherein the first threshold is higher than the second threshold
- the processor may move the second lens set in a direction corresponding to the movement direction of the first lens set and may re-evaluate the reliability periodically or aperiodically. For example, when the second lens set enters a range corresponding to the target range of the first lens set, the processor is configured to re-evaluate the reliability.
- the movement parameter of the first lens set may include a direction to a target location where the first lens set is on focus.
- the processor may be configured to evaluate the reliability based on a level of a noise included in the AF value, brightness of the image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, or a degree of a flicker phenomenon of the AF value.
- the processor may move the second lens set depending on the calculated movement parameter.
- the threshold range may be set based on an acceptable depth of field (DoF) of the second camera module.
- DoF may be determined based on at least one of, for example, a pixel pitch of a pixel included in the second camera module, an F-number of the second camera module, or lens sensitivity of the second lens set.
- the electronic device may further include a display configured to output an image received from the second camera module.
- the processor may control the first camera module such that the first lens set focuses on the subject, depending on a contrast AF scheme or a phase difference AF scheme.
- the processor may calculate a location parameter of the second lens set by converting a location parameter of the first lens set.
- the processor may reflect a difference between a specification of an optical system of the first camera module and a specification of an optical system of the second camera module.
- the first camera module may have at least part of a field of view (FoV), which is in common with an FoV of the second camera module.
- FoV field of view
- An electronic device may include a first camera module, a second camera module configured to capture an image with a deeper depth of field (DoF) than the first camera module, and a processor connected to the first camera module and the second camera module.
- the processor may be configured to select any one of the first camera module or the second camera module based on a blur level of an image obtained by the first camera module or the second camera module, move a lens set of the selected camera module such that the lens set focuses on the subject, using an AF value obtained by the selected camera module, calculate a movement parameter of a lens set included in a camera module which is not selected, corresponding to a movement parameter of a lens set included in the selected camera module, as a result of evaluating reliability of the AF value of the image obtained from the selected camera module, and move a lens set included in the camera module which is not selected, depending on the calculated movement parameter.
- the processor may select the second camera module.
- the processor may select the first camera module.
- the second camera module may correspond to a camera module with a high F-number.
- the processor may control the selected camera module such that the subject is on focus, depending on a contrast AF scheme or a phase difference AF scheme.
- the processor may be configured to evaluate the reliability based on a level of a noise included in the AF value, brightness of the image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, or a degree of a flicker phenomenon of the AF value.
- the first camera module and the second camera module may be exposed to the outside through a common surface of the electronic device.
- module used in this disclosure may represent, for example, a unit including one or more combinations of hardware, software and firmware.
- the term “module” may be interchangeably used with the terms “unit”, “logic”, “logical block”, “component” and “circuit”.
- the “module” may be a minimum unit of an integrated component or may be a part thereof.
- the “module” may be a minimum unit for performing one or more functions or a part thereof.
- the “module” may be implemented mechanically or electronically.
- the “module” may include at least one of an application-specific IC (ASIC) chip, a field-programmable gate array (FPGA), and a programmable-logic device for performing some operations, which are known or will be developed.
- ASIC application-specific IC
- FPGA field-programmable gate array
- At least a part of an apparatus (e.g., modules or functions thereof) or a method (e.g., operations) may be, for example, implemented by instructions stored in computer-readable storage media in the form of a program module.
- the instruction when executed by a processor (e.g., the processor 170), may cause the one or more processors to perform a function corresponding to the instruction.
- the computer-readable storage media for example, may be the memory 160.
- a computer-readable recording medium may include a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD), a magneto-optical media (e.g., a floptical disk)), and hardware devices (e.g., a read only memory (ROM), a random access memory (RAM), or a flash memory).
- a program instruction may include not only a mechanical code such as things generated by a compiler but also a high-level language code executable on a computer using an interpreter.
- the above hardware unit may be configured to operate via one or more software modules for performing an operation of various embodiments of the present disclosure, and vice versa.
- a module or a program module may include at least one of the above elements, or a part of the above elements may be omitted, or additional other elements may be further included.
- Operations performed by a module, a program module, or other elements according to various embodiments may be executed sequentially, in parallel, repeatedly, or in a heuristic method. In addition, some operations may be executed in different sequences or may be omitted. Alternatively, other operations may be added.
Description
- The present disclosure relates to an auto focus (AF) method of a camera and an electronic device for performing the same.
- With the advancement of information technology (IT), a camera has evolved from a device that exposes a film reel to a digital memory-based camera. Recently, cameras have been included as part of a portable electronic device such as smartphones.
- A camera may condense light incident from the outside using its lens and may convert the condensed light into an electric signal. The electric signal may configure image data, and may be stored in a memory or may be output on a display.
- Here, reference is made to known prior-art publication
US 20106/0134802 A1 - Moreover, here reference is made to known prior-art document
US 2008/0219654 A1 . - Focusing in the mentioned camera may be important to obtain a high-quality image, for example, enhance sharpness of a subject.
- An automatic focus (AF) function may be to move a lens included in a camera back and forth to focus on a subject (i.e., allow a subject to be on focus). Each of various electronic devices including the camera may include a display and may output an image received from the camera on the display in real time (a preview or a live view). Thus, due to frequent movement of a lens while the AF function is performed, a field of view (FoV) of an image output on the display may be frequently wobbled (a so-called wobbling phenomenon).
- Aspects of the present disclosure are to address at least the above-mentioned problems and/or disadvantages and to provide at least the advantages described below. Accordingly, an aspect of the present disclosure is to provide an electronic device for providing a live view finder function through a main camera module and may perform AF through a sub-camera module. Accordingly, another aspect of the present disclosure is to provide an electronic device for obtaining data associated with AF from a sub-camera module and using the obtained data for an AF operation of a main camera module.
- In accordance with an aspect of the present disclosure, an electronic device is provided in accordance with the appended independent device claim. The processor can thereby suspend moving the second lens set and prevent the wobbling phenomenon in the displayed image, until the auto-focus function in relation to the first lens set achieves and surpasses a certainty of a correct auto focus direction of movement for the first lens, while not having achieved the fully completed auto focus yet.
- An electronic device according to various embodiments of the present disclosure may remove an image wobble in a main camera module which supports a live view and may enhance a speed of an AF function a user feels. In addition, a variety of effects directly or indirectly ascertained through the present disclosure may be provided.
- Other aspects, advantages, and salient features of the disclosure will become apparent to those skilled in the art from the following detailed description, which, taken in conjunction with the annexed drawings, discloses various embodiments of the present disclosure.
- The above and other aspects, features, and advantages of certain embodiments of the present disclosure will be more apparent from the following description taken in conjunction with the accompanying drawings, in which:
-
FIG. 1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure; -
FIG. 2 is a drawing illustrating auto focus (AF) in a first camera module (or a sub-camera module) according to an embodiment of the present disclosure; -
FIG. 3 is a drawing illustrating AF in a second camera module (or a main camera module) according to an embodiment of the present disclosure; -
FIG. 4 is a drawing illustrating a selection of a camera module according to an embodiment of the present disclosure; -
FIG. 5 is a flowchart illustrating an AF method according to an embodiment of the present disclosure which does not fall into the scope of the appended claims but is considered useful for understanding the present invention. -
FIG. 6 is a flowchart illustrating an AF method according to another embodiment of the present disclosure; and -
FIG. 7 is a flowchart illustrating a selection and AF method of a camera to perform AF according to an embodiment of the present disclosure. - Throughout the drawings, it should be noted that like reference numbers are used to depict the same or similar elements, features, and structures.
- Hereinafter, various embodiments of the present invention may be described with reference to accompanying drawings. Accordingly, those of ordinary skill in the art will recognize that modification and/or alternative on the various embodiments described herein can be variously made without departing from the scope of the present invention. With regard to description of drawings, similar elements may be marked by similar reference numerals.
- In this disclosure, the expressions "have", "may have", "include" and "comprise", or "may include" and "may comprise" used herein indicate existence of corresponding features (e.g., elements such as numeric values, functions, operations, or components) but do not exclude presence of additional features.
- In this disclosure, the expressions "A or B", "at least one of A or/and B", or "one or more of A or/and B", and the like may include any and all combinations of one or more of the associated listed items. For example, the term "A or B", "at least one of A and B", or "at least one of A or B" may refer to all of the case (1) where at least one A is included, the case (2) where at least one B is included, or the case (3) where both of at least one A and at least one B are included.
- The terms, such as "first", "second", and the like used in this disclosure may be used to refer to various elements regardless of the order and/or the priority and to distinguish the relevant elements from other elements, but do not limit the elements. For example, "a first user device" and "a second user device" indicate different user devices regardless of the order or priority. For example, without departing the scope of the present disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element.
- It will be understood that when an element (e.g., a first element) is referred to as being "(operatively or communicatively) coupled with/to" or "connected to" another element (e.g., a second element), it may be directly coupled with/to or connected to the other element or an intervening element (e.g., a third element) may be present. In contrast, when an element (e.g., a first element) is referred to as being "directly coupled with/to" or "directly connected to" another element (e.g., a second element), it should be understood that there are no intervening element (e.g., a third element).
- According to the situation, the expression "configured to" used in this disclosure may be used as, for example, the expression "suitable for", "having the capacity to", "designed to", "adapted to", "made to", or "capable of". The term "configured to" must not mean only "specifically designed to" in hardware. Instead, the expression "a device configured to" may mean that the device is "capable of" operating together with another device or other components. For example, a "processor configured to (or set to) perform A, B, and C" may mean a dedicated processor (e.g., an embedded processor) for performing a corresponding operation or a generic-purpose processor (e.g., a central processing unit (CPU) or an application processor) which performs corresponding operations by executing one or more software programs which are stored in a memory device.
- Terms used in this disclosure are used to describe specified embodiments and are not intended to limit the scope of another embodiment. The terms of a singular form may include plural forms unless otherwise specified. All the terms used herein, which include technical or scientific terms, may have the same meaning that is generally understood by a person skilled in the art. It will be further understood that terms, which are defined in a dictionary and commonly used, should also be interpreted as is customary in the relevant related art and not in an idealized or overly formal unless expressly so defined in various embodiments of this disclosure. In some cases, even if terms are terms which are defined in this disclosure, they may not be interpreted to exclude embodiments of this disclosure.
- An electronic device according to various embodiments of this disclosure may include at least one of, for example, smartphones, tablet personal computers (PCs), mobile phones, video telephones, electronic book readers, desktop PCs, laptop PCs, netbook computers, workstations, servers, personal digital assistants (PDAs), portable multimedia players (PMPs), Motion Picture Experts Group (MPEG-1 or MPEG-2) Audio Layer 3 (MP3) players, mobile medical devices, cameras, or wearable devices. According to various embodiments, the wearable device may include at least one of an accessory type (e.g., watches, rings, bracelets, anklets, necklaces, glasses, contact lens, or head-mounted-devices (HMDs), a fabric or garment-integrated type (e.g., an electronic apparel), a body-attached type (e.g., a skin pad or tattoos), or a bio-implantable type (e.g., an implantable circuit).
- According to various embodiments, the electronic device may be a home appliance. The home appliances may include at least one of, for example, televisions (TVs), digital versatile disc (DVD) players, audios, refrigerators, air conditioners, cleaners, ovens, microwave ovens, washing machines, air cleaners, set-top boxes, home automation control panels, security control panels, TV boxes (e.g., Samsung HomeSync™, Apple TV™, or Google TV™), game consoles (e.g., Xbox™ or PlayStation™), electronic dictionaries, electronic keys, camcorders, electronic picture frames, and the like.
- According to another embodiment, an electronic device may include at least one of various medical devices (e.g., various portable medical measurement devices (e.g., a blood glucose monitoring device, a heartbeat measuring device, a blood pressure measuring device, a body temperature measuring device, and the like), a magnetic resonance angiography (MRA), a magnetic resonance imaging (MRI), a computed tomography (CT), scanners, and ultrasonic devices), navigation devices, Global Navigation Satellite System (GNSS), event data recorders (EDRs), flight data recorders (FDRs), vehicle infotainment devices, electronic equipment for vessels (e.g., navigation systems and gyrocompasses), avionics, security devices, head units for vehicles, industrial or home robots, automatic teller's machines (ATMs), points of sales (POSs) of stores, or internet of things (e.g., light bulbs, various sensors, electric or gas meters, sprinkler devices, fire alarms, thermostats, street lamps, toasters, exercise equipment, hot water tanks, heaters, boilers, and the like).
- According to an embodiment, the electronic device may include at least one of parts of furniture or buildings/structures, electronic boards, electronic signature receiving devices, projectors, or various measuring instruments (e.g., water meters, electricity meters, gas meters, or wave meters, and the like). According to various embodiments, the electronic device may be one of the above-described devices or a combination thereof. An electronic device according to an embodiment may be a flexible electronic device. Furthermore, an electronic device according to an embodiment of this disclosure may not be limited to the above-described electronic devices and may include other electronic devices and new electronic devices according to the development of technologies.
- Hereinafter, electronic devices according to various embodiments will be described with reference to the accompanying drawings. In this disclosure, the term "user" may refer to a person who uses an electronic device or may refer to a device (e.g., an artificial intelligence electronic device) that uses the electronic device.
-
FIG. 1 is a block diagram illustrating a configuration of an electronic device according to an embodiment of the present disclosure. - Referring to
FIG. 1 , anelectronic device 101 according to an embodiment may include afirst camera module 110, asecond camera module 120, an image sensor interface (I/F) 130, an image signal processor (ISP) 140, adisplay 150, amemory 160, and aprocessor 170. According to various embodiments, theelectronic device 101 may be implemented not to include some of the elements or may be implemented to include an element which is not shown. - The
first camera module 110 may include a first lens set 111, afirst actuator 112,aperture walls 113 defining an aperture, and afirst image sensor 114. Thefirst camera module 110 may omit some of the elements or may be implemented to include additional elements (e.g., an optical image stabilization (OIS) module, a flash, or the like) not shown. According to some embodiments, thefirst camera module 110 may be referred to as a first optical system. - The first lens set 111 may condense light reflected from the outside to the
first camera 110. The condensed light may arrive at afirst image sensor 114 via the aperture formed fromaperture walls 113. In other words, the first lens set 111 may allow light reflected or generated from a subject to reach a pixel array of thefirst image sensor 114. - The
first actuator 112 may drive the first lens set 111 depending on control of theprocessor 170. Thefirst actuator 112 may allow a subject 10 to be on focus by moving thefirst lens set 111. Thefirst actuator 112 may include a servo motor or an ultrasonic motor. According to various embodiments, thefirst actuator 112 may be referred to as a lens actuator, an AF actuator, or the like. - The
aperture walls 113 may adjust an amount of light which arrives at (or incident to) thefirst image sensor 114 by narrowing or widening the aperture formed therebetween. In general, as theaperture walls 113 are more widely opened, an amount of light which arrives at thefirst image sensor 114 increases (an effective aperture may be more increased). As theaperture walls 113 are more narrowly closed, an amount of light which arrives at thefirst image sensor 114 decreases (an effective aperture may be more decreased). According to an embodiment, theaperture walls 113 may defining an aperture having fixed width. In other words, the effective aperture may have a specified value by the fixed aperture. - The
first image sensor 114 may include a pixel array in which a plurality of pixels are arranged as a two-dimensional (2D) grating shape. In certain embodiments, millions of pixels, even tens of millions of pixels, may be included in the pixel array. One of a plurality of reference colors may be assigned to each one of the plurality of pixels. For example, the plurality of reference colors may include "red, green, and blue (RGB)". Thefirst image sensor 114 may be implemented using, for example, a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS). - According to an embodiment, the
first image sensor 114 may generate an electric signal based on light received from the outside and may generate a digital image (e.g., a Bayer image) based on the electric signal. The image generated by thefirst image sensor 114 may be transmitted to the image sensor I/F 130. - A
first shutter 119 may adjust, for example, a time when thefirst image sensor 114 is exposed to light, under control of theprocessor 170. For example, if thefirst shutter 119 is operated slowly, more light may be incident to thefirst image sensor 114. If thefirst shutter 119 is operated faster, less light may be incident to thefirst image sensor 114. A time when theshutter 119 is operated may be determined according to a shutter speed set automatically or manually. Thefirst shutter 119 may include a mechanical shutter or an electronic shutter of a manner of controlling an image sensor. - The
second camera module 120 may include the same or similar configuration to thefirst camera module 110. Thesecond camera module 120 may include a second lens set 121, a second actuator 122 for driving the second lens set 121, asecond aperture 123, asecond image sensor 124, and asecond shutter 129. A field of view (FoV) of thefirst camera 110 and an FoV of thesecond camera module 120 may be substantially the same as each other or may be within a constant range. Thefirst camera 110 and thesecond camera module 120 may be exposed to the outside through a common surface of the electronic device 101 (e.g., a surface opposite to a surface where thedisplay 150 is located). - In certain embodiment, the
second camera module 120 operates similar to thefirst camera module 110. Herein, according to an embodiment, a specification of each element included in thesecond camera module 120 may different from a specification of each element included in thefirst camera module 110. For example, focal distances of the first lens set 111 and the second lens set 121, the number of pixels of thefirst image sensor 114 and the number of pixels of thesecond image sensor 124, each pixel pitch of thefirst image sensor 114 and each pixel pitch of thesecond image sensor 124, and the like may have different values. - The image sensor I/
F 130 may serve as a medium for transmitting and receiving data between image sensor and another element (e.g., the ISP 140). For example, the image sensor I/F 130 may transmit image data generated by thefirst image sensor 114 or thesecond image sensor 124 to theISP 140. - The
ISP 140 may include a plurality of image processing (IP) functions. TheISP 140 may correct an image obtained from thefirst camera module 110 or thesecond camera module 120, through the plurality of IP functions. The plurality of IP functions may include various IP functions, such as an IP block for color interpolation, an IP block for lens shading correction, an IP block for auto white balance, an IP block for lateral chromatic aberration correction, an IP block for optical inverse correction, an IP block for noise reduction, an IP block for edge enhancement, an IP block for gamma correction, or an IP block for out-of-focus (or shallow depth of field (DoF)). The plurality of IP blocks may be referred to as an image processing filter, an image processing module, or the like. Further, according to various embodiments, theISP 140 may be included in theprocessor 170. - The
display 150 may visually output a variety of content (e.g., text, an image, a video, an icon, a widget, a symbol, or the like) to the user. For example, thedisplay 150 may be configured to output an image received from thefirst camera module 110. If an image from thefirst camera module 110 is output on thedisplay 150 in real time, an output of the image may be referred to as a so-called live view. According to an embodiment, thedisplay 150 may include a touch pad. Thus, thedisplay 150 may detect a touch, a gesture, proximity, or a hovering input using an electronic pen or part of a user's body. - The
memory 160 may include a volatile and/or non-volatile memory. Thememory 160 may store various instructions or data associated with theelements 110 to 150 and 170 included in theelectronic device 101. For example, thememory 160 may store instructions to perform various operations described in the present disclosure. The instructions may be implemented as software, for example, an application program, an operating system (OS), or firmware, may be stored in thememory 160, and may be embedded in hardware. - The
processor 170 may be electrically connected with theelements 110 to 160 included in theelectronic device 101 and may perform an arithmetic operation about control and/or communication of theelements 110 to 160 included in theelectronic device 101. - According to an embodiment, the
processor 170 may control thefirst camera module 110 to perform AF on the subject 10 in response to a specified event. In this case, an image obtained through the second camera module 120 (or thesecond image sensor 124 of the second camera module 120) may be output on thedisplay 150 in real time. - The specified event may be an event for triggering AF and may include, for example, a selection (e.g., a touch) of a user for an object of the subject 10 displayed on the
display 150, a change in scene, or the like. The change in scene may be determined from, for example, a change in brightness of a specified level or more of an image received from thefirst camera module 110 or thesecond camera module 120, a movement/change of the recognized subject 10, a change in sensitivity of a specified level or more, or the like. - According to an embodiment, the
processor 170 may move the first lens set 111 to focus on a subject, using AF value obtained by thefirst camera module 110. Theprocessor 170 may move the first lens set 111 by controlling thefirst actuator 112. For example, in performing AF at thefirst camera module 110, theprocessor 170 may use a contrast AF scheme or a phase difference AF scheme. Thus, the AF value may correspond to a contrast value or a phase difference value obtained from thefirst camera module 111. - According to an embodiment, the
processor 170 may evaluate reliability of the AF value of an image obtained by thefirst camera module 110. Theprocessor 170 may determine whether to move the second lens set 121 included in thesecond camera module 120, based on the reliability. - The reliability of the AF value may be determined by various elements. According to various embodiments, the
processor 170 may evaluate reliability based on a level of a noise included in an AF value, brightness of an image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, a degree of a flicker phenomenon of the AF value, or two or more combinations thereof. - For example, the
processor 170 may evaluate whether reliability is high or low, depending on a level of a noise included in an AF value. In general, a noise may be included in an AF value. If an AF value is small, an error may occur upon focus detection. For example, if an AF/noise value is higher than a specified threshold, theprocessor 170 may determine that reliability is high. If the AF/noise value is lower than the specified threshold, theprocessor 170 may determine that the reliability is low. - For example, the
processor 170 may evaluate whether reliability is high or low, depending on brightness of an obtained image. In general, since there is a high probability that a focus detection error will occur due to an increase in noise in a low illumination situation (a dark area), reliability may be low. Thus, for example, if brightness is higher than a specified threshold, theprocessor 170 may determine that reliability is high. If the brightness is lower than the specified threshold, theprocessor 170 may determine that the reliability is low. - For example, the
processor 170 may evaluate whether reliability is high or low, based on a contrast ratio of an image. In general, since there is a high probability that a focus detection error will occur if a contrast ratio between the subject 10 and a background is low, reliability may be low. For example, if a contrast ratio is higher than a specified threshold, theprocessor 170 may determine that reliability is high. If the contrast ratio is lower than the specified threshold, theprocessor 170 may determine that the reliability is low. - For example, the
processor 170 may evaluate whether reliability is high or low, based on a blur level (or a defocus amount, a bokeh size, or the like) of an obtained image. In general, if the blur level is high, it may difficult to detect focus. Thus, if the blur level is lower than a specific threshold, theprocessor 170 may determine that reliability is high. If the blur level is higher than the specific threshold, theprocessor 170 may determine that the reliability is low. The blur level may be calculated by a ratio of a high-frequency filter component of an AF value to a low-frequency filter component of the AF value. - For example, the
processor 170 may evaluate whether reliability is high or low, depending on the number of saturated pixels included in an obtained image. For example, if a bright subject such as a point light source is captured, since there is a high probability that a focus detection error will occur, reliability may be low. The number of the saturated pixels included in the image may be ascertained through, for example, saturated pixel information which may be obtained by theISP 140. If the number of saturated pixels is lower than a specified number, theprocessor 170 may determine that reliability is high. If the number of the saturated pixels is higher than the specified number, theprocessor 170 may determine that the reliability is low. - For example, the
processor 170 may evaluate whether reliability is high or low, depending on a degree of a flick phenomenon of an AF value. The flicker phenomenon may be a phenomenon in which an AF value wobbles. Since there is a higher probability that a focus detection error will occur as the flicker phenomenon is more intensified, reliability may be low. The degree of the flicker phenomenon may be determined through a wobble of an AF value or a period when the AF value wobbles. If the degree of the flicker phenomenon is lower than a specified level, theprocessor 170 may determine that reliability is high. If the degree of the flicker phenomenon is higher than the specified level, theprocessor 170 may determine that the reliability is low. - Each of the examples of evaluating the reliability is described independently. However, embodiments are not limited thereto. According to various embodiments, the
processor 170 may evaluate reliability in complex consideration of two or more elements. For example, in certain embodiments, the evaluation of the reliability can be the result of the sum of weighted values of the above factors. Further, the evaluating of reliability of an AF value may be performed in various methods other than the above-mentioned examples. For example, theprocessor 170 may evaluate reliability based on a repeated pattern of a high-frequency component of an image. - According to an embodiment, the
processor 170 may calculate a movement parameter of the second lens set 121, corresponding to a movement parameter of the first lens set 111, by using the movement parameter of the first lens set 111 which is performing AF as a result of evaluating the above-mentioned reliability of the AF value. According to various embodiments, if there is a specification difference between an optical system of thefirst camera module 110 and an optical system of thesecond camera module 120, theprocessor 170 may convert a movement parameter of the second lens set 121, corresponding to a movement parameter of the first lens set 111, in consideration of a design specification difference or calibration data. - Meanwhile, the term "movement parameter" may include just the movement direction to a target location where the first and second lens sets 111 and 121 should be moved, such that the subject 10 is on focus, may include the movement direction to the target location and an estimated target range, or may include the movement direction to the target location as well as an amount of movement to the target location. For example, the movement direction of the first and second lens sets 111 and 121 may be any one of a front direction (a direction approached to the subject 10) or a rear direction (a direction distant from the subject 10). The amount of movement may be represented as a discrete step of a guide mechanism configured to move the first and second lens sets 111 and 121 to the target location. The target range may be a discrete step range estimated that the target location belongs.
- According to an embodiment, if reliability of an AF value is evaluated to be higher than a first threshold, the
processor 170 may calculate a direction facing a target location to cause the first lens to focus on the subject and an amount of movement to the target location. Further, theprocessor 170 may calculate a movement direction of the second lens set 121 and an amount of movement of the second lens set 121, respectively corresponding to a movement direction of the first lens set 111 and an amount of movement of thefirst lens set 111. - Since reliability of an AF value is evaluated to be relatively high, the
processor 170 may relatively accurately calculate a direction and an amount of movement, in which the first lens set 111 is moved to a target location. Thus, a movement direction and an amount of movement for allowing the second lens set 121 to be on focus may be relatively accurately calculated. Theprocessor 170 may move the second lens set 121 based on the calculated movement direction and the calculated amount of movement. Thus, a location of the second lens set 121 may be adjusted such that the subject 10 is on focus in thesecond camera module 120. - According to an embodiment, if reliability of an AF value is evaluated to be higher than a second threshold and be lower than a first threshold (the first threshold > the second threshold), the
processor 170 may calculate a direction facing a target location where the first lens set 111 focuses on the subject and a target range estimated that the target location belongs. Further, theprocessor 170 may calculate a movement direction and a target range of the second lens set 121, corresponding to a movement direction and a target range of thefirst lens set 111. - In other words, if reliability of an AF value is evaluated as a middle degree, the
processor 170 may relatively excellently calculate a direction in which the first lens set 111 of thefirst camera module 110 which is performing AF is moved. However, a target location where the first lens set 111 purportedly focuses on the subject may be not be accurate with a high degree of confidence. Instead, theprocessor 170 may set a target range estimated that the target belongs. Theprocessor 170 may calculate a movement direction and a target range of the second lens set 121, corresponding to a movement direction and a target range of thefirst lens set 111. - Thus, for example, first of all, the
processor 170 may move the second lens set 121 in the calculated movement direction, that is, a direction close to the target range. Theprocessor 170 may evaluate reliability of an AF value again using thefirst camera module 110 on a periodic or aperiodic basis while the second lens set 121 is moved in a direction close to the target range. According to an embodiment, if the second lens set 121 enters a target range corresponding to a target range of the first lens set 111, that is, the calculated target range, theprocessor 170 may evaluate reliability of an AF value again using thefirst camera module 110. - In general, since reliability of an AF value increases as the first lens set 111 is closer to a target range, the reliability of an AF value may increases to be higher than the first threshold upon re-evaluation. As such, a location of the second lens set 121 may be adjusted to focus on the subject 10 in the
second camera module 120. - According to an embodiment, if reliability of an AF value is evaluated to be lower than a second threshold, the
processor 170 may calculate only a direction facing a target location where the first lens set 111 is on focus. - Since reliability of an AF value is evaluated to be relatively low, the
processor 170 may move the first lens set 111 in the calculated direction and may evaluate reliability of an AF value again using thefirst camera module 110 on a periodic or aperiodic basis while moving thefirst lens set 111. Thus, according to various embodiments, the second lens set 121 of thesecond camera module 120 may maintain a previous location until reliability of an AF value from thefirst camera module 110 increases to the second threshold or more. - Further, according to various embodiments, reliability of an AF value may be evaluated to be very low. In this case, since it is impossible for the
processor 170 to even calculate a direction facing a target location where the first lens set 111 focuses on the subject, it may evaluate reliability of an AF value on a periodic or aperiodic basis while moving the first lens set 111 in any direction. Thus, in this case, the second lens set 121 of thesecond camera module 120 may maintain a previous location until reliability of an AF value from thefirst camera module 110 increases to the second threshold or more. - Meanwhile, according to an embodiment, only if a difference between a current location PL2_current of the second lens set 121 and a location PL2_calculated indicated by a movement parameter of the second lens set 121, calculated based on a movement parameter of the first lens set 111, departs from a specified threshold range Th, the
processor 170 may move thesecond set 121. This may be represented as a formula using Equation 1. - The threshold range Th may be set in various manners based on, for example, an acceptable depth of field (DoF) of the
second camera 120 including thesecond lens set 121. The acceptable DoF may be determined based on a pixel pitch of thesecond image sensor 124 included in thesecond camera module 120, an F-number of thesecond camera module 120, lens sensitivity of the second lens set 121, and/or the like. - For example, in an image obtained by the
first camera module 110 and thesecond camera module 120, the subject 10 which is on focus may be stopped, but only light around the subject 10 may be bright. Since the subject 10 is stopped, the second lens set 121 (used to actually capture an image) may fail to need to be moved. - However, as the light around the subject 10 is bright, the
processor 170 may recognize a change in brightness of the light around the subject 10 as a change in scene. In this case, theprocessor 170 may move the first lens set 111 such that the subject 10 is on focus and may calculate a movement parameter of the second lens set 121, corresponding to a movement parameter of thefirst lens set 111. - According to an embodiment of the present disclosure, as shown in Equation 1, if a difference between a location indicated by a movement parameter of the second lens set 121 and a current location of the second lens set 121 is within a specified threshold range, the second lens set 121 may fail to need to be moved. Thus, if a live view is implemented using the
second camera module 120 including the second lens set 121, although light around the subject 10 which is previously on focus is rapidly changed, that is, although a change in scene is recognized, a wobbling phenomenon may fail to be generated. - For another example, the subject 10 which is on focus may be moved left and right with respect to the direction of an optical axis of the
first camera module 110. Alternatively, another subject may temporarily pass between the subject 10 and theelectronic device 101. In this case, since there is no change really in a distance between the subject 10 and theelectronic device 101, the second lens set 121 may fail to need to be moved. - However, the
processor 170 of theelectronic device 101 may recognize the above case as a change in scene. In this case, theprocessor 170 may move the first lens set 111 such that the subject 10 is on focus and may calculate a movement parameter of thesecond lens 121, corresponding to a movement parameter of thefirst lens set 111. - According to an embodiment of the present disclosure, as shown in Equation 1, if a difference between a location indicated by a movement parameter of the second lens set 121 and a current location of the second lens set 121 is within a specified threshold range, the second lens set 121 may fail to need to be moved. Thus, although the subject 10 which is previously on focus is moved left and right with respect to a direction of an optical axis or although another subject is temporarily brushed, the wobbling phenomenon may fail to be generated.
-
FIG. 2 is a graph of the AF in a first camera according to an embodiment of the present disclosure. - Referring to
FIG. 2 ,graph 201 indicating the reliability of the AF value as a function of time andgraph 202 indicating the location of the first lens set 111 included in a first camera module 110 (a sub camera module) as a function of time are shown. - An AF value may vary according to an AF scheme. For example, if AF is performed in a contrast AF scheme, the AF value may correspond to a contrast value of an image obtained by the
first camera module 110. For another example, if AF is performed in a phase difference AF scheme, the AF value may correspond to a phase difference value obtained by thefirst camera module 110. In general, as an AF value is higher, reliability of the AF value may increase and on the lens set may focus on the subject 10 with a greater degree of confidence. - Reliability of an AF value of an image obtained from the
first camera module 110 at aninitial time 0 may be evaluated to be very low (see graph 201). Thus, since it is impossible for aprocessor 170 ofFIG. 1 to even calculate a direction facing a target location dp where the first lens set 111 is on focus, as shown ingraph 202, it may evaluate reliability of an AF value again on a periodic or aperiodic basis while moving the first lens set 111 in any direction. - For example, the
processor 170 may move the first lens set 111, which is in an initial location d0 during a time between 0 to t1, in a first direction (e.g., a direction approached to the subject 10) and may move the first lens set 111 in a second direction (e.g., a direction distant from the subject 10) during a time between t1 and t2 by controlling afirst actuator 112 of thefirst camera module 110. Referring to graph 201, according to movement of the first lens set 111, an AF value may be increased during a time between 0 and t1 and may be decreased during a time between t1 and t2. - According to various embodiments, the
processor 170 may determine a direction for moving the first lens set 111 to the target location dp as the first direction by evaluating reliability of an AF value again on a periodic or aperiodic basis during the time between 0 to t2. According to another embodiment, theprocessor 170 may verify an increase and decrease in the AF value and may move the first lens set 111 in a direction in which an AF value is increased. For example, since an AF value is increased during the time between 0 and t1 and is decreased during the time between t1 and t2, theprocessor 170 may determine a direction for moving to the target location dp where the first lens set 111 is on focus as the first direction. - As shown in
graph 202, theprocessor 170 may move the first lens set 111 in the determined first direction during a time between t2 and t3 by controlling afirst actuator 112 of thefirst camera module 110. Theprocessor 170 may evaluate reliability of an AF value on a periodic or aperiodic basis during the time between t2 and t3. - According to an embodiment, as shown in
graph 201, theprocessor 170 may evaluate reliability of an AF value at a time between t3 and t4 to be higher than a second threshold Th2 and be lower than a first threshold Th1. Thus, theprocessor 170 may calculate a direction (e.g., the first direction) facing the target location dp where the first lens set 111 is on focus and a target range (e.g., a range between d1 and d2 shown in graph 202) estimated that the target location dp belongs. Further, theprocessor 170 may calculate a movement direction (e.g., the first direction) and a target range (e.g., a range between D1 and D2, seegraph 301 ofFIG. 3 ) of the second lens set 121, corresponding to a movement direction (e.g., the first direction) and the target range (e.g., the range between d1 and d2 shown in graph 202) of thefirst lens set 111. - According to an embodiment, the
processor 170 may evaluate reliability of an AF value at a time between t4 and t5 to be higher than the first threshold Th1. Thus, theprocessor 170 may calculate an amount of movement for moving to the target location dp as well as a direction (e.g., the first direction) facing the target location dp where the first lens set 111 focuses on the subject. Theprocessor 170 may move the first lens set 111 based on the calculated movement direction and the calculated amount of movement. -
FIG. 3 is a drawing illustrating AF in a second camera according to an embodiment. - Referring to
FIG. 3 ,graph 301 indicating a change in a location of a second lens set 121 included in a second camera module 120 (a main camera module) ofFIG. 1 is shown, as a function of time. - Referring to graph 301, the second lens set 121 which is in an initial location D0 may maintain a previous location during a time between 0 and t3. As shown in
graph 201 ofFIG. 2 , this is because the reliability of the AF value of an image obtained from afirst camera module 110 ofFIG. 1 during a time between 0 and t3 is evaluated to be less than or equal to a second threshold value Th2. - In
graph 301, the second lens set 121 may face a target range (a range between D1 and D2) during a time between t3 and t4 to move in a first direction. As shown ingraph 201 ofFIG. 2 , this is because the reliability of the AF value of an image obtained from thefirst camera module 110 during a time between t3 and t4 is evaluated to be higher than a second threshold Th2 and be lower than a first threshold Th1. Since theprocessor 170 calculates a target range of the second lens set 121 as a range between D1 and D2 during the time between t3 and t4, the processor may fail to determine a target location Dp while moving towards the first direction during the time between t3 and t4. - In
graph 301, the second lens set 121 may continue moving to the target location Dp in the first direction during a time between t4 and t5. As shown ingraph 201 ofFIG. 2 , this is because the reliability of the AF value of an image obtained from thefirst camera module 110 during a time between t4 and t5 is evaluated to be higher than the first threshold Th1 and because the target location Dp of the second lens set 121, corresponding to a target location dp of a first lens set 111, is calculated. - As seen in
graph 301, for example, if an image obtained from thesecond camera module 120 is displayed as a live view on adisplay 150 ofFIG. 1 , an image wobble by movement of the second lens set 121 may fail to be generated. This is because a movement direction of the second lens set 121 is kept continuous in the first direction. - As shown in
graphs FIG. 2 andgraph 301 ofFIG. 3 , the second lens set 121 of thesecond camera module 120 may fail to move in an interval between t1 and t2 when reliability of an AF value of thefirst camera module 110 falls or an interval between 0 and t3 when the reliability of the AF value of thefirst camera 110 is lower than the second threshold Th2. In other words, if the reliability of the AF value of thefirst camera module 110 falls in the interval between t1 and t2 or if the reliability of the AF value is lower than the second threshold Th2 in the interval between 0 and t3 although the reliability of the AF value rises, the second lens set 121 may fail to move although the first lens set 111 of thefirst camera module 110 moves in the interval between t1 and t2 or between 0 and t3. - As such, as a result of evaluating the reliability of the AF value of the
first camera module 110, the second lens set 121 may fail to move until at least a movement direction of the second lens set 121 is determined. When the reliability of the AF value rises to be higher than the second threshold Th2 (when a movement direction of the second lens set 121 is determined), movement may be started. - According to various embodiments, a time when the second lens set 121 is started to move is not limited to the above-mentioned example. For example, when reliability of an AF value of the
first camera module 110 rises to be higher than the first threshold Th1 (time t4 inFIGS. 2 and3 ), that is, when both of a movement direction of the second lens set 121 and an amount of movement of the second lens set 121 are determined (when a target location is determined), theprocessor 170 may start to move thesecond lens set 121. - According to the above description, the
first camera module 110 may be used for AF (i.e., as a sub-camera), and thesecond camera module 120 may be used for image capture or a live view (i.e., as a main camera). However, according to an embodiment of the present disclosure, a role each of thefirst camera module 110 and thesecond camera module 120 plays may change. In certain embodiments, the role each of thefirst camera module 110 and thesecond camera module 120 plays may be dynamically determined, and may even swap. - According to an embodiment, the
processor 170 may respectively obtain a first image and a second image from thefirst camera module 110 and thesecond camera 120 and may determine a role each of thefirst camera module 110 and thesecond camera module 120 plays, based on a blur level (or a defocus amount, a bokeh size, or the like) or a blur level of the second image. - According to an embodiment, if the blur level is higher than a specified level, the
processor 170 may select a camera module which may capture an image, a DoF of which is deep, between the first andsecond camera modules processor 170 may select a camera module which may capture an image, a DoF of which is shallow, between the first andsecond camera modules - As such, if the main camera and the sub-camera are specified, the
electronic device 101 may perform AF according to various embodiments of the present disclosure described with reference toFIGS. 2 and3 . For example, a camera module which may obtain an image with a deep DoF may be a camera module with a high F-number, and a camera module which may obtain an image with a shallow DoF may be a camera module with a low F-number. -
FIG. 4 is a graph illustrating a selection of a camera module according to an embodiment. - Referring to
FIG. 4 ,graph 401, indicating an AF value according to a lens location, of a camera module which may obtain an image with a deep DoF andgraph 402, indicating an AF value according to a lens location, of a camera module which may obtain an image with a shallow DoF are shown. The AF value may vary according to an AF scheme. For example, if AF is performed in a contrast AF scheme, the AF value may correspond to a contrast value. For another example, if AF is performed in a phase difference AF scheme, the AF value may correspond to a phase difference value. In general, as the AF value is higher, a subject may be more accurately on focus. - Referring to
graphs second camera modules second camera modules - Comparing
graph 401 withgraph 402 near the lens location l1, a change range of an AF value according to a lens location is higher ingraph 401. Thus, as verified ingraph 401, when a blur level is high, a direction of lens movement for AF may be more accurately determined if using a camera module which may obtain an image with a deep DoF. In this respect, if a blur level is high, it may be preferable that a camera module, which may obtain an image with a deep DoF, is used as a sub-camera module. - Meanwhile, if a lens location of each of the first and
second camera modules second camera modules - Comparing
graph 401 withgraph 402 near the lens location l2, a change range of an AF value according to a lens location is higher ingraph 402. Thus, as verified ingraph 402, when a blur level is low, a direction of lens movement for AF may be more accurately determined if using a camera module which may obtain an image with a shallow DoF. In this respect, if a blur level is low, it may be preferable that a camera module, which may obtain an image with a shallow DoF, is used as a sub-camera module. - Further, an acceptable DoF (a focus allowable error) of the camera module (corresponding to graph 402) which may obtain the image with the shallow DoF may be less than an acceptable DoF of the camera module (corresponding to graph 401) which may obtain the image with the deep DoF. Thus, a result value about lens movement in
graph 402 may be used in lens movement ingraph 401, whereas it is difficult to use a result value about lens movement ingraph 401 in lens movement ingraph 402. This is because blurring which departs from an error range occurs since an AF acceptable DoF (or an allowable error) is exceeded, if the result value about lens movement ingraph 401 is used in lens movement ingraph 402. -
FIG. 5 is a flowchart illustrating an AF method according to an embodiment. - Referring to
FIG. 5 , the AF method according to an embodiment may includeoperations 501 to 507.Operations 501 to 507 may be performed by, for example, anelectronic device 101 shown inFIG. 1 . Each ofoperations 501 to 507 may be implemented with, for example, instructions performable (or executable) by aprocessor 170 of theelectronic device 101. The instructions may be stored in, for example, a computer storage medium or amemory 160 of theelectronic device 101 shown inFIG. 1 . Hereinafter, reference numerals ofFIG. 1 may be used for describingoperations 501 to 507. Whileoperations 501 to 507 are performed, an image obtained from asecond camera module 120 may be output on adisplay 150 in real time. - In
operation 501, theprocessor 170 of theelectronic device 101 may move a first lens set 111 to focus on a subject 10, using an AF value obtained by thefirst camera module 110.Operation 501 may be performed in response to, for example, occurrence of a specified event for triggering an AF operation. The specified event may include, for example, a selection of a user for an object of the subject 10 displayed on thedisplay 150, a change in scene, or the like. - In
operation 503, theprocessor 170 may evaluate reliability of the AF value of the image obtained from thefirst camera module 110. The AF value may vary according to an AF scheme inoperation 501. For example, if AF is performed in a contrast AF scheme inoperation 501, the AF value may correspond to a contrast value of an image obtained by thefirst camera module 110. For another example, if AF is performed in a phase difference AF scheme inoperation 501, the AF value may correspond to a phase difference value obtained by thefirst camera module 110. - According to an embodiment, the reliability of the AF value may be determined by various elements. According to various embodiments, the
processor 170 may evaluate reliability based on a level of a noise included in an AF value, brightness of an image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, a degree of a flicker phenomenon of the AF value, or two or more combinations thereof. - In
operation 505, based on the reliability of the AF, theprocessor 170 may calculate a movement parameter of a second lens set 121, corresponding to a movement parameter of the first lens set 111, by using the movement parameter of the first lens set 111 included in thefirst camera module 110 which is performing AF inoperation 501. The movement parameter may vary as a result of evaluating the reliability in operation 503 (a description will be given in detail with reference toFIG. 6 ). - In
operation 507, theprocessor 170 may move the second lens set 121 based on the movement parameter of the second lens set 121, calculated inoperation 505. -
FIG. 6 is a flowchart illustrating an AF method according to another embodiment. - Referring to
FIG. 6 , the AF method according to an embodiment may includeoperations 601 to 613. Reference numerals ofFIG. 1 may be used for describingoperations 601 to 613. A partial description duplicated by a description with reference toFIG. 5 may be omitted. Further, an image obtained from asecond camera module 120 whileoperations 601 to 613 are performed may be output on adisplay 150 likeFIG. 5 . - In
operation 601, aprocessor 170 of anelectronic device 101 may move a first lens set 111 to focus on a subject 10 using an AF value obtained by thefirst camera module 110.Operation 601 may be performed in response to, for example, occurrence of a specified event for triggering an AF operation (e.g., a selection of an object for the subject 10, a change in scene, or the like). - In
operation 603, theprocessor 170 may evaluate reliability of the AV value of the image obtained from thefirst camera module 110. Theprocessor 170 may return tooperation 601, may proceed tooperation 605, or may proceed tooperation 609, based on the result of evaluating the reliability of the AF value. - According to an embodiment, if the reliability of the AF value is less than or equal to a second threshold in
operation 603 inoperation 603, a movement parameter of a first lens set 111 included in thefirst camera module 110 which is performing an AF operation inoperation 601 may include a movement direction to a target location where the first lens set 111 is on focus or any direction. If the reliability of the AF value is less than or equal to the second threshold, it may difficult for theprocessor 170 to calculate a movement parameter of a second lens set 121 included in asecond camera module 120 using just a movement direction of thefirst lens set 111. Thus, theprocessor 170 may return tooperation 601 to perform an AF operation according tooperation 601 again and may evaluate reliability of an AF value again inoperation 603. - According to an embodiment, if the reliability of the AF value is greater than the second threshold and is less than or equal to the first threshold in
operation 603, a movement parameter of the first lens set 111 may include a movement direction to a target location where the first lens set 111 is on focus and a target range including the target location. Theprocessor 170 may proceed tooperation 605. - According to an embodiment, if the reliability of the AF value is greater than the first threshold in
operation 603, a movement parameter of the first lens set 111 may include a direction to a target location where the first lens set 111 is on focus and an amount of movement to the target location. Theprocessor 170 may proceed tooperation 609. - In
operation 605, since theprocessor 170 evaluates that the reliability of the AF value is greater than the second threshold and is less than or equal to the first threshold, it may calculate a movement parameter of the second lens set 121, corresponding to the movement parameter of thefirst lens set 111. The calculated movement parameter of the second lens set 121 may include a movement direction of the second lens set 121, corresponding to the movement direction of the first lens set 111, and a target range of the second lens set 121, corresponding to the target range of thefirst lens set 111. - In
operation 607, theprocessor 170 may move the second lens set 121 towards the movement direction of the second lens set 121, calculated inoperation 605, that is, the target range of the second lens set 121, calculated inoperation 605. - According to an embodiment, if the movement of the second lens set 121 is started, the
processor 170 may return tooperation 601 to perform an AF operation in thefirst camera module 110 according tooperation 601 and may evaluate reliability of an AF value again on a periodic or aperiodic basis inoperation 603. For example, if the second lens set 121 enters the target range of the second lens set 121, calculated inoperation 605, theprocessor 170 may evaluate reliability of the AF value again. The evaluating again of the reliability of the AF value may be repeated until the reliability is greater than the first threshold. - In
operation 609, since theprocessor 170 evaluates that the first threshold of the reliability of the AF value is greater than the first threshold, it may calculate a movement parameter of the second lens set 121, corresponding to the movement parameter of thefirst lens set 111. The calculated movement parameter of the second lens set 121 may include a movement direction of the second lens set 121, corresponding to the movement direction of the first lens set 111 and an amount of movement to a target location of the second lens set 121, corresponding to the target location of thefirst lens set 111. - In
operation 611, theprocessor 170 may determine whether a difference between a current location of the second lens set 121 and the target location of the second lens set 121 is within a specified threshold range. According to an embodiment, the threshold range may be set based on an acceptable DoF of thesecond camera module 120. If the difference is within the specified threshold range, theprocessor 170 may end the process. If the difference is out of the specified threshold range, theprocessor 170 may proceed tooperation 613. - In
operation 613, theprocessor 170 may move the second lens set 121 to the target location based on the movement direction of the second lens set 121 and the amount of movement of the second lens set 121, calculated inoperation 609. -
FIG. 7 is a flowchart illustrating a selection and AF method of a camera to perform AF according to an embodiment. - Referring to
FIG. 7 , the selection and AF method of the camera to perform the AF according to an embodiment may includeoperations 701 to 709. Reference numerals ofFIG. 1 may be used for describingoperations 701 to 709. A partial description duplicated by a description ofFIG. 5 may be omitted. - In
operation 701, aprocessor 170 of anelectronic device 101 may select any one of afirst camera module 110 or asecond camera module 120 based on a blur level of an image obtained from thefirst camera module 110 and/or thesecond camera module 120. - According to an embodiment, if the blur level is higher than a specified level, the
processor 170 may select a camera module which may capture an image with a deeper DoF, for example, a camera module with a higher F-number, between the first andsecond camera modules processor 170 may select a camera module which may capture an image with a shallower DoF, for example, a camera module with a lower F-number, between the first andsecond camera modules - The selected camera module may be selected as a sub-camera module used for an AF operation. A camera module which is not selected may be a main camera module which provides a live view on a
display 160 whileoperations 701 to 709 are performed. Thus, the selected camera module may function as thefirst camera module 110 described with reference toFIG. 6 , and the camera module which is not selected may function as thesecond camera module 120. - In
operations 703 to 709, theprocessor 170 may perform the same operations asoperations 501 to 507 shown inFIG. 5 .Operations 703 to 709 are outlined hereafter. - In
operation 703, theprocessor 170 may move a lens set of the camera module (hereinafter referred to as "sub-camera module") selected inoperation 701 such that the lens set focuses on the subject, using an AF value obtained by the selected camera module. - In operation 705, the
processor 170 may evaluate reliability of the AF value of the image obtained from the sub-camera module. - In
operation 707, theprocessor 170 may calculate a movement parameter of a lens set included in the camera module which is not selected (hereinafter referred to as "main camera module"), corresponding to a movement parameter of a lens set included in the sub-camera module, by using the movement parameter of the lens set as a result of evaluating the reliability of the AF value. - In
operation 709, theprocessor 170 may move the lens set of the main camera module depending on the movement parameter calculated inoperation 707. - According to various embodiments of the present disclosure, a camera module which is most suitable for AF may be selected as a sub-camera module, and the camera module may be used for an AF operation. Further, since movement of a lens set included in the main camera module is determined based on reliability of an AF value in the sub-camera module, wobbling in the main camera module may be removed and an AF speed a user feels may be enhanced.
- As described above, an electronic device according to an embodiment of the disclosure may include a first camera module including a first lens set, a second camera module including a second lens set, and a processor connected to the first camera module and a second camera module. The processor is configured to move the first lens focus on a subject, resulting in an auto focus (AF) value and an move the first lens set to focus on a subject, using an auto focus (AF) value obtained by the first camera module, evaluate reliability of the AF value, based on the reliability of the AF value, calculate a movement parameter of the second lens set, corresponding to a movement parameter of the first lens set, and move the second lens set depending on the calculated movement parameter..
- According to an embodiment, when the reliability is higher than a first threshold, the movement parameter of the first lens set comprises a movement direction to a target location that causes the first lens to focus on the subject and an amount of movement to the target location. Move the second lens set depending on the calculated movement parameter comprises move the second lens set by another movement direction and another amount of movement respectively corresponding to the movement direction of the first lens set and the amount of movement of the first lens set.
- According to an embodiment, when the reliability is higher than a second threshold and lower than a first threshold, the movement parameter of the first lens set consists of a movement direction to a target location that causes the first lens to focus on the subject and a target range including the target location, and wherein the first threshold is higher than the second threshold
- According to an embodiment, the processor may move the second lens set in a direction corresponding to the movement direction of the first lens set and may re-evaluate the reliability periodically or aperiodically. For example, when the second lens set enters a range corresponding to the target range of the first lens set, the processor is configured to re-evaluate the reliability.
- According to an embodiment, when the reliability is evaluated to be lower than a second threshold lower than a first threshold, the movement parameter of the first lens set may include a direction to a target location where the first lens set is on focus.
- According to an embodiment, the processor may be configured to evaluate the reliability based on a level of a noise included in the AF value, brightness of the image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, or a degree of a flicker phenomenon of the AF value.
- According to an embodiment, when a difference between a current location of the second lens set and a location indicated by the calculated movement parameter departs from a specified threshold range, the processor may move the second lens set depending on the calculated movement parameter.
- According to an embodiment, the threshold range may be set based on an acceptable depth of field (DoF) of the second camera module. The acceptable DoF may be determined based on at least one of, for example, a pixel pitch of a pixel included in the second camera module, an F-number of the second camera module, or lens sensitivity of the second lens set.
- According to an embodiment, the electronic device may further include a display configured to output an image received from the second camera module.
- According to an embodiment, the processor may control the first camera module such that the first lens set focuses on the subject, depending on a contrast AF scheme or a phase difference AF scheme.
- According to an embodiment, the processor may calculate a location parameter of the second lens set by converting a location parameter of the first lens set.
- According to an embodiment, if calculating a location parameter of the second lens set, the processor may reflect a difference between a specification of an optical system of the first camera module and a specification of an optical system of the second camera module.
- According to an embodiment, the first camera module may have at least part of a field of view (FoV), which is in common with an FoV of the second camera module.
- An electronic device according to another embodiment may include a first camera module, a second camera module configured to capture an image with a deeper depth of field (DoF) than the first camera module, and a processor connected to the first camera module and the second camera module. The processor may be configured to select any one of the first camera module or the second camera module based on a blur level of an image obtained by the first camera module or the second camera module, move a lens set of the selected camera module such that the lens set focuses on the subject, using an AF value obtained by the selected camera module, calculate a movement parameter of a lens set included in a camera module which is not selected, corresponding to a movement parameter of a lens set included in the selected camera module, as a result of evaluating reliability of the AF value of the image obtained from the selected camera module, and move a lens set included in the camera module which is not selected, depending on the calculated movement parameter.
- According to an embodiment, if the blur level is higher than a specified level, the processor may select the second camera module.
- According to an embodiment, if the blur level is lower than the specified level, the processor may select the first camera module.
- According to an embodiment, the second camera module may correspond to a camera module with a high F-number.
- According to an embodiment, the processor may control the selected camera module such that the subject is on focus, depending on a contrast AF scheme or a phase difference AF scheme.
- According to an embodiment, the processor may be configured to evaluate the reliability based on a level of a noise included in the AF value, brightness of the image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, or a degree of a flicker phenomenon of the AF value.
- According to an embodiment, the first camera module and the second camera module may be exposed to the outside through a common surface of the electronic device.
- The term "module" used in this disclosure may represent, for example, a unit including one or more combinations of hardware, software and firmware. The term "module" may be interchangeably used with the terms "unit", "logic", "logical block", "component" and "circuit". The "module" may be a minimum unit of an integrated component or may be a part thereof. The "module" may be a minimum unit for performing one or more functions or a part thereof. The "module" may be implemented mechanically or electronically. For example, the "module" may include at least one of an application-specific IC (ASIC) chip, a field-programmable gate array (FPGA), and a programmable-logic device for performing some operations, which are known or will be developed.
- At least a part of an apparatus (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments may be, for example, implemented by instructions stored in computer-readable storage media in the form of a program module. The instruction, when executed by a processor (e.g., the processor 170), may cause the one or more processors to perform a function corresponding to the instruction. The computer-readable storage media, for example, may be the
memory 160. - A computer-readable recording medium may include a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) and a digital versatile disc (DVD), a magneto-optical media (e.g., a floptical disk)), and hardware devices (e.g., a read only memory (ROM), a random access memory (RAM), or a flash memory). Also, a program instruction may include not only a mechanical code such as things generated by a compiler but also a high-level language code executable on a computer using an interpreter. The above hardware unit may be configured to operate via one or more software modules for performing an operation of various embodiments of the present disclosure, and vice versa.
- A module or a program module according to various embodiments may include at least one of the above elements, or a part of the above elements may be omitted, or additional other elements may be further included. Operations performed by a module, a program module, or other elements according to various embodiments may be executed sequentially, in parallel, repeatedly, or in a heuristic method. In addition, some operations may be executed in different sequences or may be omitted. Alternatively, other operations may be added.
- While the present disclosure has been shown and described with reference to various embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the scope of the present disclosure as defined by the appended claims.
Claims (15)
- An electronic device (101), comprising:a first camera module (110) including a first lens (111) set;a second camera module (120) including a second lens set (121); anda processor (170) connected to the first camera module (110) and the second camera module (120),wherein the processor (170) is configured to:move the first lens set (111) to focus on a subject (10), using an auto focus, AF, value obtained by the first camera module (110);evaluate a reliability of the AF value, wherein the reliability of the AF value corresponds to a probability of a focus detection error;based on the reliability of the AF value, calculate a movement parameter of the second lens set, corresponding to a movement parameter of the first lens set, wherein the movement parameters of the first lens set (111) and the second lens set (121) respectively comprise a movement direction to a target location where the first and second lens sets should be moved, such that the subject (10) is on focus; and CHARACTERISED IN THATthe processor (170) is further configured to, while the first lens set (111) is being moved using the AF value, calculate the movement parameter of the second lens set when it is evaluated that the reliability of the AF value is greater than a first threshold; andmove the second lens set (121) depending on the calculated movement parameter.
- The electronic device (101) of claim 1, wherein, when the reliability is higher than the first threshold, the movement parameter of the first lens set (111) comprises the movement direction to a target location that causes the first lens to focus on the subject (10) and an amount of movement to the target location, and
wherein move the second lens set (121) depending on the calculated movement parameter comprises:move the second lens set (121) by another movement direction and another amount of movement respectively corresponding to the movement direction of the first lens set (111) and the amount of movement of the first lens set (111). - The electronic device (101) of claim 1, wherein, when the reliability is higher than a second threshold and lower than a first threshold, the movement parameter of the first lens set (111) consists of a movement direction to a target location that causes the first lens to focus on the subject (10) and a target range including the target location, and
wherein the first threshold is higher than the second threshold. - The electronic device (101) of claim 3, wherein the processor (170) is configured to:move the second lens set (121) in a direction corresponding to the movement direction of the first lens set (111); andre-evaluate the reliability periodically or aperiodically.
- The electronic device (101) of claim 3, wherein the processor (170) is configured to move the second lens set in a direction corresponding to the movement direction of the first lens set (111), and
wherein, the processor (170) is configured to re-evaluate the reliability when the second lens set (121) enters a range corresponding to the target range of the first lens set. - The electronic device (101) of claim 1, wherein, when the reliability is evaluated to be lower than a second threshold lower than a first threshold, the movement parameter of the first lens set (111) comprises a direction to a target location where the first lens set (111) is on focus.
- The electronic device (101) of claim 1, wherein evaluate the reliability value of the AF comprises:evaluate the reliability based on a level of a noise included in the AF value, brightness of the image, a contrast ratio of the image, a blur level of the image, the number of saturated pixels included in the image, a degree of a flicker phenomenon of the AF value, or two or more combinations thereof.
- The electronic device (101) of claim 1, wherein, when a difference between a current location of the second lens set (121) and a location indicated by the calculated movement parameter departs from a specified threshold range, the processor (170) is configured to move the second lens set (121) depending on the calculated movement parameter.
- The electronic device (101) of claim 8, wherein the threshold range is set based on an acceptable depth of field (DoF) of the second camera module (120).
- The electronic device (101) of claim 1, further comprising:a display (150) configured to output an image received from the second camera module (120).
- The electronic device (101) of claim 1, wherein the processor (170) is configured to;
control the first camera module (110) such that the first lens set (111) focuses on the subject (10) depending on a contrast AF scheme or a phase difference AF scheme. - The electronic device (101) of claim 1, wherein the processor (170) is configured to:calculate a location parameter of the second lens set by converting a location parameter of the first lens set (111).
- The electronic device (101) of claim 1, wherein the first camera module (110) has at least part of a field of view, FoV, which is in common with an FoV of the second camera module (120).
- The electronic device (101) of claim 1, wherein the second camera module (120) configured to capture an image with a deeper depth of field, DoF, than the first camera module (110).
- The electronic device (101) of claim 14, wherein a blur level obtained by the first camera module (110) or the second camera module (120) is higher than a specified level.
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR1020160183511A KR20180078596A (en) | 2016-12-30 | 2016-12-30 | Method and electronic device for auto focus |
Publications (2)
Publication Number | Publication Date |
---|---|
EP3343899A1 EP3343899A1 (en) | 2018-07-04 |
EP3343899B1 true EP3343899B1 (en) | 2021-09-01 |
Family
ID=60953571
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP17209056.5A Active EP3343899B1 (en) | 2016-12-30 | 2017-12-20 | Electronic device for performing autofocus |
Country Status (4)
Country | Link |
---|---|
US (1) | US10382665B2 (en) |
EP (1) | EP3343899B1 (en) |
KR (1) | KR20180078596A (en) |
CN (1) | CN108270967B (en) |
Families Citing this family (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11792538B2 (en) * | 2008-05-20 | 2023-10-17 | Adeia Imaging Llc | Capturing and processing of images including occlusions focused on an image sensor by a lens stack array |
US8878950B2 (en) | 2010-12-14 | 2014-11-04 | Pelican Imaging Corporation | Systems and methods for synthesizing high resolution images using super-resolution processes |
CN107277348B (en) * | 2017-06-16 | 2019-08-16 | Oppo广东移动通信有限公司 | Focusing method, device, computer readable storage medium and mobile terminal |
KR20200034276A (en) | 2018-09-21 | 2020-03-31 | 엘지이노텍 주식회사 | Camera module and method of operating the same |
CN110233962B (en) * | 2019-04-26 | 2021-04-16 | 努比亚技术有限公司 | Confidence optimization method and device and computer readable storage medium |
CN112689082B (en) * | 2019-10-17 | 2022-05-17 | 电装波动株式会社 | Imaging device with event camera |
CN113747052B (en) * | 2020-05-27 | 2023-05-12 | 深圳清华大学研究院 | Focusing tracking system and method |
US11893668B2 (en) | 2021-03-31 | 2024-02-06 | Leica Camera Ag | Imaging system and method for generating a final digital image via applying a profile to image information |
Family Cites Families (41)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003298920A (en) | 2002-03-29 | 2003-10-17 | Fuji Photo Film Co Ltd | Digital camera |
JP4299561B2 (en) | 2003-03-26 | 2009-07-22 | 富士フイルム株式会社 | Imaging device |
US8134637B2 (en) | 2004-01-28 | 2012-03-13 | Microsoft Corporation | Method and system to increase X-Y resolution in a depth (Z) camera using red, blue, green (RGB) sensing |
US8139141B2 (en) | 2004-01-28 | 2012-03-20 | Microsoft Corporation | Single chip red, green, blue, distance (RGB-Z) sensor |
US8139142B2 (en) | 2006-06-01 | 2012-03-20 | Microsoft Corporation | Video manipulation of red, green, blue, distance (RGB-Z) data including segmentation, up-sampling, and background substitution techniques |
US7676146B2 (en) * | 2007-03-09 | 2010-03-09 | Eastman Kodak Company | Camera using multiple lenses and image sensors to provide improved focusing capability |
US7859588B2 (en) * | 2007-03-09 | 2010-12-28 | Eastman Kodak Company | Method and apparatus for operating a dual lens camera to augment an image |
US7683962B2 (en) | 2007-03-09 | 2010-03-23 | Eastman Kodak Company | Camera using multiple lenses and image sensors in a rangefinder configuration to provide a range map |
US8289377B1 (en) | 2007-08-31 | 2012-10-16 | DigitalOptics Corporation MEMS | Video mode hidden autofocus |
US8866920B2 (en) | 2008-05-20 | 2014-10-21 | Pelican Imaging Corporation | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
EP4336447A1 (en) | 2008-05-20 | 2024-03-13 | FotoNation Limited | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
US8456517B2 (en) | 2008-07-09 | 2013-06-04 | Primesense Ltd. | Integrated processor for 3D mapping |
JP5185097B2 (en) * | 2008-12-19 | 2013-04-17 | 富士フイルム株式会社 | Imaging apparatus and in-focus position determination method |
CN101771816A (en) | 2008-12-27 | 2010-07-07 | 鸿富锦精密工业(深圳)有限公司 | Portable electronic device and imaging method |
KR101038815B1 (en) | 2009-08-07 | 2011-06-03 | 삼성전기주식회사 | Image capture system capable of fast auto focus |
EP2502115A4 (en) | 2009-11-20 | 2013-11-06 | Pelican Imaging Corp | Capturing and processing of images using monolithic camera array with heterogeneous imagers |
KR101616422B1 (en) | 2009-12-23 | 2016-04-28 | 삼성전자주식회사 | Method and apparatus for processing the digital image by using fast AF |
US20140030705A1 (en) | 2010-10-20 | 2014-01-30 | Bionano Genomics, Inc. | Systems and methods for assessing biomolecule characteristics |
US8836767B2 (en) | 2010-12-17 | 2014-09-16 | Samsung Electronics Co., Ltd | Imaging apparatus and imaging method |
JP2012133194A (en) | 2010-12-22 | 2012-07-12 | Nikon Corp | Imaging apparatus |
JP2013008004A (en) | 2011-05-25 | 2013-01-10 | Ricoh Co Ltd | Imaging apparatus |
JP5901935B2 (en) | 2011-10-24 | 2016-04-13 | 株式会社東芝 | Solid-state imaging device and camera module |
US20130155309A1 (en) | 2011-12-20 | 2013-06-20 | Research In Motion Limited | Method and Apparatus for Controlling Camera Functions Using Facial Recognition and Eye Tracking |
US9142010B2 (en) | 2012-01-04 | 2015-09-22 | Audience, Inc. | Image enhancement based on combining images from multiple cameras |
WO2013103523A1 (en) | 2012-01-04 | 2013-07-11 | Audience, Inc. | Image enhancement methods and systems |
US8619148B1 (en) | 2012-01-04 | 2013-12-31 | Audience, Inc. | Image correction after combining images from multiple cameras |
US8718459B2 (en) | 2012-06-01 | 2014-05-06 | Blackberry Limited | Method and digital camera having improved autofocus |
JP6175748B2 (en) | 2012-08-29 | 2017-08-09 | 株式会社ニコン | Imaging device |
US9275459B2 (en) | 2012-10-05 | 2016-03-01 | Qualcomm Incorporated | Method and apparatus for calibrating an imaging device |
CN107395975A (en) | 2013-01-07 | 2017-11-24 | 华为技术有限公司 | A kind of image processing method and device |
JP2014202771A (en) | 2013-04-01 | 2014-10-27 | 株式会社Jvcケンウッド | Imaging apparatus and method for controlling the same |
TWI471677B (en) | 2013-04-11 | 2015-02-01 | Altek Semiconductor Corp | Auto focus method and auto focus apparatus |
TWI460523B (en) | 2013-05-02 | 2014-11-11 | Altek Semiconductor Corp | Auto focus method and auto focus apparatus |
TWI538512B (en) | 2013-06-27 | 2016-06-11 | 聚晶半導體股份有限公司 | Method for adjusting focus position and electronic apparatus |
US10178373B2 (en) | 2013-08-16 | 2019-01-08 | Qualcomm Incorporated | Stereo yaw correction using autofocus feedback |
US9615012B2 (en) * | 2013-09-30 | 2017-04-04 | Google Inc. | Using a second camera to adjust settings of first camera |
KR102150890B1 (en) * | 2014-02-21 | 2020-09-02 | 삼성전자주식회사 | Method for displaying focus of image and apparatus to be applied to the same |
CN103973978B (en) * | 2014-04-17 | 2018-06-26 | 华为技术有限公司 | It is a kind of to realize the method focused again and electronic equipment |
TWI515470B (en) | 2014-04-30 | 2016-01-01 | 聚晶半導體股份有限公司 | Auto-focus system for multiple lens and method thereof |
TWI515471B (en) | 2014-04-30 | 2016-01-01 | 聚晶半導體股份有限公司 | Auto-focus system for multiple lens and method thereof |
JP6478568B2 (en) * | 2014-11-10 | 2019-03-06 | キヤノン株式会社 | CONTROL DEVICE, IMAGING DEVICE, IMAGING SYSTEM, LENS DEVICE, CONTROL METHOD, PROGRAM, AND STORAGE MEDIUM |
-
2016
- 2016-12-30 KR KR1020160183511A patent/KR20180078596A/en active IP Right Grant
-
2017
- 2017-12-14 US US15/841,521 patent/US10382665B2/en active Active
- 2017-12-20 EP EP17209056.5A patent/EP3343899B1/en active Active
- 2017-12-28 CN CN201711469108.5A patent/CN108270967B/en active Active
Also Published As
Publication number | Publication date |
---|---|
US10382665B2 (en) | 2019-08-13 |
CN108270967A (en) | 2018-07-10 |
EP3343899A1 (en) | 2018-07-04 |
KR20180078596A (en) | 2018-07-10 |
CN108270967B (en) | 2021-09-21 |
US20180191947A1 (en) | 2018-07-05 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP3343899B1 (en) | Electronic device for performing autofocus | |
CN111052727B (en) | Electronic device and control method thereof | |
JP6911192B2 (en) | Image processing methods, equipment and devices | |
CN109565551B (en) | Synthesizing images aligned to a reference frame | |
KR101345093B1 (en) | Autofocus with confidence measure | |
CN105659580B (en) | A kind of Atomatic focusing method, device and electronic equipment | |
US9196027B2 (en) | Automatic focus stacking of captured images | |
CN105096281B (en) | Image processing apparatus and image processing method | |
EP3763119B1 (en) | Method for generating depth information and electronic device supporting the same | |
US20140002618A1 (en) | Image processing device and image processing method having function for reconstructing multi-aspect images, and recording medium | |
US10516820B2 (en) | Electronic device for controlling focus of lens and method for controlling the same | |
US9706131B2 (en) | Image combining apparatus for generating a combined image from a plurality of images, image combining system, and image combining method | |
KR20150085710A (en) | Dispaly apparatus and controlling method thereof | |
US10104301B2 (en) | Image pickup apparatus that performs process of detecting change in brightness, control method therefor, and storage medium | |
US11812133B2 (en) | Image processing apparatus and method for controlling image processing apparatus | |
US9912858B2 (en) | Image capturing apparatus and control method thereof | |
US9800792B2 (en) | Image pickup apparatus capable of determining light amount change characteristic, method of determining light amount change characteristic, and storage medium | |
CN103581538B (en) | Digital photographing apparatus and the method for controlling the digital photographing apparatus | |
WO2018110282A1 (en) | Focusing apparatus, focusing method, and storage medium storing program | |
US20230188846A1 (en) | Imaging apparatus and control method thereof | |
US10681274B2 (en) | Imaging apparatus and control method thereof | |
US20220292692A1 (en) | Image processing apparatus and method of processing image | |
JP2023157200A (en) | Electronic apparatus, control method, and program | |
CN103634589B (en) | Image judgment method and object coordinates calculate device | |
JP2018116139A (en) | Imaging device, control method thereof, and control program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
AX | Request for extension of the european patent |
Extension state: BA ME |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
17P | Request for examination filed |
Effective date: 20180927 |
|
RBV | Designated contracting states (corrected) |
Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: GRANT OF PATENT IS INTENDED |
|
INTG | Intention to grant announced |
Effective date: 20210329 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: THE PATENT HAS BEEN GRANTED |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP Ref country code: AT Ref legal event code: REF Ref document number: 1427479 Country of ref document: AT Kind code of ref document: T Effective date: 20210915 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602017045092 Country of ref document: DE |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG9D |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: MP Effective date: 20210901 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211201 Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211201 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: RS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 1427479 Country of ref document: AT Kind code of ref document: T Effective date: 20210901 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20211202 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220101 Ref country code: SM Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20220103 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: AL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602017045092 Country of ref document: DE |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
26N | No opposition filed |
Effective date: 20220602 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
REG | Reference to a national code |
Ref country code: BE Ref legal event code: MM Effective date: 20211231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211220 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211220 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602017045092 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: H04N0005232000 Ipc: H04N0023600000 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211231 Ref country code: BE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211231 Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20211231 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20171220 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20231120 Year of fee payment: 7 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20231120 Year of fee payment: 7 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20210901 |