EP4258638A1 - Dispositif de traitement d'image, procédé de traitement d'image et support d'informations - Google Patents

Dispositif de traitement d'image, procédé de traitement d'image et support d'informations Download PDF

Info

Publication number
EP4258638A1
EP4258638A1 EP22211880.4A EP22211880A EP4258638A1 EP 4258638 A1 EP4258638 A1 EP 4258638A1 EP 22211880 A EP22211880 A EP 22211880A EP 4258638 A1 EP4258638 A1 EP 4258638A1
Authority
EP
European Patent Office
Prior art keywords
distance
information
layer
image
subject
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
EP22211880.4A
Other languages
German (de)
English (en)
Inventor
Koki Nakamura
Tokuro Nishida
Hiroyuki Hasegawa
Kengo Takeuchi
Kazuya Kitamura
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Canon Inc
Original Assignee
Canon Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2022183629A external-priority patent/JP2023099993A/ja
Application filed by Canon Inc filed Critical Canon Inc
Publication of EP4258638A1 publication Critical patent/EP4258638A1/fr
Pending legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/2224—Studio circuitry; Studio devices; Studio equipment related to virtual studio applications
    • H04N5/2226—Determination of depth image, e.g. for foreground/background separation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/0007—Image acquisition
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/268—Signal distribution or switching
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T1/00—General purpose image data processing
    • G06T1/20—Processor architectures; Processor configuration, e.g. pipelining
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/194—Segmentation; Edge detection involving foreground-background segmentation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H04N23/635—Region indicators; Field of view indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/704—Pixels specially adapted for focusing, e.g. phase difference pixel sets
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2621—Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/265—Mixing
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2200/00—Indexing scheme for image data processing or generation, in general
    • G06T2200/28—Indexing scheme for image data processing or generation, in general involving image processing hardware
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30196—Human being; Person
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H04N25/134—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements based on three different wavelength filter elements

Definitions

  • the present invention relates to an image processing device, an image processing method, a storage medium, and the like.
  • chroma key synthesis is video synthesis technology commonly used in movies and television, in which a specific color part of a video is made transparent and synthesized with a different video.
  • a video for chromakey synthesis it is necessary to prepare a background of a specific color, such as a green background or a blue background, and it is also necessary to perform an operation of smoothing out wrinkles so that there is no color unevenness in the background, and to adjust illumination.
  • a method of acquiring information on a distance to a subject for example, there is a method of measuring a distance to the subject by using a device equipped with a distance sensor, separate from a video processing device for video capturing.
  • a distance sensor for example, since an angle of view and a resolution differ depending on devices, it is necessary to perform calibration work before imaging in order to perform image synthesis in post-processing.
  • Japanese Patent Application Laid-Open No. 2021-48560 discloses technology for generating distance information indicating a distance from a video processing device to a subject by using an imaging element having a distance measurement function of an imaging surface phase difference system.
  • video capturing and distance measurement information generation can be performed at the same time by one imaging element, and thus the above-described calibration work is not necessary.
  • Japanese Patent Application Laid-Open No. 2016-143919 discloses technology for generating a histogram representing distribution frequency of a distance to a subject corresponding to image data to classify images of pieces of distance information according to the frequency of appearance of distance information.
  • Japanese Patent Application Laid-Open No. 2018-29270 discloses technology for performing classification for each distance information included in a subject region by using a unit that recognizes the subject region in image data.
  • distance information is output as a numerical value or the like indicating a distance to a subject, or a heat map according to the distance information is displayed, and thus it is difficult for a photographer to intuitively understand whether or not a video into which CG is easily inserted has been captured.
  • Japanese Patent Application Laid-Open Nos. 2016-143919 and 2018-29270 disclose technology for classifying images for distance information, but there is a problem that the images are classified in accordance with only distance information to a subject, and distance information with few subjects appearing so that CG can be easily inserted in post-processing is not classified. Furthermore, there is a problem that it is difficult to intuitively understand what kind of distance layer MAP can be generated depending on conditions of an optical system.
  • an object of the present invention is to provide an image processing device that can facilitate setting related to layer information based on distance information.
  • An image processing device includes at least one processor or circuit configured to function as an image acquisition unit configured to acquire an image including a subject through a lens unit, a distance information acquisition unit configured to acquire distance information indicating a distance to the subject, a layer information generation unit configured to generate layer information on a layer for each distance based on the distance information, and a setting unit configured to set a reference for generating the layer information and switch display of a setting value capable of being set in accordance with the lens information of the lens unit.
  • the image processing device includes electronic equipment having an imaging function such as digital movie cameras, network cameras, smartphones with cameras, tablet computers with cameras, in-vehicle cameras, drone cameras, and cameras mounted on robots, and the like.
  • images in the embodiments include videos such as movies in addition to still images.
  • Fig. 1 is a block diagram of an image processing device according to a first embodiment.
  • an image processing device 100 is a device capable of inputting, outputting, and even recording images.
  • a CPU 102 as a computer, a ROM 103, a RAM 104, an image processing unit 105, a lens unit 106, an imaging unit 107, a network module 108, and an image output unit 109 are connected to an internal bus 101.
  • a recording medium interface (I/F) 110 is connected to the internal bus 101.
  • the blocks connected to the internal bus 101 are configured to be able to transmit and receive data to and from each other via the internal bus 101.
  • Fig. 1 some of the blocks illustrated in Fig. 1 are implemented by causing the CPU, which is a computer included in the image processing device, to execute computer programs stored in a memory such as a ROM as a storage medium. However, some or all of them may be implemented by hardware.
  • a dedicated circuit ASIC
  • a processor a reconfigurable processor, a DSP
  • the blocks illustrated in Fig. 1 may not be built in the same housing or may be constituted by separate devices connected to each other via signal paths.
  • the lens unit 106 is a unit constituted by a lens group including a zoom lens and a focus lens, an aperture mechanism, a drive motor, and the like. An optical image that has passed through the lens unit 106 is formed on a light receiving surface of the imaging unit 107.
  • the imaging unit 107 functions as an image acquisition unit that acquires an image including a subject, and includes an imaging element such as a CCD image sensor or a CMOS image sensor. An optical image formed on a light receiving surface of the imaging element is converted into an imaging signal and is further converted into a digital signal and output.
  • the imaging element of the present embodiment is a sensor having an image surface phase difference detection function, and the details thereof will be described later.
  • the CPU 102 as a computer controls each unit of the image processing device 100 in accordance with computer programs stored in the ROM 103 and using the RAM 104 as a work memory.
  • the ROM 103 is a non-volatile semiconductor memory, and the computer program, various adjustment parameters, and the like for operating the CPU 102 are recorded therein.
  • the RAM 104 is a volatile semiconductor memory, and RAMs of which the speed and capacity lower than those of a frame memory 111 are generally used.
  • the frame memory 111 is a semiconductor element that temporarily stores image signals and can read out the image signals when necessary. Since an image signal has a huge amount of data, a high-bandwidth and large-capacity memory is required.
  • a dual data rate 4-synchronous dynamic RAM (DDR4-SDRAM) or the like is used.
  • DDR4-SDRAM 4-synchronous dynamic RAM
  • the frame memory 111 for example, processing such as synthesizing temporally different images or cutting out only a required region can be performed.
  • the image processing unit 105 performs various image processing on data from the imaging unit 107 or image data stored in the frame memory 111 or the recording medium 112 under the control of the CPU 102.
  • the image processing performed by the image processing unit 105 includes pixel interpolation of image data, encoding processing, compression processing, decoding processing, enlargement/reduction processing (resizing), noise reduction processing, color conversion processing, and the like.
  • the image processing unit 105 corrects variations in characteristics of pixels of the imaging unit 107, or performs correction processing such as correction of defective pixels, correction of white balance, correction of luminance, or correction of distortion caused by characteristics of a lens or a decrease in the amount of peripheral light.
  • the image processing unit 105 generates a distance map, but the details thereof will be described later.
  • the image processing unit 105 may be constituted by a dedicated circuit block for performing specific image processing.
  • the CPU 102 can also perform image processing in accordance with a program without using the image processing unit 105 depending on the type of image processing.
  • the CPU 102 controls the lens unit 106 to optically magnify an image and adjust an aperture for adjusting a focal length and the amount of light based on image processing results in the image processing unit 105.
  • a camera shake may be corrected by moving a portion of the lens group within a plane perpendicular to an optical axis.
  • Reference numeral 113 denotes an operation unit that receives a user's operation as an interface for the outside of the device.
  • the operation unit 113 is constituted by elements such as mechanical buttons and switches and includes a power switch, a mode changeover switch, and the like.
  • Reference numeral 114 denotes a display unit for displaying an image, and can allow, for example, an image processed by the image processing unit 105, a setting menu, and an operation state of the image processing device 100 to be confirmed.
  • a device having a small size and low power consumption such as a liquid crystal display (LCD) or an organic electroluminescence (EL) is used as a display device.
  • the display unit 114 has a touch panel structure using a thin film element of a resistive film type or an electrostatic capacitive type, and may be used as a portion of the operation unit 113.
  • the CPU 102 generates a character string for informing a user of a setting state of the image processing device 100 and a menu for setting the image processing device 100, superimposes them on an image processed by the image processing unit 105, and displays it on the display unit 114.
  • an imaging assist display such as a histogram, a vector scope, a waveform monitor, a zebra, a peaking, and a false color.
  • Reference numeral 109 denotes an image output unit, and as an interface, a serial digital interface (SDI), a high definition multimedia interface (HDMI: registered trademark), or the like is adopted as an interface. Alternatively, an interface such as a display port (registered trademark) may be used. A real-time image can be displayed on an external monitor or the like via the image output unit 109.
  • SDI serial digital interface
  • HDMI high definition multimedia interface
  • HDMI registered trademark
  • a network module 108 that can transmit not only images but also control signals is also provided.
  • the network module 108 is an interface for inputting and outputting image signals and audio signals.
  • the network module 108 can also communicate with external devices via the Internet or the like and transmit and receive various data such as files and commands.
  • the network module 108 may be a wired or wireless module.
  • the image processing device 100 also has a function of not only outputting images to the outside, but also recording them inside the device itself.
  • the recording medium 112 is a large-capacity storage device such as a hard disc drive (HDD) or a solid state drive (SSD) capable of recording image data and various setting data, and can be mounted on the recording medium I/F 110.
  • HDD hard disc drive
  • SSD solid state drive
  • the object detection unit 115 which is a block for detecting objects, performs object detection using, for example, artificial intelligence represented by deep learning using a neural network.
  • the CPU 102 transmits a program for processing stored in the ROM 103, a network structure such as an SSD and a YOLO, weight parameters, and the like to the object detection unit 115.
  • the object detection unit 115 performs processing for detecting an object from an image signal based on various parameters obtained from the CPU 102, and expands processing results to the RAM 104.
  • Fig. 2(A) is a diagram illustrating an example of color filters disposed on the light receiving surface of the imaging element.
  • Fig. 2(A) illustrates an example of a Bayer array of red (R), blue (B), and green (Gb, Gr), the imaging element has a plurality of pixels arranged in a two-dimensional pattern, any one color filter of R, B, Gb, and Gr is disposed on the front surface of each pixel as illustrated in Fig. 2(A) .
  • a microlens is disposed on the front surface of the color filter disposed on the front surface of each pixel of the imaging element, and each pixel has two photoelectric conversion units (a photodiode A and a photodiode B) disposed side by side in a horizontal scanning direction.
  • Fig. 2(B) is a diagram illustrating an example in which two photoelectric conversion units (photodiodes) are disposed in each pixel so as to correspond to the arrays of the color filters in Fig. 2(A) .
  • each pixel is constituted by a pair of photodiodes A and B, and color filters of the same color are disposed for the two paired photodiodes.
  • the photodiode A and the photodiode B receive light beams from different exit pupils of the optical system via the microlenses.
  • a image signals can be acquired from a plurality of photodiodes A of the pixels arranged in a row direction.
  • B image signals can be acquired from a plurality of photodiodes B of the pixels arranged in the row direction.
  • the A and B image signals are processed as signals for phase difference detection.
  • the CPU 102 or the image processing unit 105 performs a correlation operation between the A image signal and the B image signal, detects a phase difference between the A image signal and the B image signal, and calculates a subject distance based on the phase difference. That is, the CPU 102 or the image processing unit 105 functions as a distance information acquisition unit for acquiring distance information indicating a distance to a subject.
  • a image signal + B image signal obtained by adding the signals of the two photodiodes A and B of each pixel, and the signal for an image obtained by the addition is processed by the image processing unit 105 as a color image signal corresponding to the Bayer array illustrated in Fig. 2(A) .
  • the imaging unit 107 it is also possible to output a phase difference detection signal (an A image signal, a B image signal) for each pixel, but it is also possible to output a value obtained by adding and averaging A image signals of a plurality of adjacent pixels and adding and averaging B signals of a plurality of adjacent pixels. By outputting the value obtained by adding and averaging, it is possible to shorten a period of time required to read out a signal from the imaging unit 107 and reduce the bandwidth of the internal bus 101.
  • a phase difference detection signal an A image signal, a B image signal
  • the CPU 102 and the image processing unit 105 perform a correlation operation between two image signals by using a signal output from the imaging unit 107 having such an imaging element, and calculate information such as a defocus amount, parallax information, and various reliability based on a phase difference between the two image signals.
  • a defocus amount on a light receiving surface is calculated based on the shift (phase difference) between the A and B image signals.
  • the defocus amount has positive and negative values, and a front focus or a rear focus can be determined depending on whether the defocus amount is a positive value or a negative value.
  • the degree of in-focus can be known using an absolute value of the defocus amount, and in-focus is achieved when the defocus amount is 0. That is, the CPU 102 calculates information regarding whether being a front focus or a rear focus based on whether the defocus amount is a positive value or a negative value, and calculates in-focus degree information which is the degree of in-focus (the amount of out-of-focus) based on the absolute value of the defocus amount.
  • the information regarding whether being a front focus or a rear focus is output in a case where the defocus amount exceeds a predetermined value, and information indicating in-focus is output in a case where the absolute value of the defocus amount is within the predetermined value.
  • the CPU 102 controls the lens unit 106 in accordance with the defocus amount to perform focus adjustment. In addition, the CPU 102 calculates a distance to a subject using the principle of triangulation from the phase difference information and the lens information of the lens unit 106.
  • each pixel may be configured such that three or more photodiodes as photoelectric conversion units are disposed for one microlens.
  • all of the pixels may not be configured as described above, and for example, pixels for distance detection among a plurality of pixels for image detection disposed in a two-dimensional pattern may be disposed discretely.
  • the pixel for distance detection may have a structure having two photodiodes as described above, or each pixel for distance detection may have a structure having only one of the photodiode A and the photodiode B.
  • the photodiode A and the photodiode B are disposed such that images of different pupil regions (exit pupils) of the lens unit are incident.
  • the present embodiment may provide a configuration in which two image signals, such as the A image signal and the B image signal, allowing phase difference detection are obtained, and is not limited to the above-described pixel structure.
  • the imaging unit 107 may be a so-called stereo camera constituted by two imaging elements having parallax.
  • Fig. 3 is a flowchart illustrating distance information generation processing according to the first embodiment. Note that operations of steps in the flowchart of Fig. 3 are performed by causing the CPU 102 as a computer to execute computer programs stored in the ROM 103 or the like as a storage medium.
  • step S300 two signals of (A image signal + B image signal) for imaging and an A image signal for phase difference detection are acquired by being read out from the imaging unit 107.
  • step S301 the image processing unit 105 calculates a B image signal for phase difference detection by obtaining a difference between the (A image signal + B image signal) and the A image signal.
  • each of the A image signal and the B image signal may be read out from the imaging unit 107.
  • image signals output from the respective image sensors may be processed as an A image signal and a B image signal.
  • step S302 optical shading correction is performed for each of the A image signal for phase difference detection and the B image signal for phase difference detection.
  • step S303 filtering is performed on each of the A image signal for phase difference detection and the B image signal for phase difference detection. For example, a low-frequency range is cut with a high-pass filter constituted by a FIR. Note that the signals may pass through a band-pass filter or a low-pass filter with different filter coefficients.
  • step S304 the A image signal for phase difference detection and the B image signal for phase difference detection, which have been subjected to the filtering in step S303, are divided into minute blocks and subjected to a correlation operation. Note that there are no restrictions on the size or shapes of the minute blocks, and regions may overlap each other by adjacent blocks.
  • a correlation amount C(k) in a deviation amount k between two signal strings is arithmetically operated using the following Formula (1) while relatively shifting the signal string F(1) to F(m) of the B image signals with respect to the signal string E(1) to E(m) of the A image signals.
  • C k ⁇ E n ⁇ F n + k
  • ⁇ operation means an arithmetic operation of calculating a sum for n.
  • the range that n and n + k can take is limited to a range from 1 to m.
  • the deviation amount k is an integer value and is a relative pixel deviation amount in units of detection pitches of a pair of pieces of data.
  • Fig. 4 is a diagram illustrating arithmetic operation results of Formula (1) in a case where a correlation between a pair of image signal strings is high in an ideal state where there is no noise.
  • kj k when a discrete correlation amount C(k) is minimized.
  • x x that gives the minimum value C(x) for a continuous correlation amount is calculated.
  • the pixel deviation amount x is a real number, and the unit is pixel.
  • x kj + D
  • SLOP D C kj ⁇ 1 ⁇ C ( kj + 1 2
  • SLOP MAX C kj + 1 ⁇ C kj , C kj ⁇ 1 ⁇ C kj
  • SLOP in Formula (4) represents a slope of a change between a smallest and minimum correlation amount and a correlation amount adjacent thereto.
  • a specific example is as follows.
  • Fig. 5 is a diagram illustrating arithmetic operation results in a case where Formula (1) is applied to a minute block having noise.
  • a correlation between a signal string of A image signals and a signal string of B image signals decreases due to the influence of noise distributed randomly.
  • a minimum value of the correlation amount C(k) becomes larger than the minimum value illustrated in Fig. 4 , and the curve of the correlation amount has an overall flat shape (a shape in which a difference absolute value between a maximum value and a minimum value is small).
  • the pixel deviation amount x is far from an ideal value.
  • the reliability of the calculated pixel deviation amount x can be determined by the value of SLOP. That is, in a case where the value of SLOP is large, the correlation between the pair of image signal strings is high, and in a case where the value of SLOP is small, it can be determined that no significant correlation has been obtained between the pair of image signal strings.
  • Formula (1) is used for a correlation operation, the correlation amount C(k) is the smallest and minimum in a shift amount where the correlation between the pair of image signal strings is the highest.
  • a correlation operation method in which a correlation amount C(k) is the largest and maximum in a shift amount where the correlation between the pair of image signal strings is the highest may be used.
  • step S305 the reliability is calculated.
  • the reliability can be calculated based on C(kj), which indicates the degree of matching between two images which is calculated in step S304, and the value of SLOP.
  • step S306 interpolation processing is performed in step S306.
  • the reliability calculated in step S305 is low, and thus the reliability may not be adopted as a pixel deviation amount.
  • interpolation processing is performed using a pixel deviation amount calculated from the surrounding pixels.
  • a median filter may be applied, or an arithmetic operation of reducing data of a pixel deviation amount and then expanding the data again may be performed.
  • color data may be extracted from (A image signal + B image signal) for imaging, and a pixel deviation amount may be interpolated using the color data.
  • step S307 a defocus amount is calculated with reference to the amount x calculated in step S304.
  • the defocus amount (denoted by DEF) can be obtained by the following Formula (5).
  • DEF P ⁇ x
  • P is a conversion coefficient determined by a detection pitch (pixel arrangement pitch) and a distance between projection centers of two right and left viewpoints in a pair of parallax images, and the unit is mm/pixel.
  • step S308 the distance is calculated from the defocus amount calculated in step S307.
  • a distance to a subject is Da
  • a focal position is Db
  • a focal length is F
  • the relative distance is Da-Da', and can be obtained by the following Formula (9) from Formulas (7) and (8).
  • Da ⁇ Da ′ DEF ⁇ F 2 F ⁇ Db 0 2 + DEF ⁇ F ⁇ Db 0
  • a pixel deviation amount, a defocus amount, and distance information can be calculated from an A image signal for phase difference detection and a B image signal for phase difference detection. That is, distance information can be acquired based on a phase difference between outputs of a plurality of photoelectric conversion units.
  • the distance information in the present embodiment may be distance data itself or may be a shift amount or a defocus amount, and the distance information includes them.
  • Fig. 6 is a flowchart illustrating an example in which distance information is converted into a distance layer MAP using a histogram according to the first embodiment. Note that operations of steps in the flowchart of Fig. 6 are performed by causing the CPU 102 as a computer to execute computer programs stored in the ROM 103 or the like as a storage medium.
  • step S600 of Fig. 6 the CPU 102 initializes each of values of an internal processing variable N, a processing variable X, and a processing variable T to 1.
  • the processing variable N is a variable for counting the number of times of processing
  • the processing variable X is a temporary variable for calculation
  • the processing variable T is a variable indicating a layer number.
  • step S601 the CPU 102 acquires an in-focus position and lens aperture information as lens information and distance information from the lens unit 106 and the imaging unit 107, and calculates a distance measurable range L and a minimum resolution width M of a subject distance.
  • step S602 the CPU 102 generates a histogram showing a distribution of distances in which the subject exists in a depth direction based on the distance information.
  • Fig. 7 is a diagram illustrating an example of the histogram generated in step S602.
  • the horizontal axis is a distance in the depth direction, and the vertical axis is the number of times of appearance of distance information.
  • the histogram data is generated by accumulating distance information appearing in units of the minimum resolution width M within the distance measurable range L.
  • the CPU 102 tags the data of the generated histogram with numbers starting from 1 in ascending order of a distance from the image processing device 100 in units of the minimum resolution width M, and stores the data in the RAM 104.
  • the operation unit 113 receives an input for setting a frequency threshold value S from the user.
  • the user transmits information on the frequency threshold value S to the CPU 102 via the operation unit 113.
  • the frequency threshold value S may be set from an external device in a wireless manner through the network module 108.
  • step S604 the CPU 102 reads histogram data of a processing variable N and histogram data of a processing variable N+1 from the RAM 104. In addition, it is determined whether or not changes in the number of times of appearance of distance information of the processing variable N and the processing variable N+1 cross the frequency threshold value S. In a case where the change crosses the frequency threshold value S, the CPU 102 proceeds to step S605, and in a case where the change does not cross the frequency threshold value S, the CPU 102 proceeds to step S607.
  • step S605 the CPU 102 classifies distance information from the minimum resolution width M * the processing variable X to the minimum resolution width M * the processing variable N as a T-th layer of the distance layer MAP.
  • step S606 the CPU 102 substitutes the value of the processing variable N+1 for the processing variable X and increments the value of the processing variable T by 1.
  • step S607 the CPU 102 determines whether or not all of the histogram data stored in the RAM 104 have been read out. When all of the histogram data have been read out, the CPU 102 proceeds to a termination step, and when all of the histogram data have not been read out, the CPU 102 proceeds to step S608. Next, in step S608, the CPU 102 increments the value of the processing variable N by 1 and returns to step S604.
  • each layer can be numbered in ascending order of a distance from the image processing device 100.
  • a reference number of layers (the number of layers) and a reference layer width (the width of a layer in a distance direction) for generating layer information are changed depending on an image.
  • the CPU 102 functions as a layer information generation step (layer information generation unit) for generating a histogram based on distance information and generating layer information for each distance based on the histogram by executing steps S600 to S608.
  • a method of classifying (generating) a distance layer MAP by using a histogram has been described, but a subject may be recognized by an image recognition unit, and a distance layer MAP may be classified (generated) in accordance with the recognized subject.
  • Fig. 8 is a flowchart illustrating processing for converting distance information into a distance layer MAP by using the image recognition unit.
  • step S800 the CPU 102 initializes each of values of a processing variable P and a processing variable Q to 1.
  • the processing variable P is a variable for counting the number of recognized subjects
  • the processing variable Q is a temporary variable for executing processing.
  • step S801 the object detection unit 115 detects a subject in image data.
  • the object detection unit 115 stores the subject in the RAM 104 as coordinate data indicating in which region in the image data the subject appears.
  • the coordinate data is data representing the outline of the subject.
  • step S802 the CPU 102 determines whether all of the subjects in the image data have been detected. In a case where all of the subjects have been detected, the CPU 102 proceeds to step S803 and in a case where all of the subjects have not been detected, the CPU 102 proceeds to step S804.
  • step S803 the CPU 102 sorts the coordinate data of the detected subjects stored in the RAM 104 in ascending order of a distance from the image processing device 100 based on distance information of a subject region, numbers them in order from the first, and then proceeds to step S805.
  • step S804 the CPU 102 increments the value of the processing variable P by 1 and then returns to step S801.
  • step S805 the CPU 102 determines whether or not the same distance information is included in a subject region indicated by coordinate data of a subject of a processing variable Q and a subject region indicated by coordinate data of a subject of a processing variable Q+1 which are stored in the RAM 104. In a case where the same distance information is included, the CPU 102 proceeds to step S806, and in a case where the same distance information is not included, the CPU 102 proceeds to step S807.
  • step S806 the subject region indicated by the coordinate data of the subject of the processing variable Q and the subject region indicated by the coordinate data of the subject of the processing variable Q+1, which are stored in the RAM 104, are merged with each other as a subject region indicated by the coordinate data of the subject of the processing variable Q+1. Then, the subject region is stored in the RAM 104.
  • step S807 the CPU 102 increments the processing variable Q by 1.
  • step S808 the CPU 102 determines whether or not the value of the processing variable Q is equal to or greater than the value of the processing variable P. In a case where the value of the processing variable Q is equal to or greater than the value of the processing variable P, the CPU 102 proceeds to step S809, and otherwise, returns to step S805.
  • step S809 the setting of the number of layers of a distance layer MAP is alternately allocated in ascending order of a distance from the image processing device 100 for distance information for each subject region indicated by coordinate data of a subject stored in the RAM 104 and distance information which is not included in any subject region. Thereby, a layer where a subject exists and a layer where a subject does not exist are generated.
  • distance information can be classified into a distance layer MAP constituted by a layer with a large number of subjects in a depth direction and a layer with a small number of subjects by using the image recognition unit.
  • each layer can be numbered in ascending order of a distance from the image processing device 100.
  • a reference number of layers (the number of layers) and a reference layer width (the width of a layer in a distance direction) for generating layer information are changed depending on an image.
  • a type such as a body, a face, or a car may be selectable. That is, the width of a layer in a distance direction for each distance may be changed in accordance with the type of subject recognized. In addition, one or a plurality of subjects may be selectable.
  • steps S800 to S804 may be performed for each frame, and the accuracy of image recognition may be improved using results of a plurality of frames to perform step S805 and the subsequent steps. That is, layer information may be generated based on images of a plurality of frames.
  • the CPU 102 executes steps S800 to S809 as layer information generation steps to recognize a subject by the image recognition unit and functions as a layer information generation unit that generates layer information for each distance in accordance with the recognized subject.
  • a method of recognizing a subject by an image recognition unit and classifying (generating) a distance layer MAP in accordance with the recognized subject and a method of generating a histogram based on distance information and classifying (generating) a distance layer MAP based on the histogram may be combined.
  • a method of presenting to a user (photographer or the like) into which layer of a distance layer MAP classified in the above-described first embodiment CG is to be inserted will be described with reference to Figs. 9 to 12 .
  • a case where layer information and coordinate information for inserting CG into the image processing device 100 are transmitted from a portable terminal is described.
  • Fig. 9 is a block diagram illustrating an example of an internal configuration of a portable terminal according to the first embodiment.
  • layer information and coordinate information can be set by a user, and setting values thereof can be transmitted in a wireless manner.
  • a network module 908, an operation unit 913, a display unit 914, a CPU 902, a ROM 903, and a RAM 904 are connected to an internal bus 901.
  • the units connected to the internal bus 901 can transmit and receive data to and from each other through the internal bus 901.
  • the CPU 902 controls each unit of the portable terminal 900 in accordance with computer programs stored in the ROM 903 and using the RAM 904 as a work memory.
  • the ROM 903 is a non-volatile recording element, and programs for operating the CPU 902, various adjustment parameters, and the like are recorded therein.
  • the display unit 914 is a display for displaying various setting states, data (including digital image data and analog image signals) received from the network module 908, and the like under the control of the CPU 902.
  • the operation unit 913 is a power switch for supplying power to the portable terminal 900, or is an operation unit that receives a user's operation such as setting of layer information and coordinate information.
  • the CPU 902 detects that the touch panel has been touched with a finger or a pen (hereinafter referred to as a touch-down) or that the touch panel is being touched with the finger or the pen (hereinafter referred to as a touch-on).
  • a move it is possible to detect that the touch panel is moving while being touched with the finger or the pen (hereinafter referred to as a move), that the finger or the pen touching the touch panel is removed from the touch panel (hereinafter referred to as a touch-up), and a state where the touch panel is touched with nothing (hereafter referred to as a touch-off).
  • the CPU 902 is notified of these operations and positional coordinates where the touch panel is touched with the finger or the pen, and the CPU 902 determines what kind of operation has been performed on the touch panel based on the notified information.
  • the moving direction of the finger or the pen moving on the touch panel can also be determined for each vertical component and horizontal component on the touch panel based on changes in positional coordinates.
  • a flick is an operation of rapidly moving a finger by a certain degree of distance while touching the touch panel with the finger and then removing the finger as it is.
  • the flick is an operation of rapidly tracing the touch panel as if the touch panel is flicked with a finger.
  • any type of touch panel may be used among various types of touch panels such as a resistive film type, a capacitance type, a surface acoustic wave type, an infrared type, an electromagnetic induction type, an image recognition type, and an optical sensor type.
  • the network module 908 transmits and receives data to and from external devices such as an external camera and a personal computer by wireless communication under the control of the CPU 902.
  • the data, setting information, operation information, and the like of the image processing device 100 are received, and additional information recorded together with a command for operating the image processing device 100 and image data, and the like are transmitted.
  • the data that can be transmitted and received includes digital image data and analog image signals.
  • Fig. 10 is a flowchart illustrating an example for presenting to a photographer into which layer of a distance layer MAP CG is to be inserted. Note that operations of steps in the flowchart of Fig. 10 are performed by causing the CPU 102 of the image processing device 100 and the CPU 902 of the portable terminal 900 to execute computer programs stored in the ROM 103, the ROM 903, or the like as a storage medium.
  • step S1000 the CPU 902 of the portable terminal 900 receives setting of layer information and coordinate information from the user via the operation unit 913.
  • the layer information is a value for designating into which layer of the distance layer MAP the CG is to be inserted.
  • an image may be transmitted from the image processing device 100 to the portable terminal 900, a subject may be selected, and the front and back of the selected subject may be selected as a layer.
  • the coordinate information is coordinate information for designating where on a screen CG is to be inserted.
  • step S1001 the CPU 902 of the portable terminal 900 transmits the layer information and the coordinate information that are set in step S 1000 to the network module 108 inside the image processing device 100 via the network module 908.
  • steps S1000 and S1001 function as steps for setting a synthesized image insertion region for inserting a synthesized image and insertion layer information which is a layer for inserting a synthesized image for an image.
  • step S 1002 the CPU 102 in the image processing device 100 receives the layer information and the coordinate information via the network module 108.
  • step S1003 the CPU 102 calculates a CG insertion layer from the layer information and calculates a CG insertion region in the image data from the coordinate information.
  • step S1004 the CPU 102 synthesizes a CG insertion color with a pixel corresponding to the position of the CG insertion region of the image data.
  • the CG insertion color is a color representing the position where CG is expected to be inserted in post-processing.
  • the CG insertion color may be a color set separately by the user.
  • step S1005 the CPU 102 determines whether a layer (layer information) of a distance layer MAP of a target pixel (subject pixel) corresponding to the position of the CG insertion region of the image data is the same as the CG insertion layer (layer information). In a case where the layer is the same as the CG insertion layer, the CPU 102 proceeds to step S1006, and in a case where the layer is different from the CG insertion layer, the CPU 102 proceeds to step S1007.
  • step S1006 the CPU 102 controls the image processing unit 105 to change data of a pixel corresponding to the position of the CG insertion region of the image data of the subject to a predetermined color for warning (warning color).
  • the warning color is a color representing that a subject exists at the same position of the CG insertion region where CG is expected to be inserted in post-processing.
  • the warning color may be a color set separately by the user.
  • a predetermined pattern for example, a dot pattern, a stripe pattern, or the like
  • a predetermined pattern for example, a dot pattern, a stripe pattern, or the like
  • an overlapping region is displayed in a predetermined color (warning color) or a predetermined patter which is different from the other regions.
  • step S1007 it is determined whether or not the layer of the distance layer MAP of the pixel corresponding to the position of the CG insertion region of the image data of the subject is behind the CG insertion layer.
  • the CPU 102 proceeds to step S1008, and the layer is before the CG insertion layer, the CPU 102 proceeds to step S1009.
  • step S1008 the CPU 102 controls the image processing unit 105 to synthesize the data of the pixel of the subject corresponding to the position of the CG insertion region with a background with a transmittance according to a distance between the distance layer MAP of the pixel and the CG insertion layer.
  • the transmittance of a front image is decreased so that the image of the subject is displayed to be thinner.
  • the transmittance of the front image is increased so that a background subject is displayed darkly. In this manner, the photographer easily ascertains the sense of distance between CG to be inserted and the subject.
  • the transmittance of the overlapping region of the front image can be changed in the same manner regardless of which side is behind. That is, as a distance between the layer information of the image of the subject and the insertion layer information for inserting a synthesized image increases, the transmittance of the front image in the overlapping region may be changed.
  • the transmittance of a front image in an overlapping region is increased and displayed. In contrast, however, the transmittance may be decreased and displayed.
  • step S1009 the CPU 102 determines whether or not all of the pixels of the subject corresponding to the position of the CG insertion region have been processed. In a case where all of the pixels have been processed, the CPU 102 proceeds to step S1010, and in a case where all of the pixels have not been processed, the CPU 102 proceeds to step S1005.
  • step S1010 the CPU 102 displays, on the display unit 114, the image data in which the warning color generated in the processes of steps S1005 to S1009 and the background subject color are synthesized.
  • the steps function as a display control unit that displays an overlapping region where the synthesized image and the image of the subject overlap each other in a predetermined color or pattern corresponding to the layer information of the subject.
  • data may be output from the image output unit 109, recorded in the recording medium 112 via the recording medium I/F 110, or transmitted to an external device via the network module 108.
  • pixels of the subject are processed in the processes of steps S1005 to S1009 and then displayed on the display unit 114, the pixels may be processed and displayed pixel by pixel in a raster direction.
  • Fig. 11 is a diagram illustrating an example of an image displayed on the display unit 114 when the flowchart of Fig. 10 is performed.
  • Fig. 12 is a diagram illustrating a positional relationship between a subject displayed in Fig. 11 and the image processing device 100 in the front-back direction.
  • subjects 1100 to 1102 and a CG insertion region 1103 calculated from coordinate information are displayed.
  • the subject 1100, the subject 1101, and the subject 1102 are disposed to become more distant from the image processing device 100 in this order.
  • the subject 1100 is classified as a first layer of the distance layer MAP
  • the subject 1102 is classified as a third layer of the distance layer MAP.
  • Fig. 11 illustrates display in a case where the CG insertion layer is designated as the first layer of the distance layer MAP, and thus a pixel in a portion 1104 where the subject 1100 disposed on the first layer of the distance layer MAP and the CG insertion region 1103 overlap each other is displayed in a warning color.
  • a pixel in a portion 1105 where the subject 1101 disposed on the second layer of the distance layer MAP and the CG insertion region 1103 overlap each other the subject 1101 and the CG insertion layer are close to each other, and thus a background subject is displayed lightly.
  • the subject 1102 and the CG insertion layer are far from each other, and thus a background subject is displayed darkly.
  • the other CG insertion regions 1103 are displayed in a CG insertion color.
  • the distance layer MAP can be displayed so as to be easily understood by a photographer.
  • a program for implementing one or more functions in the present embodiment is supplied to a system or a device via a network or a storage medium, and one or more processors in a computer of the system or the device may read out and execute the program.
  • one or more functions may be implemented by a circuit (for example, an ASIC).
  • the state of the lens unit 106 will be described on the assumption that, for example, an aperture value is F5.6, a focal length is 50 mm, and a sensitivity ISO value at the time of imaging is 400.
  • a user has started an operation for setting a distance layer MAP in a state where a subject is 3 m away and is generally in focus.
  • Fig. 13 is a flowchart illustrating processing in the second embodiment
  • Fig. 14 is a flowchart illustrating a detailed flow of step S1301 in Fig. 13 . Note that operations of steps in the flowcharts of Figs. 13 and 14 are performed by causing a CPU 102 as a computer to execute computer programs stored in a ROM 103 or the like as a storage medium.
  • Fig. 13 when the user enters the mode for setting the number of layers and a layer width, and the processing starts, the user enters sub-processing for calculating a minimum resolution width M and a distance measurable range L in a distance layer MAP in step S1301.
  • this sub-processing will be described with reference to the flowchart of Fig. 14 .
  • Fig. 15 is a diagram illustrating a table in which examples of acquired lens information according to the second embodiment are collected.
  • first information (lens information 1) regarding a lens state is an aperture value of a lens (hereinafter referred to as an F value or an F number), and second information (lens information 2) is a focal length of the lens of which the unit is mm.
  • Third information (lens information 3) is positional information (in-focus position) on an image surface at the position of the focus lens, and a difference from a reference position is represented by an mm value.
  • the position of the focus lens for a subject at infinity is defined as a reference position (0 mm), and it indicates how far the current position of the focus lens is from that reference position.
  • a reference position (0 mm)
  • the position of the focus lens is 1 mm away from the reference position
  • the lens is focused on a subject at a distance of 3 m.
  • this depends on the focal length and the like of the lens and also changes depending on a zoom position.
  • Fourth information (lens information 3) regarding the lens state is the distance of the subject focused at the focus position of the focus lens, and the subject distance is acquired in m units. This can be calculated based on optical design information of the lens and is stored in the image processing device 100 as a design value. Note that, as described above, in the present embodiment, for example, a numerical value of 3 m is obtained.
  • Fifth information is relative image surface movement amount information from the current position of the focus lens to the infinite end or close end of the focus lens.
  • the information is data in units of mm with a sign, such as -1 mm to the infinite end and +9 mm to the close end.
  • a movement range of an image surface focus in the lens state is 10 mm from the infinite end of the lens to the close end.
  • the distance measurable limit information is an index indicating to what extent an image deviation amount (shift amount) can be detected as detection limit performance based on image surface phase difference technology.
  • a shift amount during a correlation operation may be changed depending on system conditions, and an image deviation of 10 shifts or more cannot be detected in a case where correlation operation processing is performed only with a shift amount of up to 20 shifts, and thus such an index is generated and stored.
  • the noise information is parameter information that greatly affects a noise level.
  • the noise information is an ISO setting value which is a sensitivity setting of a sensor.
  • step S1403 various table information as illustrated in Fig. 16 is obtained.
  • Figs. 16(A) and 16(B) are diagrams illustrating examples of tables acquired in step S1403.
  • Fig. 16(A) is a diagram in which an F value is shown in a left column, and a resolution on an image surface of a focus lens corresponding to the F value in the left column is shown in a right column in units of mm.
  • Fig. 16(B) is a diagram in which an ISO value is shown in a left column, and the degree of influence on a resolution on an image surface of a focus lens for the ISO value in the left column is shown in a right column in units of mm.
  • a minimum resolution width M of a subject distance in a distance layer MAP is calculated using the lens information in Fig. 15 and table information in Fig. 16 .
  • description is given on the assumption that an F value is 5.6, and an ISO value is 400.
  • the image surface resolution is 0.025 mm.
  • a noise component is 0.002 mm.
  • a minimum resolution width M converted into a subject distance is calculated.
  • the minimum resolution of 0.027 mm on the image surface is approximately 30 cm when converted into a distance from a subject position of 3 m and a lens state with a focal length of 50 mm.
  • a focus positional deviation on the image surface of 0.027 mm is set to be a minimum resolution, is a focus position shift of 0.027 mm on the image surface, and the minimum resolution width M converted into the subject distance is detectable as 30 cm.
  • Fig. 17 is a graph showing a resolution of an image surface in the second embodiment, in which the horizontal axis represents an aperture value of a lens (the right side is an aperture open side having a lower value), and the vertical axis represents an image surface resolution.
  • Reference numeral 281 denotes a curve plotting data in the table of Fig. 16 .
  • a resolution at an intersection A with an auxiliary line of F5.6 is 0.025 as shown in the table.
  • a point B is obtained, and a curve connecting this in an F value direction is 282, which is a curve of an image surface resolution after the noise correction in the case of ISO 400. That is, a resolution at the point B is 0.027 mm.
  • a distance measurable range L0 is calculated from lens edge information and table information.
  • the infinite end is 1 mm on an image surface
  • the close end is 9 mm.
  • the position of the image surface of each of the ends is converted into a subject distance, and the range thereof is calculated as a distance measurable range L0.
  • the conversion from the position of the image surface into the distance is calculated by the CPU 102 as described above.
  • a distance measurable range L1 of a distance layer MAP is calculated from distance measurable limit information and table information.
  • the distance measurable limit information is acquired as a correlation operation shift amount 20
  • the amount of fluctuation on the image surface becomes smaller even when the subjects moves by 50 cm, and it is not possible to distinguish whether the fluctuation is caused by noise or by the actual change in the subject distance. Thus, it is also required to add noise information.
  • the range of approximately 1.7 m to 8 m is a distance measurable range L1.
  • the subsequent description will be continued using the range of L1.
  • the above-described range of L0 is a distance measurable range due to lens restrictions
  • L1 is a distance measurable range due to correlation operation performance restrictions
  • a smaller one of them is a distance measurable range L for the image processing device 100.
  • a narrower one out of L0 and L1 is selected as a distance measurable range L.
  • the range L1 is obviously narrower, and both the infinity and close distances are within this range, and thus the distance measurable range L is also determined as a range of 6.3 m between 1.7 m and 8 m.
  • step S1301 sub-processing (step S1301) of calculating the minimum resolution width M converted into a subject distance and the distance measurable range L is performed, and layer information for each distance is generated based on lens information and distance information of a lens unit.
  • the lens information includes at least one of a focal length, an in-focus position, and aperture information.
  • step S1302 a setting mode selection screen for the user is displayed on, for example, the display unit 114. Subsequently, in step S1303, an operator (user) of the image processing device 100 is prompted to select whether to set the number of layers of a distance map, whether to set a layer width, or whether to set both.
  • steps 51302 and S1303 function as setting steps (setting units) for setting layer information. Note that, in steps S1302 and S1303 as setting units, it is only required that at least one of the number of layers and the width of a layer in a subject distance direction can be set as layer information for each distance.
  • FIG. 18(A) is a diagram illustrating an example of a selection screen for a layer parameter setting mode in the second embodiment.
  • reference numeral 1801 denotes the entire menu screen
  • reference numerals 1802 to 1804 denote options.
  • Fig. 18(B) is a diagram illustrating a menu screen in which the state of Fig. 18(A) has transitioned to a state where both the number of layers and a layer width are selected.
  • a menu screen 1805 shows a state where both the number of layers and a layer width are selected, and the option denoted by 1804 is displayed in a black-and-white reversed manner as denoted by 1806.
  • step S1303 the setting mode selection screen is displayed for the user, and when the user selects the setting mode on the menu screen, the set setting mode is determined in the next step S1304.
  • the processing proceeds to respective steps S1305 to S1307.
  • processing for validating a layer number change flag and a layer width change flag corresponding to each of the steps is performed and proceeds to each of setting processing modes.
  • Figs. 19 is a flowchart illustrating layer width setting processing in step S1305 of Fig. 13 in detail.
  • Figs. 20(A) to 20(F) are diagrams illustrating examples of display in the layer width setting mode.
  • Fig. 19 when the layer width setting mode is set, the above-described distance measurable range L and minimum resolution width M of the subject distance are acquired in step S1901. In the next step S1902, it is determined whether or not the layer width change flag is validated (On), and in a case where the layer width change flag is not validated, the flow of Fig. 19 is terminated.
  • the layer width change flag should be validated (On). Thus, a case where layer width change processing is performed and the flag is not validated results in error processing. Since the layer width change flag is validated (On) in a normal state, the processing proceeds to the next step S1903.
  • step S1903 a farthest distance value and a closest distance value at the time of displaying a menu are determined based on the distance measurable range L.
  • step S1904 only a layer width that can be designated is displayed in the menu based on the minimum resolution width M of the subject distance and the distance measurable range L.
  • a layer width for menu display is calculated using the minimum resolution width M of the subject distance, and the menu display is performed based on calculation results.
  • L/M is calculated, and thus a width when the layer width is set to be the minimum resolution width M is the layer width for performing the menu display.
  • a layer width which is larger than the minimum resolution width M and smaller than the distance measurable range L is a display candidate.
  • step S1904 functions as a step of setting a layer width as a reference for generating layer information and switching setting value display (layer width display) that can be set in accordance with lens information of a lens unit.
  • a minimum unit for pitching a layer width is set to 10 cm here, and thus, for example, when it is assumed that the minimum resolution width M is 30 cm and a step is 10 cm, candidates for the layer width can be calculated up to 30 cm, 40 cm, 50 cm,..., 310 cm.
  • the maximum value of 310 cm is half the maximum value of the distance measurable range L, and the number of layers is assumed to be 2. However, a maximum number on the menu is 20, and when the maximum number exceeds 20, a display target is limited in ascending order of a layer width.
  • the number of candidates is 20 from 30 cm to 20 in units of 10 cm.
  • the minimum number of layers may be 3, and the layer width may be the distance measurable range L/3. This is to insert a synthesized image before and after an in-focus position.
  • FIG. 20(A) A display state in this case is illustrated in Fig. 20(A) .
  • a layer width setting menu is displayed on the left side of a screen, and options for layer width setting are displayed. Note that all candidates can be selected by scroll display. A person as a subject being imaged is displayed in the vicinity of the center of the screen.
  • Fig. 20(D) is a diagram illustrating an example of display in a case where display is switched immediately after the layer width setting menu is displayed, and a depth direction at this time is schematically illustrated.
  • a closest distance of 170 cm and a farthest distance of 800 cm are displayed, and a minimum resolution of 30 cm and a subject at an in-focus position are simply displayed.
  • Figs. 20(A) and 20(D) the display is also switched by switching performed by the operation unit 113.
  • processing for switching display is performed using a layer width selected by the user.
  • a layer width which is set by moving a cursor is detected on the displayed menu.
  • a layer width that can be divided in accordance with the set layer width is displayed.
  • Fig. 20(B) when a width of 120 cm is set as illustrated in Fig. 20(B) , the location of a main subject in a layer is displayed as illustrated in Fig. 20(E) , and how to perform layer separation for a closest distance of 170 cm and a farthest distance of 800 cm is displayed in an easy-to-understand manner.
  • Fig. 20(C) is a diagram illustrating a state in a case where layer width setting is selected as 60 cm. A layer width divided in accordance with the layer width of 60 cm set in Fig. 20(F) is displayed.
  • step S1906 a layer width menu is selected in step S1905, and it is determined whether or not a layer width determination operation has been performed. In the case of No, the processing proceeds to step S1908 to check whether or not the cursor has been moved by the user in the layer width setting menu.
  • step S1906 the processing waits for the input of the layer width determination operation in step S1906.
  • step S1907 the processing proceeds to step S1907 to determine a layer width and the number of layers, and the flow of Fig. 19 is terminated.
  • step S1304 of Fig. 13 in a case where the layer number setting mode for selecting only the number of layers is selected, the processing proceeds to step S1306 of Fig. 13 to perform layer number setting processing.
  • Fig. 21 is a flowchart illustrating the layer number setting processing in step S1306 of Fig. 13 in detail, and Figs. 22(A) to 22(F) are diagrams illustrating examples of display in the layer number setting processing.
  • step S2102 it is determined whether the layer number change flag is validated (On), and in the case of No, the flow of Fig. 21 is terminated.
  • the layer number change flag should be validated (On). Thus, a case where layer number change processing is performed and the flag is not validated results in error processing. Since the layer number change flag is validated (On) in a normal state, the processing proceeds to step S2103, and farthest and closest distance values at the time of displaying the menu based on the distance measurable range L are determined.
  • the number of layers for displaying the menu is calculated based on the distance measurable range L and the minimum resolution width M of the subject distance, and only the number of layers that can be designated is displayed in the menu based on calculation results.
  • a maximum value of the number of layers can be calculated by calculating L/M.
  • L/M is 670 cm/30 cm, and a maximum of 22 layers are obtained.
  • a minimum value of the number of layers may be two. Note that there is a display limit on the menu screen, and there are a maximum of 20 candidates. Due to such restrictions, it is assumed that a display target is limited in ascending order of the number of layers. That is, the candidates are 2-layers, 3-layers, 4-layers,..., 21-layers.
  • Fig. 22(A) A display state in this case is illustrated in Fig. 22(A) .
  • a menu for setting the number of layers is displayed on the left side of the screen, and options for layer number setting are displayed. Note that all candidates can be selected by scroll display.
  • Fig. 22(A) is the same as Fig. 20 in the other respects, and thus description thereof will be omitted.
  • step S2104 functions as a step of setting a layer number as a reference for generating layer information and switching setting value display (layer number display) that can be set in accordance with lens information of a lens unit.
  • Fig. 22(D) is a diagram illustrating display immediately after the menu for setting the number of layers is displayed, and schematically illustrating a depth direction at this time. As an imageable range, a closest distance of 170 cm and a farthest distance of 800 cm are displayed, and a minimum resolution of 30 cm and a subject at an in-focus position are simply displayed.
  • Figs. 22(A) and 22(D) the display is also switched by operation performed by the operation unit 113. With such a menu display, processing for switching display is performed using the number of layers selected by the user. In the next step S2105, the number of layers which is set in association with the operation of a cursor is detected on the displayed menu. In addition, the number of layers divided in accordance with the set number of layers is displayed.
  • Fig. 22(B) when the setting is made to 5-layers as illustrated in Fig. 22(B) , the location of a main subject in a layer is displayed as illustrated in Fig. 22(E) , and how to perform layer separation for a closest distance of 170 cm and a farthest distance of 800 cm is displayed in an easy-to-understand manner.
  • Figs. 22(C) and 22(F) are diagrams illustrating a state where the setting of the number of layers is selected as three.
  • step S2106 the number of layers is selected from the menu in step S2105, and it is determined whether or not a layer number determination operation has been performed. In the case of No, the processing proceeds to step S2108, and the user checks whether or not the cursor has been moved in the layer number setting menu.
  • step S2105 the processing returns to step S2105, and the layer number setting menu is displayed again in the selected number of layers as described above. Further, in a case where the cursor has not been moved, the processing waits for the input of the layer number determination operation in step S2106.
  • step S2106 When it is determined in step S2106 that the layer number determination operation has been performed, the processing proceeds to the next step S2107 to calculate a layer width from the determined number of layers and the distance measurable range L, and the flow of Fig. 21 is terminated.
  • Fig. 23 is a flowchart illustrating layer number and layer width setting processing in step S1307 of Fig. 13 in detail
  • Figs. 24(A) to 24(E) are diagrams illustrating examples of display in a layer number and layer width setting mode.
  • step S2302 When the layer number and layer width setting processing mode is set, the flow of Fig. 23 is started, and the above-described distance measurable range L and minimum resolution width M are acquired in step S2301.
  • step S2302 first, the layer width setting menu is validated, and a cursor for setting is also moved to the layer width setting menu to select a layer width.
  • next step S2303 farthest and closest distance values at the time of displaying the menu based on the distance measurable range L are determined. Further, the number of layers for displaying the menu is calculated using the distance measurable range L and the minimum resolution width M, and only a layer width which is equal to or more than the minimum resolution and can be set within the distance measurable range is displayed in the menu.
  • a maximum value of the number of layers can be calculated by calculating L/M.
  • L/M is 670 cm/30 cm, and a maximum of 22 layers are obtained.
  • a minimum value of the number of layers may be two.
  • a layer width at the time of setting the minimum resolution width M is calculated by calculating L/M using the distance measurable range L and the minimum resolution width M. A width larger than the minimum resolution width M and smaller than the distance measurable range L is a candidate.
  • a minimum unit for pitching a layer width is set to 10 cm here, and thus, for example, when it is assumed that the minimum resolution width M is 30 cm and a step is 10 cm, candidates for the layer width at that time can be calculated up to 30 cm, 40 cm, 50 cm,..., 310 cm.
  • the maximum value of 310 cm is half the maximum value of the distance measurable range L, and the number of layers is assumed to be 2. However, a maximum number on the menu is 20, and when the maximum number exceeds 20, a display target is limited in ascending order of a layer width. That is, the number of candidates is 20 from 30 cm in units of 10 cm.
  • a menu display state at this time is illustrated in Fig. 24(A) . Both layer number setting and layer width setting are displayed side by side in the menu on the left side of the screen, and options of each of them are displayed. Note that all candidates can be selected by scroll display.
  • Fig. 24(D) is a diagram illustrating display immediately after a menu is displayed, and schematically illustrating a depth direction at this time. As an imageable range, a closest distance of 170 cm and a farthest distance of 800 cm are displayed, and a minimum resolution of 30 cm and a subject at an in-focus position are simply displayed.
  • Figs. 24(A) and 24(D) the display is also switched by operation performed by the operation unit 113.
  • processing for switching display is performed using the layer width and the number of layers selected by the user.
  • step S2304 only the number of layers that can be divided in the layer width is displayed in the layer number setting menu by using the layer width selected here.
  • a valid number of layers is displayed as bold black characters, while an invalid number of layers is not displayed or displayed in gray.
  • Fig. 24(B) illustrates a state where only a layer width is selected.
  • a menu for the number of layers of six or more is not displayed or displayed as gray characters.
  • a layer width is selected in the menu, and it is determined whether or not a layer width determination operation has been performed.
  • step S2306 the processing proceeds to step S2306 to check whether or not the cursor has been moved by the user in the layer width setting menu. In the case of Yes, it is determined that the layer width has been changed, and the processing returns to step S2304 to display the layer number menu again in the selected layer width as described above.
  • step S2305. the processing waits for the input of a layer width determination operation in step S2305.
  • step S2307 the processing proceeds to step S2307 to move a setting cursor to the layer number setting menu and set a state where the number of layers can be selected.
  • steps S2304 and S2308 function as steps for switching the display of a setting value (the display of a layer width and the number of layers) which can be set in accordance with lens information of a lens unit.
  • step S2309 it is determined whether or not a layer number determination operation has been performed. In the case of No, the processing proceeds to step S2311 to determine whether or not the cursor has been moved in the layer number setting menu. In the case of Yes, it is determined that the number of layers has been changed, and the processing returns to step S2308. In the case of No, the processing returns to step S2309.
  • step S2310 determines whether or not setting can be performed with the set number of layers and layer width. This is because the cursor can also be moved to a setting value which is not displayed or displayed in gray. In the case of No, that is, in the case of a combination that cannot be set, the processing returns to step S2302. In the case of Yes, the flow of Fig. 23 is terminated.
  • detectable distance conditions change depending on the focus position state, zoom state, and aperture state of a lens, and thus in a case where lens conditions have been changed in setting modes for the number of layers and a layer width having been described in the present embodiment, resetting of each of them is started immediately.
  • lens operations may not be accepted or may be negligible.
  • the colors of regions overlapping a subject in a synthesized image to be changed may be made different from each other.
  • the lens information has been changed by a predetermined amount or more
  • the number of layers (layer number) and a layer width (width in the distance direction) are recalculated.
  • the number of layers (layer number) and a layer width (width in the distance direction) are recalculated.
  • a subject recognized by an image recognition has moved by a predetermined amount or more and in a case where a background subject which is distant at a predetermined distance or more has moved by a predetermined value or more, it is desirable to adopt a configuration in which the number of layers (layer number) and a layer width (width in the distance direction) are recalculated by detecting the movement of the subject and the change in the subject.
  • an operation for increasing the amount of lens information such as a focal length, an aperture value, and a focus adjustment operation, to a predetermined amount or more may not be accepted.
  • a computer program realizing the function of the embodiments described above may be supplied to an image processing device or the like through a network or various storage media. Then, a computer (or a CPU, an MPU, or the like) of the image processing device or the like may be configured to read and execute the program. In such a case, the program and the storage medium storing the program configure the present invention.
  • the image processing device includes an image acquisition unit configured to acquire an image including a subject through a lens unit, a distance information acquisition unit configured to acquire distance information indicating a distance to the subject, a layer information generation unit configured to generate layer information on a layer for each distance based on the distance information, and a setting unit configured to set a reference for generating the layer information and switch display of a setting value capable of being set in accordance with the lens information of the lens unit.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Studio Devices (AREA)
EP22211880.4A 2022-01-04 2022-12-07 Dispositif de traitement d'image, procédé de traitement d'image et support d'informations Pending EP4258638A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2022000224 2022-01-04
JP2022000259 2022-01-04
JP2022000249 2022-01-04
JP2022183629A JP2023099993A (ja) 2022-01-04 2022-11-16 画像処理装置、画像処理方法、及びコンピュータプログラム

Publications (1)

Publication Number Publication Date
EP4258638A1 true EP4258638A1 (fr) 2023-10-11

Family

ID=84488554

Family Applications (1)

Application Number Title Priority Date Filing Date
EP22211880.4A Pending EP4258638A1 (fr) 2022-01-04 2022-12-07 Dispositif de traitement d'image, procédé de traitement d'image et support d'informations

Country Status (3)

Country Link
US (1) US20230216983A1 (fr)
EP (1) EP4258638A1 (fr)
CN (1) CN116402664A (fr)

Citations (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014149484A (ja) * 2013-02-04 2014-08-21 Canon Inc 撮像システムおよびその制御方法
US20160227121A1 (en) * 2015-01-29 2016-08-04 Canon Kabushiki Kaisha Image processing apparatus and image processing method
JP2018029270A (ja) 2016-08-18 2018-02-22 キヤノン株式会社 画像処理装置およびその制御方法、撮像装置、プログラム
US20190058827A1 (en) * 2017-08-18 2019-02-21 Samsung Electronics Co., Ltd. Apparatus for editing image using depth map and method thereof
JP2021048560A (ja) 2019-09-20 2021-03-25 キヤノン株式会社 画像処理装置および画像処理方法、撮像装置、プログラム
JP2022000259A (ja) 2018-06-04 2022-01-04 京楽産業．株式会社 遊技機
JP2022000249A (ja) 2015-01-30 2022-01-04 株式会社三洋物産 遊技機
JP2022000224A (ja) 2017-09-15 2022-01-04 株式会社三洋物産 遊技機
JP2022183629A (ja) 2021-05-31 2022-12-13 矢崎総業株式会社 充電装置

Patent Citations (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2014149484A (ja) * 2013-02-04 2014-08-21 Canon Inc 撮像システムおよびその制御方法
US20160227121A1 (en) * 2015-01-29 2016-08-04 Canon Kabushiki Kaisha Image processing apparatus and image processing method
JP2016143919A (ja) 2015-01-29 2016-08-08 キヤノン株式会社 画像処理装置およびその制御方法
JP2022000249A (ja) 2015-01-30 2022-01-04 株式会社三洋物産 遊技機
JP2018029270A (ja) 2016-08-18 2018-02-22 キヤノン株式会社 画像処理装置およびその制御方法、撮像装置、プログラム
US20190058827A1 (en) * 2017-08-18 2019-02-21 Samsung Electronics Co., Ltd. Apparatus for editing image using depth map and method thereof
JP2022000224A (ja) 2017-09-15 2022-01-04 株式会社三洋物産 遊技機
JP2022000259A (ja) 2018-06-04 2022-01-04 京楽産業．株式会社 遊技機
JP2021048560A (ja) 2019-09-20 2021-03-25 キヤノン株式会社 画像処理装置および画像処理方法、撮像装置、プログラム
JP2022183629A (ja) 2021-05-31 2022-12-13 矢崎総業株式会社 充電装置

Also Published As

Publication number Publication date
CN116402664A (zh) 2023-07-07
US20230216983A1 (en) 2023-07-06

Similar Documents

Publication Publication Date Title
US8224069B2 (en) Image processing apparatus, image matching method, and computer-readable recording medium
JP5809925B2 (ja) 画像処理装置、それを備えた画像表示装置及び撮像装置、画像処理方法、並びに画像処理プログラム
US9270902B2 (en) Image processing apparatus, image capturing apparatus, image processing method, and storage medium for obtaining information on focus control of a subject
JP6371553B2 (ja) 映像表示装置および映像表示システム
US20120105590A1 (en) Electronic equipment
KR101679290B1 (ko) 영상 처리 방법 및 장치
CN114245905A (zh) 深度感知照片编辑
US10585533B1 (en) Method and apparatus for creating an adaptive Bayer pattern
US20130335535A1 (en) Digital 3d camera using periodic illumination
US20120154551A1 (en) Stereo image display system, stereo imaging apparatus and stereo display apparatus
EP2757789A1 (fr) Système de traitement d'image, procédé de traitement d'image et programme de traitement d'image
KR101642402B1 (ko) 촬영 구도를 유도하는 디지털 영상 촬영 장치 및 방법
US8774551B2 (en) Image processing apparatus and image processing method for reducing noise
US10984550B2 (en) Image processing device, image processing method, recording medium storing image processing program and image pickup apparatus
WO2011014421A2 (fr) Procédés, systèmes et supports de stockage lisibles par ordinateur permettant de générer un contenu stéréoscopique par création d’une carte de profondeur
JP2019015575A (ja) 画像処理装置、測距装置および処理システム
US9172860B2 (en) Computational camera and method for setting multiple focus planes in a captured image
US11323689B2 (en) Image processing device, imaging device, image processing method, and recording medium
JP6429483B2 (ja) 情報処理装置、撮像装置、情報処理システム、情報処理方法およびプログラム
EP4258638A1 (fr) Dispositif de traitement d'image, procédé de traitement d'image et support d'informations
JP2023099993A (ja) 画像処理装置、画像処理方法、及びコンピュータプログラム
JP5741353B2 (ja) 画像処理システム、画像処理方法および画像処理プログラム
BR102022026750A2 (pt) Dispositivo de processamento de imagem, método de processamento de imagem, e meio de armazenamento
US20230300474A1 (en) Image processing apparatus, image processing method, and storage medium
US12032792B1 (en) Method and apparatus for creating an adaptive Bayer pattern

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN PUBLISHED

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20240411

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC ME MK MT NL NO PL PT RO RS SE SI SK SM TR