Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/73—Deblurring; Sharpening
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N5/00—Details of television systems
  • H04N5/76—Television signal recording
  • H04N5/765—Interface circuits between an apparatus for recording and another apparatus
  • H04N5/77—Interface circuits between an apparatus for recording and another apparatus between a recording apparatus and a television camera
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
  • H04N23/661—Transmitting camera control signals through networks, e.g. control via the Internet
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/95—Computational photography systems, e.g. light-field imaging systems
  • H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N9/00—Details of colour television systems
  • H04N9/79—Processing of colour television signals in connection with recording
  • H04N9/80—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback
  • H04N9/804—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving pulse code modulation of the colour picture signal components
  • H04N9/8042—Transformation of the television signal for recording, e.g. modulation, frequency changing; Inverse transformation for playback involving pulse code modulation of the colour picture signal components involving data reduction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20221—Image fusion; Image merging

Definitions

• the present invention relates to an image processing apparatus, a control method for the image processing apparatus, and a storage medium.
• NPL Non Patent Literature 1
• the image pickup apparatus described in NPL 1 obtains pixel signals based on light having passed through the micro lenses, and generates image data based on the pixel signals.
• the obtained pixel signals and the generated image data are stored in, for example, the image pickup apparatus.
• the image pickup apparatus described in NPL 1 also has a refocusing function of changing a focal position of generated image data to an arbitrary position based on the stored pixel signals.
• Non Patent Literature [0003] ⁇ NTL 1 ⁇ Ren Ng et al . , Light Field Photography with a Hand-Held Plenoptic Camera, Standard Tech Report CTSR 2005-02.
• the present invention provides an image
• the present invention provides an image processing apparatus comprising an obtaining unit configured to obtain a plurality of pieces of unprocessed data on which no developing process has been carried out and processed data on which a developing process has been carried out, an identifying unit configured to carry out a subject recognition process on the processed data to identify a main subject from among a plurality of
• a developing process unit configured to carry out a developing process on the plurality of pieces of unprocessed data
• a generation unit configured to, based on a phase
• a subject recognition process is carried out to automatically identify a main subject from among a plurality of
• a refocusing process is carried out so as to focus on the main subject.
• a refocusing process for an image is allowed to be carried out without any instruction from a user.
• FIG. 1 is a block diagram schematically showing an arrangement of an image processing apparatus according to an embodiment of the present invention.
• FIG. 2 is a view useful in explaining RAW data generated by an image pickup device of the image processing apparatus according to the embodiment of the present invention.
• FIG. 3 is a view useful in explaining reference data stored in a storage device of the image processing apparatus according to the embodiment of the present invention.
• FIG. 4 is a view useful in explaining a reference data management file which manages the
• FIG. 5 is a view useful in explaining image data which is subjected to refocusing in the embodiment of the present invention.
• FIG. 6 is a flowchart showing the
• FIG. 7 is a block diagram schematically showing an arrangement of an image processing system in which the image processing apparatus according to the second embodiment of the present invention is connected to a server.
• FIG. 8 is a view useful in explaining image data which is subjected to refocusing in the second embodiment of the present invention.
• FIG. 9 is a view useful in explaining an exemplary screen of a remote user' s terminal which is connected to the image processing apparatus according to the second embodiment of the present invention via the server .
• FIG. 10 is a view useful in explaining an exemplary screen of a source user' s terminal which is connected to the image processing apparatus according to the second embodiment of the present invention via the server .
• FIG. 1 is a block diagram schematically showing an arrangement of an image processing apparatus 100 according to the first embodiment of the present
• the image processing apparatus 100 is a computer
• the image processing apparatus 100 in FIG. 1 has a CPU 101, an image pickup device 102, an operating unit 103, and a display. unit 104, and these component elements are connected to one another via a bus 107.
• the image processing apparatus 100 also has a communication unit 108, an optical system 109, and a storage device 110, the communication unit 108 and the optical system 109 are connected to the CPU 101, and the storage device 110 is connected to the image pickup device 102.
• the CPU 101 is a control device for controlling the component elements of the image processing apparatus 100 and executing programs stored in the storage device 110.
• a CPU central processing unit
• the image pickup device 102 converts light having passed through the optical system 109 to pixel signals and generates image data.
• the operating unit 103 has hardware keys and an operating panel, and a user inputs instructions to the image processing apparatus 100 through the hardware keys and the operating panel.
• the display unit 104 displays live-view images, picked-up and recorded still images, picked-up and recorded moving images, operating guidance, or the like.
• the CPU 101 controls the component elements to perform various functions.
• the CPU 101 carries out a resizing process such as predetermined pixel interpolation or reduction and a color conversion process on image data.
• the CPU 101 also carries out an object recognition process (subject recognition process) to detect a subject, for example, a human face from image data.
• the CPU 101 carries out a refocusing process in which it generates image data with a focus on a detected subject and stores the image data obtained as a result of refocusing in the storage device 110.
• the communication unit 108 is connected to and sends and receives control commands and data to and from an external apparatus via, for example, a USB (Universal Serial Bus) cable or a wireless LAN. Control commands and data are sent and received using a protocol such as PTP (Picture Transfer Protocol), MTP (Media Transfer Protocol) , or NFC (Near-Field Communication) .
• the optical system 109 has a lens, a shutter, and a diaphragm and light having passed through the optical system 109 is converted to pixel signals by the image pickup device 102
• the storage device 100 stores various programs, reference data on a subject (hereafter referred to merely as "reference data") , or image data, for example, RAW data 501 and 502 and a proxy image (FIG. 5), to be described later.
• RAW data 501 and 502 are obtained by recording data substantially as they are immediately after they are output from the image pickup device 102 and have high resolution and high tone as compared to the proxy image 503.
• RAW data is obtained by converting data output from an image pickup device into a compressed state through highly-efficient coding using a technique such as wavelet conversion or difference coding.
• RAW data is lossless compressed data.
• a proxy image which is, for example, image data in a JPEG format, is obtained by not only converting data output from an image pickup device into a compressed state through highly-efficient coding using a technique such as wavelet conversion or difference coding but also subjecting the data further to a developing process and a JPGE compression process and recording the same.
• RAW data and a proxy image are
• a proxy image may be generated by carrying out a developing process on RAW data using a predetermined adjustment value.
• RAW data may also be referred to as unprocessed data
• a proxy image may also be referred to as processed data.
• RAW data is characterized by little degradation with respect to data obtained from an image pickup device but having a large data size as compared to a proxy image.
• the RAW data 501 and 502 and the proxy image 503 based on the same pixel signals are stored together in one file or stored in respective files, which are in turn associated with one another.
• FIG. 2 is a view useful in explaining RAW data generated by the image pickup device 102 of the image processing apparatus 100 according to the present
• photodiodes 203 and 204 light detectors of two types which detect light 202 obtained through a microlens 201.
• One photodiode 203 and one photodiode 204 form a pair (a picture element) .
• the photodiode 203 and the photodiode 204 function independently of each other, and each of the photodiodes 203 detects the light 202 to generate the RAW data 501, to be described later, and each of the
• photodiodes 204 detects the light 202 to generate the RAW data 502, to be described later,
• the photodiodes 203 and 204 detect the same light 202 at different positions, and hence the incident angle of the light 202 incident on the photodiode 203 and the incident angle of the light 202 incident on the photodiode 204 are different. This causes a phase difference between the light 202 detected by the photodiode 203 and the light 202 detected by the photodiode 204, and this results in a phase difference between the RAW data 501 and the RAW data 502 as well.
• FIG. 3 is a view useful in explaining reference data 300 stored in the storage device 110 in FIG. 1.
• the reference data 300 is information on main subjects.
• the main subjects are detected from the proxy image 503 by carrying out a subject recognition process on the proxy image 503 based on the reference data 300.
• each piece of the image data recorded as the reference data 300 is compared with image data on a specific subject (identified subject) included in the proximity image 503. Based on the comparison results, whether or not the identified subject in the proxy image 503 matches the face of the person 301 is determined, and when they match, it is decided that the identified subject in the proxy image 503 is a desired main subject.
• the reference data 300 may be comprised of feature quantities 405 (FIG. 4), to be described later, in place of image data.
• a feature quantity calculated by analyzing the image data on the identified subject in the proxy image 503 and the feature quantities 405 represented by the reference data 300 may be referred to, and whether or not they match each other may be determined.
• the reference data 300 as well as features of the respective five pieces of recorded image data on the face is recorded in a reference data management file 400 in FIG. 4.
• the reference data management file 400 manages not only the reference data 300 on the certain person 301 but also the reference data 300 on other persons, that is, reference data on faces of a plurality of persons.
• reference data names 401 of reference data on respective human faces, dates and times of reference data updating 402, image data names 403, dates and times of shooting 404, and feature quantities 405 on human faces are recorded .
• the reference data names 401 are names of respective pieces of reference data.
• dates and times of reference data updating 402 dates and times at which image data recorded as reference data was updated are recorded.
• the image data names 403 are names of image data recorded as reference data, and the dates and times of shooting 404 are dates and times at which image data recorded as reference data was taken by shooting.
• feature quantities 405 features of respective pieces of image data on five faces recorded as the reference data 300, for example, indexes obtained by converting ages and degrees of delight, anger, romance, and pleasure emotions on facial expressions of persons in image data into numeric values.
• FIG. 5 is a view useful in explaining a process in which a subject identified based on the reference data 300 in FIG. 3 is refocused.
• the image processing apparatus 100 generates the RAW data 501 and 502 in FIG. 5 and generates the proxy image 503 from the RAW data 501.
• the CPU 101 carries out a compression process on the RAW data 501 to generate reduced RAW data.
• the CPU 101 then subjects the reduced RAW data to a developing process in which it converts the reduced RAW data to a predetermined file format such as JPEG or TIFF, thus generating the proxy image 503.
• the developing process is carried out using a predetermined adjustment value.
• the proxy image 503 should not always be generated from reduced RAW data.
• RAW data may be generated from pixel signals output from the image pickup device 102, and the same pixel signals may be subjected directly to a compression process and a
• the same pixel signals as RAW data may be subjected directly to only a compression process to generate and store reduced RAW data in advance, and after that, as the need arises, a developing process may be carried out on the reduced RAW data to generate a proxy image.
• the developing process for generating the proxy image 503 includes a process to adjust parameters such as contrast, white balance, color balance, lightness, and saturation.
• the proxy image 503 is generated based on data obtained by subjecting the RAW data 501 to a compression process, and the data amount of the proxy image 503 is smaller than that of an image obtained by directly subjecting the RAW data 501 to a developing process. Therefore, the time and load required to display the proxy image 503 on the display unit 104 and carry out a subject recognition process on the proxy image 503 is smaller than in the case where an image obtained by as a result of a developing process is
• FIG. 6 is a flowchart showing the procedure of a refocusing process which is carried out by the CPU 101 in FIG. 1.
• the refocusing process in FIG. 6 is based on the premise that in the proxy image 503, the subject 504 in FIG. 5 is focused on, the subject 505 is out of focus, and the subject 505 is an object to be refocused on.
• image data 508 with the subject 505 focused on and the subject 504 out of focus is obtained.
• the photodiodes 203 and 204 detect the light 202 independently of each other to generate the two types of RAW data 501 and 502. And the CPU 101 develops the RAW data 501 (hereafter referred to as "RAW development") to obtain the proxy image 503 (step S601) .
• the CPU 101 automatically detects an identified subject from the proxy image 503 (step S602).
• the CPU 101 detects the subjects 504 and 505 by carrying out a subject recognition process in which it detects a human face as an identified subject from the proxy image 503.
• the CPU 101 may detect a plurality of identified subjects (not shown) as well as the subjects 504 and 505.
• the CPU 101 refers to reference data stored in the storage device 110.
• reference data 300a on a person 301a and reference data 300b on a person 301b are referred to as shown in FIG. 5.
• the CPU 101 identifies the subject 505 corresponding to the person 301b as a main subject from among the plurality of
• the CPU 101 may use a G image pickup device, a B image pickup device, an R image pickup device, and so on as well as the proxy image 503. [0036] As described above, the CPU 101 automatically identifies the subject 505 as a main subject using the reference data 300b from among the plurality of
• the CPU 101 performs refocusing on the RAW data. Moreover, the user is saved from having to manually choose the subject 505 that should be focused on, and the time and load required for the user to perform refocusing are reduced. Further, since the subject 505 to be refocused on is identified based on the reference data 300b, a subject unintended by the user is prevented from being focused on.
• the CPU 101 calculates the amount of parallax which is a phase
• step S604 performs predetermined computations on the amount of parallax to calculate the amount of defocus (step S605) .
• a method to calculate the amount of defocus is well known due to the publication of the prior art (Japanese Laid-Open Patent Publication (Kokai) No. 2008-15754), and hence
• the CPU 101 carries out a developing process on each of the RAW data 501 and 502 to obtain two types of RAW development results and synthesizes these two types of RAW development results so that the area of the subject 505 which is the main subject can be focused on.
• the CPU 101 synthesizes the two types of RAW development results so that the absolute value of the amount of defocus calculated in the step S605 can be equal to or smaller than a predetermined value, thus generating image data 508 (refocused image) in which the area of the subject 505 is refocused on (step S606) , and terminates the present process. If the amount of defocus is not equal to or smaller than the predetermined value, the two types of RAW development results are synthesized so that the amount of defocus can be minimum.
• the CPU 101 can obtain high- quality image data with a focus on the subject 504 from the RAW data 501 and 502 that have not been subjected to a compression process. It should be noted that for all subjects identified using the reference data 300, the CPU 101 may generate image data based on RAW development results. Alternatively, only for subjects in the
• reference data 300 which satisfy predetermined conditions the CPU 101 may generate image data based on RAW
• the second embodiment of the present invention is basically the same as the first embodiment described above in terms of constructions and operations, differing from the first embodiment in that an image processing apparatus is connected to a server to constitute an image processing system.
• image data taken by a certain person using an image pickup apparatus may be shared with another person using a smartphone.
• image data is transferred from the image pickup apparatus to the smartphone by way of a server, but in the image data, a subject which is in focus and a subject which is desired to be brought into focus by the other person may be different.
• a refocusing process is carried out in the present embodiment.
• FIG. 7 is a block diagram schematically showing an arrangement of the image processing system in which the image processing apparatus 100 according to the second embodiment of the present invention is connected to the server 700.
• the present embodiment differs from the first embodiment in that the CPU 101 of the image processing apparatus 100 according to the present
• the server 700 has a CPU 701, a communication unit 707, and a storage device 708, and the communication unit 707 and the storage device 708 are connected to each other via the CPU 701.
• the CPU 101 executes programs stored in the storage device 708 and controls the component elements of the server 700 to perform various functions.
• the CPU 101 executes programs stored in the storage device 708 and controls the component elements of the server 700 to perform various functions.
• Communication unit 707 is connected to and sends and receives control commands and data to and from the communication unit 108 of the image processing apparatus 100.
• Image data and personal information required to share image data with a third party such as an e-mail address are stored in the storage device 708.
• an entity file of image data to be shared with a third party may be stored in an external storage device, and in this case, a file path to the entity file of the image data is stored in the storage device 708.
• Image data to be stored in the storage device 708 is obtained by the CPU 701 via the communication unit 707.
• an OS various programs such as application programs, the reference data 300a and 300b described above, image data, for example, RAW data 802 and 803 and a proxy image 804, to be described later, are stored in the storage device 708.
• the RAW data 802 and 803 and the proxy image 804 are data and an image of which contents are the same as those of the RAW data 501 and 502 and the proxy image 503 described above.
• the CPU 701 carries out a resizing process such as predetermined pixel interpolation or reduction and a color conversion process on image data received via the communication unit 707.
• the CPU 701 also carries out a subject recognition process to detect a subject, for example, a human face from image data.
• the CPU 701 then carries out a refocusing process in which it generates image data with a focus on the detected subject and stores the image data obtained by the refocusing process in the storage device 708.
• FIG. 8 is a view useful in explaining image data which is subjected to refocusing by the server 700 in FIG. 7.
• FIG. 8 it is assumed that there are a user who performs shooting using an image pickup apparatus 801
• the source user (hereafter referred to as "the source user) and a remote user who is away from the source user and uses a
• the image pickup apparatus 801 sends RAW data 802 and 803 and a proxy image 804 to the server 700.
• the CPU 701 of the server 700 detects identified subjects from the proxy image 804.
• a plurality of subjects (not shown) as well as the subjects 805 and 806 are detected as identified subjects from the proxy image 804.
• the CPU 701 refers to the reference data 300a and 300b stored in the storage device 708 to identify subjects 805 and 806 as main subjects
• the CPU 701 of the server 700 carries out the refocusing process (FIG. 6) described above to obtain image data 807 with a focus on the subject 806.
• the CPU 701 sends the image data 807 to the smartphone 808 via the communication unit 707.
• the remote user obtains the image data 807 with a focus on the subject 806 as he or she desires without requesting the source user to perform refocusing.
• FIG. 9 is a view useful in explaining an application screen 900 which is an exemplary screen displayed on the smartphone 808 of the remote user who receives image data.
• the application screen 900 is displayed on a display unit of the smartphone 808 of the remote user when the server 700 in FIG. 7 sends image information generated based on reference data or the like stored in the storage device 708 to the smartphone 808 of the remote user.
• the application screen 900 has a tree view 901, a browser window 902, and an action button display section 903.
• a "person” folder in the tree view 901 two folders with folder names "TARO” and "HANAKO” are stored.
• Reference data on a person named “TARO” is stored in the "TARO” folder
• reference data on a person named “HANAKO” is stored in the "HANAKO” folder .
• the browser window 902 displays reference data stored in the "TARO" folder.
• the reference data corresponds to the reference data 300a described above and is used to identify a main subject from among a plurality of
• the action button display section 903 displays a plurality of buttons for use in inputting instructions to carry out predetermined processes on images displayed in the browser window 902. For example, when a RAW development button 903a is depressed with the "TARO" folder selected, the server 700 carries out the buttons
• FIG. 10 is a view useful in explaining an image sending screen 1000 which is an exemplary screen
• the image sending screen 1000 is displayed on a display unit of the source user' s terminal when the server 700 sends image information for use in accepting an instruction to send image data with each subject refocused on to a terminal of a third party.
• the image sending screen 1000 is displayed on a display unit of the terminal of the source user such as a PC or a smartphone, and for example, used when the user of the image pickup apparatus 801 issues an instruction to send image data to the remote user.
• the image sending screen 1000 has a link 1001, a send button 1002, and a link 1003
• the source user of the image pickup apparatus 801 selects the link 1001 and uploads image data to the server 700 for each of events such as a sports day and an excursion.
• the source user specifies an address of the remote user who receives the image data and depresses the send button 1002.
• the server 700 carries out the refocusing process in FIG. 6 on the uploaded image data and sends image data, in which a subject identified based on reference data corresponding to the remote user who is the destination is refocused on, or a link thereto to the smartphone 808 of the remote user.
• the address of the remote user is associated with the reference data and stored and managed in the storage device 708.
• the source user is allowed to select addresses of a plurality of remote users, and when the link 1003 is selected, a screen (not shown) for
• aspects of the present invention can also be realized by a computer of a system or apparatus (or devices such as a CPU or MPU) that reads out and executes a program recorded on a memory device to perform the functions of the above-described embodiment ( s ) , and by a method, the steps of which are performed by a computer of a system or apparatus by, for example, reading out and executing a program recorded on a memory device to perform the functions of the above-described
• the program is provided to the computer for example via a network or from a recording medium of various types serving as the memory device (e.g., computer-readable medium).

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Computing Systems (AREA)
• Studio Devices (AREA)
• Television Signal Processing For Recording (AREA)

Priority Applications (1)

Applications Claiming Priority (2)

Publications (1)

Family

ID=55399723

Family Applications (1)

Country Status (3)

Families Citing this family (2)

Citations (3)

Family Cites Families (5)

• 2014
  • 2014-08-26 JP JP2014171941A patent/JP2016046777A/ja active Pending
• 2015
  • 2015-08-19 US US15/504,713 patent/US10217196B2/en active Active
  • 2015-08-19 WO PCT/JP2015/073921 patent/WO2016031837A1/en not_active Ceased

Patent Citations (3)

Also Published As

Similar Documents

Legal Events