US7898591B2 - Method and apparatus for imaging using sensitivity coefficients - Google Patents
Method and apparatus for imaging using sensitivity coefficients Download PDFInfo
- Publication number
- US7898591B2 US7898591B2 US12/640,986 US64098609A US7898591B2 US 7898591 B2 US7898591 B2 US 7898591B2 US 64098609 A US64098609 A US 64098609A US 7898591 B2 US7898591 B2 US 7898591B2
- Authority
- US
- United States
- Prior art keywords
- image pickup
- photodetectors
- image
- imaginary light
- photodetector
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
- H04N23/81—Camera processing pipelines; Components thereof for suppressing or minimising disturbance in the image signal generation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N9/00—Details of colour television systems
- H04N9/12—Picture reproducers
- H04N9/31—Projection devices for colour picture display, e.g. using electronic spatial light modulators [ESLM]
- H04N9/3141—Constructional details thereof
- H04N9/3173—Constructional details thereof wherein the projection device is specially adapted for enhanced portability
- H04N9/3176—Constructional details thereof wherein the projection device is specially adapted for enhanced portability wherein the projection device is incorporated in a camera
Definitions
- the present disclosure relates to an image pickup technology using a solid-state image pickup element.
- a related compound-eye image pickup device using solid-state image pickup elements uses a structure in which a plurality of lenses and a set of photodetectors (disposed for the respective lenses) are disposed so as to be separated from each other by a focal length of each lens, and, in which, from a plurality of images picked up by the set of photodetectors, parallax information of each image is used to reproduce one image.
- the related compound-eye image pickup device having such a structure has the following problems:
- the image pickup device Since it is necessary to dispose the photodetectors so as to be separated from the lenses by the focal length, the image pickup device becomes thick. Although it is possible to shorten the focal length by using lenses having high refractive indices, aberrations, such as chromatic aberration, are inevitably increased, thereby reducing image quality of a picked up image;
- a step for aligning the lens system and the photodetectors when manufacturing the image pickup device is required. This step causes costs of a product to be increased.
- the image reproducing section includes part for obtaining the signals that are output by the image pickup unit; part for setting a plurality of imaginary light spots in an image pickup area so that a number m of imaginary light spots ⁇ the number n of photodetectors; part for determining the sensitivity coefficient corresponding to each imaginary light spot for each photodetector on the basis of the information stored in the sensitivity coefficient storage section; part for determining u i so that E ev determined by the following formula is equal to or less than a threshold value ⁇ :
- a light beam of an imaginary light spot i is u i
- the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is ⁇ ij
- the distance between the imaginary light spot i and the photodetector j is r ij
- the signal corresponding to the photodetector j that is output by the image pickup unit is S j ; and part for forming and outputting the image using the determined u i .
- the lenses may be superposed upon the respective photodetectors so that light is incident upon only the respective photodetectors.
- the image reproducing section includes part for obtaining the signals that are output by the image pickup unit; part for setting a plurality of imaginary light spots in an image pickup area so that a number m of imaginary light spots ⁇ the number n of photodetectors; part for determining the sensitivity coefficient corresponding to each imaginary light spot for each photodetector on the basis of the information stored in the sensitivity coefficient storage section; part for determining u i so that E ev determined by the following formula is equal to or less than a threshold value ⁇ :
- a light beam of an imaginary light spot i is u i
- the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is ⁇ ij
- the distance between the imaginary light spot i and the photodetector j is r ij
- the signal corresponding to the photodetector j that is output by the image pickup unit is S j ; and part for forming and outputting the image using the determined u i .
- the image pickup device may further include a display unit that displays the output image, wherein the photodetectors are disposed at a display screen of the display unit.
- the image pickup device may be such that the information stored in the sensitivity coefficient storage section includes at least information regarding a function f( ⁇ , ⁇ ) indicating the characteristics of the lenses, where, in a three-dimensional coordinate system in which a light-receiving plane of the photodetectors matches an XY plane, ⁇ is an angle between an XZ plane and a direction from the center of each photodetector to each imaginary light spot, and ⁇ is an angle between a YZ plane and the direction from each photodetector to each imaginary light spot.
- a function f( ⁇ , ⁇ ) indicating the characteristics of the lenses
- the part for setting the imaginary light spots may set the image pickup area or the imaginary light spots on the basis of an input of a user.
- the image pickup device may further include an image pickup signal storage section that stores the signals output by the image pickup unit, wherein, if a shooting instruction is given by a user, the part for obtaining the signals stores the signals obtained from the image pickup unit in the image pickup signal storage section, and wherein, if an instruction of image reproduction is given from the user, the part for determining u i reads out the signal S j corresponding to the photodetector j output by the image pickup unit from the image pickup signal storage section.
- a light beam of an imaginary light spot i is u i
- the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is ⁇ ij
- the distance between the imaginary light spot i and the photodetector j is r ij
- the signal corresponding to the photodetector j that is output by the image pickup unit is S j ; and part for forming and outputting an image using the determined u i .
- a light beam of an imaginary light spot i is u i
- the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is ⁇ ij
- the distance between the imaginary light spot i and the photodetector j is r ij
- the signal corresponding to the photodetector j that is output from the image pickup unit is S j ; and forming and outputting the image using the determined u i .
- part not only simply refers to “physical part,” but also refers to a case in which the function of the part is realized by software.
- function of one part or one device may be realized by two or more physical parts or physical devices, or the functions of two or more parts or devices may be realized by one physical part or one physical device.
- FIG. 1 shows the structure of hardware of an image pickup device 1 ;
- FIG. 2 shows lenses 2 and photodetectors 4 being superposed upon each other
- FIG. 3 shows a functional structure of a controller 8 ;
- FIG. 4A shows an image pickup area 401 that is set in a light receiving area 402 , and a plurality of cells C that are set in the image pickup area 401 ;
- FIG. 4B shows possible imaginary light spots that are set at a plurality of distances with respect to the cells C;
- FIG. 5 is a flowchart of operations of an image reproducing section 23 of the image pickup device 1 ;
- FIG. 6 is a flowchart of operations of the image reproducing section 23 of the image pickup device 1 ;
- FIGS. 7A and 7B illustrate the relationships between the position of a photodetector 601 and a light source L relative to each other;
- FIG. 7C shows a table that stores f( ⁇ , ⁇ ) values
- FIG. 8A illustrates an embodiment in which an image pickup unit and a display device are integrated to each other
- FIG. 8B illustrates a scene in which the embodiment shown in FIG. 8A is used.
- FIG. 8C illustrates a scene in which a related television telephone device is used.
- FIG. 1 shows the structure of hardware of the image pickup device 1 .
- the image pickup device 1 is formed so as to include an image pickup unit 5 , a controller 8 that reproduces an image on the basis of an output signal that is output by the image pickup unit 5 , a liquid crystal monitor that displays information of, for example, a reproduced image, and a transportable storage medium (such as a memory card) 11 that stores data of, for example, the reproduced image or the output signal from the image pickup unit 5 .
- the image pickup device 1 includes mechanisms (such as a shutter mechanism for a pickup operation) similar to the mechanisms of related image pickup devices, these mechanisms will not be illustrated.
- photodetectors 4 photodiodes or the like may be used similarly to related image sensors.
- the photodetectors 4 are disposed in an array in a plane on a base substrate 12 in the image pickup unit 5 .
- a mechanism for outputting a signal based on the quantity of light received by each photodetector 4 for example, a mechanism (charge coupled device (CCD) unit) that transfers a signal subjected to photoelectric conversion at each photodetector 4 , a complementary metal oxide semiconductor (CMOS) device, a circuit that performs A/D conversion on the signal subjected to the photoelectric conversion, or a circuit that removes noise may also be disposed on the base substrate.
- CCD charge coupled device
- CMOS complementary metal oxide semiconductor
- the lenses 2 are provided on a one-to-one correspondence with the photodetectors 4 . More specifically, as shown in FIG. 2 , the lenses 2 are superposed upon and disposed at their respective photodetectors 4 so that light passing through the lenses 2 is only incident upon their respective photodetectors 4 .
- the photodetectors 4 are, for example, circular
- hemispherical or plano-convex lenses having planar faces that match the circular shape are used as the lenses 2 , it is possible to place the planar faces of the lenses 2 in close contact with the surfaces of the photodetectors 4 and to dispose them so that light passing through the lenses 2 is incident upon only the respective photodetectors 4 .
- Methods of causing the lenses 2 to be in a one-to-one correspondence with the respective photodetectors 4 include, for example, a method in which the lenses 2 are directly formed on the photodetectors 4 or a method in which a photodetector array and a lens array are formed and both of them are superimposed upon each other so that the photodetectors and the lenses are in a one-to-one correspondence.
- another method is available.
- the lenses 2 and the photodetectors 4 are caused to be in a one-to-one correspondence by, after the lenses 2 and the photodetectors 4 are disposed so as to be separated from each other by a predetermined distance, surrounding gaps between the lenses 2 and the photodetectors 4 by predetermined members (such as related light-shielding blocks) so that light passing through lenses other than the lenses 2 corresponding to the respective photodetectors 4 is not incident upon the photodetectors 4 .
- predetermined members such as related light-shielding blocks
- the controller 8 may be formed as a typical information processing unit or a dedicated device including, for example, CPU 13 , a memory (such as a random access memory (RAM) 14 , a read only memory (ROM) 15 , and a rewritable nonvolatile memory 16 ).
- a memory such as a random access memory (RAM) 14 , a read only memory (ROM) 15 , and a rewritable nonvolatile memory 16 .
- FIG. 3 shows a functional structure of the controller 8 .
- the controller 8 includes, as functional means, a sensitivity coefficient storage section 21 that stores information for determining the sensitivity coefficient of each photodetector 4 of the image pickup unit 5 , an image pickup signal storage section 22 that stores signals that are output by the image pickup unit 5 , and an image reproducing section 23 that reproduces an image using the signals that are output by the image pickup unit 5 and using the sensitivity coefficients that are determined on the basis of the information stored in the sensitivity coefficient storage section 21 .
- the sensitivity coefficient storage section 21 is functionally achieved by, for example, ROM 15 .
- the image pickup signal storage section 22 is functionally achieved by, for example, the transportable storage medium 11 or the rewritable nonvolatile memory 16 .
- the image reproducing section 23 is functionally achieved when CPU 13 reads a program stored in, for example, ROM 15 out at RAM 14 and execute it.
- the image reproducing section 23 uses a model in which it is assumed that a plurality of imaginary light spots exist in an image pickup area, and in which light from the plurality of imaginary light spots is received by the respective photodetectors, the image reproducing section 23 estimates light beams of the respective imaginary light spots using the sensitivity coefficients and the output signals corresponding to the respective photodetectors. Using brightnesses corresponding to the estimated light beams as brightnesses of pixels of an image, corresponding to the image pickup area, the image reproducing section 23 reproduces the image. For the relationship between the light beams and the corresponding brightnesses, it is possible to use a linear or a nonlinear gamma curve.
- the image reproducing section 23 includes data obtaining part 201 , light spot setting part 202 , sensitivity coefficient determining part 203 , light beam calculating part 204 , and image outputting part 205 .
- the data obtaining part 201 obtains the output signals that are output from the image pickup unit 5 .
- the light spot setting part 202 sets the plurality of imaginary light spots in the image pickup area so that a number m of imaginary light spots ⁇ a number n of photodetectors.
- the sensitivity coefficient determining part 203 determines the sensitivity coefficients corresponding to the respective imaginary light spots for the respective photodetectors 4 on the basis of the information stored in the sensitivity coefficient storage section 21 .
- the light beam calculating part 204 determines u i so that a value of E ev determined by the following Formula (1) becomes smaller than a predetermined threshold value ⁇ :
- the image outputting part 205 forms and outputs an image using the determined u i .
- an image pickup area 401 is an area that is subjected to image reproduction among an entire area (hereunder referred to as “light-receiving area”) 402 where the image pickup unit 5 is capable of receiving light.
- the position and the size of the image pickup area 401 in the light-receiving area 402 may be previously determined, or may be specified by a user each time a shooting operation or image reproduction is performed.
- the imaginary light spots are set to the image pickup area 401 by dividing the image pickup area 401 into a lattice form, and setting cells C corresponding to respective pixels of a reproduction image, and performing setting operations on these cells C.
- a number m of cells C (the number m of imaginary light spots) must be less than or equal to the number n of photodetectors.
- a plurality of patterns are considered even for distances of the imaginary light spots from the image pickup unit 5 . More specifically, as shown in FIG. 4B , a plurality of possible imaginary light spots whose distances from the center of the image pickup unit 5 differ are set for the respective cells C. By a method described later, a possible imaginary light spot at which the evaluation function E ev is smallest among the possible imaginary light spots is selected as an imaginary light spot of the corresponding cell C.
- the sensitivity coefficient ⁇ ij of each photodetector 4 is a value that is determined on the basis of the relationship between the positions of a light source and the lens 2 (or the photodetector 4 ).
- the sensitivity coefficient ⁇ ij of each photodetector 4 is defined as multiplication between a function ⁇ that depends upon the distance between the light source and the photodetector 4 and a function f related to an angle of incidence of light with respect to the corresponding lens 2 .
- the function ⁇ is a factor that indicates distance decay, in which the intensity of light is reduced so as to be inversely proportional to the square of the distance from the light source.
- the function ⁇ can be expressed by the following Formula (2).
- C represents a light attenuation constant.
- ⁇ ⁇ ( x ij , y ij , z ij ) C x ij 2 + y ij 2 + z ij 2 ( 2 )
- the function f is a factor that indicates the characteristics of each lens 2 . If, for example, in the three-dimensional coordinate system, a light-receiving plane matches an XY plane, it is possible to express the function f as a function of an angle ⁇ between an XZ plane and a direction from the center of the photodetector j to the imaginary light spot i, and an angle ⁇ between a YZ plane and the direction from the photodetector j to the imaginary light spot i.
- ⁇ and ⁇ are expressed by the following Formulas (3) and (4), respectively:
- the sensitivity coefficient ⁇ ij in the embodiment can be expressed as in the following Formula (5) using the functions ⁇ and f:
- information for determining the sensitivity coefficient ⁇ ij of each photodetector 4 information concerning the function f and C in the function ⁇ are stored in the sensitivity coefficient storage section 21 .
- C and the function f can be determined by previously performing calibration. If, for each lens 2 , information of the function f is previously provided by, for example, a manufacturer, this information may be stored. If it can be thought that the lenses 2 corresponding to the respective photodetectors 4 have common characteristics, information may be thought of as being common to each of the photodetectors 4 . On the other hand, if it can be thought that the lenses 2 corresponding to the respective photodetectors 4 have different characteristics, different information may be used for each set of photodetectors 4 and lenses 2 .
- the output signal S j corresponding to the photodetector j is an output signal from the image pickup unit 5 , and the sensitivity coefficient ⁇ ij can be obtained on the basis of the relationship between the positions of the imaginary light spot and the photodetector relative to each other and the information stored in the sensitivity coefficient storage section 21 . Therefore, what is unknown is only u i .
- a i , B i , and C i are expressed as in the following Formulas (11) to (13), respectively:
- u i which represents the light beam of each imaginary light spot, is calculated using the evaluation function E ev in which a squared term r ij 2 of the variable r ij , representing the distance between the imaginary light spot i and the photodetector j, is introduced.
- An imaginary light spot that is set near the image pickup unit 5 is where distance decay when the photodetector is receiving light is small, so that the sensitivity coefficient ⁇ ij is large.
- the ⁇ ij 2 term is provided with the coefficient of u i 2 . Therefore, the closer an imaginary light spot is set to the image pickup unit 5 , the larger the coefficient of u i 2 , and the higher the sensitivity with respect to the basic evaluation function E. This means that, even if the amount of change in the light beam of the imaginary light spot is the same, the closer the imaginary light spot is set to the image pickup unit 5 , the larger the amount of change in the value of the basic evaluation function E.
- the evaluation function E ev in which the distance between the imaginary light spots and the image pickup unit is normalized is used. More specifically, the evaluation function E ev is such that the coefficient of u i 2 is provided with the multiplication terms of ⁇ ij 2 and r ij 2 by introducing the squared term r ij 2 of the distance. An imaginary light spot that is set close to the image pickup unit 5 is such that the sensitivity coefficient ⁇ ij becomes large while the distance r ij becomes small. An imaginary light spot that is set far away from the image pickup unit 5 is such that the sensitivity coefficient ⁇ ij becomes small while the distance r ij becomes large. Therefore, in the evaluation function E ev including the above, the coefficient of u i 2 is normalized, as a result of which the sensitivity of u i with respect to the evaluation function E ev can be equalized.
- FIGS. 5 and 6 are flowcharts of the content of the operations of the image reproducing section 23 of the image pickup device 1 .
- the content of the operations will hereunder be described with reference to the flowcharts.
- the steps (including partial steps that are not given reference numerals) shown in, for example, each of the flowcharts can be executed in parallel or by arbitrarily changing the order within a range that is consistent with the content of the operations.
- the initial values of the light beams u i and the initial value of a reference distance R i from the center of the image pickup unit 5 are set (ST 304 ).
- the initial values may be the same or different for the cells C i .
- the initial values of the light beams u i may be 0, and the initial value of the reference distance R i may be 5 m.
- the light spot setting part 202 selects a cell C p that is specified by the variable p (ST 306 ).
- the light spot setting part 202 sets the imaginary light spots i at the positions that are separated by the reference distance R i from the center of the image pickup unit 5 on straight lines connecting the center of the image pickup unit 5 and the cells C i (ST 307 ).
- the possible distances and the value of K may be set in accordance with the computing capability of the controller 8 . If the possible distances are negative, the corresponding possible imaginary light spots are not set.
- the sensitivity coefficient determining part 203 determines the sensitivity coefficient ⁇ pjk corresponding to each photodetector j. In addition, for the imaginary light spots corresponding to the cells C i other than the cell C p , the sensitivity coefficient determining part 203 determines the sensitivity coefficient ⁇ ij corresponding to each photodetector j (ST 308 ).
- the sensitivity coefficient ⁇ pjk can be determined as follows.
- the sensitivity coefficient storage section 21 is referred to.
- the value of the function ⁇ is determined.
- ⁇ and ⁇ which depend upon the relationship between the positions of the photodetectors j and the possible imaginary light spots i k relative to each other, the value of the function f is determined.
- the value of the function ⁇ and the value of the function f are multiplied to each other to obtain the sensitivity coefficient ⁇ pjk .
- the sensitivity coefficient ⁇ ij can be similarly determined as follows.
- the value of the function ⁇ is determined.
- the value of the function f is determined. Then, the value of the function ⁇ and the value of the function f are multiplied to each other to obtain the sensitivity coefficient ⁇ ij .
- the light beam calculating part 204 determines the updated values of the light beams u pk by the following Formula (16) (ST 309 ).
- the evaluation function E ev can be considered as a quadratic function of u pk .
- the light beam calculating part 204 calculates the value of the evaluation function E ev by the following Formula (17) (ST 310 ):
- the light beam calculating part 204 determines a possible imaginary light spot at which the calculated evaluation function E ev is smallest among the K number of possible imaginary light spots as an imaginary light spot of the corresponding cell C p (ST 311 ). Then, it stores the light beam u pk of the determined possible imaginary light spot i k as u p , and stores the distance r pjk corresponding to the determined possible imaginary light spot i k as the reference distance R p corresponding to the cell C p (ST 312 ).
- the p value is incremented by +1 (ST 313 ), and whether or not the p value is greater than m (the number of cells C) is determined (ST 314 ).
- the image reproducing section 23 proceeds to ST 306 , to execute the operations of ST 306 to ST 314 for the next cell C.
- the light beam calculating part 204 determines whether or not the value of the evaluation function E ev that is smallest in the immediately preceding ST 311 is less than the threshold value ⁇ previously provided as a completion condition (ST 315 ).
- the image reproducing section 23 proceeds to ST 305 , to execute the operations of ST 305 to ST 315 .
- the image outputting part 205 forms an image using the light beams u i of the imaginary light spots i of the respective cells C and outputs it to, for example, a liquid crystal monitor 9 (ST 316 ), to end the processing.
- the formed image may be recorded onto (output to) a memory (such as the transportable storage medium 11 or the rewritable nonvolatile memory 16 ).
- the image pickup device 1 of the embodiment since it is not necessary to separate the lenses 2 from the corresponding photodetectors 4 by the focal length, compared to the case in which the lenses and the photodetectors are separated from each other by the focal length as in the related image pickup device, it is possible to make thin the image pickup device 1 (the image pickup unit 5 ).
- the image reproducing section 23 detects a shooting instruction from a user, data obtained from the image pickup unit 5 is stored in the memory (the transportable storage medium 11 or the rewritable nonvolatile memory 16 ). If the image reproducing section 23 receives the shooting instruction from the user, the image is reproduced, so that it is possible to perform a continuous shooting operation at a high speed (obtain output data of the image pickup unit) without taking time to perform the image reproducing operation.
- the image pickup unit 5 can be formed by causing the lenses 2 and the photodetectors 4 to be in a one-to-one correspondence and bringing them close to each other (that is, contacting them to each other), it is possible to considerably reduce the burden of aligning the lens system and image sensors.
- distance information (the distances r pjk stored in ST 312 ) to the imaginary light spots can also be obtained, it is possible to use it as a three-dimensional distance sensor.
- the image pickup unit 5 is formed so that light subjected to color separation by color filters is received by the photodetectors 4 through the lenses 2 , it is possible to pick up and reproduce a color image.
- the function f( ⁇ , ⁇ ), which is a factor that represents the characteristics of the lenses 2 can be determined by performing the following calibration.
- a light source L is disposed in a three-dimensional coordinate system in which the position of a photodetector 601 is the origin and in which a light-receiving plane of the photodetector 601 is made to match an xy plane. More specifically, the light source L is disposed where the distance from the photodetector 601 is r 1 , the angle between an xz plane and a direction towards the light source L from the photodetector 601 is ⁇ 1 as shown in FIG. 7B , and the angle between a yz plane and the direction towards the light source L from the photodetector 601 is ⁇ 1 .
- the light source L emits light of a light beam u, and the light is received by the photodetector 601 through a lens.
- the sensitivity coefficient ⁇ is defined as a multiplication of ⁇ and f( ⁇ , ⁇ ). Therefore, the function f( ⁇ 1 , ⁇ 1 ) is determined by dividing the determined ⁇ by ⁇ 1 .
- the position of the light source L or the positions of the photodetector 601 and a lens 602 are changed to determine f( ⁇ 1 , ⁇ 1 ) corresponding to various ⁇ 1 and ⁇ 1 .
- variances or average values, which are parameters, of g 1 ( ⁇ ) and g 2 ( ⁇ ) may be stored in the sensitivity coefficient storage section 21 .
- the image pickup device 1 may have a calibration function in which, in accordance with the instruction of a user, the light attenuation constant C and the function f( ⁇ , ⁇ ) are determined and information thereof is stored in the sensitivity coefficient storage section 21 .
- the setting of the imaginary light spots onto the image pickup area can be specified by a user each time a shooting operation or an image reproducing operation is performed
- the setting of the cells C may also be specified by the user each time a shooting operation or an image reproducing operation is performed.
- the setting of the image pickup area and the setting of the cells C are both information required in the image reproducing operation. Therefore, it is possible to obtain output data of the image pickup unit without performing these settings during the pickup operation, and for the user to perform these settings during reproduction of an image to reproduce the image.
- the user during the reproduction of the image, it is possible for the user to select any area in the light-receiving area as the image pickup area (that is, image reproduction area), and set any resolution in the range satisfying the condition in which the number m of imaginary light spots ⁇ the number n of photodetectors with respect to the image pickup area, to reproduce the image.
- the image pickup device is applicable to an endoscopic camera or a television telephone device (television conference device). If it is applied to a television telephone device, as shown in FIG. 8A , a display device 701 that displays an image of a caller (that is, an image of the face of the caller) and the image pickup unit 5 according to the present disclosure can be combined and integrated to each other.
- an image pickup unit also serving as a display device is formed so that a plurality of sets 703 of the photodetectors 4 and the lenses 2 of the image pickup unit 5 according to the present disclosure are disposed on a display screen (such as a liquid crystal panel) of the display device 701 at a predetermined interval (for example, they are uniformly distributed in a person display area that is assumed during the call).
- a display screen such as a liquid crystal panel
- a predetermined interval for example, they are uniformly distributed in a person display area that is assumed during the call.
- a display screen such as a liquid crystal panel
- FIG. 8C since an image pickup unit 801 and a display device 802 are separately provided (for example, a camera is mounted on the top portion of a liquid crystal display), it is difficult to talk to a person displayed on the display screen while looking at his/her eyes.
- the image pickup unit 5 is disposed on the display screen. Therefore, as shown in FIG. 8B , it is possible to talk to a person displayed on the display screen while looking at his/her eyes.
- a disjunctive expression that indicates a plurality of selective words essentially means that all the words, only one of the words, either one of the words, or both of the words are included.
- the phrase “A or B” may mean “A” or “B,” or “A and B.”
Abstract
Description
where a light beam of an imaginary light spot i is ui, the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is αij, the distance between the imaginary light spot i and the photodetector j is rij, and the signal corresponding to the photodetector j that is output by the image pickup unit is Sj; and part for forming and outputting the image using the determined ui.
where a light beam of an imaginary light spot i is ui, the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is αij, the distance between the imaginary light spot i and the photodetector j is rij, and the signal corresponding to the photodetector j that is output by the image pickup unit is Sj; and part for forming and outputting the image using the determined ui.
where a light beam of an imaginary light spot i is ui, the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is αij, the distance between the imaginary light spot i and the photodetector j is rij, and the signal corresponding to the photodetector j that is output by the image pickup unit is Sj; and part for forming and outputting an image using the determined ui.
where a light beam of an imaginary light spot i is ui, the sensitivity coefficient of a photodetector j corresponding to the imaginary light spot i is αij, the distance between the imaginary light spot i and the photodetector j is rij, and the signal corresponding to the photodetector j that is output from the image pickup unit is Sj; and forming and outputting the image using the determined ui.
The
e j =S j −O j=(S j −O j+αij ·u i)−αij ·u i (8)
E=A i 2 u i 2−2A i B i u i +C i=(A i u i −B i)2 +C i −B i 2 (10)
Claims (10)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2009082395A JP4402161B1 (en) | 2009-03-30 | 2009-03-30 | Imaging apparatus, image reproducing apparatus, and imaging method |
JP2009-82395 | 2009-03-30 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20100245654A1 US20100245654A1 (en) | 2010-09-30 |
US7898591B2 true US7898591B2 (en) | 2011-03-01 |
Family
ID=41706572
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/640,986 Expired - Fee Related US7898591B2 (en) | 2009-03-30 | 2009-12-17 | Method and apparatus for imaging using sensitivity coefficients |
Country Status (2)
Country | Link |
---|---|
US (1) | US7898591B2 (en) |
JP (1) | JP4402161B1 (en) |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8830791B2 (en) | 2011-04-27 | 2014-09-09 | Empire Technology Development Llc | Measurement of 3D coordinates of transmitter |
US10348945B2 (en) | 2013-07-31 | 2019-07-09 | Samsung Electronics Co., Ltd. | Light field image capturing apparatus including shifted microlens array |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020036257A1 (en) * | 2000-08-28 | 2002-03-28 | Yuichiro Yamashita | Image pickup apparatus |
US20020094131A1 (en) * | 2001-01-17 | 2002-07-18 | Yusuke Shirakawa | Image sensing apparatus, shading correction method, program, and storage medium |
JP2004031532A (en) | 2002-06-25 | 2004-01-29 | Toppan Printing Co Ltd | Manufacturing method of solid-state image sensing device |
US20050151873A1 (en) * | 2004-01-06 | 2005-07-14 | Sony Corporation | Solid-state imaging device and signal processing circuit |
JP2006303995A (en) | 2005-04-21 | 2006-11-02 | Fuji Photo Film Co Ltd | Solid imaging device and imaging apparatus |
JP2007096825A (en) | 2005-09-29 | 2007-04-12 | Funai Electric Co Ltd | Compound eye imaging apparatus |
US20070091197A1 (en) * | 2003-11-04 | 2007-04-26 | Hiroaki Okayama | Imaging device |
US20080007804A1 (en) | 2006-07-06 | 2008-01-10 | Samsung Electronics Co., Ltd. | Image sensor having improved resolution and image sensing method using the same |
US20090122166A1 (en) * | 2003-06-04 | 2009-05-14 | Nikon Corporation | Imaging device performing color image data processing |
-
2009
- 2009-03-30 JP JP2009082395A patent/JP4402161B1/en not_active Expired - Fee Related
- 2009-12-17 US US12/640,986 patent/US7898591B2/en not_active Expired - Fee Related
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020036257A1 (en) * | 2000-08-28 | 2002-03-28 | Yuichiro Yamashita | Image pickup apparatus |
US20020094131A1 (en) * | 2001-01-17 | 2002-07-18 | Yusuke Shirakawa | Image sensing apparatus, shading correction method, program, and storage medium |
JP2004031532A (en) | 2002-06-25 | 2004-01-29 | Toppan Printing Co Ltd | Manufacturing method of solid-state image sensing device |
US20090122166A1 (en) * | 2003-06-04 | 2009-05-14 | Nikon Corporation | Imaging device performing color image data processing |
US20070091197A1 (en) * | 2003-11-04 | 2007-04-26 | Hiroaki Okayama | Imaging device |
US20050151873A1 (en) * | 2004-01-06 | 2005-07-14 | Sony Corporation | Solid-state imaging device and signal processing circuit |
JP2006303995A (en) | 2005-04-21 | 2006-11-02 | Fuji Photo Film Co Ltd | Solid imaging device and imaging apparatus |
US20060250509A1 (en) | 2005-04-21 | 2006-11-09 | Masaaki Koshiba | Solid-state image pickup element and image pickup device |
JP2007096825A (en) | 2005-09-29 | 2007-04-12 | Funai Electric Co Ltd | Compound eye imaging apparatus |
US20080007804A1 (en) | 2006-07-06 | 2008-01-10 | Samsung Electronics Co., Ltd. | Image sensor having improved resolution and image sensing method using the same |
JP2008017486A (en) | 2006-07-06 | 2008-01-24 | Samsung Electronics Co Ltd | Resolution improved image sensor and image detection method using it |
Non-Patent Citations (1)
Title |
---|
An Office Action issued in Japanese Patent Application No. 2009-082395 mailed Jun. 12, 2009 (with English abstract). |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US8830791B2 (en) | 2011-04-27 | 2014-09-09 | Empire Technology Development Llc | Measurement of 3D coordinates of transmitter |
US10348945B2 (en) | 2013-07-31 | 2019-07-09 | Samsung Electronics Co., Ltd. | Light field image capturing apparatus including shifted microlens array |
Also Published As
Publication number | Publication date |
---|---|
US20100245654A1 (en) | 2010-09-30 |
JP4402161B1 (en) | 2010-01-20 |
JP2010239220A (en) | 2010-10-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US8890972B2 (en) | Image capturing apparatus and image processing method | |
US7863550B2 (en) | Focus detection device and focus detection method based upon center position of gravity information of a pair of light fluxes | |
JP4322921B2 (en) | Camera module and electronic device including the same | |
JP4857877B2 (en) | Imaging device and camera | |
JP5163068B2 (en) | Imaging device | |
JP4770560B2 (en) | Imaging apparatus, camera, and image processing method | |
US8730374B2 (en) | Focus detection apparatus | |
US7586072B2 (en) | Correlation operation method, correlation operation device, focus detection device and imaging device | |
JP4935161B2 (en) | Imaging apparatus, camera, and image processing method | |
US20050218297A1 (en) | Optical apparatus and beam splitter | |
JP5751946B2 (en) | Focus adjustment apparatus and control method thereof | |
US20080151079A1 (en) | Imaging Device and Manufacturing Method Thereof | |
JP2007282109A (en) | Imaging apparatus, camera and image processing method | |
US20120230549A1 (en) | Image processing device, image processing method and recording medium | |
CN105323441A (en) | Image capture apparatus and method for controlling same | |
US20150264251A1 (en) | Photographing apparatus, method of calculating information for focus control, and camera system | |
US9602716B2 (en) | Focus-detection device, method for controlling the same, and image capture apparatus | |
JP6353233B2 (en) | Image processing apparatus, imaging apparatus, and image processing method | |
JP2013097154A (en) | Distance measurement device, imaging apparatus, and distance measurement method | |
US7898591B2 (en) | Method and apparatus for imaging using sensitivity coefficients | |
JP5228777B2 (en) | Focus detection apparatus and imaging apparatus | |
JP6362070B2 (en) | Image processing apparatus, imaging apparatus, image processing method, program, and storage medium | |
US20220030183A1 (en) | Infrared and non-infrared channel blender for depth mapping using structured light | |
JP2006311126A (en) | Imaging apparatus | |
JP2012080363A (en) | Imaging module, imaging device, and manufacturing method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: EMPIRE TECHNOLOGY DEVELOPMENT LLC, DELAWARE Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:UEDA, TOMOAKI;REEL/FRAME:023677/0155 Effective date: 20091211 |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
CC | Certificate of correction | ||
FPAY | Fee payment |
Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: CRESTLINE DIRECT FINANCE, L.P., TEXAS Free format text: SECURITY INTEREST;ASSIGNOR:EMPIRE TECHNOLOGY DEVELOPMENT LLC;REEL/FRAME:048373/0217 Effective date: 20181228 |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20190301 |