JP2015103157A - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging device that enables information about a subject to be handled without deteriorating appearance-looking of a photograph image.SOLUTION: An imaging device comprises: an image pickup element that includes a first photoelectric conversion layer in which a pixel photoelectrically converting a beam of non-visible light and transmitting a beam of visible light is two-dimensionally arrayed and a second photoelectric conversion layer in which a pixel arranged by being laminated on the same optical path as with the first photoelectric conversion layer and having photoelectrically converting the beam of the visible light transmitting the first photoelectric conversion layer is two-dimensionally arrayed; marker detection means for detecting a marker having a prescribed pattern reflecting the non-visible light and preliminarily installed in a surface of a subject by recognizing a pattern from a first photoelectric conversion signal; reading means for reading prescribed information embedded in the pattern of the marker detected by the marker detection means; and storage means for storing image data prepared based on a second photoelectric conversion signal output from the second photoelectric conversion layer and the prescribed information read by the reading means in association of the image data with the prescribed information in a prescribed storage medium.

Description

  The present invention relates to an imaging apparatus.

  2. Description of the Related Art Conventionally, a technique is known in which arbitrary information converted into a one-dimensional or two-dimensional image pattern is arranged on a printed matter or the like, and information is restored from the image pattern captured by a camera or the like. For example, Patent Document 1 describes a two-dimensional barcode reading apparatus that decodes basic information from an image including a so-called two-dimensional barcode.

JP 2011-175314 A

  For example, when the conventional technology is applied to photography, and an image pattern in which information on the subject is embedded is added to the subject, the image pattern is reflected in the photographed image together with the subject, so that the photographed image looks worse. there were.

The imaging apparatus according to claim 1, wherein the first photoelectric conversion layer in which pixels that photoelectrically convert invisible light and transmit visible light are two-dimensionally arranged, and the first photoelectric conversion layer are stacked on the same optical path. And an image sensor having a second photoelectric conversion layer in which pixels that photoelectrically convert visible light transmitted through the first photoelectric conversion layer are two-dimensionally arranged, and a predetermined pattern that reflects invisible light. And a marker detecting unit that detects a marker previously set on the surface of the subject by recognizing the pattern from the first photoelectric conversion signal output from the first photoelectric conversion layer, and the marker detecting unit detects the marker. Reading means for reading predetermined information embedded in the pattern from the pattern of the marker, and an image created based on the second photoelectric conversion signal output from the second photoelectric conversion layer And data, in association with the predetermined information read by the reading means, characterized in that it comprises a storage means for storing in a predetermined storage medium.
The imaging device according to claim 2, wherein a first photoelectric conversion layer in which pixels that photoelectrically convert visible light and transmit invisible light are two-dimensionally arranged, and the first photoelectric conversion layer are stacked on the same optical path. And an image sensor having a second photoelectric conversion layer in which pixels that photoelectrically convert invisible light that has passed through the first photoelectric conversion layer are two-dimensionally arranged, and a predetermined pattern that reflects the invisible light. And a marker detecting means for detecting a marker previously set on the surface of the subject by recognizing the pattern from the second photoelectric conversion signal output by the second photoelectric conversion layer, and the marker detecting means Readout means for reading predetermined information embedded in the pattern from the pattern of the marker, and a first photoelectric conversion signal output from the first photoelectric conversion layer An image data, in association with the predetermined information read by the reading means, characterized in that it comprises a storage means for storing in a predetermined storage medium.

  According to the present invention, information about a subject can be acquired without deteriorating the appearance of a captured image.

It is a figure which illustrates the structure of the digital camera 1 by one embodiment of this invention. It is a figure showing the outline of image sensor 21 concerning this embodiment. 3 is a diagram illustrating a pixel arrangement of an upper photoelectric conversion layer 31 and a lower photoelectric conversion layer 32. FIG. 2 is a diagram illustrating a part of a cross section of an image sensor 21. FIG. It is a figure which shows an example of the pattern formed in the marker surface. It is a figure which shows the example of utilization of the marker 80 typically. 3 is a flowchart of a photographing process executed by a control unit 11.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration of a digital camera 1 according to an embodiment of the present invention. The digital camera 1 includes a photographing optical system 10, a control unit 11, an imaging unit 12, an operation unit 13, an image processing unit 14, a liquid crystal monitor 15, and a buffer memory 16. In addition, a memory card 17 is attached to the digital camera 1.

  The photographic optical system 10 includes a plurality of lenses, and forms a subject image on an imaging surface of an imaging element 21 (described later). The plurality of lenses constituting the photographing optical system 10 include a focus lens that is driven in the optical axis direction in order to adjust the focus position. The focus lens is driven in the optical axis direction by an actuator (not shown) (for example, an ultrasonic motor).

  The control unit 11 includes a microprocessor and its peripheral circuits, and performs various controls of the digital camera 1 by executing a control program stored in a ROM (not shown). The control unit 11 functionally includes a focus adjustment unit 11a, a marker detection unit 11b, an information reading unit 11c, and a storage unit 11d. Each of these functional units is implemented in software by the control program described above. Each of these functional units can also be configured by an electronic circuit.

  The focus adjusting unit 11a adjusts the focus of the photographing optical system 10 based on an image signal output from an image sensor 21 (described later). The marker detection unit 11b detects a marker described later based on the infrared light component of the subject image formed by the photographing optical system 10. The information reading unit 11c reads marker information described later from the marker detected by the marker detection unit 11b. The storage unit 11d stores the captured image data and the marker information read by the information reading unit 11c in the memory card 17 in association with each other.

  The imaging unit 12 includes an imaging device 21, an amplification circuit 22, and an AD conversion circuit 23. The image sensor 21 is composed of a plurality of pixels, receives a light beam from a subject via a photographing optical system (not shown), performs photoelectric conversion, and outputs an analog image signal. The amplifier circuit 22 amplifies the analog image signal output from the image sensor 21 with a predetermined amplification factor (gain) and outputs the amplified signal to the AD conversion circuit 23. The AD conversion circuit 23 performs AD conversion on the analog image signal and outputs a digital image signal. The control unit 11 stores the digital image signal output from the imaging unit 12 in the buffer memory 16.

  The digital image signal stored in the buffer memory 16 is subjected to various image processing in the image processing unit 14 and displayed on the liquid crystal monitor 15 or stored in the memory card 17. The memory card 17 is composed of a non-volatile flash memory or the like and is detachable from the digital camera 1.

  The operation unit 13 includes various operation buttons such as a release button, a mode switching button, and a power button, and is operated by a photographer. The operation unit 13 outputs an operation signal corresponding to the operation of each operation button by the photographer to the control unit 11. The image processing unit 14 is configured by an ASIC or the like. The image processing unit 14 performs various types of image processing such as interpolation, compression, and white balance on the image data captured by the imaging unit 12.

(Description of the image sensor 21)
FIG. 2 is a diagram showing an outline of the image sensor 21 according to the present embodiment. 2 shows a state in which the light incident side of the image sensor 21 is the upper side. Therefore, in the following description, the direction on the light incident side of the image sensor 21 is “upper” or “upper”, and the direction opposite to the light incident side is “lower” or “lower”. The image sensor 21 includes an upper photoelectric conversion layer 31 and a lower photoelectric conversion layer 32. The upper photoelectric conversion layer 31 and the lower photoelectric conversion layer 32 are stacked on the same optical path. The upper photoelectric conversion layer 31 is composed of an organic photoelectric film that absorbs (photoelectric conversion) light in a predetermined wavelength range (details will be described later). Light in a wavelength region that has not been absorbed (photoelectric conversion) by the upper photoelectric conversion layer 31 passes through the upper photoelectric conversion layer 31 and enters the lower photoelectric conversion layer 32, and is photoelectrically converted by the lower photoelectric conversion layer 32. The lower photoelectric conversion layer 32 performs photoelectric conversion using a photodiode.

  The upper photoelectric conversion layer 31 photoelectrically converts invisible light (for example, infrared light). The lower photoelectric conversion layer 32 photoelectrically converts visible light. The upper photoelectric conversion layer 31 and the lower photoelectric conversion layer 32 are formed on the same semiconductor substrate, and each pixel position corresponds to one to one. For example, the pixel in the first row and the first column of the upper photoelectric conversion layer 31 corresponds to the pixel in the first row and the first column of the lower photoelectric conversion layer 32.

  FIG. 3A is a diagram illustrating a pixel arrangement of the upper photoelectric conversion layer 31. In FIG. 3A, the horizontal direction is the x-axis, the vertical direction is the y-axis, and the coordinates of the pixel P are expressed as P (x, y). In the example of the upper photoelectric conversion layer 31 shown in FIG. 3A, each pixel is an organic photoelectric film that photoelectrically converts invisible light (infrared light). Then, light that is not received by each pixel, that is, visible light is transmitted. For example, the pixel P (1, 1) photoelectrically converts infrared light (IR) and transmits visible light.

  FIG. 3B is a diagram illustrating a pixel arrangement of the lower photoelectric conversion layer 32. Each pixel position shown in FIG. 3B is the same as that in FIG. For example, the pixel P (1, 1) of the lower photoelectric conversion layer 32 corresponds to the pixel P (1, 1) of the upper photoelectric conversion layer 31. In FIG. 3B, the lower photoelectric conversion layer 32 is provided with three color filters of red (R), green (G), and blue (B) in a so-called Bayer arrangement. Visible light that passes through the upper photoelectric conversion layer 31 (that is, light in a wavelength region other than invisible light that is absorbed and photoelectrically converted by the organic photoelectric film of the upper photoelectric conversion layer 31) is photoelectrically converted. Therefore, the lower photoelectric conversion layer 32 outputs an image signal of G and B color components for pixels in odd rows, and an image signal of R and G color components for each pixel in even rows. For example, a G component image signal is obtained at the pixel P (1, 1). Similarly, a B component image signal is obtained from the pixel P (2, 1), and an R component image signal is obtained from the pixel P (1, 2).

  Thus, in the imaging device 21 according to the present embodiment, the upper photoelectric conversion layer 31 formed of an organic photoelectric film serves as an infrared filter with respect to the lower photoelectric conversion layer 32. In the imaging device 21 according to the present embodiment, an image of an object by infrared light (infrared light image) can be acquired from the upper photoelectric conversion layer 31, and an image of the object by visible light from the lower photoelectric conversion layer 32. (RGB image composed of three colors of R, G, and B) can be acquired. In the following description, an image signal output from the upper photoelectric conversion layer 31 is referred to as an infrared light signal. The image signal output from the lower photoelectric conversion layer 32 is referred to as a visible light signal.

  FIG. 4 is a diagram illustrating a part of a cross section of the image sensor 21. As shown in FIG. 4, in the imaging element 21, a lower photoelectric conversion layer 32 formed on a silicon substrate and an upper photoelectric conversion layer 31 using an organic photoelectric film are stacked via a wiring layer 40. Above the upper photoelectric conversion layer 31, one microlens ML is formed for one pixel. Between the upper photoelectric conversion layer 31 and the lower photoelectric conversion layer 32, one color filter CF is formed for one pixel. For example, in the upper photoelectric conversion layer 31, the light receiving unit PC (1,1) by the organic photoelectric film constituting the photoelectric conversion unit of the pixel P (1,1) is the subject incident from the micro lens ML (1,1). Visible light is transmitted by photoelectrically converting infrared light in the light. Of the visible light transmitted through the light receiving unit PC (1, 1), the G component transmits through the color filter CF (1, 1), and the R component and B component are shielded by the color filter CF (1, 1). In the lower photoelectric conversion layer 32, the photodiode PD (1, 1) constituting the pixel P (1, 1) receives and photoelectrically converts the G light transmitted through the color filter CF (1, 1).

(Description of automatic focus adjustment process)
When the photographer performs a predetermined focus adjustment operation (for example, an operation of pressing the release button halfway), the control unit 11 executes a so-called contrast type automatic focus adjustment process. Briefly explaining the contrast type auto-focusing process, the subject lens is picked up periodically while the focus lens is driven, the contrast amount of the subject image at each focus lens position is examined, and the focus lens that maximizes the contrast amount This position is specified as the in-focus position.

  The focus adjustment unit 11a included in the control unit 11 can perform a contrast-type automatic focus adjustment process based on a visible light signal, that is, a subject image by visible light. The focus adjustment unit 11a can also perform contrast-type automatic focus adjustment processing based on an infrared light signal, that is, an object image by infrared light. The control unit 11 selects one of a visible light signal and an infrared light signal in accordance with, for example, the illuminance of the subject or the detection result of a marker to be described later, and performs automatic focus adjustment processing based on the selected image signal. 11a is executed. For example, when the illuminance of the subject is low or a marker to be described later is detected, since the infrared light is more advantageous for focus detection, the automatic focus adjustment process based on the infrared light signal is executed. On the other hand, when the illuminance of the subject is high, or when a marker to be described later is not detected, automatic focus adjustment processing based on the visible light signal is executed.

(Description of marker)
The digital camera 1 of this embodiment can read information from the marker. The marker has a predetermined pattern formed on the surface, and is affixed to the subject surface as a seal, for example, or printed on the subject surface. This pattern is formed to transmit visible light and reflect infrared light. That is, even if this marker is imaged by the lower photoelectric conversion layer 32, the above pattern does not appear in the visible light signal. On the other hand, when the upper photoelectric conversion layer 31 captures an image, the above pattern appears in the infrared light signal.

  FIG. 5 is a diagram illustrating an example of a pattern formed on the marker surface. A pattern 81 is formed on the surface of the marker 80. The pattern 81 is formed of a material that transmits visible light and reflects infrared light with a constant reflectance.

  The pattern 81 has two roles. First, in the pattern 81, marker information, which is information about a subject (for example, a person, a building, etc.) to which this pattern is attached, is embedded. Specifically, a character string representing the name of the subject and a URL (Uniform Resource Locator) character string related to the subject are encoded. Specifically, for example, a bit pattern of marker information is expressed by a two-dimensionally arranged dot pattern, such as a well-known two-dimensional barcode. The information reading unit 11c decodes the pattern 81 detected by the marker detection unit 11b and reads marker information.

  Next, the pattern 81 has a shape advantageous for focus detection. Specifically, the shape has features such as high contrast, clear edges, and no periodicity. The pattern 81 having such a characteristic has characteristics that are advantageous for focus detection, such as being able to detect contrast with a certain degree of reliability even when the focus is greatly deviated, and preventing false focusing. Have

  As described above, the pattern 81 included in the marker 80 has both a role of holding marker information, which is information related to the subject, and a role of assisting focus adjustment. When the digital camera 1 is used, a marker 80 is formed in advance on clothes, accessories, a building surface, or the like of a person to be focused. The marker detection unit 11b detects the marker 80 from the infrared light signal by recognizing the pattern 81 by a method such as pattern matching. The focus adjusting unit 11a specifies, for example, the position of the subject, sets a focus detection area, and calculates a focus evaluation value (contrast value) from the detected marker 80.

  FIG. 6 is a diagram schematically illustrating a usage example of the marker 80. FIG. 6A shows an example of a visible light signal output from the lower photoelectric conversion layer 32 by photographing, and FIG. 6B shows an example of an infrared light signal output from the upper photoelectric conversion layer 31 by the same photographing. is there. The visible light signal 83 in FIG. 6A includes a person 84 and an advertisement 85 as subjects. 6B includes the same person 84 and advertisement 85 as subjects. The infrared light signal 86 further includes a marker 80 a attached to the clothes of the person 84 and a marker 80 b attached to the surface of the advertisement 85.

  At the time of shooting, the marker detection unit 11b detects the markers 80a and 80b from the infrared light signal 86. The information reading unit 11c reads marker information from each of the detected markers 80a and 80b. The control unit 11 performs various image processing on the visible light signal 83 to create digital image data. The storage unit 11d associates the created digital image data, the marker information 82a read from the marker 80a, and the marker information 82b read from the marker 80b, and stores them in the memory card 17.

  When the digital camera 1 is set to the reproduction mode, the control unit 11 reproduces the image data stored in the memory card 17 on the liquid crystal monitor 15. FIG. 6C is a diagram showing an example of the display screen of the liquid crystal monitor 15 during the reproduction of image data. When the image data is reproduced, the marker information 82a and 82b associated with the image data is displayed on the liquid crystal monitor 15 so as to be superimposed on the image data. The marker information 82a is superimposed and displayed in a form such as a balloon at the position where the marker 80a is detected. Similarly, the marker information 82b is superimposed and displayed at the position where the marker 80b is detected.

  FIG. 7 is a flowchart of the photographing process executed by the control unit 11. The imaging process shown in FIG. 7 is included in a control program executed by the control unit 11. When a predetermined photographing operation (for example, full pressing operation of a release switch (not shown)) is performed by the photographer, the control unit 11 starts executing the photographing process illustrated in FIG.

  First, in step S100, the image sensor 21 captures a subject image. Thereby, a visible light signal is output from the upper photoelectric conversion layer 31 and an infrared light signal is output from the lower photoelectric conversion layer 32. In step S110, the control unit 11 performs various image processing on the visible light signal output by the imaging in step S100, and creates digital image data. In step S120, the marker detection unit 11b attempts to detect the marker 80 from the infrared light signal output by the imaging in step S100. In step S130, the control unit 11 determines whether or not the marker detection unit 11b has successfully detected at least one marker 80 in step S120.

  If no marker 80 is detected in step S130, the control unit 11 advances the process to step S160. In step S160, the storage unit 11d stores the image data created in step S110 in the memory card 17.

  On the other hand, when at least one marker 80 is detected in step S130, the control unit 11 advances the process to step S140. In step S140, the information reading unit 11c reads marker information represented by the pattern of the marker 80 from the one or more markers 80 detected in step S120. In step S150, the storage unit 11d stores the image data created in step S110 and the marker information read in step S150 in association with each other in the memory card 17. For example, the marker information is stored in the EXIF (registered trademark) header of the image file.

  When displaying the image data stored in the memory card 17, the liquid crystal monitor 15 displays the marker information superimposed on the image data if there is marker information associated with the image data. When there is a plurality of marker information for one image data, all the marker information is displayed.

According to the digital camera according to the first embodiment described above, the following operational effects can be obtained.
(1) The image pickup device 21 includes an upper photoelectric conversion layer 31 (first photoelectric conversion layer) and a lower photoelectric conversion layer 32 (second photoelectric conversion layer) disposed on the same optical path as the upper photoelectric conversion layer 31. And have. In the upper photoelectric conversion layer 31, pixels that photoelectrically convert invisible light and transmit visible light are two-dimensionally arranged. In the lower photoelectric conversion layer 32, pixels that photoelectrically convert visible light transmitted through the upper photoelectric conversion layer 31 are two-dimensionally arranged. The marker detection unit 11b has a pattern 81 that reflects an invisible ray and has a marker 80 that is set in advance on the surface of the subject from the infrared light signal (first photoelectric conversion signal) output by the upper photoelectric conversion layer 31. It is detected by recognizing 81. The information reading unit 11c reads marker information embedded in the pattern 81 from the pattern 81 of the marker 80 detected by the marker detection unit 11b. The storage unit 11d associates the image data created based on the visible light signal (second photoelectric conversion signal) output from the lower photoelectric conversion layer 32 and the marker information read by the information reading unit 11c, Store in the memory card 17. Since it did in this way, the information regarding a to-be-photographed object can be acquired, without deteriorating the appearance of a picked-up image. In addition, since the imaging element 21 has a layer for imaging visible light and a layer for imaging infrared light, it is necessary to separately provide an element for imaging visible light and an element for imaging infrared light. Therefore, the digital camera 1 can be downsized.

(2) On the liquid crystal monitor 15, image data is displayed and marker information is displayed superimposed on the image data. The marker information is information regarding the subject on which the marker 80 is installed. Since it did in this way, the additional information regarding a to-be-photographed object can be shown to a photographer clearly.

  The following modifications are also within the scope of the present invention, and one or a plurality of modifications can be combined with the above-described embodiment.

(Modification 1)
As an installation destination of the marker 80, a person's clothes and a signboard advertisement have been described as examples. However, the marker 80 can be installed on an object other than these. For example, if the marker 80 is formed on the surface of the contact lens, it is possible to easily focus on the face of the person wearing the contact lens. Since the marker 80 is made of a material that transmits visible light, it does not hinder the function of the contact lens.

(Modification 2)
In the first embodiment, so-called contrast type automatic focus adjustment control is performed, but other types of automatic focus adjustment control may be adopted. For example, automatic focus adjustment control using a so-called phase difference detection method that detects a focus adjustment state from a phase difference between a pair of light beams that have passed through a pair of pupil regions of the photographing optical system 10 using visible light or infrared light. May be executed.

(Modification 3)
The digital camera 1 may be provided with an infrared light irradiation device so that the subject is irradiated with infrared light when the marker detection unit 11b detects the marker. By doing in this way, the detection accuracy of a marker can be improved.

(Modification 4)
The shape of the marker 80, the contents of the marker information, the method of embedding the marker information in the pattern 81, and the like are not limited to those exemplified in the above-described embodiment, and may be anything.

(Modification 5)
In the embodiment described above, the digital camera 1 using the imaging element 21 in which the upper photoelectric conversion layer 31 that photoelectrically converts infrared light and transmits visible light and the lower photoelectric conversion layer 32 that photoelectrically converts visible light is stacked. explained. The stacking order may be reversed. For example, an imaging element that photoelectrically converts visible light into the upper layer and transmits infrared light may be provided in the upper layer, and an imaging element that photoelectrically converts infrared light may be used in the lower layer.

  Although various embodiments and modifications have been described above, the present invention is not limited to these contents. Other embodiments conceivable within the scope of the technical idea of the present invention are also included in the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 ... Digital camera, 11 ... Control part, 11a ... Focus adjustment part, 11b ... Marker detection part, 11c ... Information reading part, 11d ... Memory | storage part, 21 ... Image sensor, 31 ... Upper photoelectric conversion layer, 32 ... Lower photoelectric conversion layer

Claims (4)

A first photoelectric conversion layer in which pixels that perform photoelectric conversion of invisible light and transmit visible light are two-dimensionally arranged, and are stacked on the same optical path as the first photoelectric conversion layer, and the first photoelectric conversion layer An image sensor having a second photoelectric conversion layer in which pixels that photoelectrically convert visible light transmitted through the second photoelectric conversion layer are two-dimensionally arranged;
Marker detection means for detecting a marker that has a predetermined pattern that reflects invisible light and is previously set on the surface of the subject by recognizing the pattern from the first photoelectric conversion signal output from the first photoelectric conversion layer When,
Reading means for reading predetermined information embedded in the pattern from the pattern of the marker detected by the marker detection means;
Storage that associates the image data created based on the second photoelectric conversion signal output from the second photoelectric conversion layer with the predetermined information read by the reading means, and stores it in a predetermined storage medium Means,
An imaging apparatus comprising: A first photoelectric conversion layer in which pixels that photoelectrically convert visible light and transmit invisible light are two-dimensionally arranged, and is stacked on the same optical path as the first photoelectric conversion layer, and the first photoelectric conversion layer An image sensor having a second photoelectric conversion layer in which pixels that photoelectrically convert invisible light that has passed through are arranged two-dimensionally;
Marker detection means for detecting a marker that has a predetermined pattern that reflects invisible light and is previously set on the surface of the subject by recognizing the pattern from the second photoelectric conversion signal output from the second photoelectric conversion layer When,
Reading means for reading predetermined information embedded in the pattern from the pattern of the marker detected by the marker detection means;
Storage that associates the image data created based on the first photoelectric conversion signal output from the first photoelectric conversion layer with the predetermined information read by the reading unit and stores the image data in a predetermined storage medium Means,
An imaging apparatus comprising: The imaging device according to claim 1 or 2,
An image pickup apparatus comprising: display means for displaying the image data and displaying the predetermined information superimposed on the image data. In the imaging device according to any one of claims 1 to 3,
The imaging apparatus according to claim 1, wherein the predetermined information is information related to a subject on which the marker is installed.

Images