JP2015087428A - Multi-focus image photograph method, multi-focus image photograph device and digital pen - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a multi-focus image photograph method and device that allow an in-focus image to be always obtained without depending upon a change in distance between an imaging device and a subject.SOLUTION: An image pickup element 11 and an image formation lens 12 are positioned so as to focus on a subject P1 at a prescribed distance A0. Between the subject and the image formation lens 12, mirrors B1 and B2 are disposed that fold an optical path. For the subject P1 located at a position of the distance A0, out of images formed by the image pickup element 11, a part of an image formed through an optical path A is selected, while for a subject P3 located at a position of a shorter distance B0 than the distance A0, out of the images formed on the image pickup element 11, by selecting a part of an image formed through an optical path B passing though the mirrors B1, B2 and the image formation lens 12 is selected, and a sharper image can be obtained.

Description

  The present invention relates to a multifocal image capturing method, a multifocal image capturing apparatus, and a digital pen equipped with the multifocal image capturing apparatus for capturing close-up images of a plurality of subjects at different distances from an image sensor.

  A multifocal imaging device that captures a plurality of images with different distances between an objective lens and a subject by one imaging operation is disclosed in Patent Document 1 listed below.

  Further, Patent Document 2 listed below discloses an image pickup apparatus that can take an image without using a wide-angle lens when performing close-up photography such as taking a picture of itself by holding a portable device by hand. Has been.

  In the multifocal imaging device of Patent Document 1, the light passing through the objective lens is divided into transmitted light and reflected light on the spectral surface of the beam splitter, and after the optical path length of each of the divided light is made different, It is configured to reach the imaging device. As a result, Patent Document 1 discloses that an image focused on different positions of a subject such as a fine part having a three-dimensional shape can be obtained by one imaging operation.

  Further, Patent Document 2 listed below discloses an image pickup apparatus that can take an image without using a wide-angle lens when performing close-up photography such as taking a picture of itself by holding a portable device by hand. Has been.

  The imaging device of Patent Document 2 has a first optical path and a second optical path through which image light is incident on one imaging element. A plurality of mirrors for reflecting an image are provided in any one of these two optical paths. As a result, the first optical path and the second optical path can be physically separated even if the imaging device that enters the two optical paths is small, and the subject can be entirely separated without using a wide-angle lens. Patent Document 2 describes that it is possible to capture an image.

JP 2008-59121 A (published March 13, 2008) JP 2003-69867 A (published March 7, 2003)

  However, in each of the above conventional techniques, when the imaging device is brought close to or away from the subject, it is impossible to focus at high speed in accordance with the change in the distance between the imaging device and the subject. There is.

  In the case of the multifocal imaging device of Patent Literature 1, one imaging is performed with the imaging device being stationary with respect to a plurality of subjects having different subject distances. Therefore, when the imaging apparatus is brought close to or away from the subject, it is necessary to focus on the imaging apparatus side. Needless to say manual focusing, a certain amount of time is required for focusing even with autofocusing.

  Further, in the case of the imaging device of Patent Document 2, the same subject such as a human face is photographed through the first optical path and the second optical path, and images obtained through the respective optical paths are converted into one image. Therefore, it is not intended to focus at high speed in accordance with changes in the subject distance.

  The present invention has been made in view of the above problems, and an object of the present invention is to propose a multi-focus image imaging technique capable of performing high-speed focusing in accordance with a change in the distance between the imaging device and the subject. It is in.

In order to solve the above problems, a multifocal image capturing method according to an aspect of the present invention includes:
(1) In order to form an image of a subject in focus on the image sensor through an imaging lens system that keeps the distance between the image side principal point and the light receiving surface of the image sensor constant,
(2) When the actual distance between the subject and the subject-side principal point of the imaging lens system is relatively long, the shooting distance that is the distance between the light-receiving surface of the image sensor and the subject is constant. While taking a picture of the subject,
(3) When the actual distance is relatively short, the subject is photographed through an optical path that is physically bent so as to have the same optical path length as the fixed photographing distance. To do.

In order to solve the above problem, a multifocal image capturing device according to an aspect of the present invention includes:
(1) an image sensor;
(2) an imaging lens system that forms an image of a subject on the imaging device;
(3) optical path bending means for physically bending the optical path from the subject to the subject-side principal point of the imaging lens system;
(4) When the actual distance between the subject and the principal point on the subject side of the imaging lens system is relatively long, the image of the subject is captured at a fixed shooting distance without using the optical path bending means. While the imaging element and the imaging lens system are arranged so as to form an image in the focused state on the imaging element,
(5) When the actual distance is relatively short, the image of the subject is combined with the image sensor through a physically bent optical path so that the optical path length is the same as the fixed shooting distance. The optical path bending means is arranged with respect to the imaging element and the imaging lens system so as to form an image in a focused state.

  According to the above aspect of the present invention, there is an effect that focusing can be performed at high speed in accordance with a change in the distance between the imaging device and the subject.

It is explanatory drawing which shows the structure of the multifocal image pick-up device which concerns on Embodiment 1 to 3 of this invention. It is explanatory drawing which shows the light-receiving surface of the image pick-up element which the said multifocal image imaging device comprises. In the multifocal image pick-up device concerning Embodiment 4 of the present invention, it is a mimetic diagram showing overlap of depth of field. FIG. 10 is an explanatory diagram illustrating a configuration example of a mirror unit that changes a distance between an imaging device and a subject in a multifocal image capturing device according to Embodiment 5 of the present invention. It is a perspective view which shows the external appearance of the said digital pen. (A) is a figure which shows the image of a paper fiber pattern, (b) is a figure which shows the image after performing the feature point extraction process to the said image. It is a functional block diagram which shows the structure in connection with the production | generation concerning handwriting data in the digital pen which concerns on Embodiment 6 of this invention with which the said multifocal image imaging device was mounted as a proximity camera. It is a flowchart which shows the flow of the production | generation process of the handwriting data in the said digital pen. (A) is explanatory drawing which shows the locus | trajectory of the said digital pen when writing one character, (b) is explanatory drawing which shows two adjacent frame images image | photographed with the said proximity camera.

Embodiment 1
(Key points of multifocal imaging method)
First, the main points of the multifocal image capturing method according to the embodiment of the present invention will be described with reference to FIG. FIG. 1 is an explanatory diagram illustrating a configuration of a multifocal image capturing apparatus 10 according to an embodiment of the present invention. In this multifocal image capturing method, from the subject side principal point (main point M) of the imaging lens system (imaging lens 12) that forms an image of the subject (paper P1, P2, P3) on the image sensor 11 to the subject. Is suitable for close-up photography with a distance of about 10 mm.

  In addition, this multifocal image capturing method fixes the distance between the image side principal point (principal point M) of the imaging lens system and the image sensor 11 to a constant value, and changes the actual distance from the subject side principal point to the subject. This is a method corresponding to the selected shooting. In the following description, the actual distance is obtained by drawing a perpendicular from a certain point of the subject to a plane that includes the subject-side principal point of the imaging lens system and is perpendicular to the optical axis of the imaging lens system. , Meaning the length of the perpendicular.

  For example, the optical path A from the paper P1 through the imaging lens system to the image sensor 11 is a straight optical path. On the other hand, the actual path of the paper P1 is changed, and the optical path C from the same paper P2 to the image sensor 11 through the imaging lens system as the subject physically follows the optical path L1 from the paper P1 to the subject side principal point. It is a bent optical path. Further, although the actual distance of the paper P1 is changed and is different from the actual distance of the paper P2, as an object, the optical path B from the same paper P3 through the imaging lens system to the image pickup device 11 follows the straight optical path L1. The optical path is physically bent by a different bending method from C. However, the optical path lengths of the optical paths A, B, and C are all equal.

  Thus, according to the present multifocal image capturing method, the actual distance can be changed by physically bending the optical path from the subject to the subject side principal point of the imaging lens system. As a result, an in-focus (in-focus) image can be obtained regardless of changes in the actual distance, and focusing can be performed at high speed in accordance with changes in the distance between the imaging device and the subject.

(Configuration of multifocal imaging device)
A configuration of the multifocal image capturing apparatus 10 for performing the multifocal image capturing method will be described below. As the first embodiment, a configuration including mirrors (reflecting mirrors) C1 and C2 will be described, and a configuration further including mirrors B1 and B2 will be described later as a second embodiment.

  As shown in FIG. 1, the multifocal image capturing apparatus 10 forms, as a basic configuration, an image sensor 11 and images of paper P1 and P2 (paper fiber pattern images described later) as subjects on the image sensor 11. The imaging lens 12 and mirrors (reflecting mirrors) C1 and C2 as optical path bending means for physically bending the optical path L1 from the paper P1 to the principal point M of the imaging lens 12 are provided.

  Since the imaging lens 12 is a single lens, the subject side principal point and the image side principal point coincide with each other and become the principal point M. The principal point M is represented by the center position of the imaging lens 12 on the optical axis of the imaging lens 12. The imaging lens 12 may be replaced with a combination lens as an imaging lens system that eliminates aberrations. In the case of a combination lens, the subject side principal point and the image side principal point do not coincide with each other and have a distance between principal points.

  The actual distance A0 (equal to the subject distance) from the principal point M of the imaging lens 12 to the paper P1 is a, and the distance (image distance) between the principal point M and the light receiving surface of the image sensor 11 is b. Further, the optical path A (first optical path) from the paper P1 through the imaging lens 12 to the image sensor 11 is linear. The optical path A can be divided into an optical path L1 from the paper P1 to the principal point M and an optical path L2 from the principal point M to the light receiving surface of the image sensor 11. The optical path length of the optical path L1 is equal to the distance a. The optical path length of the optical path L2 is equal to the distance b. Therefore, the optical path length of the entire optical path A is a + b.

  Note that the distance between the light receiving surface of the image sensor 11 and the subject (equal to a + b in the form of FIG. 1) is called a photographing distance. When a combination lens is used instead of the imaging lens 12, the optical path length a + b includes the distance between the principal points between the subject side principal point and the image side principal point.

  On the other hand, in the optical path C (second optical path) from the paper P2 through the imaging lens 12 to the image sensor 11, the optical path from the paper P2 to the principal point M is physically bent by the mirrors C1 and C2. . In other words, the optical path from the paper P2 to the principal point M is the same as when the optical path L1 is physically bent. However, the optical path length of the optical path C is the same as the optical path length of the optical path A, and is a + b.

  Here, what is important is that the actual distance of the paper P1 is a, whereas the actual distance of the paper P2 is smaller than a by bending the optical path. Therefore, the imaging device 11 can capture a clear image regardless of whether the subject is located relatively far or relatively close. In addition, the image sensor 11 can capture both a subject located relatively far away and a subject located relatively far away clearly and simultaneously.

(Detailed configuration of optical path bending means)
The configuration of the mirrors C1 and C2 that form the bent optical path C will be described in more detail. The mirror C1 is composed of two mirrors having an inverted V shape, and the top of the inverted V shape is opposed to the paper P2. One mirror constituting the mirror C1 receives light from the paper P2 first and reflects it to the other mirror constituting the mirror C1. The other mirror reflects incident light to the mirror C2.

  In addition, the angle which the said one mirror which comprises the mirror C1 makes with respect to the other mirror is adjustable. As the adjusting means, for example, a piezoelectric element can be adopted. Thereby, the incident angle of the light from the paper P2 with respect to the one mirror can be changed, and as a result, the distance between the optical paths can be changed by moving the optical path C closer to or away from the optical path A. . The angle of the one mirror can be adjusted so that the one mirror receives the light from the paper P1. In this case, the image sensor 11 can capture substantially the same image via the optical path A and the optical path C.

  The merit of capturing substantially the same image is that, in a digital pen (to be described later) equipped with the multifocal image capturing apparatus 10, the accuracy of processing for identifying the same feature point between temporally continuous frame images can be improved. It is. The reason will be described in detail in the sixth embodiment.

  The mirror C2 is constituted by two V-shaped mirrors, and one mirror constituting the mirror C2 receives light reflected by the other mirror of the mirror C1, and is directed to the other mirror constituting the mirror C2. reflect. The other mirror of the mirror C2 reflects the incident light toward the imaging lens 12.

  According to the above configuration, the light traveling parallel to (but not limited to) the optical axis of the imaging lens 12 from the paper P2 reaches the mirror C1 through the optical path L10 and the optical path L11, and configures the mirror C1. Each time the light is reflected by the two mirrors, the traveling direction is bent. As a result, the traveling direction of the light passing through the optical path L10 and the optical path L11 is reversed by the mirror C1 through the optical path L12, and reaches the mirror C2 through the optical path L13.

  Furthermore, every time the light reaching the mirror C2 is reflected by the two mirrors constituting the mirror C2, the traveling direction is bent again. As a result, the traveling direction of the light passing through the optical path L13 is reversed again via the optical path L14 by the mirror C2, and reaches the imaging lens 12 via the optical path L15. The light that has reached the imaging lens 12 is collected by the imaging lens 12 through the optical path L16 onto the image sensor 11, and an image of the paper P2 is formed on the image sensor 11.

  The total of the optical path lengths of the optical paths L10 to L16 included in the optical path C is equal to the optical path length (a + b) of the optical path A.

  Since the total of the optical path lengths of the optical paths L10 to L16 is equal to the optical path length of the optical path A, when the imaging lens 12 forms a clear image of the paper P1 on the imaging device 11, the actual distance is changed. A clear image of the paper P2 can also be formed on the image sensor 11.

  FIG. 2 is an explanatory diagram illustrating a light receiving surface of the image sensor 11 included in the multifocal image capturing apparatus 10. As shown in FIG. 2, the light receiving surface of the image sensor 11 is divided into a plurality of light receiving regions corresponding to different types of actual distances. FIG. 2 shows a case of Embodiment 2 described later in which the number of types of actual distances is three. In this embodiment, since the number of types of actual distances is two, the light receiving surface may be divided into two. That's fine. From each light receiving area, a signal representing an image taken through an optical path having a corresponding actual distance is output.

(Example of imaging)
As shown in FIG. 1, when a clear image is formed on the image sensor 11 through the optical path A, the actual distance A0 is, for example, 50 mm, and a clear image is formed on the image sensor 11 through the optical path C. Is set to 30 mm, for example. The depth of field is constant regardless of the optical path A and the optical path C, but is set to ± 7 mm as an example.

  In this case, the multifocal image capturing apparatus 10 clearly displays the subject even when the actual distance is changed so that there is a difference of 20 mm in the actual distance far exceeding ± 7 mm of the depth of field. Can be imaged.

  More specifically, for example, the paper P1 and the paper P2 on which the subject is placed on the same plane, and the actual distance A0 of the paper P1 arranged on the optical path A is the paper arranged on the optical path C. It is assumed that the actual distance C0 of P2 is equal to 30 mm. In this case, a clear image of the paper P2 is formed on the image sensor 11, but a clear image is not formed on the paper P1.

  On the other hand, if the multifocal image capturing apparatus 10 is separated from the paper P1 and the paper P2, and the actual distance A0 of the paper P1 and the actual distance C0 of the paper P2 are both 50 mm, a clear image of the paper P1 is captured this time. An image is formed on the element 11, but a clear image is not formed on the paper P2.

  As described above, when a plurality of subjects are placed on the same plane, a clear image and a non-clear image can be obtained by moving the multifocal imaging device 10 closer to or away from the plane. Can be obtained at the same time. Therefore, by always selecting a clear image according to the change in the actual distance, it is possible to focus at high speed in accordance with the change in the distance between the imaging device and the subject. As will be described later, this image capturing method is suitable for use in generating handwriting data by a digital pen equipped with the multifocal image capturing apparatus 10.

  As described above, the image sensor 11 can capture a clear image both when the subject is located relatively far and when the subject is located relatively close. Moreover, in order to physically bend the optical path serving as a reference, it is only necessary to use reflection of light, so that it is possible to avoid light attenuation as in the case of using a beam splitter.

  Therefore, according to the multifocal image capturing method and the multifocal image capturing apparatus 10, focusing can be performed at high speed in accordance with a change in the distance between the image capturing apparatus and the subject.

  Since the multifocal image capturing method and the multifocal image capturing apparatus 10 do not use a special lens such as a compound eye lens, optical adjustment is easy and the cost is low. In addition, since it is not necessary to move the lens or the like in order to cope with a plurality of types of actual distances, it is possible to switch imaging corresponding to a plurality of types of actual distances at high speed.

[Embodiment 2]
In the first embodiment, the multifocal image capturing apparatus 10 corresponding to two types of actual distances has been described. In the present embodiment, a multifocal image capturing apparatus 10 corresponding to three types of actual distances will be described with reference to FIG.

  The multifocal image pickup apparatus 10 according to the present embodiment further includes mirrors B1 and B2 as optical path bending means, and determines the distance between the mirror B1 and the paper P3 that receives light from the paper P3 as a subject, and the paper P2 and the mirror. The distance from C1 is different. Instead of changing the distance between the mirror B1 and the paper P3, the distance between the mirror B1 and the mirror B2 may be changed (for example, the mirror B2 is moved away from the imaging lens 12).

  As a result, the optical path B from the paper P3 through the mirrors B1 and B2 and the imaging lens 12 to the image sensor 11 is formed, and the actual distance B0 at which a clear image is formed is both the actual distance A0 and the actual distance C0. The difference is 40 mm. The depth of field is the same as the optical path A and the optical path C, and is ± 7 mm as an example.

  The configuration and operation of the mirrors B1 and B2 are the same as the configuration and operation of the mirrors C1 and C2.

  Thus, the multifocal image pickup apparatus 10 corresponding to a plurality of actual distances can be obtained in correspondence with the number of installed optical path bending means provided so that the distance between the subject and the optical path bending means is different.

  The advantages of the configuration as in the present embodiment are exhibited when handwriting data is generated by a digital pen equipped with the multifocal image capturing apparatus 10 as will be described later. For example, when a user attempts to write characters on paper using a digital pen, the operation of bringing the digital pen into contact with the paper and the operation of floating the digital pen from the paper are repeated.

  In order to generate handwriting data, it is necessary to track both of the two operations as described later. On the other hand, in a fixed focus image pickup apparatus corresponding to one type of actual distance, the depth of field is about ± 5 mm, so that if the digital pen is lifted by 5 mm or more from paper, a focused image cannot be acquired. For this reason, in order to generate handwriting data, a multifocal image pickup device corresponding to at least two types of actual distances is required.

  However, since there are individual differences in the operation of floating the digital pen from the paper, there may occur a case where the multi-focal image pickup apparatus corresponding to two kinds of actual distances cannot track the operation of floating the digital pen from the paper well. For this reason, when a multifocal imaging device corresponding to three or more types of actual distances is configured, it is possible to deal with individual differences in the operation of lifting the digital pen from the paper.

  The number of installed optical path bending means is limited by the installation space. Even when the installation space is limited and the number of installed optical path bending means is limited to one, the multifocal imaging device 10 corresponding to a plurality of actual distances is obtained by the configuration of the following third embodiment. be able to.

[Embodiment 3]
As shown in FIG. 1, the multifocal image capturing apparatus 10 according to the present embodiment is a telescopic unit that advances and retracts a mirror C <b> 1 with the inverted V-shaped top and the paper P <b> 2 facing each other in parallel with the optical axis of the imaging lens 12. CU is provided. In the mirror B1 corresponding to the mirror C1, the telescopic unit BU can be similarly provided.

  With the above configuration, the actual distance C0 can be changed from 30 mm to 40 mm by moving the mirror C1 to the same position as the mirror B1, that is, the same position on the optical axis of the imaging lens 12. Therefore, the installation of the mirrors B1 and B2 can be omitted.

  In addition, it is possible to cope with individual differences of users who use the digital pen while saving space by setting the number of optical path bending means as one.

[Embodiment 4]
Next, in the multifocal image capturing apparatus 10 capable of dealing with a plurality of real distances, an embodiment in which the depths of field at a plurality of real distances are overlapped will be described based on FIG. . FIG. 3 is a schematic diagram illustrating the overlap of the depth of field in the multifocal image capturing apparatus 10 according to the fourth embodiment of the present invention.

  As shown in FIG. 3B, the depths of field at the actual distances A0, B0, C0 are all ± 7 mm, but the actual distances A0, B0 and between the actual distances B0, C0. Each of these depths of field has an overlap in the optical axis direction of the imaging lens 12. In other words, when the coordinates representing the position on the optical axis are considered, the range that the coordinates of the depth of field at the actual distance A0 can take, and the range that the coordinates of the depth of field at the actual distance B0 can take. There is an overlap. The same applies to the actual distances B0 and C0.

  In this way, when the subject is photographed in the vicinity of the boundary value between different real distances and an attempt is made to capture the image at the actual distance that is in focus, the hysteresis control is performed with a wider range for the actual distance switching target value. It becomes possible. By performing the hysteresis control, the operation of selecting the output signal of the light receiving region in focus among the output signals from the light receiving region shown in FIG. 2 is frequently switched around the boundary value between different actual distances. (Chattering) can be prevented.

  In addition, when compared with the actual distance (50 mm ± 7 mm) of the conventional fixed focus camera shown in FIG. 3A, the actual distance of the multifocal image capturing apparatus 10 is substantially in a range from 30 mm−7 mm to 50 mm + 7 mm. It can be seen that it can be expanded.

[Embodiment 5]
Next, a new configuration example of the mirror unit that changes the actual distance will be described with reference to FIG. FIG. 4 is an explanatory diagram illustrating a configuration example of a mirror unit that changes the actual distance in the multifocal image capturing device according to the fifth embodiment of the present invention.

  As shown in FIG. 4A, the multifocal image capturing apparatus according to this embodiment includes a mirror unit 30 having a polygonal columnar housing, and half of the whole is accommodated in the mirror unit 30. Is provided with a camera module 31 divided into two.

  The polygonal columnar casing has a plurality of (for example, seven) side surfaces. Among them, the side surface facing the light receiving surface of the camera module 31 is opened so that light from the subject enters the mirror unit 30, and the side surface is inclined with respect to the optical axis of the imaging lens built in the camera module 31. The inner surface is a mirror surface.

  As shown in FIG. 4B, the plurality of side surfaces whose inner surfaces are mirror surfaces are inclined at various angles with respect to the optical axis, and the inclination angle is determined by the light incident on the housing from the camera module 31. It is set so that it will be reflected so that it may go around and may inject into the camera module 31 finally.

  According to the above configuration, the camera module 31 exposed outside the mirror unit 30 directly captures a subject, and the camera module 31 accommodated in the mirror unit 30 passes through the optical path bent by the mirror unit 30. It is possible to perform shooting with a shorter actual distance than in direct shooting. For example, the actual distance A0 in direct photographing can be set to 50 mm as described above, and the actual distance C0 in photographing through the mirror unit 30 can be set to 30 mm as described above.

  Therefore, it can be said that the multifocal image capturing apparatus including the mirror unit 30 according to the present embodiment is a bifocal image capturing apparatus.

  Since the camera module 31 is united with the mirror unit 30, compared with a combination in which the image sensor 11 and the mirrors C 1 and C 2 shown in FIG. Can do.

  In addition, if the way of reflection by the mirror unit 30 is changed by changing the inclination angle of a plurality of side surfaces whose inner surfaces are mirror surfaces, the length of the optical path parallel to the optical axis changes, so that the actual distance is changed. Can do. Therefore, by providing a configuration in which the angle of inclination of a plurality of side surfaces whose inner surfaces are mirror surfaces can be changed, a multifocal image pickup device that supports three or more types of actual distances and saves space is provided. Can do.

[Embodiment 6]
(Application to digital pen)
(1. Features of the digital pen)
One application example of the multifocal image pickup apparatus according to the first to fifth embodiments is mounting on a digital pen that has a shape similar to a ballpoint pen and can be used as a writing instrument like a ballpoint pen. The main concept of this digital pen is, “As with conventional pens, it has the convenience of writing on plain paper with ink, and it is a pen that converts handwritten characters or pictures written by the user into digital data. ".

Furthermore, the following four points can be given as specific features of the digital pen.
(1) A sensing device such as a tablet that makes up the digitizer is not required, or paper for digital pens is not required, and ink is drawn from the pen tip, so characters can be written anytime, anywhere with only one digital pen. it can.
(2) There are no restrictions on the type of paper on which characters and the like are written and the size of the paper.
(3) Handwriting such as letters or pictures written on paper by the user can be actually visually recognized. At the same time, the handwriting at that time, including the user's eyelids, is directly converted into digital data and stored in the digital pen. Is done.
(4) The handwritten digital data is further converted into image data such as JPEG format and stored in the digital pen. Therefore, since dedicated software is not required to display handwriting such as letters or pictures written by the user on the display, a personal computer or a smartphone can be used as a display device. Image data that the user wants to confirm is selectively transferred from the digital pen to the display device, and characters or pictures written by the user can be easily confirmed on the screen of the display device.

  Note that text data may be generated from handwritten digital data, and the text data may be sent to a display device so that not only character confirmation but also character editing can be performed. The pen function of the digital pen is not limited to the configuration realized by the ballpoint pen. For example, a fountain pen, mechanical pencil, or the like may be provided instead of or in addition to the ballpoint pen.

(2. Appearance of digital pen)
FIG. 5 is an external view of the digital pen. As shown in FIG. 5, the digital pen 1 includes a distal end portion 2, a main body portion 3, a clip portion 4, and a cap portion 5.

  In the main body 3 (pen component), an ink cartridge similar to a normal ballpoint pen is accommodated, and a pen tip integrated with the ink cartridge advances and retreats from the tip 2. The advancement / retraction of the pen tip is linked to the user's knocking operation on the clip unit 4 or the cap unit 5. Further, the knocking operation of the user for moving the pen tip forward / backward is also a switch operation for turning on / off the power of the digital pen 1. The ink cartridge can be replaced.

  Since the tip 2 is open, the pen tip can be advanced and retracted as described above, and the light L from the subject can be taken in.

  The main body 3 is further generated with the multifocal image capturing device 10 as a proximity camera, and a data processing unit 20 (see FIG. 7) that generates handwriting data from the output of the image sensor 11 as described later. A memory for storing handwritten data and a rechargeable battery as a power source are incorporated.

  The cap unit 5 is provided with, for example, a micro USB connector, and handwriting data stored in the memory can be output to an external display device via the micro USB connector. Note that a communication unit that transmits and receives handwriting data between an external display device and the digital pen 1 may be incorporated in the digital pen 1 instead of or in addition to the micro USB connector.

(3. Digital pen subject)
Here, for the digital pen 1, as shown in FIG. 6A, the subject is a pattern woven by paper fibers on which characters or pictures are written. This pattern is hereinafter referred to as a paper fiber pattern.

  Paper is made by tangling fine fibers (generally fibers having a thickness of 4 to 70 μm and a length of about 0.25 to 50 mm) made of wood pulp at random. The number of patterns in which paper fibers of various thicknesses and lengths are intertwined may be infinite. Therefore, even if a paper fiber pattern is a single sheet of paper, the same paper fiber pattern will not appear so that it will change depending on the position on the paper surface, and if the paper changes, it will become another pattern. equal.

  Since the digital pen 1 is close-up of such a fine paper fiber pattern and extracts feature points from the paper fiber pattern image as shown in FIG. Requires fine imaging and high resolution.

(4. Configuration to generate handwriting data)
FIG. 7 is a functional block diagram showing a configuration relating to generation of handwriting data in the digital pen 1.

  The output of the multifocal image capturing apparatus 10, that is, the output of the image sensor 11 is sent to the data processing unit 20. The data processing unit 20 includes a determination unit 21, a frame memory 22, and a generation unit 23.

  The multifocal imaging device 10 receives image light Li1 and Li10 corresponding to at least two types of actual distances, and determines an image signal Q1 based on the image light Li1 and an image signal Q10 based on the image light Li10. To 21. For example, the image light Li1 corresponds to the paper P1, and the image light Li10 corresponds to the paper P2.

  The determination unit 21 generates image data of the paper fiber pattern of the paper P1 and the paper P2 from the image signal Q1 and the image signal Q10, respectively, and the image data that is more focused (for example, the contrast is relatively high) Image data) is output frame by frame. Of the two consecutive frames of image data output by the determination unit 21, the image data of the frame output first is output to the frame memory 22, and the image data of the frame output later is output to the generation unit 23. The

  The generation unit 23 receives the continuous two frames of image data output from the determination unit 21, and generates and outputs handwriting data as will be described later.

(5. Overview of processing to generate handwriting data)
FIG. 8 is a flowchart showing an outline of processing (handwriting data generation processing) for generating handwriting data from a paper fiber pattern image. Specifically, the handwriting data generation process includes the following processes S1 to S5.
S1: A plurality of paper fiber patterns having different actual distances are simultaneously photographed frame by frame by the multifocal image capturing apparatus 10 at a period (frame rate) of 1/30 seconds. The determination means 21 specifies the image with the highest contrast among the images of the plurality of paper fiber patterns photographed at the same time.
S2: The generation unit 23 detects and extracts feature points from the paper fiber pattern image specified in S1. As algorithms for this processing, there are known SIFT (Scale-Invariant Feature Transform), SURF (Speed-Up Robust Features), ORB (Oriented-BREF) and the like.
S3: The generation unit 23 verifies the matching of the feature points of the paper fiber pattern between the previous frame and the current frame.
S4: The generation unit 23 calculates the moving direction and moving distance of the proximity camera (digital pen 1) from the movement of the matched feature points.
S5: The generation unit 23 stores the calculated movement direction and movement distance as stroke information of the digital pen 1, and then returns to the process of S1.

  By repeating the above processing, as a result of accumulating stroke information, handwriting data corresponding to a character or a picture written by the user can be generated.

  In order to improve the accuracy of the process of S3, that is, the process of identifying the same feature point between temporally consecutive frame images, as described in the first embodiment, for example, the one mirror constituting the mirror C1 However, it is preferable to adjust the angle formed with respect to the other mirror so that the image sensor 11 can capture substantially the same image via the optical path A and the optical path C. The reason is as follows.

  For example, when the imaging area of the optical path A and the imaging area of the optical path C are different, the feature point matching can be performed only by comparing the previous frame image and the current frame image in each area. On the other hand, when the imaging area of the optical path A and the imaging area of the optical path C are substantially the same or the same, in addition to the comparison between the previous frame image and the current frame image of the same imaging area, It is possible to compare the previous frame image with the current frame image of the imaging area of the optical path C, and compare the current frame image of the imaging area of the optical path A with the previous frame image of the imaging area of the optical path C. Therefore, since the number of comparison patterns can be increased, the accuracy of the feature point matching process can be improved. Furthermore, if the imaging area of the optical path B is made substantially the same as the imaging area of the optical path A, the number of comparison patterns increases, and the accuracy is further increased.

(6. Supplement 1 for processing to generate handwriting data)
FIG. 9A is an explanatory diagram showing the trajectory of the digital pen 1 when the character “A” is written, and FIG. 9B is an explanatory diagram showing two adjacent frame images taken by the proximity camera. FIG.

  As shown in FIG. 9A, when the user tries to write one character on paper, when the digital pen 1 is moved from one image (stroke) to the next image (stroke), the digital pen 1 Will be lifted from the page. The locus of the digital pen 1 in a lifted state is indicated by a dotted line in FIG.

  While the digital pen 1 is moving in a lifted state, the multifocal image capturing apparatus 10 continues to capture the paper fiber pattern along the trajectory of the digital pen 1. This is because it is not possible to generate handwriting data for one entire character unless the locus of the digital pen 1 in a floating state is acquired as locus data. In other words, if the locus of the raised digital pen 1 is not acquired as the locus data, the matching of the feature points of the paper fiber pattern by the generating means 23 is interrupted, so that one image (stroke) and the next image (stroke) are The generation means 23 cannot recognize the positional relationship. In this case, for example, it is impossible to generate a batch of handwriting data representing the character “A”, and the next image (stroke) is output as another handwriting data of another character.

  In addition, even when the digital pen 1 is moved too quickly, it is impossible to generate a batch of handwriting data representing one character. This is because, in order to track the trajectory of the digital pen 1 regardless of whether the digital pen 1 is floating or not, as shown in FIG. 9B, two adjacent frame images F1 and F2 are connected to each other. This is because an overlap of a certain threshold (for example, about 75%) or more is necessary as a ratio to the image area of one frame. That is, when the overlap between two adjacent frame images F1 and F2 becomes less than the threshold, recognition of a series of handwriting is interrupted, and recognition of a new handwriting (for example, a new character) is started.

  As shown in FIG. 9B, when acquiring one frame image, the range on the paper surface taken by the proximity camera is about 4 mm square.

  As described above, the multifocal imaging device 10 has a subject surface (for example, a paper surface) regardless of whether the digital pen 1 is in contact with the subject and draws a locus, or when the digital pen 1 is drawn from a subject in a floating state. Can be taken brightly and with high definition.

  At this time, since the multifocal imaging device 10 simultaneously performs imaging with different actual distances between the principal point M of the imaging lens 12 and the paper surface, the paper fiber pattern when the digital pen 1 is in contact with the paper surface. Image (referred to as a near image) and a paper fiber pattern image (referred to as a far image) when the digital pen 1 is floating from the paper surface can be obtained immediately.

  Therefore, since the lens can be physically moved by autofocus control and the near image and the far image can be switched much faster than switching between the near image and the far image, the generation means 23 can be used for the digital pen 1. It is possible to generate highly accurate handwriting data that accurately follows a fast movement.

  In addition, since the multifocal imaging device 10 has a configuration that can handle three or more types of actual distances, the digital pen 1 can be adapted to individual differences in how the digital pen 1 is lifted from the paper surface. .

(7. Supplement 2 for processing to generate handwriting data)
In the above-described example, the near image when the digital pen 1 is in contact with the paper surface and the far image when the digital pen 1 is floating from the paper surface are obtained by switching immediately, that is, the focus is better. In order to select an image, a contrast determination process was used.

However, the present invention is not limited to this method, and other methods as described below can be adopted.
○ A pressure sensor is attached to the pen tip, and the near image and the far image are switched based on the output of the pressure sensor indicating whether or not the pen tip is in contact with the paper.
○ Attach a distance sensor (infrared rays, etc.) to the pen tip, and determine near and far images by distance.
○ When extracting feature points of a paper fiber pattern image, the one with more feature points extracted is adopted as a near image or a far image.

  Note that a method using contrast determination processing or feature point extraction processing is advantageous in terms of downsizing and cost reduction of a digital pen, compared to a method of providing a sensor in a digital pen.

[Summary]
A multifocal image capturing method according to aspect 1 of the present invention includes:
(1) The image of the subject is focused on the image sensor through the imaging lens system imaging (lens 12) that keeps the distance between the image side principal point (principal point M) and the light receiving surface of the image sensor 11 constant. In order to form an image in the state,
(2) When the actual distance between the subject and the subject-side principal point (principal point M) of the imaging lens system is relatively long (actual distance A0), the light-receiving surface of the image sensor and the above-mentioned While shooting the subject so that the shooting distance, which is the distance to the subject, is constant (a + b),
(3) When the actual distance is relatively short (in the case of the actual distance B0 or C0), the optical path (B or C) that is physically bent so as to have the same optical path length as the fixed photographing distance. ), The subject is photographed.

  In the above configuration, the actual distance is defined as a perpendicular line extending from a certain point of the subject to a plane including the subject-side principal point of the imaging lens system and perpendicular to the optical axis of the imaging lens system. It means the length of perpendicular. When the subject is photographed, an image of the subject imaged in the focused state on the image sensor is obtained.

  According to the above method, when photographing a subject at the fixed photographing distance, a physically bent optical path (hereinafter referred to as a bent optical path) is not used. The actual distance at this time is A0. On the other hand, if the actual distance when the subject is photographed through the bent optical path is set to B0 so that the optical path length is the same as the above-mentioned fixed photographing distance, the actual distance B0 is a natural result by bending the optical path. It becomes smaller than the actual distance A0.

  Therefore, even when the actual distance changes to the actual distance A0 or the actual distance B0, it is possible to obtain an image of the subject that is imaged in the focused state on the image sensor. That is, it is possible to provide a multifocal image capturing method capable of performing high-speed focusing in accordance with a change in the distance between the imaging device and the subject.

  The actual distance that can be changed is not limited to two. By preparing two or more bent optical paths with different ways of bending the optical path, the actual distance can be changed to three or more.

  The imaging lens system may be a combination lens composed of a plurality of types of lenses, or may be a single lens only. When the imaging lens system is composed of only a single lens, the subject side principal point and the image side principal point are the same, and the principal point is represented by the center position on the optical axis of the single lens.

A multifocal image capturing apparatus 10 according to aspect 2 of the present invention includes:
(1) the image sensor 11;
(2) an imaging lens system (imaging lens 12) that forms an image of the subject (paper P1, P2) on the imaging element;
(3) optical path bending means (mirrors C1, C2) for physically bending the optical path from the subject to the subject side principal point (principal point M) of the imaging lens system;
(4) When the actual distance between the subject and the subject-side principal point of the imaging lens system is relatively long (actual distance A0), a fixed photographing distance (without the optical path bending means) While the imaging element and the imaging lens system are arranged so that the image of the subject is focused on the imaging element in a + b),
(5) When the actual distance is relatively short (in the case of the actual distance B0 or C0), the optical path is physically bent so as to have the same optical path length as the fixed shooting distance. The optical path bending means is arranged with respect to the imaging element and the imaging lens system so that an image of a subject is focused on the imaging element in a focused state.

  In the above configuration, the actual distance can be changed by the optical path bending means, as already described for the multifocal image capturing method.

  According to the above configuration, even when the actual distance is long and short, the image sensor is in focus without changing the mutual arrangement of the image sensor and the imaging lens system. It is possible to obtain an image of the subject imaged in step. The fact that there is no need to change the mutual arrangement of the image sensor and the imaging lens system, and that only the optical path bending means is not required means that it is not necessary to adjust the focusing state such as autofocusing. That is. Therefore, it is possible to provide a multifocal image capturing apparatus capable of performing high-speed focusing in accordance with a change in the distance between the image capturing apparatus and the subject.

  In the aspect 2, the optical path bending means of the multifocal image capturing apparatus 10 according to aspect 3 of the present invention may be configured by a combination of a plurality of reflecting mirrors (mirrors C1, C2).

  Thus, the bending optical path from the subject (paper P1) to the image sensor 11 is different from the optical path that does not pass through the optical path bending means, that is, the actual distance between the subject side principal point (main point M) and the subject (paper P2). The close optical path can be formed while avoiding light attenuation as in the case of using a beam splitter.

  In the aspect 3, the plurality of reflecting mirrors (mirrors C1, C2) of the multifocal image capturing device 10 according to aspect 4 of the present invention move along the optical axis of the imaging lens system (imaging lens 12). It may be configured to be possible.

  As a result, the actual distance between the subject-side principal point and the subject can be variously changed according to the position where the plurality of reflecting mirrors are moved along the optical axis.

  Even if the actual distance between the subject-side principal point and the subject changes, the optical path length is always maintained at the same optical path length as the fixed shooting distance, so that the focus is not affected by the position of the plurality of reflecting mirrors. You can always get the right image. That is, it is possible to obtain the same effect that enables high-speed focusing in accordance with the change in the distance between the imaging device and the subject.

  In the multifocal imaging device 10 according to the aspect 5 of the present invention, in the aspect 3 or 4, the angle of at least the first reflecting mirror that receives light from the subject can be changed among the plurality of reflecting mirrors. It may be configured.

  According to said structure, the incident angle of the light from a to-be-photographed object with respect to the reflective mirror which receives the light from a to-be-photographed first can be changed. As a result, the distance between the optical paths can be changed by moving the optical path through the optical path bending means closer to or away from the optical path not through the optical path bending means. In addition, the image sensor can capture images of substantially the same or the same part of the subject via both optical paths. In this case, it is possible to improve the accuracy of the process of identifying the same feature point between two images obtained through both optical paths.

A digital pen 1 according to an aspect 6 of the present invention includes:
(1) A digital pen that functions as a pen of a writing instrument and includes a pen component having a pen-shaped housing (main body 3),
(2) The multifocal image pickup device 10 according to any one of the above aspects 2 to 5 mounted in the casing so as to capture light from a subject incident from the pen tip (tip portion 2);
(3) It is characterized by comprising generating means 23 for generating handwriting data representing a handwriting using the digital pen by a user based on an output signal of the image sensor 11.

  According to said structure, the multifocal image imaging device 10 mounted in the digital pen 1 can focus at high speed according to the change of the distance of an imaging device and a to-be-photographed object. Furthermore, since the digital pen 1 includes the generation unit 23, the user generates handwriting data representing the handwriting using the digital pen based on the output signal of the image pickup device 11 provided in the multifocal image pickup apparatus 10. can do.

  At this time, the multifocal image capturing apparatus 10 focuses on the subject surface (for example, a paper surface) regardless of whether the digital pen 1 is in contact with the subject and draws a locus, or when the digital pen 1 is drawn while the digital pen 1 is floating from the subject. You can shoot with

  In general, when a user writes a character on paper with a writing tool such as a pen, for example, the writing tool is lifted from the paper once after writing the first stroke of the character and before writing the second stroke. That is, one character is written on the paper surface as the writing instrument repeatedly touches or leaves the paper.

  Therefore, in order for the generating means 23 to generate handwriting data corresponding to handwriting (for example, one character) when the user uses the digital pen 1, the subject surface when the digital pen 1 is in contact with the subject, It is necessary to image both the subject surface and the subject surface when the digital pen 1 is floating from the subject in a focused state.

  The multifocal imaging device 10 can respond to this request very quickly.

  The multifocal image capturing device 10 of the digital pen 1 according to the aspect 7 of the present invention is configured to capture the paper fiber pattern of paper to be written using the digital pen 1 in the aspect 6. preferable.

  As described above, the paper fiber pattern changes when the paper changes, and even if the paper is the same, the paper fiber pattern changes when the position on the paper surface changes. Therefore, a vector representing a handwriting can be detected by tracking a change in an image obtained by photographing a paper fiber pattern.

  At this time, since the multifocal image capturing apparatus 10 simultaneously performs imaging with different actual distances between the subject-side principal point of the imaging lens system and the paper surface, the paper fiber when the digital pen 1 is in contact with the paper surface. A pattern image (referred to as a near image) and a paper fiber pattern image (referred to as a far image) when the digital pen 1 is floating from the paper surface can be obtained immediately.

  Therefore, since the near image and the far image can be switched at a much higher speed than switching between the near image and the far image by the autofocus control, the generation unit 23 accurately follows the fast movement of the digital pen 1. Highly accurate handwriting data can be generated.

  The digital pen 1 according to aspect 8 of the present invention is the digital pen 1 according to aspect 6 or 7, wherein the image sensor 11 generates image data of the subject generated by the imaging device 11 through the optical path bending means, and the optical path bending means. A determination unit 21 that determines the image data that is relatively in focus among the image data of the subject that is generated by imaging without passing through the image and outputs the image data to the generation unit 23; Preferably, the generation means uses image data that is relatively focused on for generating the handwriting data.

  As a result, among the plurality of image data obtained simultaneously by the multifocal image capturing apparatus 10 via different optical paths, the focal point is relatively focused, in which the actual distance between the subject-side principal point of the imaging lens system and the subject is different. The determination means 21 selects the image data that matches. Therefore, it is possible to switch the image data used by the generating unit 23 to generate handwriting data at high speed.

  The multifocal imaging device 10 according to the ninth aspect of the present invention is the multifocal imaging device 10 according to any one of the second to fifth aspects, wherein the depth of field for each of the plurality of subjects having different actual distances is an optical axis of the imaging lens system. The actual distances for each of the plurality of subjects may be set so as to partially overlap each other in a direction parallel to.

  According to the above configuration, the operation of selecting the signal output of the light receiving region in focus among the signal outputs from the light receiving region corresponding to the actual distance of the image sensor near the boundary value between different real distances, It is possible to prevent frequent switching (chattering).

  In the above-described aspect 2 of the multifocal image capturing apparatus 10 according to the aspect 10 of the present invention, the optical path bending means has a polygonal columnar shape and has a housing that accommodates a part of the image sensor and the imaging lens system therein. And the inner surface of the plurality of side surfaces is configured as a mirror unit, and the plurality of side surfaces serving as the mirror surfaces have different inclination angles with respect to the optical axis of the imaging lens system, and The tilt angle may be set so that light incident on the housing is reflected around the imaging element and the imaging lens system, and finally enters the imaging lens system.

  According to the above configuration, it is possible to provide a bifocal image pickup device that performs direct shooting without using the optical path bending means and shooting with a reduced actual distance through the bent optical path. Furthermore, since the image pickup element and the imaging lens system are housed inside the mirror unit, the multifocal image pickup apparatus can be reduced in size.

  The multifocal imaging device 10 according to the aspect 11 of the present invention includes the optical path bending means (mirrors B1, B2, mirrors C1, C2) according to any one of the aspects 2 to 5, and the actual distance is the same. It may be different in three ways.

  According to the above configuration, the multifocal imaging device 10 has the same number of bent optical paths as the radix provided with the optical path bending means, and the actual distance between the subject-side principal point and the subject is different. The optical path can be formed separately from the optical path not through the optical path bending means.

  The number of installation bases of the optical path bending means is limited by the allowable space for installing the optical path bending means. In consideration of this point, it is possible to configure the number of installation bases of the optical path bending means to be one, and the plurality of reflecting mirrors constituting the optical path bending means to be movable along the optical axis of the imaging lens system. This is advantageous for space saving of the multifocal image pickup apparatus. That is, the actual distance between the subject-side principal point and the subject can be changed variously while minimizing the installation space for the optical path bending means.

  The digital pen according to the above aspects 6 to 8 may include any of the multifocal image capturing apparatuses 10 according to the above aspects 9 to 11.

  The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope shown in the claims, and embodiments obtained by appropriately combining technical means disclosed in different embodiments. Is also included in the technical scope of the present invention. Furthermore, a new technical feature can be formed by combining the technical means disclosed in each embodiment.

  The present invention can be used for an application of imaging corresponding to a plurality of subject distances, and can be particularly preferably used for generating handwriting data of a digital pen.

1 Digital Pen 2 Tip (Pen Tip)
3 Main body (housing, pen component)
DESCRIPTION OF SYMBOLS 10 Multifocal image pick-up device 11 Image pick-up element 12 Imaging lens (imaging lens system)
21 determination means 23 generation means 30 mirror unit 31 camera module (imaging device and imaging lens system)
B1, B2, C1, C2 mirrors (optical path bending means, reflecting mirrors)
M principal point (subject principal point and image principal point)
P1 paper (subject)
P2 paper (subject)
P3 paper (subject)
L1, L2, L10, L11, L12, L13 Optical path

Claims (8)

In order to form an image of a subject in focus on the image sensor through an imaging lens system that maintains a constant distance between the image side principal point and the light receiving surface of the image sensor,
When the actual distance between the subject and the subject side principal point of the imaging lens system is relatively long, the shooting distance, which is the distance between the light receiving surface of the image sensor and the subject, is constant. While shooting the above subject,
When the actual distance is relatively short, the subject is photographed through an optical path that is physically bent so as to have the same optical path length as the fixed photographing distance. Image capturing method. An image sensor;
An imaging lens system for forming an image of a subject on the image sensor;
Optical path bending means for physically bending the optical path from the subject to the subject-side principal point of the imaging lens system;
When the actual distance between the subject and the subject-side principal point of the imaging lens system is relatively long, the image of the subject is transferred to the image sensor at a fixed shooting distance without using the optical path bending means. While the imaging element and the imaging lens system are arranged so as to form an image in a focused state,
When the actual distance is relatively short, the image of the subject is focused on the image sensor through the optical path that is physically bent so that the optical path length is the same as the fixed shooting distance. A multifocal image pickup apparatus, wherein the optical path bending means is arranged with respect to the image pickup element and the image forming lens system so as to form an image. The multi-focus image capturing apparatus according to claim 2, wherein the optical path bending means is configured by a combination of a plurality of reflecting mirrors. The multi-focus image capturing apparatus according to claim 3, wherein the plurality of reflecting mirrors are configured to be movable along an optical axis of the imaging lens system. 5. The multifocal image capturing apparatus according to claim 3, wherein among the plurality of reflecting mirrors, at least a reflecting mirror that first receives light from the subject is configured to be capable of changing an angle. . A digital pen that functions as a pen of a writing instrument and includes a pen component having a pen-shaped housing,
The multifocal image pickup device according to any one of claims 2 to 5, which is mounted in the housing so as to take light from a subject incident from a pen tip,
A digital pen comprising: generation means for generating handwriting data representing a handwriting using the digital pen by a user based on an output signal of the imaging device. The digital pen according to claim 6, wherein the multifocal image capturing device is configured to capture a paper fiber pattern of paper to be written using the digital pen. Among the image data of the subject generated by the image pickup device imaging through the optical path bending means and the image data of the subject generated by imaging without passing through the optical path bending means, A determination unit that determines image data of a relatively focused image and outputs the image data to the generation unit;
8. The digital pen according to claim 6, wherein the generation unit uses image data that is relatively focused on for generating the handwriting data.

Images