JP2011058854A - Portable terminal - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a portable terminal capable of determining the relative position between shooting points by photographing a same object at a plurality of shooting points separated in space. <P>SOLUTION: The portable terminal includes a camera 16 for generating a still image by photographing the object, a posture detection section 32 for detecting the direction of the optical axis and the inclination centered at the optical axis of the camera 16, a feature point extraction section 24a for extracting feature points from the still image, a distance measuring section 25 for measuring the distance between the points on the object corresponding to the feature points in the still image, and the shooting points where the still image is photographed, and a position determination section 26 for determining the relative position of a second shooting position to a first shooting position based on the measured distance for the feature points in a first image photographed at the first shooting point, the measured distance for the feature points in a second image photographed at the second shooting point, and the detection result of the direction and inclination of the camera 16 at the first and the second shooting points. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a portable terminal, and more particularly to a portable terminal that photographs the same object as a subject at a plurality of spatially separated photographing points and determines a relative position between the photographing points.

  As a method of optically measuring the three-dimensional shape of an object, the same object is photographed as a subject at a plurality of spatially separated photographing points, and the surface of the subject is obtained using still images photographed at the respective photographing points. A technique for calculating a shape has been conventionally known (for example, Patent Document 1). In this method, for each point on the still image, matching between images taken at different shooting points is examined, and each point matched from the position of each shooting point and the orientation and tilt of the camera at each shooting point is checked. By specifying the position, the surface shape of the object is obtained. Therefore, in order to calculate the surface shape of the object by this method, at least the amount of change in position between the photographing points must be identified in advance.

  Therefore, in Patent Document 1, a signal from a GPS (Global Positioning System) satellite is received, and a displacement calculation result by an INS (Inertial Navigation System) is calculated based on the reception result. It is corrected. However, in position detection using a GPS satellite, the time required to transmit and receive radio waves is calculated by specifying the reception time of the signal from the GPS satellite with high accuracy, and the distance to the GPS satellite is calculated. For this reason, there has been a problem in that the amount of change in position cannot be accurately identified because the error is too large for movement within a range of several meters. In addition, since the detection device that detects the position using GPS and INS is expensive and has a large size, there is a problem that it is difficult to reduce the size of the terminal.

JP 2006-234703 A

  The present invention has been made in view of the above circumstances, and is inexpensive as a portable terminal capable of photographing the same object as a subject at a plurality of spatially separated photographing points and determining the relative position between the photographing points. The purpose is to provide. In particular, it is an object to provide a mobile terminal capable of accurately determining the relative position of the second shooting point with respect to the first shooting point. Moreover, it aims at providing the portable terminal which can measure the three-dimensional shape of a target object, suppressing the increase in manufacturing cost.

  A portable terminal according to a first aspect of the present invention includes an imaging unit that shoots a subject to generate a still image, an orientation detection unit that detects an orientation of the optical axis of the imaging unit and an inclination around the optical axis, and Feature point extracting means for extracting a feature point from a still image, and distance measuring means for measuring a distance between a point on the subject corresponding to the feature point in the still image and a shooting point at which the still image was shot A distance measured for the feature point in the first image taken at the first shooting point, a distance measured for the feature point in the second image taken at the second shooting point, and the first Position determining means for determining the relative position of the second shooting point with respect to the first shooting point based on the detection results of the orientation and tilt of the imaging means at the shooting point and the second shooting point.

  In this portable terminal, the distance between the point on the subject corresponding to the feature point in the still image and the shooting point of the still image is measured, the distance measured for the feature point in the first image, and the second The relative position of the second shooting point with respect to the first shooting point is determined based on the distance measured for the corresponding feature point in the image and the orientation and inclination of the imaging means at the first shooting point and the second shooting point. Is done. According to such a configuration, the relative position between the shooting points is determined by measuring the distance to the point on the subject corresponding to the feature point in the still image, so compared to the position detection using the GPS satellite, The relative position can be accurately determined while suppressing an increase in manufacturing cost.

  A mobile terminal according to a second aspect of the present invention includes, in addition to the above configuration, an acceleration sensor that detects acceleration, and a displacement amount calculation that calculates a relative position of the second shooting point with respect to the first shooting point based on the output of the acceleration sensor. And a position correction means for correcting the calculation result by the displacement amount calculation means based on the determination result by the position determination means.

  According to such a configuration, since the relative position calculation result based on the output of the acceleration sensor is corrected using the determination result of the relative position between the photographing points, the relative position determination accuracy can be improved.

  In addition to the above-described configuration, the mobile terminal according to a third aspect of the present invention is configured so that the imaging unit changes the distance between the imaging element that receives light incident from the subject via the lens and the lens and the imaging element. A focusing mechanism for adjusting the focus of the still image, and the distance measuring unit is configured to select a point on the subject corresponding to a feature point in the still image based on the position of the lens in the optical axis direction and the still image. It is comprised so that the distance between the imaging | photography points may be determined.

  According to such a configuration, the distance to the point on the subject corresponding to the feature point in the still image is determined based on the position of the lens at the time of focus adjustment for the area around the feature point. As compared with position detection, the terminal can be downsized.

  In addition to the above-described configuration, the mobile terminal according to a fourth aspect of the present invention includes a first image, a second image, each detection result of the orientation and inclination of the imaging means at the first imaging point and the second imaging point, and the relative A surface shape calculating means for calculating the surface shape of the subject based on the position determination result is provided.

  According to such a configuration, the surface shape of the subject is based on the first image and the second image, the orientation and inclination of the imaging means at the first imaging point and the second imaging point, and the determination result of the relative position. Therefore, it is possible to measure the three-dimensional shape of the object while suppressing an increase in manufacturing cost as compared with the case where the position is detected using a GPS satellite.

  According to the portable terminal of the present invention, the relative position between the photographing points is determined by measuring the distance to the point on the subject corresponding to the feature point in the still image, so that it is compared with the position detection using the GPS satellite. The relative position of the second shooting point with respect to the first shooting point can be accurately determined while suppressing an increase in manufacturing cost. Therefore, it is possible to realize a portable terminal that can photograph the same object as a subject at a plurality of spatially separated photographing points and determine the relative position between the photographing points at low cost. In addition, the three-dimensional shape of the object can be measured while suppressing an increase in manufacturing cost.

It is the figure which showed an example of schematic structure of the portable terminal by embodiment of this invention, and the mobile telephone 100 with a camera is shown as an example of a portable terminal. FIG. 2 is a block diagram illustrating a configuration example of a main part of the mobile phone 100 in FIG. 1. It is explanatory drawing which showed typically an example of operation | movement of the mobile telephone 100 of FIG. 1, and shows an example of the camera coordinate in imaging | photography point C1, C2 spaced apart spatially. FIG. 2 is an explanatory diagram schematically showing an example of the operation of the mobile phone 100 in FIG. 1, and shows the relationship among a point M on an object, a point m on an imaging surface, and an optical center C. 3 is a flowchart showing an example of an operation at the time of shape measurement of the mobile phone 100 of FIG. 3 is a flowchart showing an example of an operation at the time of shape measurement of the mobile phone 100 of FIG.

  1A and 1B are diagrams showing an example of a schematic configuration of a mobile terminal according to an embodiment of the present invention, and a mobile phone 100 with a camera 16 is shown as an example of the mobile terminal. FIG. 1A shows the mobile phone 100 viewed from the front, and FIG. 1B shows the mobile phone 100 viewed from the back.

  The mobile phone 100 is a small portable electronic device that includes a thin casing 10 that is long in the vertical direction. The mobile phone 100 can measure the three-dimensional shape of the object 3 by photographing the object while changing its position. Dimensional shape measuring device.

  On the front surface of the thin casing 10, a receiver 11 for receiving, a display 12, operation keys 13, and a microphone 14 for transmitting are provided. On the other hand, a camera 16 is disposed on the back surface of the thin housing 10, that is, on the surface opposite to the surface on which the display 12 of the thin housing 10 is provided. An antenna 15 for mobile communication is disposed on the upper side surface of the thin casing 10.

  The display 12 is a display device having a display screen. For example, a still image taken by the camera 16 is displayed as a finder image in the display screen. The receiver 11 for reception is an audio output device for reproducing an audio signal received during a call. The transmitting microphone 14 is a voice input device for inputting voice during a call.

  The camera 16 is an imaging unit that captures a subject and generates a still image. The imaging device receives light incident from the subject via a lens, and changes the distance between the lens and the imaging device to focus the still image. Consists of a focusing mechanism that performs adjustment.

  As the imaging device, for example, a CCD (Charge Coupled Devices) image sensor is used. Further, as the focusing mechanism, for example, by moving the lens in the optical axis direction, a plurality of still images taken with different focal positions in the optical axis direction are compared, and focus adjustment is performed based on the comparison result. Thus, a passive AF (autofocus) mechanism that automates focusing is used.

  A still image captured by the camera 16 is captured as a captured image for storage in a memory by operating a predetermined operation key 13, for example, an operation key to which a shutter function is assigned.

  FIG. 2 is a block diagram illustrating a configuration example of a main part of the mobile phone 100 of FIG. In addition to the CCD image sensor 16a and the focusing mechanism 16b constituting the camera 16, the cellular phone 100 includes a key input unit 21, an imaging control unit 22, a captured image storage unit 23, an image processing unit 24, a distance measuring unit 25, a position The determination unit 26, the position correction unit 27, the surface shape calculation unit 28, the triaxial geomagnetic sensor 31, the posture detection unit 32, the triaxial acceleration sensor 33, and the displacement amount calculation unit 34 are configured.

  The key input unit 21 generates a key input signal based on a shutter operation, that is, an operation of a predetermined operation key 13, and outputs the key input signal to the imaging control unit 22. The imaging control unit 22 is an image acquisition unit that controls the focusing mechanism 16b based on a shutter operation and takes a still image generated by the CCD image sensor 16a into the captured image storage unit 23.

  The image processing unit 24 includes a feature point extraction unit 24a and matching processing units 24b and 24c. The image processing unit 24 performs predetermined image processing on a still image captured as a captured image for storage in the captured image storage unit 23.

  The feature point extraction unit 24a performs edge detection on the still image, and extracts an apex angle or the like at which the edges intersect from the still image as a feature point. As a specific method for extracting feature points from a captured image, a known technique such as a Harris corner detector method may be used.

  The matching processing units 24b and 24c compare still images captured at two spatially separated shooting points, and extract points on the second still image corresponding to the points in the first still image. Verification processing is performed. Here, the still image captured at the first shooting point and captured in the captured image storage unit 23 is defined as a first image, and the still image captured at the second shooting point and captured in the captured image storage unit 23 is defined as a still image. Let it be a second image.

  In the matching processing unit 24b, for one or more feature points extracted from the first image, corresponding feature points are extracted from the second image. As a specific method of extracting feature points corresponding to feature points in the first image from the second image, for example, the position information of the first shooting point and the second shooting point, and the posture of the camera 16 at each shooting point A method is used in which an epipolar constraint line on the second image is obtained from the information, and a matching region is determined by comparing a predetermined region including a feature point on the first image with an image region on the epipolar constraint line.

  The distance measurement unit 25 measures the distance between a point on the subject corresponding to the feature point in the still image and the shooting point of the still image, and outputs the measurement result to the position determination unit 26. Specifically, the distance between the point on the subject corresponding to the feature point in the still image and the shooting point is determined based on the position of the lens in the optical axis direction during the focus adjustment by the focusing mechanism 16b.

  The triaxial acceleration sensor 33 is a sensor for detecting the acceleration generated in the terminal, and detects acceleration in three directions orthogonal to each other, for example, the X axis direction, the Y axis direction, and the Z axis direction, and the detection result Is output to the displacement amount calculation unit 34. As the triaxial acceleration sensor 33, for example, a piezoresistive type composed of a piezoresistive element or a capacitance type semiconductor sensor composed of a capacitive element is used.

  The displacement amount calculation unit 34 calculates the relative position of the second shooting point with respect to the first shooting point based on the output of the triaxial acceleration sensor 33, and outputs the calculation result to the position correction unit 27. This relative position, that is, the amount of change in the position from the first shooting point to the second shooting point, is obtained by integrating each component of acceleration in the X-axis direction, Y-axis direction, and Z-axis direction twice over time, or It is obtained by double integration with time.

  The triaxial geomagnetic sensor 31 is a magnetic sensor that detects inclination angles with respect to geomagnetism in three directions orthogonal to each other in order to detect the direction of the optical axis of the camera 16 and the inclination about the optical axis. It consists of a magnetoresistive element. The posture detection unit 32 detects the direction of the optical axis of the camera 16 and the inclination around the optical axis based on the output of the triaxial geomagnetic sensor 31, and uses the detection results as the position determination unit 26 and the surface shape calculation unit 28. Output to.

  In this posture detection unit 32, for example, as the orientation of the optical axis of the camera 16, a rotation angle (pitch angle ξ) centered on the X axis and a rotation angle (yaw angle φ) centered on the Y axis are detected, and the camera A rotation angle (roll angle ζ) about the Z axis is detected as the inclination about the 16 optical axes.

  The position determination unit 26 measures the distance measured for the feature point in the first image, the distance measured for the corresponding feature point in the second image, and the camera 16 at the first shooting point and the second shooting point. The relative position of the second shooting point with respect to the first shooting point is determined based on the detection results of the direction and the tilt, and the determination result is output to the position correction unit 27. The relative position determination accuracy can be improved by performing the determination of the relative position by matching such feature points with respect to two or more feature points extracted from the first image.

  The position correction unit 27 corrects the calculation result by the displacement amount calculation unit 34 based on the determination result by the position determination unit 26, and outputs the corrected relative position data to the surface shape calculation unit 28. For example, the displacement amount calculated by the displacement amount calculation unit 34 is corrected based on the displacement amount estimated from the distance between the point on the subject and each shooting point and the direction and inclination of the camera 16 at each shooting point. Is done.

  The surface shape calculation unit 28 uses the first image and the second image, the detection results of the orientation and tilt of the camera 16 at the first shooting point and the second shooting point, and the relative position data from the position correction unit 27. Based on this, the surface shape of the subject is calculated, and three-dimensional position data is output. As a method for calculating the surface shape of the subject, a known technique such as a stereo method may be used.

  For example, the matching processing unit 24c extracts points corresponding to the same location on the object between the first image and the second image. The surface shape calculation unit 28 obtains the surface shape of the object by repeatedly calculating the three-dimensional position of the point on the object using the principle of three-point surveying for these points.

  Alternatively, the space containing the object is divided into small cubes called voxels, and whether or not the voxel corresponds to the surface of the object is determined based on the degree of matching of pixels when each voxel is projected onto each image. By doing so, a method of calculating the surface shape of the object is also conceivable.

  FIG. 3 is an explanatory diagram schematically showing an example of the operation of the mobile phone 100 of FIG. 1, and shows an example of camera coordinates at photographing points C1 and C2 that are spatially separated. The camera coordinates are an orthogonal coordinate system in which the position of the camera 16 is the origin (optical center) and the optical axis direction of the camera 16 is the Z-axis direction.

  When measuring the three-dimensional shape of the target object, still images are used that are respectively shot with the same target object as subjects at the spatially separated shooting points C1 and C2. At this time, camera coordinates having the respective imaging points C1 and C2 as the optical center of the camera 16 are used.

FIG. 4 is an explanatory view schematically showing an example of the operation of the mobile phone 100 of FIG. 1, and shows the relationship among the point M on the object, the point m on the imaging surface, and the optical center C. . When an arbitrary point on the object is referred to as a target point M, the coordinates of the target point M in camera coordinates with the optical center C as the origin are represented as (Xc, Yc, Zc). The coordinates (x, y) of the projection point m on the imaging surface separated by a distance from the imaging surface are expressed by the following equation (1) with the position of the optical axis of the camera 16 as the origin and λ as a parameter. .

Further, the coordinates (u, v) on the final image data are such that the scale factor of the CCD image sensor 16a is k _u , k _v , the distortion is θ, and the optical axis position of the camera 16 is (u ₀ , v _0). ) Is expressed by the following equation (2).

In the above equation (2), when the projection point m is a feature point extracted from the captured image, u, v, u ₀ , and v ₀ are the position of the feature point in the captured image and the time of capturing the still image. Can be calculated on the basis of the orientation and inclination of the camera 16.

The general coordinates (X, Y, Z) in the space of the target point M are obtained by rotating and translating the camera coordinates so that the optical center C is moved to the origin. That is, the coordinates (X, Y, Z) are obtained by the following equation (3), where R is a 3 × 3 matrix representing rotational movement, and 3 × 1 matrix representing parallel movement is a vector t.

  In the above equation (3), the matrix R is determined from the direction and inclination of the camera 16 at the photographing point. The vector t is a displacement vector indicating the relative position of the second shooting point with respect to the first shooting point, with the first shooting point as the start point and the second shooting point as the end point.

From the above equations (2) and (3), the following equations (4) to (7) are obtained.

  From the above equation (4), it can be seen that the symmetry point M corresponding to the projection point m on the imaging surface is located on a half line connecting the optical center C of the camera 16 and the projection point m. Further, when considering the above formula (4) for the first shooting point and the second shooting point, if the camera coordinates at the first shooting point are regarded as general coordinates, the conversion of the above formula (3) is omitted for the first shooting point. For the second imaging point, the detection result by the posture detection unit 32 can be used for the matrix R.

Specifically, assuming that the projection point on the first image is m ₁ , the projection point on the second image is m _2, and the parameter at each photographing point is λ ₁ and λ ₂ , the following equation (8) holds. .

When the above equation (8) is solved for the vector t, the following equation (9) is obtained.

For the parameter λ, the following equation (10) holds.

Therefore, if the distance measurement unit 25 obtains the distance from the optical center C to the target point M, λ ₁ and λ ₂ can be obtained from the above equation (10), and thus the vector t using the above equation (9). Can be calculated.

That is, if the distance measurement unit 25 obtains the distance to the target point M for each of the matched feature points among the feature points extracted from the first image and the feature points extracted from the second image, the above formula is obtained. Λ ₁ and λ ₂ can be calculated from (10).

  Here, an example in which the relative position between these shooting points is determined based on still images shot at two spatially separated shooting points has been described, but shooting is performed at three or more shooting points. It is also possible to determine the relative position using the still image thus obtained. For example, for the shooting points after the third shooting point, the relative position with respect to the immediately preceding shooting point is obtained by the method described above, and the rotation matrix R is added to the previous movement, and the vector t is added and accumulated. Thus, the displacement amount from the first photographing point can be calculated.

  Steps S101 to S114 in FIGS. 5 and 6 are flowcharts showing an example of the operation at the time of shape measurement of the mobile phone 100 in FIG. First, if there is a shutter operation, the imaging control unit 22 controls the focusing mechanism 16b based on the shutter operation, and the still image generated by the CCD image sensor 16a is stored in the captured image storage unit 23 as a first image. Capture (steps S101, S102).

  The feature point extraction unit 24a extracts feature points from the first image captured in the captured image storage unit 23 (step S103). The distance determination unit 25 measures the distance to the corresponding point on the subject, that is, the distance between the optical center C and the target point M, for the feature points extracted from the first image (step S104).

  Next, if there is a shutter operation, the imaging control unit 22 controls the focusing mechanism 16b based on the shutter operation, and uses the still image generated by the CCD image sensor 16a as the second image in the captured image storage unit 23. (Steps S105 and S106).

  At this time, based on the output of the triaxial acceleration sensor 33, the displacement amount calculation unit 34 calculates the relative position of the second shooting point with respect to the first shooting point, that is, the displacement amount from the first shooting point to the second shooting point. Calculate (step S107).

  The feature point extraction unit 24a extracts feature points from the second image captured in the captured image storage unit 23 (step S108), and the matching processing unit 24b associates the feature points with the feature points extracted from the first image. This is performed (step S109). Then, the distance determination unit 25 measures the distance to the point on the subject for the feature point that matches the feature point extracted from the first image among the feature points extracted from the second image (step S110). .

  The position determination unit 26 uses the distance measured for the feature point in the first image, the distance measured for the feature point corresponding to the feature point in the second image, and the first shooting point and the second shooting point. The relative position of the second shooting point with respect to the first shooting point is determined based on the detection results of the direction and tilt of the camera 16, and a vector t is calculated (step S111). The position correction unit 27 corrects the displacement amount calculated by the displacement amount calculation unit 34 based on the vector t (step S112).

  The surface shape calculation unit 28 uses the first image and the second image, the detection results of the orientation and tilt of the camera 16 at the first shooting point and the second shooting point, and the relative position data from the position correction unit 27. Based on this, the surface shape of the subject is calculated, and three-dimensional position data is output (steps S113 and S114).

  According to the present embodiment, since the relative position between the shooting points is determined by measuring the distance to the point on the subject corresponding to the feature point in the still image, compared with the position detection using the GPS satellite, The relative position can be accurately determined while suppressing an increase in manufacturing cost. In addition, since the distance to the point on the subject corresponding to the feature point in the still image is determined based on the lens position at the time of focus adjustment in the area around the feature point, it is compared with position detection using a GPS satellite. The terminal can be miniaturized.

  In the present embodiment, an example in which the direction and inclination of the camera 16 are detected by detecting an inclination angle with respect to geomagnetism has been described, but the present invention is not limited to this. For example, the direction and inclination of the camera 16 may be detected by detecting gravitational acceleration.

DESCRIPTION OF SYMBOLS 10 Thin housing | casing 11 Receiver 12 Display 13 Operation key 14 Transmission microphone 15 Antenna 16 Camera 16a CCD image sensor 16b Focusing mechanism 21 Key input part 22 Imaging control part 23 Captured image memory | storage part 24 Image processing part 24a Feature point extraction Units 24b, 24c matching processing unit 25 distance measuring unit 26 position determining unit 27 position correcting unit 28 surface shape calculating unit 31 triaxial geomagnetic sensor 32 posture detecting unit 33 triaxial acceleration sensor 34 displacement amount calculating unit 100 mobile phone C optical center C1 , C2 Shooting point M Target point m Projection point

Claims (4)

Imaging means for capturing a subject and generating a still image;
Posture detection means for detecting the orientation of the optical axis of the imaging means and the inclination around the optical axis;
Feature point extracting means for extracting feature points from the still image;
Distance measuring means for measuring a distance between a point on the subject corresponding to the feature point in the still image and a shooting point at which the still image was shot;
The distance measured for the feature point in the first image taken at the first shooting point, the distance measured for the feature point in the second image taken at the second shooting point, and the first shooting point And a position determination means for determining the relative position of the second shooting point with respect to the first shooting point based on the detection results of the direction and inclination of the imaging means at the second shooting point. Terminal. An acceleration sensor that detects acceleration;
A displacement amount calculating means for calculating a relative position of the second shooting point with respect to the first shooting point based on the output of the acceleration sensor;
The mobile terminal according to claim 1, further comprising: a position correction unit that corrects a calculation result by the displacement amount calculation unit based on a determination result by the position determination unit. The imaging means includes an imaging device that receives light incident from a subject via a lens, and a focusing mechanism that adjusts a focus of the still image by changing a distance between the lens and the imaging device.
The distance measuring means determines a distance between a point on the subject corresponding to a feature point in the still image and a shooting point of the still image based on a position of the lens in the optical axis direction. The portable terminal according to claim 1 or 2.   The surface shape of the subject is calculated based on the first image, the second image, the detection results of the orientation and inclination of the imaging means at the first shooting point and the second shooting point, and the determination result of the relative position. The portable terminal according to any one of claims 1 to 3, further comprising surface shape calculation means.

Images