JP5928228B2 - Subject detection apparatus, subject detection method, and program - Google Patents

Description

  The present invention relates to a subject detection device, a subject detection method, and a program.

Conventionally, as an augmented reality (AR) technique, an image processing apparatus that displays a virtual object (for example, a three-dimensional model) in association with an image of a marker (real object) in a captured image is known. (For example, refer to Patent Document 1).
In this image processing apparatus, based on the result of image analysis of the captured frame image, the inclination of the image of the marker in the frame image (for example, the rotation angle from the normal viewing state in the display screen) is detected, and the marker Is recorded as an initial value. In addition, the image processing apparatus calculates a relative difference between the inclination of the marker in the sequentially captured frame image and the inclination recorded as the initial value, specifically, a difference in coordinate values. Then, the image processing apparatus performs a process of sequentially displaying the virtual object image tilted along a predetermined vector from the calculated coordinate value difference.

JP 2006-72667 A

  By the way, when the coordinates of each point of the marker in the three-dimensional space are converted into corresponding coordinates in the frame image using a predetermined coordinate conversion formula, from the coordinate data of a plurality of points constituting the marker image in the frame image When the initial value is set, there is a possibility that the posture of the marker and the positional relationship with respect to the imaging unit cannot be properly estimated. In this case, the virtual object cannot be displayed properly.

  The present invention has been made in view of such a problem, and an object of the present invention is to provide a subject detection apparatus, a subject detection method, and a program capable of appropriately estimating the posture of a specific subject with respect to an imaging unit. Is to provide.

In order to solve the above problems, an aspect of the subject detection apparatus according to the present invention is as follows.
An acquisition unit that sequentially acquires images of a specific subject imaged by the imaging unit; a first detection unit that detects an inclination of the imaging unit with respect to the gravitational direction when one image is acquired by the acquisition unit; the specific subject relative to the imaging unit associated with a plurality of points constituting the image of a specific subject included in tilt-out及 beauty the one image with respect to the gravity direction of the imaging unit, which is detected by the first detection means A first calculating unit that calculates the inclination of the specific subject with respect to the gravitational direction based on the provisional posture of the image, and the gravity of the imaging unit when another image different from the one image is acquired by the acquiring unit. second detection means, the second said specific object scene calculated by the inclination-out及 beauty said first calculating means with respect to the gravity direction of the imaging unit, which is detected by the detection means for detecting a tilt with respect to the direction On the basis on the inclination with respect to the direction of gravity, the spatial coordinates of each point of the plurality of points constituting the image of the specific object as a coordinate conversion formula initial value to be converted to plane coordinates in the other image, the imaging a second calculating means for calculating a relative posture of the specific subject for parts, Ru provided with, characterized in that.

In order to solve the above problem , an aspect of the subject detection method according to the present invention is:
A method for detecting a subject using a subject detection device, which sequentially acquires an image of a specific subject imaged by an imaging unit, and detects an inclination of the imaging unit relative to the gravitational direction when one image is acquired a process for the detected tilt-out with respect to the gravity direction of the imaging unit 及 beauty the of the specific subject with respect to the imaging unit associated with a plurality of points constituting the image of the particular subject included in one image A process of calculating the inclination of the specific subject with respect to the gravitational direction based on a provisional posture; and a process of detecting the inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired. , based on the inclination with respect to the detected tilt-out及 beauty said gravity direction of the specific subject with respect to the gravity direction of the imaging unit, the space locus of points of the plurality of points constituting the image of the specific object As the initial value of the coordinate conversion formula for converting the plane coordinates in the other image, including a process of calculating the relative attitude of the specific subject with respect to the imaging unit, it is characterized.

Moreover, in order to solve the said subject, the one aspect | mode of the program which concerns on this invention is:
An acquisition unit that sequentially acquires an image of a specific subject captured by the imaging unit, and a tilt of the imaging unit relative to the gravitational direction when one image is acquired by the acquisition unit. first detecting means, wherein said image pickup unit associated with a plurality of points constituting the image of a specific subject included in tilt-out及 beauty the one image with respect to the gravity direction of the imaging unit, which is detected by the first detection means The first calculation means for calculating the inclination of the specific subject with respect to the gravitational direction based on the provisional posture of the specific subject with respect to the above, when the acquisition means acquires another image different from the one image second detecting means for detecting an inclination relative to the gravity direction of the imaging unit, the inclination-out及 beauty said first calculating means with respect to the gravity direction of the image pickup unit detected by the second detecting means Based on the inclination with respect to the gravity direction of the specific subject issued, coordinate conversion formula for converting the spatial coordinates of each point of the plurality of points constituting the image of the specific object in the plane coordinates in the other image as an initial value, second calculating means for calculating a relative posture of the specific subject with respect to the imaging unit, to function as, and wherein the.

  According to the present invention, it is possible to appropriately estimate the posture of a specific subject with respect to the imaging unit.

It is a block diagram which shows schematic structure of the portable terminal of one Embodiment to which this invention is applied. It is a flowchart which shows an example of the operation | movement which concerns on the marker detection process by the portable terminal of FIG. It is a figure which shows typically the positional relationship of the portable terminal of FIG. 1, and a marker.

  Hereinafter, specific embodiments of the present invention will be described with reference to the drawings. However, the scope of the invention is not limited to the illustrated examples.

FIG. 1 is a block diagram showing a schematic configuration of a mobile terminal 100 according to an embodiment to which the present invention is applied.
As shown in FIG. 1, the mobile terminal 100 includes a central control unit 1, a memory 2, an imaging unit 3, an imaging control unit 4, an image data generation unit 5, an inclination detection unit 6, and a marker detection processing unit. 7, a display unit 8, a display control unit 9, a transmission / reception unit 10, a communication control unit 11, an operation input unit 12, and the like.
The mobile terminal 100 includes, for example, an imaging device having a communication function, a mobile station used in a mobile communication network such as a mobile phone or a PHS (Personal Handy-phone System), a PDA (Personal Data Assistants), and the like. Has been.

  The central control unit 1 controls each unit of the mobile terminal 100. Specifically, the central control unit 1 includes a CPU (not shown) that controls each part of the mobile terminal 100, and performs various control operations according to various processing programs (not shown).

  The memory 2 is composed of, for example, a DRAM (Dynamic Random Access Memory) or the like, and is a buffer memory that temporarily records image information, a working memory such as the central control unit 1, and various programs related to the functions of the mobile terminal 100 And a program memory (not shown) in which data is stored.

The imaging unit 3 captures a specific subject (for example, the marker M) and generates an image signal of the frame image F (see FIG. 3).
Here, the specific subject may be a two-dimensional image or a three-dimensional solid. For example, when a specific subject is a two-dimensional image, a marker M having a substantially rectangular frame shape (frame shape) may be used (see FIG. 3).

The imaging unit 3 includes a lens unit 3a and an electronic imaging unit 3b.
The lens unit 3a includes a plurality of lenses such as a zoom lens and a focus lens.
The electronic imaging unit 3b is composed of, for example, an image sensor such as a charge coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS), and converts an optical image that has passed through various lenses of the lens unit 3a into a two-dimensional image signal. To do.
In addition, although illustration is abbreviate | omitted, the imaging part 3 may be provided with the aperture_diaphragm | restriction which adjusts the quantity of the light which passes the lens part 3a.

The imaging control unit 4 controls the imaging of the subject by the imaging unit 3. That is, the imaging control unit 4 includes a timing generator, a driver, and the like, although not illustrated. Then, the imaging control unit 4 scans and drives the electronic imaging unit 3b with a timing generator and a driver, converts the optical image into a two-dimensional image signal with the electronic imaging unit 3b at a predetermined period, and the electronic imaging unit 3b. The frame image F is read out from the imaging area for each screen and is output to the image data generation unit 5.
In addition, the imaging control unit 4 performs adjustment control of imaging conditions of the subject such as AF (automatic focusing process), AE (automatic exposure process), AWB (automatic white balance), and the like.

The image data generation unit 5 appropriately adjusts the gain for each RGB color component with respect to the analog value signal of the frame image F transferred from the electronic image pickup unit 3b, and then samples and holds it by a sample hold circuit (not shown). The digital data is converted into digital data by an A / D converter (not shown), color processing including pixel interpolation processing and γ correction processing is performed by a color process circuit (not shown), and then a digital luminance signal Y and color difference signal Cb, Cr (YUV data) is generated.
Further, the image data generation unit 5 reduces the YUV data of the generated frame image F at a predetermined magnification in both horizontal and vertical directions, and generates low-resolution (for example, VGA, QVGA size, etc.) image data for live view display. Is generated. Specifically, the image data generation unit 5 has low resolution image data for live view display from the YUV data of the frame image F at a predetermined timing according to a predetermined display frame rate of the live view image by the display unit 8. Is generated.

  Then, the image data generation unit 5 sequentially outputs the generated YUV data of each frame image F to the memory 2 and stores it in the memory 2.

  The inclination detection unit 6 detects the posture (inclination) of the terminal main body including the imaging unit 3 with respect to the gravity axis direction. Specifically, the inclination detection unit 6 includes a first detection unit 6a and a second detection unit 6b.

The first detection unit (first detection means) 6a detects the first posture Pac0 with respect to the gravity axis direction of the terminal body when the first image (one image) is acquired by the image acquisition unit 7a (described later).
That is, the first detection unit 6a includes, for example, a triaxial acceleration sensor and the like, specifies the gravity axis direction based on the frequency component of 0 Hz of the detection signal of each axis by the triaxial acceleration sensor, and with respect to the gravity axis direction. The inclination of each axis is detected as the first posture Pac0 with respect to the gravity axis direction of the terminal body. Specifically, the first detection unit 6a acquires, from the plurality of frame images F,..., The first image related to the calculation of the posture Pam (described later) of the marker M with respect to the gravity axis direction by the image acquisition unit 7a. At this time, the first posture Pac0 with respect to the gravity axis direction of the terminal body is detected.
Here, the first posture Pac0 with respect to the direction of the gravity axis of the terminal body is specifically represented by an n-by-m matrix (n and m are natural numbers, respectively), for example, a 3 × 3 rotation matrix or the like. Can be mentioned.

The second detection unit (second detection unit) 6b has a second posture with respect to the gravity axis direction of the terminal body when a second image (other image) different from the first image is acquired by an image acquisition unit 7a (described later). Detect Pac1.
The second detection unit 6b has substantially the same configuration as the first detection unit 6a. That is, the second detection unit 6b includes, for example, a triaxial acceleration sensor and the like, specifies the gravitational axis direction based on the frequency component of 0 Hz of the detection signal of each axis by the triaxial acceleration sensor, and with respect to the gravitational axis direction. The inclination of each axis is detected as the second posture Pac1 with respect to the gravity axis direction of the terminal body. Specifically, the second detection unit 6b acquires, from the plurality of frame images F,..., The second image related to the calculation of the relative posture Prm1 (described later) of the marker M with respect to the terminal body by the image acquisition unit 7a. When this is done, the second posture Pac1 with respect to the gravitational axis direction of the terminal body is detected.
Here, the second posture Pac1 with respect to the gravitational axis direction of the terminal body is specifically represented by a matrix of n rows and m columns (n and m are natural numbers, respectively), for example, a 3 × 3 rotation matrix or the like. Can be mentioned.

Note that the above-described detection method of the gravity axis direction by the three-axis acceleration sensor and the detection method of the postures Pac0 and Pac1 with respect to the gravity axis direction of the terminal body by the first detection unit 6a and the second detection unit 6b are examples. The present invention is not limited to this, and can be arbitrarily changed as appropriate.
In addition, the inclination (posture) of the terminal main body including the imaging unit 3 with respect to the gravity axis direction is detected, but is not limited to this, and the inclination of the imaging unit 3 itself with respect to the gravity axis direction is not limited thereto. May be detected. Further, instead of the imaging unit 3, a tilt of a predetermined member or element (for example, a liquid crystal panel provided on an extension line of the optical axis) with respect to the gravity axis direction may be detected.

The marker detection processing unit 7 includes an image acquisition unit 7a, a first calculation unit 7b, a second calculation unit 7c, a coordinate conversion unit 7d, and an image generation unit 7e.
In addition, although each part of the marker detection process part 7 is comprised from the predetermined | prescribed logic circuit, for example, the said structure is an example and is not restricted to this.

The image acquisition unit 7a sequentially acquires a plurality of frame images F,.
That is, the image acquisition unit ( acquisition unit ) 7a sequentially acquires a plurality of images in which the marker (specific subject) M is continuously captured. Specifically, the image acquisition unit 7 a stores image data with a predetermined resolution of a plurality of frame images F,..., Which are continuously captured by the image data generation unit 5 after the marker M is continuously captured by the image capturing unit 3. Obtain sequentially from 2.

ここで、端末本体に対するマーカーＭの仮の姿勢Prm0及びマーカーＭの重力軸方向に対する姿勢Pamは、具体的には、ｎ行ｍ列の行列（ｎ、ｍは、それぞれ自然数）で表され、例えば、３×３の回転行列等が挙げられる。 The first calculator 7b calculates the posture Pam of the marker M with respect to the gravity axis direction.
That is, the first calculation unit (first calculation unit) 7b includes the first posture Pac0 with respect to the gravity axis direction of the terminal body detected by the first detection unit 6a, and the marker image included in the first image (one image). Based on the provisional posture Prm0 of the marker M with respect to the terminal body (imaging unit 3) associated with a plurality of points constituting the (specific subject image) S, the posture (inclination) Pam of the marker M with respect to the gravity axis direction is calculated. To do.
Specifically, the first calculation unit 7b performs a predetermined feature extraction process on the image data of one of the plurality of frame images F acquired by the image acquisition unit 7a,. A marker image S having a substantially rectangular frame shape is extracted from the first image.
Subsequently, the first calculation unit 7b detects a plurality of points (for example, four corner points) constituting the marker image S, and based on the plurality of points, determines the temporary posture Prm0 of the marker M with respect to the terminal body. Identify. For example, the first calculation unit 7b refers to a database (not shown) or the like recorded in a predetermined recording unit, and the marker for the terminal body having a geometric positional relationship with the four corner points of the marker image S. The provisional posture Prm0 of M is specified.
Then, the first calculation unit 7b is based on the provisional posture Prm0 of the marker M with respect to the identified terminal body and the first posture Pac0 with respect to the gravity axis direction of the terminal body detected by the first detection unit 6a. In accordance with Expression (1), the posture Pam of the marker M with respect to the gravity axis direction is calculated.

Here, the temporary posture Prm0 of the marker M with respect to the terminal body and the posture Pam of the marker M with respect to the gravity axis direction are specifically represented by a matrix of n rows and m columns (n and m are natural numbers, respectively), for example, For example, a 3 × 3 rotation matrix is used.

Further, the first calculation unit 7b uses each frame image F as a first image each time a plurality of frame images F,... Sequentially generated by the image data generation unit 5 are acquired by the image acquisition unit 7a. Posture Pam with respect to the gravity axis direction is calculated. Then, the first calculation unit 7b may calculate a representative value (for example, an average value or a median value) of the posture Pam based on the posture Pam of the marker M with respect to the gravity axis direction calculated a predetermined number of times. Specifically, for example, the first calculation unit 7b averages a predetermined number of postures Pam of the marker M with respect to the gravity axis direction, and sets the result as a representative value of the posture Pam of the marker M with respect to the gravity axis direction.
The number of times of calculating the posture Pam of the marker M with respect to the direction of the gravity axis can be arbitrarily changed as appropriate. For example, it can be a user-specified value specified in advance or a value set as a default. There may be.

  In addition, the first calculation unit 7b determines the marker M with respect to the gravity axis direction every time the coordinate conversion unit 7d specifies matrices R and T (described later) representing the relative posture and positional relationship of the marker M with respect to the terminal body. The posture Pam may be calculated. That is, the first calculation unit 7b sets the matrix R representing the relative posture of the marker M with respect to the terminal body sequentially identified by the coordinate conversion unit 7d as the relative posture Prm1 of the marker M with respect to the terminal body, and the posture Prm1. Based on the second posture Pac1 with respect to the gravity axis direction of the terminal body detected by the second detection unit 6b, a predetermined calculation substantially similar to the above equation (1) is performed, and the posture Pam of the marker M with respect to the gravity axis direction is performed. May be calculated and updated.

ここで、端末本体に対するマーカーＭの相対的な姿勢Prm1は、具体的には、ｎ行ｍ列の行列（ｎ、ｍは、それぞれ自然数）で表され、例えば、３×３の回転行列等が挙げられる。 The second calculator 7c calculates the relative posture Prm1 of the marker M with respect to the terminal body.
That is, the second calculation unit (second calculation unit) 7c is configured such that the second posture Pac1 with respect to the gravity axis direction of the terminal body detected by the second detection unit 6b and the gravity of the marker M calculated by the first calculation unit 7b. Based on the posture Pam with respect to the axial direction, a coordinate conversion expression (for example, an expression) that converts the coordinates of each point of the marker M in the three-dimensional space into the corresponding two-dimensional plane coordinates in the second image (other image). As an initial value of (3) (see below), a relative posture Prm1 of the marker M with respect to the terminal body (imaging unit 3) is calculated.
Specifically, each time the second posture Pac1 with respect to the gravity axis direction of the terminal body is detected by the second detection unit 6b, the second calculation unit 7c detects the second posture Pac1 with respect to the detected gravity direction of the terminal body. Based on the posture Pam of the marker M with respect to the gravity axis direction, the relative posture Prm1 of the marker M with respect to the terminal body is calculated according to the following equation (2). The calculated relative posture Prm1 of the marker M with respect to the terminal body is an initial value of the relative posture R of the marker M with respect to the terminal body in a coordinate conversion formula described later.

Here, the relative posture Prm1 of the marker M with respect to the terminal body is specifically represented by an n-by-m matrix (n and m are natural numbers, respectively), for example, a 3 × 3 rotation matrix or the like. Can be mentioned.

なお、式（３）の座標変換式は、同次座標変換行列である。また、当該式（３）中、「Ｘ_ｗ」、「Ｙ_ｗ」及び「Ｚ_ｗ」は、互いに直交するＸ軸、Ｙ軸及びＺ軸により規定される三次元空間でのマーカーＭを構成する各点の座標位置を表し、「ｘ_ｉ」及び「ｙ_ｉ」は、Ｘ軸及びＹ軸により規定される二次元平面である第２画像内でマーカーＭの各点の対応する座標位置を表している。
また、端末本体に対するマーカーＭの相対的な姿勢を表す行列Ｒは、具体的には、ｎ行ｍ列の行列（ｎ、ｍは、それぞれ自然数）で表され、例えば、３×３の回転行列等が挙げられる。また、端末本体に対するマーカーＭの相対的な位置関係を表す行列Ｔは、具体的には、ｎ行ｍ列の行列（ｎ、ｍは、それぞれ自然数）で表され、例えば、１×３の並進ベクトル等が挙げられる。
また、式（３）中の「Ｃ」は、撮像部３の焦点距離等の内部パラメータを表し、「ｈ」は、第２画像のスケールを表している。 The coordinate conversion unit 7d converts the coordinates of each point of the marker M in the three-dimensional space into corresponding two-dimensional plane coordinates in the second image.
That is, the coordinate conversion unit (coordinate conversion unit) 7d calculates a coordinate conversion formula of the following formula (3) based on the relative posture Prm1 of the marker M with respect to the terminal body calculated as the initial value by the second calculation unit 7c. solve. Specifically, the coordinate conversion unit 7d uses the relative posture Prm1 of the marker M with respect to the terminal body as an initial value of a matrix R representing the relative posture of the marker M with respect to the terminal body, and the relative position of the marker M with respect to the terminal body. Matrixes R and T representing a proper posture and positional relationship are estimated and specified by bundle adjustment.
Here, bundle adjustment is a method for estimating geometric model parameters from an image in order to achieve high estimation accuracy, for example, a method for minimizing the total reprojection error of unknown parameters. Etc. are used.

In addition, the coordinate conversion formula of Formula (3) is a homogeneous coordinate conversion matrix. Moreover, in the said Formula (3), " _Xw ", " _Yw ", and " _Zw " comprise the marker M in the three-dimensional space prescribed | regulated by the mutually orthogonal X-axis, Y-axis, and Z-axis. Represents the coordinate position of each point, and “x _i ” and “y _i ” represent the corresponding coordinate position of each point of the marker M in the second image which is a two-dimensional plane defined by the X axis and the Y axis. ing.
Further, the matrix R representing the relative posture of the marker M with respect to the terminal body is specifically represented by a matrix of n rows and m columns (n and m are natural numbers, respectively), for example, a 3 × 3 rotation matrix. Etc. Further, the matrix T representing the relative positional relationship of the marker M with respect to the terminal body is specifically represented by a matrix of n rows and m columns (n and m are natural numbers, respectively), for example, 1 × 3 translation Vector etc. are mentioned.
Further, “C” in Expression (3) represents an internal parameter such as a focal length of the imaging unit 3, and “h” represents a scale of the second image.

The image generation unit 7e displays a virtual object (for example, a three-dimensional model; not shown) corresponding to the marker image S in the second image among the plurality of frame images F acquired by the image acquisition unit 7a. A superimposed virtual image (not shown) is generated.
Specifically, for example, every time a live view image or the like is sequentially acquired as the second image, the image generation unit 7e determines the relative posture and positional relationship of the marker M with respect to the terminal body specified by the coordinate conversion unit 7d. Based on the above, the posture and position of the marker image S in the live view image are adjusted, a virtual object corresponding to the marker image S is acquired from a predetermined recording means, and image data of the virtual image is generated. Then, the image generation unit 7 e sequentially outputs the generated image data of each virtual image to the memory 2 and stores it in the memory 2.
Note that the image data of the virtual image generated by the image generation unit 7e is encoded in, for example, a predetermined compression format (eg, JPEG format), and is a recording medium (not shown) such as a nonvolatile memory (flash memory). May be recorded.

  The display unit 8 is composed of, for example, a liquid crystal display panel, and displays an image (for example, a REC view image) captured by the imaging unit 3 based on a video signal from the display control unit 9 on a display screen.

The display control unit 9 performs control to read out display image data temporarily stored in the memory 2 and display it on the display unit 8.
Specifically, the display control unit 9 includes a VRAM (Video Random Access Memory), a VRAM controller, a digital video encoder, and the like. The digital video encoder reads the luminance signal Y and the color difference signals Cb and Cr read from the memory 2 and stored in the VRAM (not shown) under the control of the central control unit 1 through the VRAM controller. Are read out at a predetermined playback frame rate (for example, 30 fps), and a video signal is generated based on these data and output to the display unit 8.
For example, the display control unit 9 updates the plurality of frame images F captured by the imaging unit 3 and the imaging control unit 4 and generated by the image data generation unit 5 to the display unit 8 while sequentially updating at a predetermined display frame rate. Display live view. For example, the display control unit 9 sequentially acquires image data of virtual images from the memory 2 and sequentially displays the virtual images on the display unit 8 in live view.

The transmitter / receiver unit 10 performs a call with an external user of an external device connected via the communication network N.
Specifically, the transmission / reception unit 10 includes a microphone 10a, a speaker 10b, a data conversion unit 10c, and the like. The transmission / reception unit 10 performs A / D conversion processing on the user's transmission voice input from the microphone 10a by the data conversion unit 10c and outputs the transmission voice data to the central control unit 1. Under the control, voice data such as received voice data outputted and inputted from the communication control unit 11 is D / A converted by the data conversion unit 10c and outputted from the speaker 10b.

The communication control unit 11 transmits and receives data via the communication network N and the communication antenna 11a.
That is, the communication antenna 11a is connected to a predetermined communication method (for example, W-CDMA (Wideband Code Division Multiple Access) method, GSM (Global System)) used by the mobile terminal 100 for communication with a radio base station (not shown). for Mobile Communications (registered trademark) system, etc.). The communication control unit 11 transmits / receives data to / from the radio base station via the communication antenna 11a through a communication channel set in the communication method according to a communication protocol corresponding to a predetermined communication method.
That is, the communication control unit 11 transmits / receives voice during a call with an external user of the external device to the external device of the communication partner based on the instruction signal output from the central control unit 1 and input, Send and receive e-mail data.
Note that the configuration of the communication control unit 11 is an example and is not limited thereto, and can be arbitrarily changed as appropriate. For example, although not shown, a wireless LAN module is mounted and an access point (Access Point) is provided. It is good also as a structure which can access the communication network N via this.

  The communication network N is a communication network that connects the mobile terminal 100 via a wireless base station, a gateway server (not shown), or the like. The communication network N is a communication network constructed by using a dedicated line or an existing general public line, and applies various line forms such as a LAN (Local Area Network) and a WAN (Wide Area Network). Is possible. The communication network N includes, for example, various communication network networks such as a telephone line network, ISDN line network, dedicated line, mobile communication network, communication satellite line, CATV line network, IP network, VoIP (Voice over Internet Protocol). ) Gateways, Internet service providers, etc. are included.

The operation input unit 12 is for inputting various instructions to the terminal body.
Specifically, the operation input unit 12 executes a shutter button related to a subject photographing instruction, up / down / left / right cursor buttons and a determination button related to a selection instruction of a mode, a function, etc., making / receiving a call, sending / receiving an e-mail, etc. Various buttons (not shown) such as communication-related buttons related to instructions and numeric buttons and symbol buttons related to text input instructions are provided.
When various buttons are operated by the user, the operation input unit 12 outputs an operation instruction corresponding to the operated button to the central control unit 1. The central control unit 1 causes each unit to execute a predetermined operation (for example, imaging of a subject, incoming / outgoing calls, transmission / reception of an e-mail, etc.) according to an operation instruction output from the operation input unit 12 and input.

  The operation input unit 12 may include a touch panel provided integrally with the display unit 8, and an operation instruction corresponding to the predetermined operation is given to the central control unit based on a predetermined operation of the touch panel by the user. 1 may be output.

Next, marker detection processing by the mobile terminal 100 will be described with reference to FIGS.
FIG. 2 is a flowchart illustrating an example of an operation related to the marker detection process. FIG. 3 is a diagram schematically showing the positional relationship between the mobile terminal 100 and the marker M.
It is assumed that the marker M detected in the following marker detection process is previously arranged (fixed) in a state where it cannot move to a predetermined position.

  As shown in FIG. 2, first, the imaging control unit 4 causes the imaging unit 3 to image the marker M, and the image data generation unit 5 sets the image data of a plurality of frame images F,... Transferred from the electronic imaging unit 3b. Is generated (step S1). Then, the image data generation unit 5 sequentially outputs the generated YUV data of each frame image F to the memory 2 and stores it in the memory 2.

  The image acquisition unit 7a of the marker detection processing unit 7 sequentially acquires, from the memory 2, image data of any one of the plurality of frame images F,... Sequentially generated by the image data generation unit 5. Step S2). And the 1st detection part 6a of the inclination detection part 6 detects the 1st attitude | position Pac0 with respect to the gravity-axis direction of a terminal main body when the image data of a 1st image is acquired (step S3). Specifically, the first detection unit 6a detects the inclination of each axis with respect to the gravity axis direction specified based on the detection signal of each axis by the triaxial acceleration sensor as the first posture Pac0 with respect to the gravity axis direction of the terminal body. To do.

Next, the first calculation unit 7b of the marker detection processing unit 7 performs a predetermined feature extraction process on the image data of the first image acquired by the image acquisition unit 7a, and the marker image is extracted from the first image. S is extracted (step S4). Subsequently, after the first calculation unit 7b specifies the provisional posture Prm0 of the marker M with respect to the terminal body having a geometric positional relationship with the four corner points of the marker image S (step S5), Based on the provisional posture Prm0 of the marker M and the first posture Pac0 with respect to the gravity axis direction of the terminal body, the posture Pam of the marker M with respect to the gravity axis direction is calculated according to the following equation (1) (step S6).

Next, the first calculator 7b determines whether or not the posture Pam of the marker M with respect to the direction of the gravity axis has been calculated a predetermined number of times (step S7).
Here, when it is determined that the posture Pam of the marker M with respect to the direction of the gravity axis has not been calculated a predetermined number of times (step S7; NO), the CPU of the central control unit 1 returns the process to step S2, and thereafter Execute the process.
That is, the processes in steps S2 to S6 are repeated until it is determined in step S7 that the posture Pam of the marker M with respect to the gravity axis direction has been calculated a predetermined number of times (step S7; YES).

  If it is determined in step S7 that the posture Pam of the marker M with respect to the gravity axis direction has been calculated a predetermined number of times (step S7; YES), the first calculation unit 7b determines the predetermined number of times of the posture Pam of the marker M with respect to the gravity axis direction. The minutes are averaged, and the result is calculated as a representative value of the posture Pam with respect to the gravity axis direction of the marker M (step S8).

  Next, the image acquisition unit 7a acquires the image data of the second image from the memory 2 among the plurality of frame images F generated by the image data generation unit 5 (step S9). And the 2nd detection part 6b of the inclination detection part 6 detects the 2nd attitude | position Pac1 with respect to the gravity-axis direction of a terminal main body at the time of the image data of a 2nd image being acquired (step S10). Note that the processing content of the second detection unit 6b is substantially the same as the processing content of the first detection unit 6a, and a description thereof is omitted.

Next, the second calculation unit 7c includes the second posture Pac1 with respect to the gravity axis direction of the terminal body detected by the second detection unit 6b, and the posture Pam of the marker M calculated by the first calculation unit 7b with respect to the gravity axis direction. Based on the representative value, a relative posture Prm1 of the marker M with respect to the terminal body is calculated according to the following equation (2) (step S11).

これにより、座標変換部７ｄにより特定された端末本体に対するマーカーＭの相対的な姿勢及び位置関係に基づいて、第２画像内での当該マーカー画像Ｓの姿勢や位置が調整される。 Next, the coordinate conversion unit 7d uses the relative orientation Prm1 of the marker M with respect to the terminal body calculated by the second calculation unit 7c as the initial value of the matrix R representing the relative orientation of the marker M with respect to the terminal body, as follows. According to Equation (3), matrices R and T representing the relative posture and positional relationship of the marker M with respect to the terminal body are estimated and specified by bundle adjustment (step S12).

Thereby, the attitude | position and position of the said marker image S in a 2nd image are adjusted based on the relative attitude | position and positional relationship of the marker M with respect to the terminal main body specified by the coordinate transformation part 7d.

After that, the first calculation unit 7b sets the matrix R representing the relative posture of the marker M with respect to the terminal body specified by the coordinate conversion unit 7d as the relative posture Prm1 of the marker M with respect to the terminal body, and the posture Prm1 and the terminal Based on the second posture Pac1 with respect to the gravity axis direction of the main body, the posture Pam of the marker M with respect to the gravity axis direction is calculated and updated (step S13).
Then, the image generation unit 7e generates image data of a virtual image in which a virtual object corresponding to the marker image S is superimposed on the second image, and the display control unit 9 performs live view of the virtual image on the display unit 8. It is displayed (step S14).

  The processes in steps S9 to S14 are repeatedly executed every time the second image is acquired in step S9.

  As described above, according to the mobile terminal 100 of the present embodiment, the inclination (first posture Pac0) of the imaging unit 3 (terminal body) with respect to the gravity axis direction when the first image is acquired, and the marker image S are obtained. Based on the temporary posture (provisional posture Prm0) of the marker M with respect to the imaging unit 3 associated with a plurality of constituent points, the inclination (posture Pam) of the marker M with respect to the gravitational axis direction is calculated, and the calculated marker M 3D spatial coordinates of each point of the marker M based on the inclination with respect to the gravity axis direction and the inclination with respect to the gravity axis direction of the imaging unit 3 when the second image is acquired (second posture Pac1). Since the relative posture (posture Prm1) of the marker M with respect to the imaging unit 3 is calculated as the initial value of the coordinate conversion equation for conversion into two-dimensional plane coordinates in the two images, an appropriate value as the initial value of the coordinate conversion equation The imaging unit 3 can be used. The posture of the marker M with respect to can be estimated appropriately.

That is, for example, when the coordinates of each point of the marker M in the three-dimensional space are converted into the coordinates of the corresponding two-dimensional space in the second image using a predetermined coordinate conversion formula, an appropriate value is set as an initial value. Although it is necessary to estimate the posture and positional relationship of the marker M with respect to the imaging unit 3 so as to minimize the projection error, depending on the value set as the initial value of the coordinate conversion formula, There is a possibility that the positional relationship cannot be estimated properly and the virtual object corresponding to the marker M cannot be displayed properly.
Therefore, the inclination (posture Pam) of the marker M with respect to the gravity axis direction is calculated in advance using the inclination of the imaging unit 3 with respect to the gravity axis direction when the first image is acquired, and the inclination of the marker M with respect to the gravity axis direction is calculated. By calculating the relative posture (posture Prm1) of the marker M with respect to the imaging unit 3 using the above, the relative posture of the marker M with respect to the imaging unit 3 can be used as the initial value of the coordinate conversion formula. And the attitude | position of the marker M with respect to the imaging part 3 can be estimated by solving the said coordinate conversion formula by making the initial value of a coordinate conversion formula into an appropriate value. Specifically, by solving the coordinate conversion formula, a matrix R representing the relative posture of the marker M with respect to the imaging unit 3 and a matrix T representing the relative positional relationship of the marker M with respect to the imaging unit 3 can be specified. .
Thus, by considering the inclination of the marker M with respect to the direction of the gravity axis, the initial value of the coordinate conversion equation can be set to a more appropriate value, and the posture of the marker M with respect to the imaging unit 3 is appropriately estimated. be able to. As a result, the virtual object corresponding to the marker M can be properly displayed.

  Each time the matrix R representing the relative posture of the marker M with respect to the imaging unit 3 or the matrix T representing the relative positional relationship of the marker M with respect to the imaging unit 3 is specified, the matrix and the gravity axis of the imaging unit 3 are specified. Since the inclination (posture Pam) of the marker M with respect to the gravity axis direction is calculated based on the inclination with respect to the direction (second posture Pac1), the inclination of the marker M with respect to the imaging unit 3 using the inclination of the marker M with respect to the gravity axis direction is calculated. The relative posture (posture Prm1) can be calculated sequentially. That is, for example, even if the inclination of the imaging unit 3 with respect to the gravitational axis direction when the second image is acquired due to camera shake, imaging position displacement, or the like, the marker M with respect to the gravitational axis direction of the marker M corresponds to the change of the inclination. The inclination can be updated, and the relative posture of the marker M with respect to the imaging unit 3 can be updated.

  In addition, since the representative value of the inclination is calculated based on the inclination (posture Pam) of the marker M with respect to the gravity axis direction calculated a predetermined number of times, the detection unit 3 detects the marker image S included in the first image and the like. Even if there is a possibility that a detection error of the provisional posture (Prm0) of the marker M may occur, a more appropriate value for the posture Pam of the marker M with respect to the gravity axis direction is obtained by obtaining a representative value of the inclination of the marker M with respect to the gravity axis direction. Can be used to calculate the relative posture (posture Prm1) of the marker M with respect to the imaging unit 3.

The present invention is not limited to the above-described embodiment, and various improvements and design changes may be made without departing from the spirit of the present invention.
For example, the matrix R representing the relative posture of the marker M with respect to the terminal body (imaging unit 3) is specified by solving the coordinate conversion formula, but this is an example of the relative posture of the marker M with respect to the terminal body. is not limited to Re lever, it can be appropriately changed.
Further, the matrix T representing the relative positional relationship of the marker M with respect to the terminal body is specified by solving the coordinate conversion formula, but it is not always necessary to specify the relative positional relationship of the marker M with respect to the terminal body. Absent.

Further, when calculating the relative posture (posture Prm1) of the marker M with respect to the terminal body, it is specified in advance which direction the one side of the marker M is facing up, down, left and right, and a predetermined value is set as an initial value. As a result of solving the coordinate conversion formula, it may be determined whether the direction is different from the designated direction, and the determination result may be reflected in the initial value.
For example, when the normal direction of one surface of the marker M is previously designated as a downward direction, predetermined elements (for example, the second row) of the matrix R representing the relative posture of the marker M with respect to the terminal body estimated as a result of bundle adjustment By checking the sign of the element in the third column), it is determined whether the direction is different, that is, whether the marker M is downward or upward. Here, when it is determined that the marker M is pointing in a direction different from the designated direction (the marker M is upward), a predetermined position of the terminal body, an axis passing through the center of the marker M, and a method of one surface of the marker The outer product of the line vectors is calculated, and is corrected by rotating the matrix R around the calculated outer product as an axis so as to match the direction of one surface of the marker M. Thereafter, the coordinate transformation equation is solved using the corrected matrix R, and the matrix R representing the relative posture of the marker M with respect to the terminal body is estimated again.

  Furthermore, the configuration of the mobile terminal 100 is merely an example illustrated in the above embodiment, and is not limited thereto. At least the acquisition unit, the first detection unit, the first calculation unit, the second detection unit, and the second Any configuration provided with calculation means can be arbitrarily changed. That is, the mobile terminal 100 does not necessarily include the imaging unit 3, and may acquire the image captured by an external imaging device and perform processing for detecting a specific subject (marker M). .

In addition, in the above embodiment, the functions as the acquisition unit, the first detection unit, the first calculation unit, the second detection unit, and the second calculation unit are controlled under the control of the central control unit 1 of the mobile terminal 100. The image acquisition unit 7a, the first detection unit 6a, the first calculation unit 7b, the second detection unit 6b, and the second calculation unit 7c are driven. However, the present invention is not limited to this. The CPU may be configured such that a predetermined program or the like is executed by the CPU of the central control unit 1.
That is, a program including an acquisition process routine, a first detection process routine, a first calculation process routine, a second detection process routine, and a second calculation process routine is stored in a program memory that stores the program. Then, the CPU of the central control unit 1 may function as means for sequentially acquiring images of a specific subject imaged by the imaging unit 3 by an acquisition process routine. Further, the CPU of the central control unit 1 may function as a means for detecting the inclination of the imaging unit 3 with respect to the gravity direction when one image is acquired by the first detection processing routine. Also, the central control unit 1 of the CPU by the first calculation processing routine, the inclination with respect to the detected direction of gravity of the imaging unit 3, and associated with a plurality of points constituting the image of a specific subject included in one image Based on the provisional posture of the specific subject with respect to the imaging unit 3, it may function as a means for calculating the inclination of the specific subject with respect to the direction of gravity. Further, the CPU of the central control unit 1 is caused to function as a means for detecting the inclination of the imaging unit 3 with respect to the gravity direction when another image different from the one image is acquired by the acquisition unit by the second detection processing routine. May be. In addition, the CPU of the central control unit 1 causes the CPU of the central control unit 1 to tilt with respect to the gravity direction of the imaging unit 3 detected by the second detection unit, and the tilt of the specific subject with respect to the gravity direction calculated by the first calculation unit. Based on the above, the relative value of the specific subject with respect to the imaging unit 3 is used as an initial value of a coordinate conversion formula that converts the spatial coordinates of each of a plurality of points constituting the image of the specific subject into plane coordinates in another image. You may make it function as a means to calculate a general attitude | position.

  Similarly, the coordinate conversion means may be realized by executing a predetermined program or the like by the CPU of the central control unit 1.

  Furthermore, as a computer-readable medium storing a program for executing each of the above processes, a non-volatile memory such as a flash memory or a portable recording medium such as a CD-ROM is applied in addition to a ROM or a hard disk. Is also possible. A carrier wave is also used as a medium for providing program data via a predetermined communication line.

[Appendix]
Although several embodiments of the present invention have been described, the scope of the present invention is not limited to the above-described embodiments, but includes the scope of the invention described in the claims and equivalents thereof.
The invention described in the scope of claims attached to the application of this application will be added below. The item numbers of the claims described in the appendix are as set forth in the claims attached to the application of this application.
<Claim 1>
Acquisition means for sequentially acquiring images of a specific subject imaged by the imaging unit;
First detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when one image is acquired by the acquisition means;
The inclination of the image pickup unit with respect to the direction of gravity detected by the first detection unit and the provisional image of the specific subject with respect to the image pickup unit associated with a plurality of points constituting the specific object image included in the one image. First calculating means for calculating the inclination of the specific subject with respect to the direction of gravity based on the posture of
Second detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired by the acquisition means;
The space of each point of the specific subject based on the inclination of the imaging unit detected by the second detection unit with respect to the direction of gravity and the inclination of the specific subject calculated with respect to the direction of gravity calculated by the first calculation unit. Second calculation means for calculating a relative posture of the specific subject with respect to the imaging unit, as an initial value of a coordinate conversion formula for converting coordinates into plane coordinates in the other image;
A subject detection apparatus comprising:
<Claim 2>
2. The subject according to claim 1, further comprising coordinate conversion means for solving the coordinate conversion formula based on a relative posture of the specific subject calculated as the initial value by the second calculation means. Detection device.
<Claim 3>
The subject detection apparatus according to claim 2, wherein the coordinate conversion unit further specifies a matrix representing a relative posture of the specific subject with respect to the imaging unit by solving the coordinate conversion formula.
<Claim 4>
4. The subject according to claim 2, wherein the coordinate conversion unit further specifies a matrix representing a relative positional relationship of the specific subject with respect to the imaging unit by solving the coordinate conversion formula. 5. Detection device.
<Claim 5>
The first calculating unit is further configured to identify the specific matrix based on the matrix and the inclination of the imaging unit with respect to the gravitational direction detected by the second detecting unit each time the matrix is specified by the coordinate converting unit. The subject detection apparatus according to claim 2, wherein an inclination of the subject with respect to the direction of gravity is calculated.
<Claim 6>
The first calculation means further calculates an inclination of the specific subject with respect to the gravity direction every time an image is acquired by the acquisition means, and based on the inclination of the specific subject with respect to the gravity direction calculated a predetermined number of times. 6. The subject detection apparatus according to claim 1, wherein a representative value of the inclination is calculated.
<Claim 7>
A subject detection method using a subject detection device,
A process of sequentially acquiring images of a specific subject imaged by the imaging unit;
Processing for detecting the inclination of the imaging unit with respect to the gravitational direction when one image is acquired;
Based on the detected inclination of the imaging unit with respect to the direction of gravity and the provisional posture of the specific subject with respect to the imaging unit associated with a plurality of points constituting the specific subject image included in the one image, Processing for calculating the inclination of the specific subject with respect to the direction of gravity;
A process of detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired;
Coordinate transformation that converts the spatial coordinates of each point of the specific subject to plane coordinates in the other image based on the detected inclination of the imaging unit with respect to the gravity direction and the inclination of the specific subject with respect to the gravity direction A process of calculating a relative posture of the specific subject with respect to the imaging unit as an initial value of the equation;
A method for detecting a subject, comprising:
<Claim 8>
The computer of the subject detection device
Acquisition means for sequentially acquiring images of a specific subject imaged by the imaging unit;
First detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when one image is acquired by the acquisition means;
The inclination of the image pickup unit with respect to the direction of gravity detected by the first detection unit and the provisional image of the specific subject with respect to the image pickup unit associated with a plurality of points constituting the specific object image included in the one image. First calculating means for calculating the inclination of the specific subject with respect to the direction of gravity based on the posture of
Second detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired by the acquisition means;
The space of each point of the specific subject based on the inclination of the imaging unit detected by the second detection unit with respect to the direction of gravity and the inclination of the specific subject calculated with respect to the direction of gravity calculated by the first calculation unit. Second calculating means for calculating a relative posture of the specific subject with respect to the imaging unit, as an initial value of a coordinate conversion formula for converting coordinates into plane coordinates in the other image;
A program characterized by functioning as

DESCRIPTION OF SYMBOLS 100 Portable terminal 1 Central control part 3 Imaging part 6 Inclination detection part 6a 1st detection part 6b 2nd detection part 7 Marker detection process part 7a Image acquisition part 7b 1st calculation part 7c 2nd calculation part 7d Coordinate conversion part

Claims (8)

Acquisition means for sequentially acquiring images of a specific subject imaged by the imaging unit;
First detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when one image is acquired by the acquisition means;
The specific with respect to the imaging unit associated with a plurality of points constituting the image of a specific subject included in tilt-out及 beauty the one image with respect to the gravity direction of the image pickup unit detected by said first detection means First calculating means for calculating an inclination of the specific subject with respect to the direction of gravity based on a provisional posture of the subject;
Second detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired by the acquisition means;
Based on the inclination with respect to the second detecting means the direction of gravity of the specific object which is calculated by the inclination-out及 beauty said first calculating means with respect to the gravity direction of the image pickup unit detected by, construct an image of said specific object A second calculation for calculating a relative posture of the specific subject with respect to the imaging unit as an initial value of a coordinate conversion formula for converting the spatial coordinates of each of the plurality of points to the plane coordinates in the other image Means,
Ru with a,
An object detection apparatus characterized by the above. Based on the relative orientation of the specific subject that is calculated as the initial value by the second calculating means, according to claim 1, further comprising a coordinate transformation means for solving the coordinate transformation formula Subject detection device. The coordinate conversion means further solving the coordinate transformation equation, subject detection apparatus according to claim 2, wherein the identifying a matrix representing the relative position of the specific subject with respect to the imaging unit .   4. The subject according to claim 2, wherein the coordinate conversion unit further specifies a matrix representing a relative positional relationship of the specific subject with respect to the imaging unit by solving the coordinate conversion formula. 5. Detection device. It said first calculating means further each time the matrix is identified by the coordinate transformation means, said matrix and on the basis of the inclination with respect to the gravity direction of the image pickup unit detected by the second detection means, of the specific subject detection apparatus according to claim 3 or 4, characterized in that calculating a slope with respect to the direction of gravity of the object. The first calculation means further calculates an inclination of the specific subject with respect to the gravity direction every time an image is acquired by the acquisition means, and based on the inclination of the specific subject with respect to the gravity direction calculated a predetermined number of times. calculating the tilt Kino representative value each subject detection apparatus according to any one of claims 1 to 5, wherein. A subject detection method using a subject detection device,
A process of sequentially acquiring images of a specific subject imaged by the imaging unit;
Processing for detecting the inclination of the imaging unit with respect to the gravitational direction when one image is acquired;
Wherein the specific temporary position of the object relative to the imaging unit associated with a plurality of points constituting the image of a specific subject included in tilt-out及 beauty the one image for the detected direction of gravity of the imaging unit Based on the process of calculating the inclination of the specific subject with respect to the direction of gravity;
A process of detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired;
Based on the inclination with respect to the detected tilt-out with respect to the gravity direction of the imaging unit 及 beauty the gravity direction of the specific subject, the plurality of points of the spatial coordinates of the other points constituting the image of the specific object A process of calculating a relative posture of the specific subject with respect to the imaging unit as an initial value of a coordinate conversion formula for converting into plane coordinates in an image;
Including,
An object detection method characterized by the above. The computer of the subject detection device
Acquisition means for sequentially acquiring images of a specific subject imaged by the imaging unit;
First detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when one image is acquired by the acquisition means;
The specific with respect to the imaging unit associated with a plurality of points constituting the image of a specific subject included in tilt-out及 beauty the one image with respect to the gravity direction of the image pickup unit detected by said first detection means First calculating means for calculating an inclination of the specific subject with respect to the direction of gravity based on a provisional posture of the subject;
Second detection means for detecting an inclination of the imaging unit with respect to the gravitational direction when another image different from the one image is acquired by the acquisition means;
Based on the inclination with respect to the second detecting means the direction of gravity of the specific object which is calculated by the inclination-out及 beauty said first calculating means with respect to the gravity direction of the image pickup unit detected by, construct an image of said specific object A second calculation for calculating a relative posture of the specific subject with respect to the imaging unit as an initial value of a coordinate conversion formula for converting the spatial coordinates of each of the plurality of points to the plane coordinates in the other image means,
To function as,
A program characterized by that.

Images