JP5772540B2 - Portable terminal device, area designation method, and area designation program - Google Patents

Description

  The disclosed technology relates to, for example, a portable terminal device, an area designation method, and an area designation program.

  For example, when a portable terminal device such as a smartphone is linked with a peripheral device, the peripheral device to be targeted must be specified by inputting the IP address or MAC address of the peripheral device into the portable terminal device. In order to omit such work, an object designation technique has been developed in which an object that a user sees in the real world, that is, a peripheral device is directly pointed and information associated with the object is acquired.

  As an object designating technique, for example, an ID reader is disclosed that irradiates an optical ID tag attached to an object with a light beam and reads identification information of the optical ID tag based on reflected light of the optical ID tag. ing. The optical ID tag according to the prior art modulates the light beam emitted from the optical ID reader with the identification information stored in the memory of the optical ID tag, thereby adding the identification information to the reflected light of the optical ID tag. For this reason, the ID reader can acquire the identification information of the optical ID tag by receiving the reflected light of the optical ID tag.

JP 2006-11948 A

  By the way, according to the ID reader according to the prior art, the optical ID tag must be irradiated with a light beam. However, it is not easy to accurately irradiate a light beam to a remote optical ID tag.

  The disclosed technology provides a portable terminal device, an area designation method, and an area designation program that can designate a target area from a captured image.

  According to one aspect of the disclosed technology, a determination unit that determines a display image to be displayed on a display screen among a plurality of captured images captured by a camera in a portable terminal device; A first acquisition unit that acquires a first angular velocity and a second angular velocity centered on a first axis and a second axis defined in a plane intersecting a normal line of the display screen; A first rotation amount and a second rotation amount about the first axis and the second axis of the portable terminal device are calculated based on an angular velocity of 1 and the second angular velocity. A calculation unit; a second calculation unit that calculates coordinates that are included in the range of the display screen and that are linked to the optical axis of the camera based on the first rotation amount and the second rotation amount; and the display A superimposing unit that superimposes a pointer on the coordinates of the image, and before the pointer is superimposed A display unit for displaying a display image on the display screen, an extraction unit that extracts an image area including the pointer from the display image, the portable terminal device with a provided.

  According to the disclosed technology, a target area can be designated from a captured image.

It is a perspective view of the smart phone concerning a 1st embodiment. It is the schematic of the hardware of the smart phone concerning a 1st embodiment. It is the schematic of the functional block of the controller concerning 1st Embodiment. It is a flowchart of the process by the controller concerning 1st Embodiment. It is the schematic of the locus | trajectory of the pointer displayed on a display screen by the motion of the smart phone concerning 1st Embodiment. It is the schematic of the extraction process which moves the smart phone concerning 1st Embodiment and extracts an image area | region from a display image. It is the schematic of the functional block of the display image determination part concerning 1st Embodiment. It is a graph of the SMA value calculated from the motion component of the acceleration component derived from the motion of the smart phone concerning a 1st embodiment, and the acceleration component of a smart phone. It is a flowchart of the process by the display image determination part concerning 1st Embodiment. It is the schematic of the functional block of the image comparison part concerning 1st Embodiment. It is the schematic of the database of the template memorize | stored in the template memory | storage part concerning 1st Embodiment. It is the schematic of the matching by the image comparison part concerning 1st Embodiment. It is the schematic of the functional block of the controller concerning 2nd Embodiment. It is the schematic of the functional block of the size adjustment part concerning 2nd Embodiment. It is a flowchart of the process by the size adjustment part concerning 2nd Embodiment. It is the schematic of the size adjustment process which adjusts the size of the image area | region concerning 2nd Embodiment. It is the schematic of the size adjustment process which adjusts the size of the image area | region concerning 2nd Embodiment. It is the schematic of the functional block of the controller concerning 3rd Embodiment. It is the schematic of the functional block of the size adjustment part concerning 3rd Embodiment. It is the schematic of the functional block of the controller concerning 4th Embodiment. It is the schematic of the functional block of the image area extraction part concerning 4th Embodiment. It is a flowchart of the process by the image area extraction part concerning 4th Embodiment. It is the schematic of the extraction process which extracts the optimal image area | region concerning 4th Embodiment. It is the schematic of the functional block of the controller concerning 5th Embodiment. It is the schematic of the functional block of the display image determination part concerning 5th Embodiment. It is a flowchart of the process by the display image determination part concerning 5th Embodiment. It is the schematic of the smart phone and server concerning 6th Embodiment.

[First Embodiment]
The first embodiment will be described below with reference to FIGS.

(Outline of smartphone 10)
FIG. 1 is a perspective view of a smartphone 10 according to the first embodiment.

  As shown in FIG. 1, the smartphone 10 according to the present embodiment includes a rectangular box-shaped casing 11, a rectangular display screen 12, and a round operation button 13. In addition, although this embodiment demonstrates the smart phone 10, if it is a portable terminal carrying a camera function, it will not be limited to the smart phone 10.

  The smartphone 10 according to the present embodiment has three axes orthogonal to each other, that is, an x axis, a y axis, and a z axis. As illustrated, the x-axis extends in the longitudinal direction of the display screen 12, the y-axis extends in the short direction of the display screen 12, and the z-axis extends in the orthogonal direction of the display screen 12. .

(Smartphone 10 hardware)
FIG. 2 is a schematic diagram of hardware of the smartphone 10 according to the first embodiment.

  As shown in FIG. 2, the smartphone 10 according to the present embodiment includes a controller 21, a memory 22, a flash memory 23, a communication module 24, a real time clock 25, a display 26, a camera 27, and an inertial sensor 28.

  The controller 21, the memory 22, the flash memory 23, the communication module 24, the real time clock 25, the display 26, the camera 27, and the inertial sensor 28 are connected to each other by a bus B, for example.

  The controller 21 reads out various programs stored in the flash memory 23 and develops them in the memory 22, and implements various functions by executing the various programs developed in the memory 22. Details of the various functions will be described later.

  The memory 22 stores various programs read from the flash memory 23 by the controller 21. Furthermore, the memory 22 functions as, for example, the image storage unit 250, the template storage unit 251, the display image storage unit 252, and the feature amount storage unit 253, and various data used when various functions are executed by the controller 21. Memorize etc. Details of the image storage unit 250, the template storage unit 251, the display image storage unit 252, and the feature amount storage unit 253 will be described later.

  The flash memory 23 stores various programs for controlling the operation of the smartphone 10. The area designation program according to the present embodiment is also stored in the flash memory 23. In the present embodiment, the flash memory 23 is used. However, the present invention is not limited to this as long as it is a nonvolatile memory.

  The communication module 24 transmits the image information of the image region R extracted from the display image I described later to, for example, an external server S by using the communication function of the smartphone 10, for example. Further, the communication module 24 sends information associated with each of the peripheral devices A, B..., For example, identification information of the peripheral devices A, B. Receive from S.

  The real-time clock 25 is used, for example, to calibrate an event timer (not shown) of the controller 21 and to acquire the display image I from a plurality of captured images, for example.

  The display 26 is disposed inside the housing 11 and exposes the display screen 12 from the front surface of the housing 11. The type of the display 26 is not particularly limited, but a touch screen (not shown) is pasted on the display screen 12.

  The camera 27 is disposed inside the housing 11, exposes a lens unit (not shown) from the rear surface of the housing 11, and photographs the rear side of the housing 11, that is, the side opposite to the display screen 12 of the display 26. However, the present embodiment is not limited to this. For example, the lens unit (not shown) of the camera 27 may be exposed from the front surface of the housing 11 and the front side of the housing 11 may be photographed.

  The inertial sensor 28 includes an acceleration sensor 28a and a gyro sensor 28b. In this embodiment, the acceleration sensor 28a and the gyro sensor 28b are incorporated in one package of the inertial sensor 28, but may be separate packages.

The acceleration sensor 28a detects the x-axis, y-axis, and z-axis accelerations of the smartphone 10, that is, the first acceleration a _x , the second acceleration a _y , and the third acceleration a _z , respectively. However, depending on the posture of the smartphone 10, the first acceleration a _x , the second acceleration a _y , and the third acceleration a _z include the gravity component of the earth in addition to the movement component derived from the movement of the smartphone 10. Sometimes.

The gyro sensor 28b detects angular velocities around the x-axis, y-axis, and z-axis of the smartphone 10, that is, the first angular velocity ω _x , the second angular velocity ω _y , and the third angular velocity ω _z . Note that the first angular velocity ω _x , the second angular velocity ω _y , and the third angular velocity ω _z include a drift component derived from the drift of the gyro sensor 28 b in addition to the motion component derived from the movement of the smartphone 10. There is also.

(Function block of controller 21)
FIG. 3 is a schematic diagram of functional blocks of the controller 21 according to the first embodiment.

  As shown in FIG. 3, the controller 21 according to the present embodiment includes an acceleration acquisition unit 211, an angular velocity acquisition unit 212, an angle calculation unit 213, a coordinate calculation unit 214, an image acquisition unit 215, a display image determination unit 216, and a pointer superimposition unit. 217, an image display unit 218, an image region extraction unit 219, an image comparison unit 220, an information acquisition unit 221, a time management unit 222, an image storage unit 250, a template storage unit 251, a display image storage unit 252, and a feature amount storage unit 253. Prepare.

  Acceleration acquisition unit 211, angular velocity acquisition unit 212, angle calculation unit 213, coordinate calculation unit 214, image acquisition unit 215, display image determination unit 216, pointer superimposition unit 217, image display unit 218, image region extraction unit 219, image comparison unit 220, the information acquisition unit 221, the time management unit 222, the image storage unit 250, the template storage unit 251, the display image storage unit 252, and the feature amount storage unit 253 are all based on an area designation program developed in the memory 22. 21 is realized.

The acceleration acquisition unit 211 detects the acceleration detected by the acceleration sensor 28a of the inertial sensor 28, that is, the first acceleration a _x in the axial direction of the _x axis, the second acceleration a _y in the axial direction of the y axis, and the axis of the z axis. The third acceleration a _{y in} the direction is acquired every predetermined time, for example, every 20 [ms].

The angular velocity acquisition unit 212 is centered on the angular velocity detected by the gyro sensor 28b of the inertial sensor 28, that is, the first angular velocity ω _x centered on the x axis, the second angular velocity ω _y centered on the y axis, and the z axis. a third angular velocity omega _z that, every predetermined time, acquires for example, every 20 [ms].

The angle calculation unit 213 uses the first angular velocity ω _x , the second angular velocity ω _y , and the third angular velocity ω _z acquired by the angular velocity acquisition unit 212 to perform the first angle θ, y centered on the x axis. A second angle φ centered on the axis and a third angle ψ centered on the z axis are calculated.

  1st angle (theta), 2nd angle (phi), and 3rd angle (psi) are based on the attitude | position (henceforth an initial attitude | position) of the smart phone 10 when the display image I is determined by the display image determination part 216. , The rotation amount (rotation angle) about the x axis, the y axis, and the z axis. Therefore, when the first angle θ, the second angle φ, and the third angle ψ are calculated, the posture of the smartphone 10 based on the initial posture is determined. Hereinafter, a method for calculating the first angle θ, the second angle φ, and the third angle ψ will be described.

In the present embodiment, a quaternion is used to calculate the first angle θ, the second angle φ, and the third angle ψ. The quaternion is Q (q ₀ , q ₁ , q ₂ , q ₃ ), and the acceleration centered on the x, y, and z axes acquired by the angular velocity acquisition unit 212 is Ω (ω _x , ω _y , Assuming that ω _z ), the posture update motion differential equation is expressed as the following mathematical formula (1).

The initial value of the quaternion Q is defined as Q ₀ (1, ₀ , ₀ , ₀ ).

  In Expression (1), when the sampling period of the gyro sensor 28b is fixed, the solution of the posture update motion differential equation of Expression (1) is expressed as Expression (2) below.

  In the formula (2), if the angular velocity within the sampling interval of the gyro sensor 28b is constant, the integral term of the formula (2) is approximated as the following formula (3).

In Equation (3), Δθ _x , Δθ _y , and Δθ _z are increases in angle in the section [t _k , t _{k + 1} ], respectively.

  When Equation (1) is Taylor-expanded using Equation (3), a solution to the posture update motion differential equation can be obtained. The solution of the posture update motion differential equation is expressed as the following mathematical formula (4).

  However, in actuality, when a numerical operation is performed in the smartphone 10, the sin term and the cos term in the mathematical formula (4) may be expanded in series. Note that the first order approximate expression of Expression (4) by series expansion is expressed as Expression (5) below, and the second order approximation expression is expressed as Expression (6) below. In this embodiment, Formula (5) is used.

According to Equation (5), if the quaternion Q (t _k ) at a certain time t _k is known, the four elements Q (t _{k + 1} ) at the next time t _{k + 1} can be obtained. In the present embodiment, the initial value of the quaternion Q is Q ₀ (1, _{0, 0, 0} ). Therefore, when the mathematical expression (5) is used, the display image I to be displayed on the display screen 12 is determined. The quaternion Q (t) when an arbitrary time t has elapsed can be calculated.

  Further, if the posture expression by the quaternion Q is converted into the posture expression by Euler angles (θ, φ, ψ), the x-axis, y-axis, and z-axis from the time of determining the display image I to be displayed on the display screen 12 are changed. The rotation angle around the center, that is, the first angle θ, the second angle φ, and the third angle ψ can be calculated. The conversion formula for converting the quaternion into the Euler angle is expressed as the following formulas (7), (8), and (9).

The coordinate calculation unit 214 is centered on the first angle θ centered on the x axis and the y axis among the first angle θ, the second angle φ, and the third angle ψ calculated by the angle calculation unit 213. The coordinates (x _i , y _i ) included in the range of the display screen 12 are calculated based on the second angle φ. The coordinates (x _i , y _i ) are linked to the movement of the optical axis of the camera 27, that is, the central axis of the lens (not shown) of the camera 27. However, when the display image I to be displayed on the display screen 12 is determined, the coordinates (x _i , y _i ) are assumed to be located at the center of the display screen 12, that is, C ₀ (W / 2, H / 2). Therefore, when the smartphone 10 is tilted forward after the display image I to be displayed on the display screen 12 is determined, the optical axis of the camera 27 is also tilted forward, so that the coordinates (x _i , y _i ) calculated by the coordinate calculation unit 214 are. Moves below the center C ₀ (W / 2, H / 2) of the display screen 12. Similarly, when the smartphone 10 is tilted backward after the display image I to be displayed on the display screen 12 is determined, the optical axis of the camera 27 is also tilted backward, so that the coordinates (x _i , y _i) calculated by the coordinate calculation unit 214 are displayed. ) Moves to the upper side of the center C ₀ (W / 2, H / 2) of the display screen 12. W represents the length of the display screen 12 in the axial direction of the x axis, and H represents the length of the display screen 12 in the axial direction of the y axis.

  The image acquisition unit 215 acquires a plurality of captured images captured by the camera 27. The plurality of captured images acquired by the image acquisition unit 215 are stored in the image storage unit 250 constructed in the memory 22, respectively. Note that a plurality of captured images acquired by the image acquisition unit 215 may be stored in the image storage unit 250 in association with respective shooting times.

  The display image determination unit 216 determines the display image I to be displayed on the display screen 12 of the display 26 from the plurality of captured images acquired by the image acquisition unit 215. The display image I determined by the display image determination unit 216 is fixed while being displayed on the display screen 12. That is, the display image I is displayed on the display screen 12 as a so-called background image. The process for determining the display image I will be described later.

The pointer superimposing unit 217 superimposes the pointers p _i on the coordinates (x _i , y _i ) calculated by the coordinate calculating unit 214 in the display image I determined by the display image determining unit 216, respectively. Form of the pointer P _i is not particularly limited, in the present embodiment, a red round dots. The coordinate calculation unit 214 calculates the coordinates (x _i , y _i ) every predetermined time, for example, every 100 [ms]. Therefore, the pointer superimposing unit 217 additionally superimposes the pointer p _i on the display image I every time the coordinates (x _i , y _i ) are calculated by the coordinate calculating unit 214. For this reason, when a certain amount of time elapses after the display image I to be displayed on the display screen 12 is determined, the coordinates C ₀ (W / 2, H / 2) that are the center of the display screen 12 are displayed on the display screen 12. the a starting point, the trajectory T _p of the plurality of pointer P _i is displayed.

The image display unit 218 displays the display image I on which the pointer _Pi is superimposed by the pointer superimposing unit 217 on the display screen 12 of the display 26. However, during the period until the display image I is determined, a plurality of captured images captured by the camera 27 are displayed on the display screen 12 of the display 26.

The image region extraction unit 219 extracts the image region R from the display image I based on the coordinates (x _i , y _i ) of the plurality of pointers P _i superimposed on the display image I by the pointer superimposing unit 217. The image region R is preferably smaller than at least the display image I. Extraction of the image region R, but may take various forms, in the present embodiment, for example, a circumscribed rectangle of the trajectory T _p of the plurality of pointer P _i superimposed on the display image I, the so-called AABB (Axis Aligned Bounding Box) Are extracted as an image region R.

  The image comparison unit 220 compares the image information of the image region R extracted by the image region extraction unit 219 with a plurality of pattern information prepared in advance. The comparison process between the image information of the image region R and the pattern information will be described later.

  The information acquisition unit 221 acquires information related to the object included in the image region R from the server S, for example, based on the comparison result by the image comparison unit 220. When the peripheral device A is included in the image region R, for example, identification information of the peripheral device A or an operation program of the peripheral device A may be acquired.

  Based on the output from the real-time clock 25, the time management unit 222 manages the elapsed time from the determination of the display image I, the elapsed time from the start of the area designation program, and the like.

(Processing flow by the controller 21)
FIG. 4 is a flowchart of processing by the controller 21 according to the first embodiment. Figure 5 is a schematic diagram of the trajectory T _p of the pointer P _i that is displayed on the display screen 12 by the movement of the smartphone 10 according to the first embodiment. FIG. 6 is a schematic view of an extraction process for extracting the image region R from the display image I by moving the smartphone 10 according to the first embodiment.

  The processing flow according to the present embodiment is started by starting an area designation program. When the processing flow is started, as shown in FIG. 4, the image acquisition unit 215 acquires a plurality of captured images captured by the camera 27 (step S101). The plurality of captured images acquired by the image acquisition unit 215 are stored in the image storage unit 250 constructed in the sequential memory 22.

  Next, the display image determination unit 216 determines one display image I from the plurality of captured images acquired by the image acquisition unit 215 (step S102). The display image I determined by the display image determination unit 216 is displayed as a background image on the display screen 12 of the display 26. The process for determining the display image I will be described later.

Next, the angle calculation unit 213 converts the first angular velocity ω _x , the second angular velocity ω _y , and the second angular velocity ω _z around the x axis, the y axis, and the z axis acquired by the angular velocity acquisition unit 212. Based on this, the rotation amount of the smartphone 10 around the x-axis, y-axis, and z-axis, that is, the first angle θ, the second angle φ, and the third angle ψ are calculated (step S103).

Next, the coordinate calculation unit 214 is based on the first angle θ and the second angle φ among the first angle θ, the second angle φ, and the third angle ψ calculated by the angle calculation unit 213. The coordinates (x _i , y _i ) included in the range of the display screen 12 of the display 26 and interlocking with the movement of the optical axis of the camera 27 are calculated every predetermined time (step S104).

Next, the pointer superimposing unit 217 superimposes the pointers p _i on the coordinates (x _i , y _i ) calculated by the coordinate calculating unit 214 in the display image I determined by the display image determining unit 216, respectively. (Step S105).

For example, as shown in FIG. 5A, when the user U rotates the smartphone 10 around the x axis, the display displayed on the display screen 12 by the function of the area designation program according to the present embodiment. the image I, a plurality of pointer P _i is superimposed one after another, in a specific area of the display image I, the trajectory T _p of zigzag back and forth in the x-direction is depicted.

Further, as shown in FIG. 5B, when the user U rotates the smartphone 10 around the y axis, the display displayed on the display screen 12 by the function of the area designation program according to the present embodiment. A plurality of pointers _Pi are successively superimposed on the image I, and a zigzag locus T _p that reciprocates in the y direction is drawn in a specific area of the display image I.

  Next, based on the output from the real-time clock 25, the time management unit 222 determines whether the elapsed time from the time when the display image I is determined by the display image determination unit 216 exceeds a predetermined threshold ( Step S106).

  When it is not determined that the elapsed time from the time when the display image I is determined exceeds the threshold (No in step S106), the angle calculation unit 213 determines the first angle θ, the second angle φ, and the third angle ψ. Is continuously calculated (step S103).

On the other hand, when the elapsed time from when the display image I is determined is determined to have exceeded the threshold value (Yes in step S106), the image area extracting unit 219, a rectangle circumscribing the trajectory T _p of the plurality of pointer P _i An area is extracted from the display image I as an image area R (step S107).

  Next, the image comparison unit 220 performs matching between the image information of the image region R extracted by the image region extraction unit 219 and a plurality of pieces of pattern information prepared in advance (step S108). The matching process between the image information of the image region R and the image pattern information will be described later.

  If it is not determined that the matching between the image area R and any of the plurality of pattern information is successful (No in step S109), the image acquisition unit 215 continues to acquire the captured image captured by the camera 27. (Step S101).

  On the other hand, if it is determined that the matching between the image information of the image region R and any of the plurality of pattern information is successful (Yes in step S109), the information acquisition unit 221 corresponds to the pattern information when the matching is successful. The related information is acquired from the server S, for example. For example, when the peripheral device A is included in the image region R, the identification information of the peripheral device A or the operation program of the peripheral device A may be acquired as related information.

The smartphone 10 according to the present embodiment is equipped with an area designation program for realizing the above processing flow. For this reason, when the smartphone 10 is moved so as to rotate the optical axis of the camera 27, a pointer as shown in FIG. 6A is displayed on the display image I displayed on the display screen 12 by the function of the area designation program. the trajectory _{T p} of P _i is depicted. Further, when the trajectory T _p of the pointer P _i as shown in FIG. _6A is drawn, the function of the area designation program changes the trajectory T _p of the pointer P _i as shown in FIG. 6B. A circumscribed rectangular area is generated as the image area R.

Thus, the peripheral device A as a target displayed on the display image I, if moving the smartphone 10 as the trajectory T _p of the pointer P _i is superimposed, the peripheral device A is included, is smaller than the display image I A rectangular area can be designated as the image area R.

  Through the processing flow described above, the smartphone 10 can capture the object displayed in the captured image, for example, related information of the peripheral device A, into the smartphone 10. Thereby, the smart phone 10 can function-link with the peripheral device A very easily, for example.

(Details of display image determination unit 216)
FIG. 7 is a schematic diagram of functional blocks of the display image determination unit 216 according to the first embodiment. FIG. 8A is a graph of motion components a _xm , a _ym , and a _zm of acceleration derived from the motion of the smartphone 10 according to the first embodiment. FIG. 8B is a graph of the SMA value calculated from the motion component of acceleration of the smartphone 10 according to the first embodiment. Note that the three graphs in FIG. 8A indicate any one of the motion components a _xm , a _ym , and a _zm , respectively. A two-dot chain line in FIG. 8B indicates a threshold value of the SMA value.

  As shown in FIG. 7, the display image determination unit 216 according to this embodiment includes an AC component acquisition unit 216a, an SMA calculation unit 216b, a stillness determination unit 216c, a shooting instruction unit 216d, and a display image capture unit 216e.

The AC component acquisition unit 216a obtains the gravity component from the first acceleration a _x , the second acceleration a _y , and the third acceleration a _{z in} the axial directions of the x axis, the y axis, and the z axis acquired by the acceleration acquisition unit 211. , And acceleration components derived from the movement of the smartphone 10, that is, the movement components a _xm , a _ym , and a _zm are acquired. A so-called high pass filter is used as the AC component acquisition unit 216a.

As shown in FIG. 8A, in the sections A and C in which the smartphone 10 is operating, the acceleration motion components a _xm , a _ym , and a _zm are high, and the sections B and sections in which the smartphone 10 is stationary. In D, it can be seen that acceleration motion components a _xm , a _ym , and a _zm are low.

The SMA calculation unit 216b calculates an SMA (Signal Magnitude Area) value based on the acceleration motion components a _xm , a _ym , and a _zm acquired by the AC component acquisition unit 216a. The formula for calculating the SMA value is not particularly limited, but in the present embodiment, the following formula (10) is used.

In the equation (10), i is a sampling identification number (meaning the number of times of sampling), and T is a motion component a _xm (i), a _ym (i), a _This is the sampling period of _zm (i).

  As shown in FIG. 8B, it is understood that the SMA value is high in the sections A and C where the smartphone 10 is operating, and the SMA value is low in the sections B and D where the smartphone 10 is stationary. .

  The stationary determination unit 216c compares the SMA value calculated by the SMA calculation unit 216b with a preset threshold value. When the SMA value is lower than the threshold value for a predetermined time, the smartphone 10 Determined to be stationary.

  For example, as shown in FIG. 8B, if the predetermined time is 50 [ms], the SMA value is over 50 [ms] in the sections B and D where the smartphone 10 is stationary. Since it is lower than the threshold value, the smartphone 10 is determined to be stationary.

  When the stationary determination unit 216c determines that the smartphone 10 is stationary, the imaging instruction unit 216d instructs the display image capturing unit 216e to capture a captured image.

  When the capture instruction unit 216d is instructed to capture a captured image, the display image capture unit 216e captures the captured image immediately after the capture of the captured image is instructed as a display image I. The display image I captured by the display image capturing unit 216e is stored in the display image storage unit 252 constructed in the memory 22.

  Thereby, for example, when the user U stops the smartphone 10 while displaying the peripheral device A on the display screen 12, one captured image is automatically captured as the display image I. For this reason, for example, even when the smartphone 10 is held with one hand, the display image I including the target peripheral device A can be easily acquired.

(Processing flow by the display image determination unit 216)
FIG. 9 is a flowchart of processing by the display image determination unit 216 according to the first embodiment. In the present embodiment, the processing flow by the display image determination unit 216 is used, but it goes without saying that the controller 21 is actually realized based on the area designation program.

The processing flow according to the present embodiment starts first when the area designation program is activated. When the processing flow is started, as shown in FIG. 9, the AC component acquisition unit 216 a uses the first acceleration a _x , the second acceleration a _{y, and} the third acceleration a _z acquired by the acceleration acquisition unit 211 to generate a smartphone. The acceleration motion components a _xm , a _ym , and a _zm derived from the ten motions are acquired (step S201).

  In parallel with this, based on the output from the real-time clock 25, the time management unit 222 starts counting the elapsed time from the start of the area designation program according to the present embodiment (step S202).

Next, the SMA calculation unit 216b calculates an SMA value based on the acceleration motion components a _xm , a _ym , and a _zm extracted by the AC component acquisition unit 216a (step S203).

  Next, the stillness determination unit 216c determines whether the SMA value calculated by the SMA calculation unit 216b is lower than a predetermined threshold over a predetermined time (step S204).

When it is not determined that the SMA value is lower than the predetermined threshold for a predetermined time (No in step S204), the AC component acquisition unit 216a determines the acceleration motion component a derived from the motion of the smartphone 10. Acquisition of _xm , _aym , and _azm is continued (step S201).

  On the other hand, if it is determined that the SMA value is lower than the predetermined threshold for a predetermined time (Yes in step S204), the imaging instruction unit 216d captures the captured image in the display image capturing unit 216e. Is instructed (step S205).

  Next, the display image capturing unit 216e captures a captured image acquired immediately after the capturing instruction unit 216b instructs capturing, among a plurality of captured images continuously acquired by the camera 27 (step S206). . The captured image captured by the display image capturing unit 216e is stored in the display image storage unit 252 as the display image I.

(Details of the image comparison unit 220)
FIG. 10 is a schematic diagram of functional blocks of the image comparison unit 220 according to the first embodiment.

  As illustrated in FIG. 10, the image comparison unit 220 according to the present embodiment includes a first feature value calculation unit 220a, a second feature value calculation unit 220b, and a feature value comparison unit 220c.

  The first feature amount calculation unit 220a calculates a feature amount based on the image information of the image region R extracted by the image region extraction unit 219. The feature amount is not particularly limited. For example, an index obtained by quantifying the characteristic position of an object included in the image region R may be used.

  The second feature amount calculation unit 220b calculates each feature amount based on a plurality of pieces of template information stored in the template storage unit 251. Similar to the feature amount calculated by the first feature amount calculation unit 220a, the feature amount may be an index obtained by quantifying the characteristic position of each template. The feature amount of each template calculated by the second feature amount calculation unit 220b is stored in the feature amount storage unit 253 constructed in the memory 22 in association with the template identification information. Details of the template information will be described later.

  The feature amount comparison unit 220c includes the feature amounts of the image information of the image region R calculated by the first feature amount calculation unit 220a and the feature amounts of the plurality of templates calculated by the second feature amount calculation unit 220b. , And the mutual matching degree is calculated. Furthermore, the feature amount comparison unit 220c determines whether there is a template whose feature amount matching level exceeds a predetermined threshold.

(Outline of database DB)
FIG. 11 is a schematic diagram of a database DB of templates T _m1 , T _m2 , T _m3 ... _Stored in the template storage unit 251 according to the first embodiment.

As shown in FIG. 11, the database DB according to this embodiment, correspondence with a plurality of templates _{T m1, T m2, T m3} ... , and each template _{T m1, T m2, T m3} ... identification number, the ing.

For example, the template T _m1 is associated with the identification number 0001, the template T _m2 is associated with the identification number 0002, and the template T _m3 is associated with the identification number 0003. In the present embodiment employs a PC monitor as a copy machine, the template _{T m3} as a template _{T m1} projector, as the template _{T m @ 2.}

(Outline of matching)
FIG. 12 is a schematic diagram of matching by the image comparison unit 220 according to the first embodiment. When the image region R is extracted by the image region extraction unit 219, as shown in FIG. 12, the feature amount of the image information of the image region R is stored in each template T _{m1 of} the database DB stored in the template storage unit 251. It is collated with the feature values of T _m2 and T _m3 . Note that since the feature values of the templates T _m1 , T _m2 , and T _m3 are stored in the feature value storage unit 253, the feature value storage unit 253 is actually referred to.

If there is a template whose matching degree with the feature amount of the image information of the image region R exceeds a predetermined threshold in the plurality of templates T _m1 , T _m2 , T _m3 , the template is associated with the template Information such as an identification number or an operation program is acquired from the server S, for example. In the present embodiment, for example, if the degree of matching between the feature amount of the image information in the image region R and the feature amount of the template T _m1 exceeds the threshold value, the identification information 0001 corresponding to the template T _m1 is obtained. It will be acquired by the smartphone 10.

Thus, according to the present embodiment, the feature amount of the image information in the image region R smaller than the display image I is compared with the feature amounts of the plurality of templates T _m1 , T _m2 , T _m3 . For this reason, the calculation amount can be reduced as compared with the case where the entire feature amount of the display image I is calculated.

[Second Embodiment]
Hereinafter, the second embodiment will be described with reference to FIGS. 13 to 17. However, descriptions of configurations and processes equivalent to those of the first embodiment will be omitted.

  The controller 21A according to the second embodiment is different from the controller 21 according to the first embodiment in that it further includes a size adjusting unit 223.

(Function block of controller 21A)
FIG. 13 is a schematic diagram of functional blocks of the controller 21A according to the second embodiment.

  As illustrated in FIG. 13, the controller 21 </ b> A according to the present embodiment further includes a size adjustment unit 223. The size adjustment unit 223 is realized by the controller 21 based on the area designation program developed in the memory 22.

(Details of the size adjustment unit 223)
FIG. 14 is a schematic diagram of functional blocks of the size adjusting unit 223 according to the second embodiment.

  As shown in FIG. 14, the size adjustment unit 223 according to the present embodiment includes a comparison unit 223a, a magnification calculation unit 223b, an enlargement / reduction unit 223c, and an enlargement / reduction determination unit 223d.

Comparing unit 223a compares calculated by the angle calculator 213, the absolute value of the third angle [psi about the z-axis, the threshold [psi ₀ which is determined in advance, the.

The magnification calculator 223b enlarges / reduces the image region R based on the absolute value of the third angle ψ calculated by the angle calculator 213 and the third angular velocity ω _z acquired by the angular velocity acquirer 212. The magnification of is calculated. In the present embodiment, the magnification for enlarging / reducing the image region R is proportional to the increase in the third angle ψ. For this reason, if the rotation amount of the smart phone 10 centering on az axis is increased, the magnification for enlarging / reducing the image region R also increases. Further, in the present embodiment, or to enlarge the image area R, or is either reduced, determined from the orientation of the third angular velocity omega _z. For this reason, when the rotation direction of the smartphone 10 around the z axis is reversed, the enlargement and reduction of the image region R are also reversed. However, if the third angle [psi is less than the threshold [psi _0, scale factor calculation section 223b does not calculate the magnification of the image area R. This is not intended by the user U, for example, to remove the rotation of the smartphone 10 due to camera shake.

  The enlargement / reduction unit 223c adjusts the size of the image region R based on the magnification determined by the magnification calculator 223b. The size-adjusted image region R is displayed on the display image I in real time. That is, how the size of the image region R changes is displayed on the display screen I.

The _{enlargement / reduction} determination unit 223d determines whether the absolute value of the third angular velocity ω _z is smaller than a predetermined threshold value ω _z0 over a predetermined time. This is because if the absolute value of the third angular velocity ω _z is smaller than a predetermined threshold value ω _z0 over a predetermined time, it is estimated that the size adjustment of the image region R is not aimed. For example, it is estimated that the smartphone 10 is rotating due to camera shake or the like.

(Processing flow by the size adjustment unit 223)
FIG. 15 is a flowchart of processing by the size adjustment unit 223 according to the second embodiment. In the present embodiment, the processing flow by the size adjustment unit 223 is used, but it goes without saying that the controller 21A is actually realized based on the area designation program.

When the display image I including the image region R is displayed on the display screen 12, as illustrated in FIG. 15, the comparison unit 223a acquires the third angle ψ calculated by the angle calculation unit 213 (step S301). Then, it is determined whether the third angle ψ is larger than the threshold ψ ₀ (step S302).

If it is not determined that the magnitude of the third angle ψ is larger than the threshold ψ ₀ (No in step S302), the comparison unit 223a continues to acquire the third angle ψ (step S301). On the other hand, when it is determined that the magnitude of the third angle ψ is larger than the threshold ψ ₀ (Yes in step S302), the magnification calculator 223b scales the image region R based on the magnitude of the third angle ψ. the absolute value of magnification for, on the basis of the third angular velocity omega _z, the enlargement or reduction of the image region R, respectively calculated (step S303).

  Next, the enlargement / reduction unit 223c adjusts the size of the image region R based on the magnification calculated by the magnification calculation unit 223b (step S304).

Next, the scaling determination unit 223d determines whether or not the third angular velocity ω _z is smaller than a predetermined threshold ω _z0 over a predetermined time (step S305). If it is not determined that the third angular velocity ω _z is smaller than the threshold value ω _z0 over a predetermined time (No in step S305), the comparison unit 223a continues to acquire the third angle ψ (step S305). S301). On the other hand, when it is determined that the third angular velocity ω _z is smaller than the threshold value ω _z0 over a predetermined time (Yes in step S305), it is considered that the size adjustment of the image region R is finished, and the third The image region R when the angular velocity ω _z is determined to be smaller than the threshold value ω _z0 is fixed (step S306).

  FIG. 16 is a schematic diagram of a size adjustment process for adjusting the size of the image region R according to the second embodiment, where (a) shows the display image I before size adjustment, and (b) shows the size after adjustment. Display images I are shown respectively. FIG. 17 is a schematic diagram of a size adjustment process for adjusting the size of the image region R according to the second embodiment, where (a) shows the display image I before size adjustment, and (b) shows the size after adjustment. Display images I are shown respectively.

As shown in FIG. 16 (a), the display image I before size adjustment is encompasses the rectangular image region R circumscribing the trajectory T _p of the pointer P _i, a smart phone 10 about the z axis When the image is rotated leftward by an angle ψ (> ψ ₀ ), the image region R is enlarged as shown in FIG. For this reason, as shown in FIG. 17A, even if the image area R is smaller than the display area of the target peripheral device A, the display of the peripheral device A can be performed only by rotating the smartphone 10 around the z axis. An image region R that encompasses the entire region can be generated. That is, if the locus T _p of the pointer P _i is drawn on a very small part of the display area of the peripheral device A, the peripheral device A can be specified by enlarging the image region R. Therefore, the locus of the pointer P _i it is possible to shorten the time required to render the T _p.

[Third Embodiment]
Hereinafter, the third embodiment will be described with reference to FIGS. 18 and 19. However, descriptions of configurations and processes equivalent to those of the second embodiment will be omitted.

  The controller 21B according to the third embodiment is different from the controller 21A according to the second embodiment in that it includes a size adjustment unit 224 that is different from the size adjustment unit 223 according to the second embodiment.

(Function block of controller 21B)
FIG. 18 is a schematic diagram of functional blocks of the controller 21B according to the third embodiment.

  As illustrated in FIG. 18, the controller 21 </ b> B according to the present embodiment includes a size adjustment unit 224 that is different from the size adjustment unit 223 according to the second embodiment. The size adjustment unit 224 is realized by the controller 21 based on the area designation program developed in the memory 22.

(Details of the size adjustment unit 224)
FIG. 19 is a schematic diagram of functional blocks of the size adjusting unit 224 according to the third embodiment.

  As shown in FIG. 19, the size adjustment unit 224 according to the present embodiment further includes a first filter unit 224a, a second filter unit 224b, an angle conversion unit 224c, and an angle index calculation unit 224d.

The first filter portion 224a, the first acceleration _{a x} obtained by the acceleration obtaining unit 211, the first acceleration _{a y,} among the first acceleration _{a z,} a first acceleration a in the axial direction of the x-axis _x is received, and the gravity component a _xg derived from the gravity of the earth is acquired by removing the motion component a _xm derived from the motion of the smartphone 10 from the first acceleration a _x . In the present embodiment, the gravitational component a _xg of the first acceleration a _x is acquired, but the present embodiment is not limited to this. For example, the movement component a _ym derived from the movement of the smartphone 10 may be removed from the second acceleration a _{y in} the axial direction of the y axis, and the gravity component a _ig derived from the gravity of the earth may be acquired. Further, both the gravity component a _{xg of} the first acceleration a _x and the component a _ym of the second acceleration a _y may be acquired. A so-called low-pass filter is used as the first filter unit 224a.

Angle conversion unit 224b is obtained by the first filter portion 224a, based on the gravity component a _xg of first acceleration a _x in the axial direction of the x-axis, and the gravitational acceleration g, which is pre-stored, The gravity component a _xg of the first acceleration a _x is converted into _a rotation angle ψ _a of the smartphone 10 around the z axis.

The second filter unit 224c receives a third angle ψ centered on the z-axis among the first angle θ, the second angle φ, and the third angle ψ calculated by the angle calculation unit 213, By removing the drift angle ψ _d derived from the drift of the gyro sensor 28 b from the third angle ψ, the rotation angle ψ _b of the smartphone 10 around the z axis is acquired. A so-called high-pass filter is used as the second filter unit 224c.

Angle index calculator 224d includes a rotation angle [psi _a converted by the angle conversion unit 224b, by using the rotation angle [psi _b obtained by the second filter section 224c, an expression of the degree of inclination of the smartphone 10 A rotation angle index ψ _index is calculated. The calculation formula of the rotation angle index ψ _index is expressed as the following formula (11).

In Expression (11), “α” and “1-α” are weighting multipliers for the rotation angle ψ _a and the rotation angle ψ _b , respectively.

In the present embodiment, the rotation angle index ψ _index calculated by Expression (11) is regarded as the third angle ψ, and the size of the image region R displayed on the display screen 12 is set as in the second embodiment. It will be adjusted.

According to this embodiment, in order to adjust the size of the image region R displayed on the display screen 12, not only the third angle ψ calculated by the angle calculation unit 213 but also the acceleration acquisition unit 211 acquired. first acceleration a _x are also used. For this reason, the reliability of the size adjustment of the image region R can be improved.

Furthermore, according to the present embodiment, in order to adjust the size of the image region R displayed on the display screen 12, the drift angle ψ derived from the drift of the gyro sensor 28b from the third angle ψ centered on the z axis. _The rotation angle ψ _b from which _d is removed is used. For this reason, it is possible to prevent size adjustment unintended by the user U from being affected by the drift of the gyro sensor 28b.

[Fourth Embodiment]
Hereinafter, the fourth embodiment will be described with reference to FIGS. 20 to 23. However, descriptions of configurations and processes equivalent to those of the first embodiment will be omitted.

  The controller 21 </ b> C according to the fourth embodiment is different from the controller 21 according to the first embodiment in that the controller 21 </ b> C includes an image region extraction unit 225 different from the image region extraction unit 219 according to the first embodiment.

(Function block of controller 21C)
FIG. 20 is a schematic diagram of functional blocks of a controller 21C according to the fourth embodiment.

  As illustrated in FIG. 20, the controller 21 </ b> C according to the present embodiment includes an image area extraction unit 225 that is different from the image area extraction unit 219 according to the third embodiment. The image area extraction unit 225 is realized by the controller 21 </ b> C based on the area designation program developed in the memory 22.

(Details of image region extraction unit 225)
FIG. 21 is a schematic diagram of functional blocks of the image area extraction unit 225 according to the fourth embodiment.

  As shown in FIG. 21, the image area extraction unit 225 according to the present embodiment includes a first extraction unit 225a, a pointer centroid calculation unit 225b, a distance calculation unit 225c, and a second extraction unit 225d.

First extraction portion 225a, from the display screen I multiple pointers _{P i} is superimposed by the pointer superimposing unit 217, extracts a first rectangular region _{R 1} encompasses trajectory Tp of a plurality of pointer _{P i.} Multiple pointers P _i first extraction of the rectangular region R ₁ encompasses trajectory Tp of, but are not particularly limited, in the present embodiment, the circumscribed rectangle which circumscribes the trajectory Tp of the plurality of pointer P _i and extracted first as a rectangular region _{R 1.}

Pointer centroid calculating unit 225b, based on the display screen I multiple pointers P _i is superimposed by the pointer superimposing unit 217, calculates the coordinates of the center of gravity C _g plurality of pointer P _i in the display screen I.

The distance calculation unit 225c uses the first distance d ₁ to the first boundary B ₁ located in the positive direction of the x-axis of the first rectangular region R ₁ with the center of gravity C _g of the plurality of pointers P _i as the starting point. The second distance d ₂ to the second boundary B ₂ located in the negative x-axis direction of the first rectangular region R ₁ is calculated.

The second extraction unit 225d compares the first distance _{d 1} and the second distance _{d 2} calculated by the distance calculation unit 225c. Further, the second extracting unit 225d by a distance smaller one of the first distance d ₁ and the second distance d _2, at a position spaced from the center of gravity C _g plurality of pointer P _i, respectively borderline a second rectangular region R ₂ having extracted.

(Processing flow by the image region extraction unit 225)
FIG. 22 is a flowchart of processing by the image region extraction unit 225 according to the fourth embodiment. FIG. 23 is a schematic diagram of an extraction process for extracting an optimal image region R according to the fourth embodiment. In the present embodiment, the processing flow by the image region extraction unit 225 is used, but it goes without saying that the controller 21C is actually realized based on the area designation program.

The processing flow according to the present embodiment is performed when the elapsed time from the time when the display image I is determined exceeds the predetermined threshold, that is, when the pointer superimposing unit 217 sets a plurality of pointers _Pi . It starts when it is determined that superimposition has been completed.

When the process flow starts, first extraction portion 225a, as shown in FIG. 23 (a), a circumscribed rectangle which circumscribes the trajectory Tp of the plurality of pointer P _i, the display screen I as the first rectangular region R ₁ (Step S401).

Next, the pointer centroid calculating unit 225b, as shown in FIG. 23 (b), based on the image information of the display image I, calculates the coordinates of the center of gravity C _g plurality of pointer P _i in the display image I ( Step S402). In the display image I, the coordinates of the center of gravity C _g plurality of pointer P _i, the mark of the center of gravity, for example may be displayed like a dot.

Next, as illustrated in FIG. 23C, the distance calculation unit 225 c has a first position located in the positive direction of the x-axis of the _first rectangular region R ₁ with the centroids C _g of the plurality of pointers P _i as starting points. calculating a first distance d ₁ to the boundary B _1, a second distance d ₂ to the second boundary B ₂ is located in the negative direction of the first x-axis of the rectangular region R _1, respectively (Step S403).

Then, the second extraction unit 225d compares the first distance _{d 1} and the second distance _{d 2} calculated by the distance calculating unit 225c (step S404). Further, the second extraction unit 225d, as shown in FIG. 23 (d), the smaller the distance of the first distance _{d 1} and the second distance _{d 2,} in this embodiment the first distance _{d 1} Thus, the second rectangular region R ₂ having a boundary line at each position separated from the centroid C _g of the plurality of pointers P _i in the positive and negative directions of the x-axis is extracted as the image region R (step S405). Incidentally, the hatched region in the figure, when the first rectangular region R ₁ for extracting a second rectangular region R _2, is a region that is removed from the first rectangular region R _1.

According to the present embodiment, the first distance d _{1 to} the first boundary B ₁ and the second boundary B ₂ of the first rectangular region R ₁ with the center of gravity C _g of the plurality of pointers P _i as the starting point, and The second distance d ₂ is calculated, and each boundary is set at a position separated from the centroids C _g of the plurality of pointers P _{i by} the smaller one of the first distance d ₁ and the second distance d _2. An image region R having the same is generated.

Therefore, while the user U has specified the image region R, for example, a smart phone 10 will be inadvertently shaking, as shown in FIG. 23 (a), a low density of the plurality of pointer P _i region There is also generated, as shown in FIG. 23 (d), it is possible to extract an image region R deleting the density of a plurality of pointer P _i is lower region (hatched region in the drawing). That is, only the area that the user U wants to specify can be extracted as the image area R.

[Fifth Embodiment]
Hereinafter, the fifth embodiment will be described with reference to FIGS. However, descriptions of configurations and processes equivalent to those of the first embodiment will be omitted.

  The controller 21D according to the fifth embodiment is different from the controller 21 according to the first embodiment in that it includes a display image determination unit 226 different from the display image determination unit 216 according to the first embodiment.

(Function block of controller 21D)
FIG. 24 is a schematic diagram of functional blocks of a controller 21D according to the fifth embodiment.

  As illustrated in FIG. 24, the controller 21D according to the present embodiment includes a display image determination unit 226 that is different from the display image determination unit 216 according to the first embodiment. The display image determination unit 226 is realized by the controller 21D based on the area designation program developed in the memory 22.

(Details of display image determination unit 226)
FIG. 25 is a schematic diagram of functional blocks of the display image determination unit 226 according to the fifth embodiment.

  As shown in FIG. 25, the display image determination unit 226 according to the present embodiment includes a minimum SMA specifying unit in addition to the AC component acquisition unit 216a and the SMA calculation unit 216b of the display image determination unit 216 according to the first embodiment. 226a and a display image capturing unit 226b. However, the display image determination unit 226 according to the present embodiment does not include the stillness determination unit 216c, the imaging instruction unit 216d, and the display image capture unit 216e of the display image determination unit 216 according to the first embodiment.

  The minimum SMA specifying unit 226a specifies the minimum SMA value in a predetermined time interval from the plurality of SMA values calculated by the SMA calculation unit 216b. The minimum SMA value is specified at predetermined time intervals.

  The display image capturing unit 226b captures a captured image corresponding to the minimum SMA value in a predetermined time interval specified by the minimum SMA specifying unit 226a from a plurality of captured images stored in the image storage unit 250. Select and capture as display image I. Note that when selecting a captured image corresponding to the minimum SMA value, it is only necessary to use a capturing time associated with each captured image. The display image I captured by the display image capturing unit 226 b is stored in the display image storage unit 252 constructed in the memory 22.

(Processing flow by the display image determination unit 226)
FIG. 26 is a flowchart of processing by the display image determination unit 226 according to the fifth embodiment. In the present embodiment, the processing flow by the display image determination unit 226 is used, but it goes without saying that the controller 21D is actually realized based on the area designation program.

  The processing flow according to the present embodiment starts first when the area designation program is activated. When the processing flow is started and the SMA value is calculated by the SMA calculation unit 216b as shown in FIG. 26 (step S203), the time management unit 222 according to the present embodiment based on the output from the real-time clock 25. It is determined whether the elapsed time from the start of the area designation program exceeds a predetermined threshold (step S501).

If it is not determined that the elapsed time from the start of the area designation program exceeds a predetermined threshold (No in step S501), the AC component acquisition unit 216a determines the acceleration motion component derived from the motion of the smartphone 10. _{a xm,} a ym, to continue the acquisition of _{a zm} (step S201).

  On the other hand, if it is determined that the elapsed time from the start of the area designation program has exceeded a predetermined threshold (Yes in step S501), the minimum SMA specifying unit 226a sets the predetermined time interval by the SMA calculation unit 216b. The minimum SMA value is specified from the calculated plurality of SMA values (step S502).

  Next, the display image capturing unit 226b captures one captured image corresponding to the minimum SMA value specified by the minimum SMA specifying unit 225a from the plurality of captured images stored in the image storage unit 250 ( Step S503). The captured image captured by the display image capturing unit 226b is stored as the display image I in the display image storage unit 252.

  According to the present embodiment, a plurality of photographed images photographed by the camera 27 are stored in the image storage unit 250, and the SMA value is obtained from the photographed images photographed in the immediately preceding time interval each time a predetermined time elapses. The captured image that minimizes is set as the display image I. For this reason, for example, even the user U who has symptoms such as camera shake can reliably set the display image I.

[Sixth Embodiment]
Hereinafter, the sixth embodiment will be described with reference to FIG. However, descriptions of configurations and processes equivalent to those of the first embodiment will be omitted.

  FIG. 27 is a schematic diagram of the smartphone 10 and the server S according to the sixth embodiment.

  As illustrated in FIG. 27, the controller 21E of the smartphone 10 according to the present embodiment does not include the image comparison unit 220, and instead, the controller 300 of the server S includes the image comparison unit 220. That is, the image comparison unit 220 according to the present embodiment has the same function as the image comparison unit 220 according to the first embodiment, but differs from the first embodiment in that it is constructed on the server S side. . Along with this, the template storage unit 251 is also constructed in the memory 310 of the server S.

  If the image comparison unit 220 is constructed on the server S side as in the present embodiment, the calculation load necessary for matching can be assigned to the server S, so that the load on the smartphone 10 can be reduced.

DESCRIPTION OF SYMBOLS 10: Smart phone 12: Display screen 27: Camera 211: Acceleration acquisition part 212: Angular velocity acquisition part 213: Angle calculation part 214: Coordinate calculation part 216: Display image determination part 217: Pointer superimposition part 218: Image display part 219: Image area | region Extraction unit 220: image comparison unit 221: information acquisition unit 223: size adjustment unit 224: size adjustment unit 225: image region extraction unit 225a: first extraction unit 225b: pointer centroid calculation unit 225c: distance calculation unit 225d: second Extraction unit 226: display image determination unit B ₁ : first boundary line B ₂ : second boundary line C _g : center of gravity d ₁ : first distance d ₂ : second distance I: display image P _i : Pointer R: Image area R ₁ : First rectangular area R ₂ : Second rectangular area θ: First angle φ: Second angle ψ: Third angle

Claims (13)

In portable terminal devices,
A determining unit that determines a display image to be displayed on the display screen among a plurality of captured images captured by the camera;
The first angular velocity about the first axis and the second axis defined in the plane intersecting the normal line of the display screen of the portable terminal device is acquired. An acquisition unit;
Based on the first angular velocity and the second angular velocity, a first rotation amount and a second rotation amount about the first axis and the second axis of the portable terminal device are calculated. 1 calculation unit;
A second calculation unit that calculates coordinates that are included in the range of the display screen and that are linked to the optical axis of the camera, based on the first rotation amount and the second rotation amount;
A superimposing unit that superimposes a pointer on the coordinates of the display image;
A display unit for displaying the display image on which the pointer is superimposed on the display screen;
An extraction unit for extracting an image area including the pointer from the display image;
A portable terminal device comprising: The portable terminal device according to claim 1, further comprising:
A portable terminal device comprising a designation unit for designating an object included in the image area based on image information of the image area. In the portable terminal device according to claim 2,
The designation unit is:
A comparison unit that compares the image information of the image area with the image information prepared in advance;
An acquisition unit that acquires information corresponding to the image information of the image area based on a comparison result of the comparison unit;
A portable terminal device comprising: The portable terminal device according to any one of claims 1 to 3,
The first acquisition unit further acquires a third angular velocity centered on a third axis that intersects the first axis and the second axis,
The first calculation unit calculates a third rotation amount around the third axis of the portable terminal device based on the third angular velocity,
The portable terminal device further includes:
A portable terminal device comprising an adjustment unit that adjusts the size of the image area based on the third rotation amount. In the portable terminal device according to claim 4,
The said adjustment part is a portable terminal device which determines enlargement and reduction of the size of the said image area based on the rotation direction of the said portable terminal device centering on the said 3rd axis | shaft. In the portable terminal device according to claim 4 or 5,
The adjustment unit compares the third rotation amount with a reference rotation amount set in advance, and when the third rotation amount exceeds the reference rotation amount, the size of the image area A portable terminal device that adjusts. The portable terminal device according to any one of claims 1 to 3, further comprising:
A second acquisition unit that acquires acceleration in at least one of the first axis and the second axis of the portable terminal device;
An adjustment unit that adjusts the size of the image area based on a change in acceleration in the axial direction;
A portable terminal device comprising: The portable terminal device according to any one of claims 1 to 7,
The extraction unit is a portable terminal device that determines the image area based on the density of the pointer in the display image. The portable terminal device according to any one of claims 1 to 8,
The extraction unit includes:
A first extraction unit that extracts a first rectangular area that includes a first boundary line and a second boundary line that are parallel to each other and that includes the pointer, from the display image;
A third calculator for calculating the center of gravity of the pointer in the first area;
A fourth calculation unit for calculating a distance from the center of gravity of the pointer to the first boundary line and the second boundary line;
The boundary line is located at a position separated from the center of gravity by the smaller one of the distance from the center of gravity of the pointer to the first boundary line and the distance from the center of gravity of the pointer to the second boundary line. A second extraction unit that extracts a rectangular second area having the image area,
A portable terminal device comprising: In an area designation method for designating an area by a portable terminal device,
Determining a display image to be displayed on a display screen among a plurality of captured images captured by the camera of the portable terminal device; and
Obtaining a first angular velocity and a second angular velocity centered on a first axis and a second axis defined in a plane intersecting a normal line of the display screen of the portable terminal device;
Calculating a first rotation amount and a second rotation amount around the first axis and the second axis of the portable terminal device based on the first angular velocity and the second angular velocity; When,
Calculating a coordinate included in the range of the display screen and interlocked with the optical axis of the camera based on the first rotation amount and the second rotation amount;
Superimposing a pointer on the coordinates of the display image;
Extracting an image area including the pointer from the display image;
An area designation method comprising: The area designation method according to claim 10, further comprising:
An area designation method comprising a step of designating an object included in the image area based on image information of the image area. In an area designation program that designates an area with a portable terminal device,
The area specifying program is stored in a computer.
Determining a display image to be displayed on a display screen among a plurality of photographed images photographed by the camera of the portable terminal device;
Obtaining a first angular velocity and a second angular velocity centered on a first axis and a second axis defined in a plane intersecting a normal line of the display screen of the portable terminal device;
Calculating a first rotation amount and a second rotation amount around the first axis and the second axis of the portable terminal device based on the first angular velocity and the second angular velocity; ,
Calculating specific coordinates that are included in the range of the display screen and that are linked to the optical axis of the camera, based on the first rotation amount and the second rotation amount;
Superimposing a pointer on the coordinates of the display image; and extracting an image area including the pointer from the display image;
Area specification program to execute In the area designation program according to claim 12,
The area specifying program is further stored in a computer.
An area designation program for executing a step of designating an object included in the image area based on image information of the image area.

Images