JP2012129731A - Portable terminal and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable terminal and an image processing method capable of stably composing data on a virtual object, etc. to imaging data without influences of changing an imaging range or occurrence of hand shake, etc.SOLUTION: In a portable terminal 100, a continuity detection part 103 detects a sensor value indicating the imaging range of an imaging part 101 and detects an imaging state, and a smoothing processing part 106 smooths the detected sensor value, using a smoothing parameter fixed on the basis of the detected photographing state. Then, a virtual object position determination part 108 calculates a composing position to the imaging data, on the basis of a smoothed sensor value and composes image data to the calculated composing position in the imaging data. A display part 102 is capable of displaying the imaging data to which the image data is composed.

Description

  The present invention relates to a portable terminal and an image processing method for combining and displaying image data such as a virtual object on captured image data.

  In recent years, services using AR (Augmented Reality) technology have been developed and provided. For example, a technique is known in which additional information associated with an object in an image acquired by a camera is acquired, and the acquired additional information is combined with the object and displayed. In this technology, the additional information is posted by the mobile terminal of each user in association with the location of the user, for example, on a server device or the like. Shared. Further, as a technique common to such a technique, a technique is known in which data such as the name of a structure is stored in advance and the name of the structure is displayed in association with the structure photographed by the camera. (For example, refer to Patent Document 1).

JP 2000-50156 A

  As described in Patent Document 1 described above, when combining predetermined data with image data, it is necessary to combine the data in accordance with a predetermined position of the video data. On the other hand, mobile terminals are generally small and easily affected by camera shake. In addition, the user can easily change the shooting range. At that time, depending on the speed of changing the shooting range, the data combining process cannot catch up, and the combining position of the data to be combined varies.

  Therefore, in the present invention, in order to solve the above-described problem, even if the shooting range is changed or camera shake occurs, the data of the virtual object or the like can be stably converted into the shooting data without being affected by it. An object is to provide a portable terminal and an image processing method that can be combined.

  In order to solve the above-described problems, a mobile terminal according to the present invention detects a photographing unit that captures photographing data, a sensor unit that detects a sensor value indicating a photographing range of the photographing unit, and a photographing state of the photographing unit. Continuity detecting means, smoothing means for smoothing the sensor value detected by the sensor means using a smoothing parameter determined based on the photographing state detected by the continuity detecting means, and the smoothing Calculation means for calculating a composite position with respect to the photographing data captured by the photographing means based on the sensor value smoothed by the conversion means; and image data at the composite position in the photographing data calculated by the calculation means. Combining means for combining, and display means for displaying photographed data obtained by combining image data by the combining means.

  Further, the image processing method of the present invention includes a photographing step for capturing photographing data, a sensor step for detecting a sensor value indicating a photographing range of the photographing step, a continuity detecting step for detecting a photographing state of the photographing step, A smoothing step for smoothing the sensor value detected by the sensor step using a smoothing parameter determined based on the photographing state detected by the continuity detecting step, and a smoothing by the smoothing step. A calculation step for calculating a combined position with respect to the shooting data captured by the shooting step based on the sensor value; a combining step for combining image data with a combined position in the shooting data calculated by the calculation step; A display for displaying the shooting data obtained by combining the image data in the combining step. Is provided with a step, a.

  According to the present invention, the sensor value indicating the photographing range is detected, the photographing state is detected, and the detected sensor value is smoothed using the smoothing parameter determined based on the detected photographing state. . Then, based on the smoothed sensor value, a composite position with respect to the shooting data is calculated, and the image data is combined with the calculated combined position in the shooting data, and the shooting data with the combined image data is displayed. it can.

  As a result, depending on the shooting state, it is possible to eliminate the fact that the image data is shifted from the position to be originally synthesized, and it is possible to synthesize the image data at the synthesis position with little error. In particular, if smoothing processing is performed in all states in the photographing state, there is a problem that the quick response is reduced. That is, when the shooting range is moving greatly, even if smoothing processing is performed using past values, the movement of the shooting range cannot be followed and the position relative to the shooting range that has already passed is calculated. Resulting in. In the present invention, since the smoothing process can be performed according to the photographing state, it is possible to calculate an appropriate composite position, improve the response, and reduce the error as much as possible.

  In the mobile terminal of the present invention, it is preferable that the sensor means detects at least one of an acceleration sensor, a geomagnetic sensor, and a position sensor as a sensor value indicating a shooting range.

  According to the present invention, the sensor value indicating the imaging range is preferably a sensor value detected by at least one of an acceleration sensor, a geomagnetic sensor, and a position sensor. As a result, the shooting range can be acquired easily and accurately, and the image data and shooting data can be combined easily and accurately.

  The mobile terminal according to claim 1 or 2, wherein the continuity detecting unit detects a photographing state based on a sensor value detected by the sensor unit.

  According to this invention, it is preferable to detect the photographing state based on the sensor value detected by the sensor means. For example, it is possible to determine whether the shooting state is a still state, a state in which the shooting range has greatly changed, or a state in which the shooting range has changed slowly depending on changes in sensor values. As a result, it is possible to determine the shooting state with a simple configuration by using the sensor means used for grasping the shooting range without providing any special means for determining the shooting state.

  In the mobile terminal of the present invention, it is preferable that the continuity detecting unit detects a shooting state according to a change state of shooting data captured by the shooting unit.

  According to the present invention, it is preferable to detect the shooting state according to the change state of the shooting data. For example, it is possible to determine whether the shooting state is a still state, a state in which the shooting range has changed significantly, or a state in which the shooting range has changed slowly, depending on the variation of the shooting data for each pixel. Accordingly, it is possible to determine the shooting state with a simple configuration by using the shooting unit for taking shooting data without providing a special unit for determining the shooting state.

  In the portable terminal of the present invention, the sensor means sequentially detects sensor values at predetermined intervals and holds them in time series, and the smoothing processing means is a sensor held in the time series. Of the values, it is preferable to perform weighting processing according to the shooting state, with a predetermined number of sensor values extracted according to the shooting state as a smoothing target.

  According to the present invention, sensor values are sequentially detected at predetermined intervals, held in time series, and among sensor values held in time series, a predetermined number of sensor values extracted from the nearest according to the shooting state As a smoothing target, it is preferable to perform a weighting process according to the shooting state. Thereby, it is possible to be affected by a predetermined number of sensor values in the past according to the shooting state, and the influence of the latest error can be reduced.

  Further, in the portable terminal of the present invention, the continuity detecting unit detects the shooting state based on whether or not the shooting state based on the terminal operation by the user has fluctuated by a predetermined value or more in a predetermined time unit, Furthermore, it is preferable to determine the shooting state by detecting whether or not the variation pattern is a predetermined variation pattern in a predetermined time unit.

  According to the present invention, based on whether or not the shooting state based on the terminal operation by the user has fluctuated by a predetermined value or more, the shooting state is detected, and further, the amplitude of the shooting state is determined in a predetermined time unit. It is preferable to detect the presence or absence. Thereby, it is possible to appropriately determine the shooting state. For example, when the shooting state changes by a predetermined value or more, it can be determined that the shooting range has fluctuated greatly. Moreover, when it does not change more than a predetermined value, it can be judged that it changed slowly by operation by the user. Furthermore, when the variation pattern is a predetermined variation pattern, it can be determined that the variation pattern has been varied due to camera shake or disturbance. And an appropriate smoothing process can be performed by using the smoothing parameter according to these three states.

  According to the present invention, it is possible to eliminate the fact that the image data is shifted from the position to be originally synthesized depending on the shooting state, and it is possible to synthesize the image data at the synthesis position with little error. In particular, if smoothing processing is performed in all states in the photographing state, there is a problem that the quick response is reduced. That is, when the shooting range is moving greatly, even if smoothing processing is performed using past values, the movement of the shooting range cannot be followed and the position relative to the shooting range that has already passed is calculated. Resulting in. In the present invention, since the smoothing process can be performed according to the photographing state, it is possible to calculate an appropriate composite position, improve the response, and reduce the error as much as possible.

It is a block diagram which shows the function structure of the portable terminal 100 of this embodiment. 2 is a hardware configuration diagram of the mobile terminal 100. FIG. It is explanatory drawing which showed the fluctuation | variation of the sensor value of the sensor part 105 (for example, geomagnetic sensor). It is explanatory drawing which showed the movement amount (fluctuation) of the pixel by image-analyzing imaging | photography data. It is a specific example of the database memorize | stored in smoothing parameter DB104. It is a figure which shows the management table which memorize | stores a virtual object. 5 is a flowchart showing processing of the mobile terminal 100. 5 is a flowchart showing processing in a continuity detection unit 103. It is explanatory drawing for comparing the synthetic | combination position when the imaging | photography state of the portable terminal 100 is a stationary state. It is explanatory drawing for comparing the synthetic | combination position when the imaging | photography state of the portable terminal 100 is changing a lot.

  Embodiments of the present invention will be described with reference to the accompanying drawings. Where possible, the same parts are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram illustrating a functional configuration of the mobile terminal 100 according to the present embodiment. As shown in FIG. 1, the mobile terminal 100 includes an imaging unit 101 (imaging unit), a display unit 102 (display unit), a continuity detection unit 103 (continuity detection unit), a smoothing parameter DB unit 104, and a sensor unit 105 ( Sensor means), a smoothing processing unit 106 (smoothing means), a virtual object storage unit 107, and a virtual object position determination unit 108 (calculation means, composition means). The portable terminal 100 is configured by hardware shown in FIG.

  FIG. 2 is a hardware configuration diagram of the mobile terminal 100. As shown in FIG. 2, the portable terminal 100 shown in FIG. 1 physically includes a CPU 11, a RAM 12 and a ROM 13 that are main storage devices, an input device 14 such as a keyboard and a mouse that are input devices, and an output such as a display. The computer 15 includes a device 15, a communication module 16 that is a data transmission / reception device such as a network card, an auxiliary storage device 17 such as a hard disk, and the like. Each function described in FIG. 1 has an input device 14, an output device 15, and a communication module 16 under the control of the CPU 11 by reading predetermined computer software on hardware such as the CPU 11 and the RAM 12 shown in FIG. 2. This is realized by reading and writing data in the RAM 12 and the auxiliary storage device 17. Hereinafter, each functional block will be described based on the functional blocks shown in FIG.

  The photographing unit 101 is a part that acquires photographing data photographed by a camera. The acquired shooting data is output to the display unit 102.

  The display unit 102 is a part that displays the shooting data. The display unit 102 further displays shooting data obtained by combining a virtual object with shooting data captured by the shooting unit 101 in a virtual object position determination unit 108 described later.

  The continuity detection unit 103 is photographed by the photographing unit 101 based on a sensor value obtained from the sensor unit 105 (at least one of a gyro sensor, an acceleration sensor, a geomagnetic sensor, or a position sensor (GPS)). This is a part for detecting continuity indicating the degree of change of the photographing data. That is, the continuity detection unit 103 determines whether the shooting range is moved greatly, the shooting range is moved slowly, or not moved when the user performs shooting by the shooting unit 101. Whether the variation of the sensor value is greater than or equal to the predetermined value, less than the predetermined value (not including 0 or its approximate value), or does not vary based on the sensor value from the sensor unit 105 described above. (Including 0 or an approximate value thereof). If it is determined that the image capturing range is greater than or equal to the predetermined value, the user determines that the image capturing range has been moved greatly. If it is determined that it is not, the user determines that the shooting range is not moved.

  The processing of the continuity detection unit 103 will be described in detail. FIG. 3 is an explanatory diagram showing changes in the sensor value of the sensor unit 105 (for example, a geomagnetic sensor). In this explanatory diagram, the vertical axis indicates the azimuth and the horizontal axis indicates time. In the time zone indicated by time (1), the sensor value fluctuates so that the azimuth increases from a position slightly below the azimuth 180 degrees to the right. For such a fluctuation pattern in which the fluctuation range of the sensor value is small (below a predetermined value) and does not show a pattern that fluctuates regularly in the vertical direction, the user determines that the shooting range is moved slowly. Can do. In addition, this direction shall show the angle from the predetermined reference value.

  In the time zone indicated by time (2), the sensor value fluctuates by a predetermined value or more from the azimuth of 180 degrees. About such a fluctuation | variation, the user can judge that the imaging | photography range is moved rapidly.

  And in the time slot | zone shown by time (3), the sensor value is fluctuate | varied regularly up and down with the amplitude of the predetermined range. With respect to such fluctuations, the user can determine that the user has not moved the shooting range and is affected by camera shake or disturbance.

  In FIG. 3, the determination based on the fluctuation of the sensor value is performed. However, the determination is not limited to this, and the determination of the movement of the imaging range of the imaging unit 101 is performed based on the imaging data captured by the imaging unit 101. It can be carried out. FIG. 4 is an explanatory diagram showing the movement amount (fluctuation) of the pixel by image analysis of the photographic data.

  FIG. 4A is a diagram illustrating a movement transition of pixels of image data captured by the image capturing unit 101. As shown in the figure, when the shooting range is moved (or rotated) to the right, it can be seen that the position of the mark m indicating the corner of the roof of the building has moved to the left. By determining the amount of movement, it is determined how much the movement has been made.

  FIG. 4B is an explanatory diagram showing fluctuations in the movement amount of the pixel position. In the time zone indicated by time (4) in FIG. 4B, the movement amount (variation) of the pixel position has a predetermined fluctuation pattern, and the time zone indicated by time (5) is a predetermined value. It indicates that there is the above movement amount (variation), and in the time zone indicated by time (6), it indicates that there is a movement amount (variation) less than the predetermined value of the sensor value, and the time indicated by time (7). The band indicates that there is a movement amount (variation) greater than or equal to a predetermined value of the sensor value. By performing image analysis in this way and determining the amount of pixel movement, the shooting state can be determined. In FIG. 4B, the time (4) is affected by camera shake and the like, and the times (5) and (7) are cases where the photographing range is greatly changed. 6) shows a case where the photographing range is changed slowly.

  Returning to FIG. The smoothing parameter DB unit 104 is a part that stores the weighting coefficient A (t) and the retroactive value N as smoothing parameters in association with the shooting state. FIG. 5 shows a specific example of the database stored in the smoothing parameter DB 104. As shown in FIG. 5, there are “moving slowly”, “moving greatly”, and “still” as shooting states, and the weighting factor A (t) and the retroactive value N are associated with each other. Furthermore, a weighting factor A (t) corresponding to the number of retroactive values N is set. For example, in FIG. 5, since the retroactive value N is 2, two weighting factors A (t) are set as the weighting factors A (t). This usage will be described later.

  The sensor unit 105 includes at least one of an acceleration sensor, a geomagnetic sensor, and a position sensor (GPS module), and acquires a sensor value as information for grasping the imaging range of the imaging unit 101. It is.

ここでは、ある時間tのセンサ値をf(t)とし、ある時間tの平滑化したセンサ値をF(t)とする。また、どこまで遡ってセンサ値を利用するかを示す値をnとする。例えば、ある時間tにセンサ取得した前回の情報を利用する場合はn=1となる。遡ったセンサ値に対する重み付け係数をa(n)とする。ｉは、ｔを基準にして、遡った過去のセンサ値を示す。そして、F(t)の値から仮想オブジェクトをディスプレイ部１０２に表示する位置をX(t)とする。なお、上述のF(t)は、加速度センサ、地磁気センサ、または位置センサのうちの一つを示すものであるが、これらすべてのセンサのセンサ値に対して上述の式（１）を適用して、平滑化処理を行う。 The smoothing processing unit 106 is a part that performs a smoothing process on the sensor value of the sensor unit 105 using the weighting coefficient A (t) and the retroactive value N stored in the smoothing parameter DB unit 104. Specifically, the following arithmetic processing (1) is performed.

Here, a sensor value at a certain time t is defined as f (t), and a smoothed sensor value at a certain time t is defined as F (t). A value indicating how far the sensor value is used is assumed to be n. For example, when using the previous information acquired by the sensor at a certain time t, n = 1. The weighting coefficient for the retroactive sensor value is a (n). i indicates a past sensor value traced back with respect to t. Then, the position where the virtual object is displayed on the display unit 102 from the value of F (t) is defined as X (t). The above F (t) indicates one of an acceleration sensor, a geomagnetic sensor, or a position sensor, and the above formula (1) is applied to the sensor values of all these sensors. Smoothing processing is performed.

  In the smoothing parameter DB unit 104 in FIG. 5, 0.2 and 0.1 are set as the weighting coefficient A (t) in the “slow movement” pattern, and the retroactive value N is set to 2. .

And when applying this to the above equation (1),
F (t) = f (t) + a (t-1) * f (t-1) + a (t-2) * f (t-2) = f (t) + 0.2 * f (t-1 ) + 0.1 * f (t-2)
It becomes.

  The virtual object storage unit 107 is a part that stores virtual objects. FIG. 6 shows a management table for storing virtual objects. In this management table, a file name, a distance (meter), and an arrangement direction (degree) of image data of the virtual object are stored in association with each other. Based on this management table, the virtual object position determination unit 108 can determine in which direction and in which position of the shooting data the virtual object is to be combined.

  The virtual object storage unit 107 acquires a virtual object existing in a predetermined range around the position of the mobile terminal 100 in advance via a network or the like, and virtually generates it based on the shooting range of the mobile device. Management table. Therefore, this management table is rewritten every time the shooting range of the mobile terminal 100 changes.

  The virtual object position determination unit 108 reads out a virtual object according to the management table stored in the virtual object storage unit 107, and performs a process of synthesizing the image data. Prior to the synthesis process, the synthesis position of the virtual object is determined. In this determination, the sensor value for determining the synthesis position is smoothed in advance in the smoothing processing unit 106, and the synthesis position is calculated and determined using the smoothed sensor value. The virtual object position determination unit 108 can repeatedly perform the combined display of the virtual object according to the shooting state, even if the shooting range is moved, by repeatedly performing the combined position determination process in predetermined time units. A method for calculating a composite position from sensor values from an acceleration sensor, a geomagnetic sensor, a position sensor, and the like is generally known.

  The display unit 102 is a part that displays shooting data in which a virtual object is combined at a predetermined combining position.

  Next, processing of the mobile terminal 100 configured as described above will be described. FIG. 7 is a flowchart showing processing of the mobile terminal 100.

  First, the photographing unit 101 of the mobile terminal 100 is activated by a user operation, and photographing data is captured (S101). In response to the activation, the sensor unit 105 is activated (S102), the current position, the photographing range, and the photographing range are grasped, and a virtual object corresponding to the position of the mobile terminal 100 is acquired via the wireless network. Stored in the storage unit 107.

  Next, the virtual object position determination unit 108 determines whether or not there is a virtual object to be synthesized in the virtual object storage unit 107 (S103). If it is determined here that there is a virtual object to be synthesized, various sensor values by the continuity detecting unit 103 are acquired by the smoothing processing unit 106 (S104). Then, the continuity detection unit 103 determines the continuity of the shooting data, that is, the speed of the state change of the shooting range (S105).

  In the smoothing processing unit 106, the weighting coefficient A (t) and the retroactive value N, which are the smoothing parameters stored in the smoothing parameter DB unit 104, are extracted based on continuity (S106). Then, various sensor values are smoothed in the smoothing processing unit 106 using the extracted smoothing parameters (S107). Then, based on the smoothed sensor value, a composite position in the photographing data is calculated. Then, the virtual object is combined at the combined position of the shooting data and displayed on the display unit 102 (S108).

  The processes from S104 to S108 are repeated until the activation state of the photographing unit 101 is turned off. Therefore, according to the sensor value acquisition timing by the sensor unit 105, the synthesis position of the virtual object in the photographic data is determined, and the photographic data and the virtual object are synthesized.

  Next, processing in the continuity detection unit 103 will be described in detail. FIG. 8 is a flowchart showing processing in the continuity detection unit 103.

  In the continuity detection unit 103, a difference from the latest sensor value is measured (S201). Then, it is compared with a predetermined threshold value X, and if it is determined that the most recent sensor value is equal to or greater than the threshold value X, it is determined that the pattern is an imaging state B (S205). Here, the shooting state pattern B indicates an operation for greatly changing the shooting range.

  If it is determined that the sensor value is smaller than the threshold value X, the sensor value per unit time is further measured (S203). Then, a variation pattern of the sensor value per unit time is derived, and it is determined whether or not a predetermined variation pattern is taken (S204). Here, if it is determined that a predetermined variation pattern is taken, it is determined that the pattern C is a shooting state. That is, it is determined that the amplitude is due to camera shake or disturbance (S206). The predetermined variation pattern may be, for example, a pattern in which the difference in amplitude of sensor values is less than or equal to a predetermined value, and the sensor values regularly vary up and down. Good. If it is determined that the pattern is not a predetermined variation pattern, it is determined that the pattern is a shooting state pattern A, and it is determined that the user is slowly changing the shooting range (S207).

  Note that it is preferable to reduce the power consumption if the timing of composing the virtual object composition position is changed depending on the pattern.

  For example, when it is determined that the pattern A is the continuity pattern, the user is performing an operation of slowly changing the shooting range, so the synthesis position of the virtual object synthesized on the display unit 102 changes slowly. To do. Further, when the pattern C is determined, it is determined that the influence is due to camera shake or disturbance, and the change in the combined position of the virtual object combined with the display unit 102 is small. For this reason, the pattern A and the pattern C may have a longer time interval for updating the virtual object combining position on the display unit 102.

  That is, a predetermined reference value T1 is set, and if it is determined in S105 that the pattern A or C is the continuity pattern, the continuity detection unit 103 starts from the time updated last time before proceeding to S106. It is determined whether the time determined by the predetermined reference value T1 has elapsed. If the time has elapsed, the extraction process and the smoothing process in S106 are performed. If it has not elapsed, the extraction process and the smoothing process are not performed.

  On the other hand, when the pattern B is determined as the continuity pattern, the user performs an operation of changing the shooting range greatly, and the change in the combination position of the virtual object combined on the display unit 102 is large. For this reason, when it is determined as the pattern B, the time interval for updating the composite position of the virtual object on the display unit 102 may be shortened.

  For example, when a predetermined reference value T2 (<T1) is set and it is determined in S105 that the pattern B is the continuity pattern, the continuity detection unit 103 performs the time previously updated before the process proceeds to S106. It is determined whether the time determined by the predetermined reference value has elapsed, and if it has elapsed, the extraction process and the smoothing process in S106 are performed. If it has not elapsed, the extraction process and the smoothing process are not performed.

  In this way, it is possible to reduce power consumption by changing the time interval for updating the composite position of the virtual object. Note that the time for updating the composite position may be stored in the virtual object position determination unit 108, or may be stored in the continuity detection unit 103 as the time when the detection processing is performed.

  In this way, a shooting state pattern is determined, and a smoothing parameter corresponding to the shooting state pattern is extracted from the smoothing parameter DB 104 and processed by the smoothing processing unit 106.

  Therefore, when the mobile terminal 100 combines the virtual object with the shooting data and causes the display unit 102 to display the virtual object, the mobile terminal 100 may display the virtual object combining position according to the moving state of the shooting range of the mobile terminal 100. it can. That is, the composite position is smoothed according to the state transition speed of the shooting range, and the quick response to the composite processing and the reduction of the composite position error can be processed in a balanced manner according to the speed. Therefore, even if the shooting range is changed quickly, the compositing process is performed without much influence of the previous combining position, and if the shooting range is changed slowly, the compositing process is affected by the previous combining position. And is not affected by the error of the latest sensor value.

  Next, functions and effects of the mobile terminal 100 of the present embodiment will be described. In the mobile terminal 100 of the present embodiment, the continuity detection unit 103 detects a sensor value indicating the shooting range of the shooting unit 101 and also detects a shooting state, and the smoothing processing unit 106 changes to the detected shooting state. The detected sensor value is smoothed using the smoothing parameter determined on the basis thereof. Then, the virtual object position determination unit 108 calculates a combined position for the shooting data based on the smoothed sensor value, combines the image data with the calculated combined position in the shooting data, and the display unit 102 Shooting data obtained by combining image data can be displayed.

  As a result, depending on the shooting state, it is possible to eliminate the fact that the image data is shifted from the position to be originally synthesized, and it is possible to synthesize the image data at the synthesis position with little error. On the other hand, if the smoothing process is performed in all states in the photographing state, there is a problem that the quick response is lowered. That is, when the shooting range is moving greatly, even if smoothing processing is performed using past values, the movement of the shooting range cannot be followed and the position relative to the shooting range that has already passed is calculated. Resulting in. However, since the mobile terminal 100 according to the present embodiment can perform the smoothing process according to the shooting state, it is possible to calculate an appropriate composite position and improve the response, and minimize the error as much as possible. Can be reduced.

  Here, a description will be given of the position where the virtual object and the shooting data are combined when the embodiment is applied and when the embodiment is not applied.

  The case where the fluctuation | variation of the imaging | photography range of the portable terminal 100 is small is demonstrated. For example, it is a case where the mobile terminal 100 is moved slowly or is affected by hand shake. FIG. 9 is an explanatory diagram for comparing the synthesis positions.

  In the synthesis example shown in FIG. 9A, a synthesis example in the case where the smoothing process is increased is shown. For example, a predetermined number (for example, n) of sensor values are selected as the target of the sensor value of the smoothing process. This is an example. As shown in FIG. 9A, a virtual object M is synthesized and displayed on the display unit 102 at substantially the center of the photographing data S. In this example, it is assumed that the combined position at the appropriate position is a combination of the virtual object M at the center (cross mark C) of the shooting data. If the smoothing process is increased in this way, it will be affected by past sensor values, so even if an error is found in the most recent sensor value, the effect will be reduced, and as a result, the appropriate synthesis position will be reduced. Virtual objects are synthesized.

  The synthesis example shown in FIG. 9B is an example in which the synthesis position is bad. For example, the target sensor values for the smoothing process are targeted for a predetermined number (for example, m <n) of sensor values. This is an example. As shown in FIG. 9B, the display unit 102 displays the virtual object M combined with the shooting data S. The virtual object M is slightly left of the center of the shooting data (cross mark C). It is synthesized at the position shifted to. If the smoothing process is reduced in this way, it will be greatly affected by the most recent sensor value, and if the most recent sensor value contains a lot of errors, the combined position will also shift. Become.

  In the mobile terminal 100 of the present embodiment, the virtual object is not synthesized at the synthesis position in the shooting data as shown in FIG. 9B, but is virtual at the appropriate synthesis position shown in FIG. The object will be synthesized.

  Next, a case where the photographing range of the mobile terminal 100 varies greatly will be described. For example, this is a case where the mobile terminal 100 has been moved abruptly and its shooting range has changed greatly. FIG. 10 is an explanatory diagram for comparing the synthesis positions. In this FIG. 10, it is a figure which shows the imaging | photography data in the moment which stopped by rotating the imaging | photography range from the left to the right.

  The synthesis example shown in FIG. 10A is an example in which the synthesis position when the technique of the present embodiment is not applied is bad. For example, the target of the sensor value of the smoothing process is determined from the nearest. This is an example for a number (for example, n) of sensor values. In this example, as shown in FIG. 10A, a virtual object M is synthesized and displayed on the display unit 102 slightly on the right side from the center (cross mark C) of the shooting data S. In this example, if the smoothing process is enlarged because the shooting range has been changed abruptly, if the smoothing process is performed using past sensor values, the combined position will affect the past sensor values. As a result, the composite position of the virtual object is shifted.

  In contrast, the synthesis example shown in FIG. 10B is an example in which the synthesis position is good when the technique of the present embodiment is applied. For example, the sensor value target of the smoothing process is an example in which a predetermined number (n <m) of sensor values are the target. In this example, as shown in FIG. 10B, the virtual object M is displayed on the display unit 102 in a synthesized manner at substantially the center (cross mark C) of the shooting data S. This is because the smoothing process is deliberately reduced to limit the use of past sensor values, so that the combining process can be performed without delaying the combining position due to the change in the photographing range.

  In this manner, the mobile terminal 100 according to the present embodiment can perform an appropriate synthesis process between the shooting data and the virtual object by performing the smoothing process according to the shooting state.

  Furthermore, in the mobile terminal 100, the sensor value indicating the imaging range is preferably a sensor value detected by at least one of an acceleration sensor, a geomagnetic sensor, and a position sensor. As a result, the shooting range can be acquired easily and accurately, and the image data and shooting data can be combined easily and accurately.

  In portable terminal 100, it is preferable to detect a shooting state based on a sensor value detected by sensor unit 105 for determining a shooting range. For example, it is possible to determine whether the shooting state is a still state, a state in which the shooting range has greatly changed, or a state in which the shooting range has changed slowly depending on changes in sensor values. As a result, it is possible to determine the shooting state with a simple configuration by using the sensor unit 105 that is used for grasping the shooting range without providing a special means for determining the shooting state. .

  In the mobile terminal 100, it is preferable to detect the shooting state according to the change state of the shooting data captured by the shooting unit 101. For example, it is possible to determine whether the shooting state is a still state, a state in which the shooting range has changed significantly, or a state in which the shooting range has changed slowly, depending on the variation of the shooting data for each pixel. Accordingly, it is possible to determine the shooting state with a simple configuration by using the shooting unit for taking shooting data without providing a special unit for determining the shooting state.

  Further, in the mobile terminal 100, the sensor unit 105 sequentially detects sensor values at predetermined intervals and holds them in time series, and among the sensor values held in time series, the smoothing processing unit 106 takes an image. It is preferable to perform a weighting process according to the photographing state, with a predetermined number of sensor values extracted from the latest depending on the state as a smoothing target. Thereby, it is possible to be affected by a predetermined number of sensor values in the past according to the shooting state, and the influence of the latest error can be reduced.

  In the mobile terminal 100, the continuity detecting unit 103 detects the shooting state based on whether or not the shooting state based on the terminal operation by the user has fluctuated by a predetermined value or more in a predetermined time unit. It is preferable to detect the presence or absence of the amplitude in predetermined time units. Thereby, it is possible to appropriately determine the shooting state. For example, when the shooting state changes by a predetermined value or more, it can be determined that the shooting range has fluctuated greatly. Moreover, when it does not change more than a predetermined value, it can be judged that it changed slowly by operation by the user. Further, when the fluctuation pattern is a predetermined fluctuation pattern (regular fluctuation pattern having a predetermined amplitude difference), it can be determined that the fluctuation pattern has fluctuated due to camera shake or disturbance. And an appropriate smoothing process can be performed by using the smoothing parameter according to these three states.

DESCRIPTION OF SYMBOLS 100 ... Portable terminal, 101 ... Imaging | photography part, 102 ... Display part, 102 ... Display part, 103 ... Continuity detection part, 104 ... Smoothing parameter DB part, 105 ... Sensor part, 106 ... Smoothing process part, 107 ... Virtual Object storage unit 108... Virtual object position determination unit.

Claims (7)

Photographing means for capturing photographing data;
Sensor means for detecting a sensor value indicating a photographing range of the photographing means;
Continuity detecting means for detecting the photographing state of the photographing means;
Smoothing means for smoothing the sensor value detected by the sensor means using a smoothing parameter determined based on the photographing state detected by the continuity detecting means;
Calculation means for calculating a composite position for the photographing data captured by the photographing means based on the sensor value smoothed by the smoothing means;
Combining means for combining the image data at the combining position in the photographing data calculated by the calculating means;
Display means for displaying photographing data obtained by combining image data by the combining means;
A mobile terminal comprising:   The mobile terminal according to claim 1, wherein the sensor unit uses at least one of a gyro sensor, an acceleration sensor, a geomagnetic sensor, and a position sensor as a sensor value indicating an imaging range.   The portable terminal according to claim 1, wherein the continuity detecting unit detects a photographing state based on a sensor value detected by the sensor unit.   The portable terminal according to claim 1, wherein the continuity detecting unit detects a shooting state in accordance with a change state of shooting data captured by the shooting unit. The sensor means sequentially detects sensor values at predetermined intervals and holds them in time series,
The smoothing means performs a weighting process according to the shooting state, with a predetermined number of sensor values extracted from the latest according to the shooting state among the sensor values held in time series as a smoothing target. The portable terminal according to any one of claims 1 to 4, wherein the portable terminal is characterized in that: The continuity detecting means detects the shooting state based on whether or not the shooting state based on the terminal operation by the user has fluctuated by a predetermined value or more in a predetermined time unit,
6. The mobile phone according to claim 1, wherein the photographing state is determined by detecting whether or not the fluctuation pattern is a predetermined fluctuation pattern in a predetermined time unit. Terminal. Shooting steps to capture shooting data,
A sensor step for detecting a sensor value indicating a photographing range of the photographing step;
A continuity detection step for detecting a shooting state of the shooting step;
A smoothing step of smoothing the sensor value detected by the sensor step using a smoothing parameter determined based on the imaging state detected by the continuity detection step;
Based on the sensor value smoothed by the smoothing step, a calculation step for calculating a combined position for the shooting data captured by the shooting step;
A synthesis step of synthesizing image data at a synthesis position in the photographing data calculated by the calculation step;
A display step for displaying photographing data obtained by combining the image data in the combining step;
An image processing method comprising:

Images