JP5350427B2 - Image processing apparatus, image processing apparatus control method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To support operation by a user to which augmented reality is provided. <P>SOLUTION: Display control means (74) aligns and combines a real space image (62) and a virtual space image (64) to display the combined image on a screen (60). Movement control means (78) moves an operation target (52) on the basis of a basic direction corresponding to direction indication operation performed by a user. Determination means (80) determines whether an imaging range indicated by the real space image (62) is changed. When the imaging range indicated by the real space image (62) is determined to be changed and the direction indication operation is performed in a predetermined period, the movement control means (78) changes the association between the direction indication operation and the basic direction on the basis of the content of the change in the imaging range indicated by the real space image (62) and moves the operation target (52) on the basis of the changed association and the direction indication operation performed in the predetermined period. <P>COPYRIGHT: (C)2013,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus, a control method for the image processing apparatus, and a program.

  Conventionally, augmented reality (AR) is displayed to the user by superimposing an image obtained by photographing a real space with a camera and an image obtained by viewing the virtual space from a given viewpoint on the screen. The technology to provide is known.

  For example, Patent Literature 1 and Patent Literature 2 describe a technique for determining a display position or the like of an object placed in a virtual space by photographing a marker placed in a real space with a camera. In such a technique, the user can feel that the user's operation target (for example, a car) arranged in the virtual space moves in the real space.

JP 2006-72903 A JP 2010-170316 A

  In the augmented reality technology as described above, when the user moves the camera and changes the shooting range of the real space, the visual field of the virtual space also changes with this change. The user operates the operation target while changing the shooting range in the real space. It is difficult for the user to perform these two operations at the same time, and there is a possibility that the operation target advances in a direction not intended by the user.

  The present invention has been made in view of the above problems, and an object thereof is to provide an image processing apparatus capable of assisting a user who is provided with augmented reality, a control method for the image processing apparatus, and a program. It is to provide.

  In order to solve the above problems, an image processing apparatus according to the present invention includes a photographing unit whose photographing range is changed by a user operation, a real space image generated by photographing the real space by the photographing unit, Display control means for aligning and synthesizing a virtual space image showing a view of the virtual space where the user's operation target is arranged from a viewpoint with a field of view corresponding to the shooting range and displaying the virtual space image on the display screen And an operation means for the user to perform a direction instruction operation of the operation object, a movement control means for moving the operation object based on a basic direction corresponding to the direction instruction operation performed by the user, Determining means for determining whether or not the shooting range indicated by the real space image has changed, wherein the movement control means changes the shooting range indicated by the real space image. When the direction instruction operation is performed within a predetermined period, the association between the direction instruction operation and the basic direction is changed based on the change content of the shooting range indicated by the real space image, The operation target is moved based on the association after the change and the direction instruction operation performed within the predetermined period.

  According to the control method of the image processing apparatus according to the present invention, a real space image generated by photographing a real space by a photographing unit whose photographing range is changed by a user operation, and a field of view corresponding to the photographing range are set. A display control step of aligning and synthesizing the virtual space image showing a view of the virtual space in which the operation target of the user is viewed from a viewpoint and displaying the virtual space on the display screen; and a direction of the operation target by the user The movement control step for moving the operation target based on the basic direction corresponding to the direction instruction operation performed by the user from the operation means for performing the instruction operation, and the photographing range indicated by the real space image has changed. A determination step of determining whether or not the shooting range indicated by the real space image has changed in the movement control step. When the direction instruction operation is performed within a predetermined period, the association between the direction instruction operation and the basic direction is changed based on the change content of the shooting range indicated by the real space image, and the change The operation target is moved based on a later association and a direction instruction operation performed within the predetermined period.

  The program according to the present invention includes a photographing unit whose photographing range is changed by a user's operation, a real space image generated by photographing the real space by the photographing unit, and a viewpoint in which a field of view corresponding to the photographing range is set. Display control means for aligning and synthesizing the virtual space image showing a view of the virtual space in which the user's operation target is arranged and displaying it on the display screen; Operation means for performing the movement, movement control means for moving the operation target based on a basic direction corresponding to the direction instruction operation performed by the user, whether or not the shooting range indicated by the real space image has changed Determination means for determining, the movement control means is determined that the shooting range indicated by the real space image has changed, and the direction within a predetermined period When the display operation is performed, the association between the direction instruction operation and the basic direction is changed based on the change content of the shooting range indicated by the real space image, and the association after the change is performed within the predetermined period. The computer is caused to function as an image processing apparatus that moves the operation target based on the direction instruction operation performed in step (b).

  An information storage medium according to the present invention is a computer-readable information storage medium storing the above program.

  According to the present invention, it is possible to assist a user who is provided with augmented reality.

  In the aspect of the invention, the movement control unit may be configured such that a change in a positional relationship between the shooting range indicated by the real space image in the real space and the position of the operation target in the virtual space is a predetermined condition. If the condition is satisfied, the association between the direction instruction operation and the basic direction is changed.

  In the aspect of the invention, the movement control unit may change a correspondence between the direction instruction operation and the basic direction indicated by a coordinate system different from the viewpoint coordinate system of the viewpoint in the virtual space. It is characterized by that.

  In one aspect of the present invention, the movement control means includes the movement direction of the operation target determined based on the basic direction corresponding to the direction instruction operation performed within the predetermined period, and the real space image. Means for determining a deviation from the change direction of the photographing range shown, and when the deviation is a reference range, the operation target is moved based on a basic direction corresponding to the direction indicating operation performed by the user. And when the deviation is not a reference range, the association between the direction instruction operation and the basic direction is changed based on the change content of the shooting range indicated by the real space image, and the association after the change and the And a means for moving the operation object based on a direction instruction operation performed within a predetermined period.

It is a figure which shows the hardware constitutions of the portable terminal which concerns on this embodiment. It is a figure which shows a mode that a user plays the game using an augmented reality. It is a figure which shows an example of virtual space. It is a figure which shows an example of the game screen displayed on a display part. It is a figure for demonstrating the basic direction matched with direction instruction | indication operation. It is a functional block diagram which shows the function implement | achieved by a portable terminal. It is a figure which shows the example of data storage of marker data. It is a figure for demonstrating the method of the alignment by a display control part. It is a figure for demonstrating the process by which the change direction of an imaging | photography range is acquired. It is a figure for demonstrating matching with the changed direction instruction | indication operation and a basic direction. It is a figure which shows a mode that the position of a tank is controlled when matching with direction instruction | indication operation and a basic direction is changed. It is a flowchart which mainly shows the process relevant to this invention among the processes which a portable terminal performs. It is a flowchart which mainly shows the process relevant to this invention among the processes which a portable terminal performs. It is a figure for demonstrating the display position of the tank in a game screen. It is a figure which shows the conditions for the matching of direction instruction | indication operation and a basic direction being changed. FIG. 10 is a diagram for explaining processing of the third embodiment. It is a figure for demonstrating the markerless system augmented reality.

[1. Embodiment]
Hereinafter, embodiments according to the present invention will be described in detail with reference to the drawings. The image processing according to the present invention is realized by using, for example, a home game machine (stationary game machine), a portable game machine, an arcade game machine, a portable terminal such as a cellular phone, or a personal computer. Here, a case where the image processing apparatus according to the embodiment of the present invention is realized using a mobile terminal will be described.

[1-1. Hardware configuration of mobile terminal]
FIG. 1 is a diagram illustrating a hardware configuration of the mobile terminal 10 according to the present embodiment. As shown in FIG. 1, the mobile terminal 10 includes a control unit 12, a storage unit 14, a main storage unit 16, a display unit 18, an operation key unit 20, a touch panel 22, a CCD camera 24, a sensor unit 26, a communication unit 28, and the like. For example, each part is connected via the bus 30. Note that the hardware configuration of the mobile terminal 10 is not limited to the example of FIG. 1, and the mobile terminal 10 may include various hardware (for example, an SD card slot) included in a known mobile terminal.

  The control unit 12 is based on an operating system stored in the storage unit 14, a program stored in various externally connected storage media (for example, an SD card) and various data acquired via the communication unit 28. Each part of the portable terminal 10 is controlled.

  The storage unit 14 includes a nonvolatile storage medium such as a flash memory. The storage unit 14 stores an operating system and the like. The main storage unit 16 includes, for example, a RAM. Programs and the like stored in the storage unit 14 are written into the main storage unit 16 as necessary. The main storage unit 16 is also used as a working memory for the control unit 12.

  The display unit 18 is one or a plurality of known liquid crystal display panels. For example, an image drawn in a VRAM (not shown) is displayed on the display unit 18. The operation key unit 20 includes a cross key and various buttons. The touch panel 22 is disposed so as to be superimposed on the display unit 18 and detects a contact position from the outside.

  The CCD camera 24 captures a real space and generates an image. In the present embodiment, a case will be described in which the CCD camera 24 is included in the housing of the mobile terminal 10 and the shooting range of the CCD camera 24 changes when the user changes the position and orientation of the mobile terminal 10. . When the portable terminal 10 and the CCD camera 24 are connected by wire, the shooting range of the CCD camera 24 may be changed based on a user operation from the operation key unit 20 or the touch panel 22.

  The sensor unit 26 includes a GPS sensor, a gyro sensor, an electronic compass (geomagnetic sensor), an acceleration sensor, and the like. The communication unit 28 performs data transmission / reception with an external device via a network.

[1-2. Augmented reality provided to users]
The mobile terminal 10 provides augmented reality to the user. In the present embodiment, an image of a real space (hereinafter referred to as a real space image) generated when the CCD camera 24 images the real space, and an image (see below) showing the virtual space viewed from a given viewpoint. Hereinafter, the virtual space image) is superimposed and displayed on the display unit 18, thereby providing the user with augmented reality. Moreover, in this embodiment, the case where a user plays the game using an augmented reality is demonstrated.

  FIG. 2 is a diagram illustrating how a user plays a game using augmented reality. As shown in FIG. 2, for example, the CCD camera 24 photographs a marker 40 (detection target) on which a predetermined pattern 44 is drawn. The marker 40 is paper, film, or the like, and is disposed at an arbitrary position within the field of view of the CCD camera 24, for example. For example, in the case of FIG. 2, the marker 40 is disposed on a desk in the real space.

  In the present embodiment, the marker 40 has a frame 42 (for example, a square frame) of a predetermined size, and a predetermined pattern 44 (for example, an asymmetric pattern 44) is drawn in the frame 42. Explain the case. Here, as shown in FIG. 2, an “L-shaped” pattern 44 is drawn on the marker 40.

  Data indicating the pattern 44 drawn on the marker 40 is stored in the storage unit 14 in advance. The portable terminal 10 compares the pattern 44 included in the real space image with the pattern 44 stored in the storage unit 14 based on a known image comparison algorithm, thereby identifying the type of the marker 40 arranged in the real space. The position, size, and orientation of the pattern 44 are specified.

  This specified information (for example, the type, position, size, and direction of the pattern 44 of the marker 40) indicates the size and direction of the object placed in the virtual space on the screen of the display unit 18. Used to determine whether to display with. In other words, the information indicating the type, position, size, and orientation of the pattern 44 of the marker 40 is used to perform alignment for the real space image and the virtual space image.

  FIG. 3 is a diagram illustrating an example of a virtual space. In this embodiment, the case where the game space 50 (virtual three-dimensional space) imitating a battlefield is produced | generated as an example of virtual space is demonstrated. When the game is started, a game space 50 is generated in the main storage unit 16. In the game space 50, three axes (virtual space coordinate system described later) orthogonal to each other are set. The position of each object arranged in the game space 50 and the virtual camera 58 described later is determined based on the three-dimensional coordinates set in the virtual space.

  As shown in FIG. 3, in the game space 50, for example, a tank 52 that is an object indicating a tank provided for a user's operation, a fort 54 that is an object indicating an attack target of the tank 52, and an enemy An enemy 56 that is an object representing a character is arranged. The tank 52 moves in the game space 50 based on a user operation.

  Further, for example, when the user performs a predetermined operation (for example, tapping the touch panel 22), the tank 52 fires a bullet. When a bullet is fired from the tank 52, an object (not shown) indicating a bullet is generated, and when the object collides with the enemy 56, the fort 54 and the enemy 56 disappear. In this way, the user aims to destroy the fort 54 and the enemy 56 by operating the tank 52.

  A virtual camera 58 (viewpoint) is set in the game space 50. A virtual space image showing how the game space 50 is viewed from the virtual camera 58 is generated at a predetermined time interval (for example, the same interval determined based on the frame rate of the CCD camera 24). For example, a virtual space image indicating an object in the field of view (view frustum) of the virtual camera 58 among the objects arranged in the game space 50 is generated. The user plays the game while looking at the game screen in which the virtual space image and the real space image are synthesized. When the tank 52 and the enemy 56 move in the game space 50, display control is performed on the game screen 60 so that the tank 52 and the enemy 56 move on the desk in the real space.

  FIG. 4 is a diagram illustrating an example of a game screen displayed on the display unit 18. As shown in FIG. 4, a real space image 62 and a virtual space image 64 are synthesized and displayed on the game screen 60. For example, when the marker 40 is arranged on a desk in the real space, the game of each object arranged in the game space 50 is displayed on the game screen 60 so that the battlefield is developed on the desk in the real space. A display position on the screen 60 is determined. In other words, the alignment for the real space image 62 and the virtual space image 64 is performed.

  For example, when the shooting range (region) displayed in the real space image 62 moves, the range (region) of the virtual space displayed in the virtual space image 64 also moves. For example, when the user changes the position, orientation, or orientation of the mobile terminal 10 to change the shooting range of the CCD camera 24, the shooting range of the virtual camera 58 also changes with this change.

  In the present embodiment, the position and line-of-sight direction of the CCD camera 24 correspond to the position and line-of-sight direction of the virtual camera 58. That is, when the position of the CCD camera 24 changes, the position of the virtual camera 58 also changes in the same way as this change in position. When the line-of-sight direction of the CCD camera 24 changes, the line-of-sight direction of the virtual camera 58 also changes in the same direction as the changed direction.

  For example, in the case of the game screen 60 shown in FIG. 4, in order for the user to check whether the enemy 56 is behind the tank 52, for example, the user rotates the direction of the mobile terminal 10 to the left, The line-of-sight direction of the CCD camera 24 is rotated to the left. When the user rotates the viewing direction of the CCD camera 24 to the left, the viewing direction of the virtual camera 58 also rotates to the left.

  Further, as shown in FIG. 4, the game screen 60 includes a forward button 66a, a backward button 66b, a left rotation button 66c, and a right rotation button 66d (which are simply put together) for the user to perform a direction instruction operation of the tank 52. Also referred to as button 66). Whether or not each button 66 has been pressed is determined based on a detection signal from the touch panel 22.

  Each button 66 is associated with a basic direction. The basic direction is a direction previously associated with the direction instruction operation. The tank 52 moves based on the basic direction corresponding to the direction instruction operation. That is, the moving direction of the tank 52 is determined based on the basic direction associated with the button 66.

  In the present embodiment, a case where the basic direction is represented by a three-dimensional vector will be described. However, the basic direction may be represented by a two-dimensional vector. Here, a case where the basic direction is defined not by a coordinate system based on the virtual camera 58 but by a coordinate system based on the tank 52 (local coordinate system; also referred to as a character coordinate system) will be described.

FIG. 5 is a diagram for explaining the basic direction associated with the direction instruction operation. As shown in FIG. 5, the direction represented by the local coordinate system (X _LW -Y _LW -Z _LW coordinate system) set based on the tank 52 is set as the reference direction. The origin O _LW of the local coordinate system is determined based on the position of the tank 52.

  When the direction instruction operation is performed by the user, the basic direction represented in the local coordinate system is converted into a vector in the virtual space coordinate system (world coordinate system) by a known coordinate conversion process. The moving direction is specified.

For example, the front direction of the tank 52 is associated with the forward button 66a as the reference direction Va (for example, the positive direction of the Z _LW axis). For example, when the forward button 66 a is pressed, the tank 52 moves in the front direction viewed from the tank 52. For example, the backward direction of the tank 52 is associated with the reverse button 66b as the reference direction Vb (for example, the negative direction of the Z _LW axis). For example, when the retreat button 66 b is pressed, the tank 52 moves in the back direction viewed from the tank 52.

For example, the left rotation button 66c is associated with the left direction as viewed from the tank 52 as the reference direction Vc (for example, the negative direction of the X _LW axis). For example, when the left rotation button 66 c is pressed, the tank 52 rotates counterclockwise as viewed from the tank 52. For example, the right direction as viewed from the tank 52 is associated with the right rotation button 66d as the reference direction Vd (for example, the positive direction of the X _LW axis). For example, when the right rotation button 66 d is pressed, the tank 52 rotates clockwise as viewed from the tank 52.

  For example, when the user wants to move the tank 52 back and forth, the user presses the forward button 66a or the backward button 66b. For example, when the user wants to move the tank 52 to the left or right, the user presses the left rotation button 66c or the right rotation button 66d to rotate the tank 52, and then presses the forward button 66a or the backward button 66b. Further, for example, when the user wants to move the tank 52 while turning it obliquely, the user moves the tank by pressing the forward button 66a or the backward button 66b and the left rotation button 66c or the right rotation button 66d.

  As described above, when the basic direction is defined in the local coordinate system based on the tank 52, if the positional relationship between the tank 52 and the virtual camera 58 is different, even if the user performs the same direction instruction operation, the game screen 60 The movement direction of the tank 52 is different. Therefore, the user must perform a direction instruction operation while considering the direction of the tank 52, and there is a possibility that the direction in which the tank 52 is moved is wrong. For example, the tank 52 may go out of the field of view of the virtual camera 58, and the tank 52 may disappear from the game screen 60.

  The portable terminal 10 according to the present embodiment changes the association between the direction instruction operation and the basic direction when the shooting range of the CCD camera 24 is changed to expand the real space image 62 and the button 66 is pressed. Thus, the tank 52 is moved in the direction in which the real space image 62 is expanded. That is, the user's operation assistance is performed by regarding the direction in which the real space image 62 spreads as the direction in which the user wants to move the tank 52.

  Here, as a method for assisting the user's operation, for example, the tank 52 may be moved in a direction in which the mobile terminal 10 is tilted by using a gyro sensor or an acceleration sensor of the sensor unit 26. However, since the virtual camera 58 does not follow the movement of the tank 52 due to the nature of the game using augmented reality, the tank 52 may still be out of the display area of the game screen 60 in this method. . That is, the tank 52 is out of the field of view of the virtual camera 58. In this case, since the user cannot visually recognize the tank 52, it becomes difficult to play the game.

  Therefore, the mobile terminal 10 according to the present embodiment performs operation assistance for the user while preventing the tank 52 from being detached from the game screen 60 by moving the tank 52 in the direction in which the real space image 62 spreads. Is called. Hereinafter, the technique will be described in detail.

[1-3. Functions realized in game device]
FIG. 6 is a functional block diagram showing functions realized by the mobile terminal 10. As illustrated in FIG. 6, the mobile terminal 10 includes an augmented reality data storage unit 70, an imaging unit 72, a display control unit 74, an operation unit 76, a movement control unit 78, and a determination unit 80. Each of these functions is realized, for example, when the control unit 12 executes a program stored in the storage unit 14.

[1-3-1. Augmented reality data storage]
The augmented reality data storage unit 70 is realized mainly by the storage unit 14 and the main storage unit 16. The augmented reality data storage unit 70 stores data necessary to provide augmented reality to the user. For example, the augmented reality data storage unit 70 stores marker data related to the marker 40.

  FIG. 7 is a diagram illustrating a data storage example of marker data. As shown in FIG. 7, for example, image data indicating the pattern 44 of the marker 40 and data relating to augmented reality associated with the marker 40 are associated with the marker data. The image data stored in the marker data is used as a comparison target with the marker 40 arranged in the real space. That is, the marker data is data that defines the marker 40 necessary for the user to use augmented reality.

  The data related to augmented reality is information for identifying the type of augmented reality to be provided to the user when the marker 40 stored in the marker data is detected. For example, when the marker 40 having the “L-shaped” pattern 44 is detected as in this actual form, augmented reality using a game in which the user operates the tank 52 is provided. Further, for example, when a marker 40 of another pattern 44 is detected, augmented reality is provided by displaying a stereoscopic image of a character dancing while singing a song around the marker 40.

  The augmented reality data storage unit 70 stores virtual space data indicating the current state of the game space 50. In the virtual space data, for example, information on the position, posture, orientation, and moving direction of each object arranged in the game space 50 and information on the visual field of the virtual camera 58 (for example, position and line-of-sight direction) are stored. The

  The contents of the positions of the objects stored in the virtual space data are updated based on the user's operation or a predetermined algorithm. In the present embodiment, the position and line-of-sight direction of the virtual camera 58 stored in the virtual space data are based on the display contents (for example, display position, size, and orientation) of the marker 40 in the real space image 62. It is determined.

  The control unit 12 functions as a means for acquiring various data stored in the augmented reality data storage unit 70. The control unit 12 functions as a means for changing (updating) various data stored in the augmented reality data storage unit 70. Further, the data stored in the augmented reality data storage unit 70 is not limited to the above example. For example, data necessary for the mobile terminal 10 to execute a game (data indicating a user's score or a tank 52) For example, data indicating the remaining physical strength of the child may be stored.

[1-3-2. Shooting part]
The photographing unit 72 is realized mainly by the control unit 12 and the CCD camera 24. The shooting range of the shooting unit 72 is changed by a user operation. For example, the shooting range changes when the user changes the position, orientation, and orientation of the shooting unit 72. The imaging range is determined based on, for example, the position of the imaging unit 72 and the line-of-sight direction.

  The imaging unit 72 images, for example, a real space where a detection target (for example, the marker 40) is arranged. Image data indicating a real space image generated by photographing the real space by the photographing unit 72 is temporarily stored in, for example, the augmented reality data storage unit 70.

[1-3-3. Display control unit]
The display control unit 74 is realized mainly by the control unit 12. The display control unit 74 operates the user's operation target (for example, the virtual camera 58) from the real space image 62 generated by the photographing unit 72 photographing the real space and the viewpoint (for example, the virtual camera 58) in which the visual field corresponding to the photographing range is set. For example, the virtual space image 64 showing a state in which the tank 52) is placed and the virtual space image 64 showing a state of viewing the virtual space (for example, the game space 50) is aligned and combined and displayed on the display screen (eg, the game screen 60). .

  The alignment relating to the real space image 62 and the virtual space image 64 is a process for determining a display position when an object placed in the virtual space is displayed on the game screen 60. In other words, the alignment for the real space image 62 and the virtual space image 64 is an alignment process between the real space and the virtual space that is performed to provide the user with augmented reality.

  The field of view corresponding to the shooting range indicates that the field of view of the virtual camera 58 is reduced based on the shooting range of the shooting unit 72. For example, the display control unit 74 controls the virtual camera 58 so that the relationship between the coordinate system of the real space and the coordinate system of the photographing unit 72 corresponds to the relationship between the coordinate system of the virtual space and the coordinate system of the virtual camera 58. By performing the control, alignment with respect to the real space image 62 and the virtual space image 64 is performed.

  That is, the display control unit 74 determines the positional relationship between the reference position in the real space (for example, the position of the marker 40) and the shooting range (field of view) of the shooting unit 72, and the reference position in the virtual space (for example, in the virtual space coordinate system). The virtual viewpoint is controlled so that the positional relationship between the position of the origin) and the visual field of the virtual camera 58 corresponds to perform alignment with respect to the real space image 62 and the virtual space image 64. When the user changes the shooting range of the shooting unit 72, the display control unit 74 changes the field of view of the virtual camera 58 based on this change.

  In the present embodiment, a case will be described in which the display control unit 74 performs alignment between the real space image 62 and the virtual space image 64 based on the position, size, and orientation of the marker 40 in the real space image 62. .

FIG. 8 is a diagram for explaining a positioning method by the display control unit 74. As shown in FIG. 8, for example, in the real space, the real space coordinate system (X _RW -Y _RW -Z _RW coordinate system) is set based on the position, size, and orientation of the marker 40, and the photographing unit 72. The actual viewpoint coordinate system (X _RC -Y _RC -Z _RC coordinate system) is set based on the position, orientation, and orientation (that is, the position, orientation, and orientation of the CCD camera 24).

  The real space coordinate system is a coordinate system for representing the position of an object placed in the real space, and is a coordinate system based on the marker 40. On the other hand, the real viewpoint coordinate system is a coordinate system for representing the relationship between the viewpoint of the photographing unit 72 and an object placed in the real space, and is a coordinate system based on the photographing unit 72.

For example, the origin O _RW of the real space coordinate system is set to the center point of the marker 40. The X _RW axis direction corresponds to the short direction of the “L-shaped” pattern 44, and the Y _RW axis direction corresponds to the long direction of the “L-shaped” pattern 44. The Z _RW axis direction is the outer product direction of the X _RW axis direction and the Y _RW axis direction.

Further, for example, the origin O _RC reality viewpoint coordinate system is set to the position of the viewpoint of the imaging unit 72. For example, the _XRC axis direction corresponds to the longitudinal direction of the real space image 62 (that is, the horizontal direction when viewed from the photographing unit 72), and the _YRC axis direction corresponds to the short direction of the real space image 62 (that is, the photographing unit). Corresponds to the vertical direction as viewed from 72). Further, for example, the Z _RC axis direction is set as the line-of-sight direction of the imaging unit 72.

The real space coordinate system (X _RW -Y _RW -Z _RW coordinate system) can be converted into a real viewpoint coordinate system (X _RC -Y _RC -Z _RC coordinate system) by rotating and translating. The display control unit 74 calculates a coordinate conversion example in the case of performing conversion from the real space coordinate system to the real viewpoint coordinate system based on the display contents (position, size, and orientation) of the marker 40 in the real space image 62. To do.

  Specifically, first, the display control unit 74 extracts the frame 42 of the marker 40 of the real space image 62 based on a known contour extraction process. When the frame 42 of the marker 40 is extracted, the positions of the four corners of the frame 42 in the real space image 62 are specified, whereby the positions of the markers 40 appearing in the real space image 62 are specified.

  Then, the display control unit 74 performs a pattern matching process for comparing the pattern 44 drawn in the extracted frame 42 with the image data of the pattern 44 stored in the marker data. , The size of the marker 40 appearing in the real space image 62, and the orientation of the pattern 44 are specified.

  Then, the position of the marker 40 in the real space image 62, the deviation between the size of the marker 40 in the real space image 62 and the size of the marker 40 stored in the marker data, and the direction of the pattern 44 of the marker 40 in the real space image 62 And a rotation component and a translation component in the conversion from the real space coordinate system to the real viewpoint coordinate system are calculated based on the deviation of the direction of the pattern 44 of the marker 40 stored in the marker data. A coordinate transformation matrix is obtained.

Based on this coordinate transformation matrix, the virtual viewpoint coordinate system (X _VC -Y _VC -Z _VC coordinate system) is set by transforming the virtual space coordinate system (X _VW -Y _VW -Z _VW coordinate system). . The virtual space coordinate system is a coordinate system for representing the position of each object arranged in the game space 50. The virtual viewpoint coordinate system is a coordinate system for representing the positional relationship between the virtual camera 58 and each object. For example, the position of the virtual camera 58 is the origin O _VC of the virtual viewpoint coordinate system, and the line-of-sight direction of the virtual camera 58 is the Z _vc axis direction of the virtual viewpoint coordinate system.

When the virtual space coordinate system is rotated and translated as indicated by the coordinate transformation matrix, the virtual viewpoint coordinate system is determined. That is, the positional relationship between the position of the viewpoint of the photographing unit 72 (for example, the origin O _RC ) and the position of the marker 40 (for example, the origin O _RW ), the position of the virtual camera 58 (for example, the origin O _VC ), and the virtual space coordinates. The position and line-of-sight direction of the virtual camera 58 are determined so that the positional relationship with the system origin O _vw is the same.

  As described above, the virtual space image 64 showing the state of viewing the game space 50 from the virtual camera 58 whose position and line-of-sight direction are determined and the real space image 62 are superimposed, so that the tank on the game screen 60 52 display positions are determined.

  Note that the method of determining the display position of the object in the virtual space in augmented reality using the marker 40 is not limited to the above example, and various known methods can be applied. In addition, for example, the display position of the object in the virtual space may be determined based on the marker 40 (for example, the marker 40 attached to a wall or a window) arranged in the vertical direction of the real space. The display position of the object in the virtual space may be determined by detecting a plurality of markers 40 simultaneously.

[1-3-4. Operation unit]
The operation unit 76 is mainly realized by the control unit 12 and the touch panel 22. The operation unit 76 is for a user to perform a direction instruction operation on an operation target (for example, a tank 52). For example, the user's direction instruction operation is detected based on the detection signal from the operation unit 76.

  For example, the operation unit 76 detects a direction instruction operation by the user based on a signal detected from the touch panel 22. In the present embodiment, a case will be described in which the user performs a plurality of direction instruction operations by pressing any of the plurality of buttons 66. Each of the plurality of direction instruction operations is associated with a basic direction.

  For example, which direction instruction operation corresponds to which basic direction is predetermined by a game creator or the like. The basic direction in the present embodiment is a direction for specifying the movement direction of the operation target, and is represented by a coordinate system (for example, a local coordinate system) different from the virtual viewpoint coordinate system. That is, when the position of the virtual camera 58 and the line-of-sight direction are different, the basic direction is defined in a coordinate system in which the direction of the basic direction seen from the virtual camera 58 is different.

  In the present embodiment, a case where a direction instruction operation is performed by an operation from the touch panel 22 will be described. However, for example, a direction instruction operation is performed based on an input signal from the operation key unit 20. May be. In this case, the operation unit 76 is realized by the operation key unit 20. For example, a plurality of buttons of the operation key unit 20 are associated with a plurality of basic directions.

[1-3-5. Movement control unit]
The movement control unit 78 is realized mainly by the control unit 12. The movement control unit 78 moves the operation target (for example, the tank 52) based on the basic direction corresponding to the direction instruction operation performed by the user. For example, the movement control unit 78 moves the tank 52 based on the basic direction corresponding to the direction instruction operation. In the present embodiment, a case where the tank 52 moves on a predetermined plane of the game space 50 (for example, on the X _VW -Y _VW plane) will be described.

  For example, as in the present embodiment, when the basic direction is defined in a local coordinate system based on the tank 52, a coordinate transformation matrix for transforming the local coordinate system and the virtual space coordinate system is included in the virtual space data. It is appropriately calculated based on the stored position, orientation, and posture of the tank 52. The basic direction is converted from the local coordinate system to the virtual space coordinate system based on this coordinate conversion matrix, whereby the moving direction of the tank 52 in the virtual space coordinate system is specified. Note that various known coordinate conversion processes can be applied to the method of converting between the local coordinate system and the virtual space coordinate system.

[1-3-6. Judgment unit]
The determination unit 80 is realized mainly by the control unit 12. The determination unit 80 determines whether or not the shooting range indicated by the real space image 62 has changed. Whether or not the shooting range indicated by the physical space image 62 has changed is determined based on, for example, time-series changes in the physical space image 62. For example, the real space image 62 is stored in the augmented reality data storage unit 70 in time series, and one real space image 62, a real space image 62 taken before the real space image 62, Are compared (pattern matching) to determine whether or not the shooting range indicated by the real space image 62 has changed.

  In the present embodiment, for example, in a predetermined period (for example, 1 second), an area in which the real space image 62 spreads (that is, a real space area newly appearing in the real space image 62) has a predetermined area (predetermined pixels). A case will be described in which it is determined whether or not the shooting range indicated by the real space image 62 has been changed by determining whether or not it is greater than or equal to (number). That is, it is determined whether or not the area of the real space newly appearing in the real space image 62 is greater than or equal to the predetermined area during the predetermined period.

  The method for determining whether or not the shooting range indicated by the real space image 62 has changed is not limited to the above example, and the shooting range indicated by the real space image 62 has changed based on various image change detection methods. It may be determined whether or not. For example, by tracking an object included in the physical space image 62, it may be determined whether or not the shooting range indicated by the physical space image 62 has changed. That is, for example, based on whether or not the display position of the marker 40 in the real space image 62 has changed by a predetermined distance or more, it may be determined whether or not the shooting range indicated by the real space image 62 has changed.

  The mobile terminal 10 is configured to change the association between the direction instruction operation and the basic direction described with reference to FIG. 5 based on the determination result of the determination unit 80 performed as described above.

  Specifically, the movement control unit 78 determines that the shooting range indicated by the real space image 62 has changed, and when a direction instruction operation is performed within a predetermined period, the change contents of the shooting range indicated by the real space image 62 Based on this, the association between the direction instruction operation and the basic direction is changed, and the operation target (for example, the tank 52) is moved based on the association after the change and the direction instruction operation performed within a predetermined period.

  The predetermined period is a period from before the present predetermined interval to the present time, for example, a period from when it is determined that the imaging range indicated by the real space image 62 has changed to the present time.

  The change content of the shooting range indicated by the real space image 62 is a change in the shooting range of the shooting unit 72 indicated by the real space image 62, for example, a change direction of the shooting range or a change amount of the shooting range. Here, a case will be described in which the association between the direction instruction operation and the basic direction is changed based on the change direction of the shooting range.

  The change direction of the shooting range is, for example, a direction in which the shooting range that appears as the real space image 62 (that is, the area displayed on the game screen 60) is expanded. In other words, the change direction of the shooting range is the direction in which the shooting range of the CCD camera 24 has moved, and is the same as the direction in which the position or line-of-sight direction of the CCD camera 24 has changed. It can also be said that the (position or line-of-sight direction) has changed.

  The change direction of the shooting range may be expressed by a two-dimensional vector (a vector in the screen coordinate system) set on the game screen 60, or by a three-dimensional vector in a real space coordinate system or a virtual space coordinate system. May be. Here, a case will be described in which the change direction of the shooting range indicated by the real space image 62 is expressed by a three-dimensional vector in the real space coordinate system.

The change direction of the shooting range indicated by the real space image 62 is acquired based on the time-series change of the real space image 62. For example, when the user changes the shooting range of the shooting unit 72 to the left as viewed from the user, the real space is newly displayed in the area corresponding to the left end portion of the real space image 62. The change direction of the shooting range indicated by the spatial image 62 is the left direction (for example, the _XRW axis direction).

FIG. 9 is a diagram for explaining processing for acquiring the change direction of the imaging range. As shown in FIG. 9, the change direction V _RC of the shooting range indicated by the real space image 62 is the line-of-sight direction of the shooting unit 72 after the change of the shooting range indicated by the real space image 62 (ie, the Z _RC axis direction indicated by the solid line). ) and line-of-sight direction of change before the shutter portion 72 (i.e., the direction indicated vector difference between the Z _RC axis direction) indicated by a broken line.

  For example, the association between the direction instruction operation and the basic direction is changed based on the change direction of the shooting range specified as described above. For example, the direction instruction operation and the direction determined based on the change direction of the shooting range indicated by the real space image 62 are changed so as to be associated with each other. The direction determined based on the change direction of the shooting range is a direction for moving the display position of the tank 52 in the direction in which the real space image 62 spreads on the game screen 60. For example, even when the user presses the right rotation button 66d, the association between the direction instruction operation and the basic direction is changed so that the tank 52 rotates to the left.

  FIG. 10 is a diagram for explaining the association between the changed direction instruction operation and the basic direction. As shown in FIG. 10, the association before the shooting range indicated by the real space image 62 changes and the association after the change are changed. For example, as shown in FIG. 10, the correspondence between the direction instruction operation and the basic direction is changed so that the tank 52 moves in the direction of change of the shooting range regardless of which button 66 the user presses. Since the change direction of the shooting range is represented by, for example, a three-dimensional vector in the real space coordinate system, the vector of the corresponding “change direction” shown in FIG. The

FIG. 11 is a diagram illustrating a state in which the position of the tank 52 is controlled when the association between the direction instruction operation and the basic direction is changed. As shown in FIG. 11, the real space image 62 expands in the left direction (for example, the _XRW axis direction) as viewed from the user (the expanded area is indicated by hatching as 62a), and the left direction (for example, the X _VW axis is positive Direction) or right direction (for example, the negative direction of the _XVW axis) is input, the tank 52 moves, for example, to the left in which the real space image 62 spreads. By moving the tank 52 in this way, the tank 52 can be moved in the direction in which the real space image 62 is expanded.

[1-4. Processing executed in portable terminal]
Next, the process which the portable terminal 10 performs is demonstrated. FIG.12 and FIG.13 is a flowchart which mainly shows the process relevant to this invention among the processes which the portable terminal 10 performs. For example, the control unit 12 executes the processes shown in FIGS. 12 and 13 according to a program stored in the storage unit 14. For example, when a game start instruction is input from the operation key unit 20, the processes shown in FIGS. 12 and 13 are executed.

  As shown in FIG. 12, the control unit 12 first activates the CCD camera 24 to display the real space image 62 on the display unit 18 (S1). Next, the control unit 12 generates a game space 50 in the main storage unit 16 (S2). In S2, for example, each object in the game space 50 is arranged at a predetermined initial position. Further, the virtual camera 58 may not be arranged in the game space 50 in the stage of S2.

  The control unit 12 determines whether or not the marker 40 is included in the real space image 62 (S3). In S3, for example, the real space image 62 obtained appropriately from the CCD camera 24 is referred to, and whether or not the marker 40 is included is determined based on whether or not the frame 42 of the marker 40 is extracted.

  When it is determined that the marker 40 is included (S3; Y), the control unit 12 compares the pattern 44 drawn in the frame 42 of the marker 40 with the pattern 44 stored in the marker data. Then, it is determined whether or not a predetermined marker 40 (marker 40 stored in the marker data) is detected (S4). In the present embodiment, it is determined whether or not the “L-shaped” marker 40 is detected.

  When it determines with the predetermined marker 40 having been detected (S4; Y), the control part 12 acquires the display position of the marker 40 in the real space image 62 (S5). In S5, for example, the display positions of the four corners of the frame 42 of the marker 40 are specified.

  The control unit 12 acquires the size and orientation of the marker 40 in the real space image 62 (S6). In S6, for example, the size of the marker 40 is compared by comparing the positional relationship of the four corners of the frame 42 of the marker 40 specified in S5 with the positional relationship of the four corners of the marker 40 stored in the marker data. Information about is obtained. Further, for example, information related to the orientation of the marker 40 is acquired based on the difference between the pattern 44 of the marker 40 in the real space image 62 and the pattern 44 stored in the marker data.

The control unit 12 determines the position and line-of-sight direction of the virtual camera 58 based on the position, size, and orientation of the marker 40 in the real space image 62 (S7). For example, a coordinate transformation matrix from the real space coordinate system to the real viewpoint coordinate system is calculated based on the position, size, and orientation of the marker 40. Then, by applying the coordinate transformation matrix to the virtual space coordinate system, the position of the origin O _VC and the direction of the three axes of the virtual viewpoint coordinate system are determined. For example, the origin O _VC of the virtual viewpoint coordinate system is determined as the position of the virtual camera 58, and the Z _VC axis direction is determined as the viewing direction of the virtual camera 58.

  The control unit 12 synthesizes the real space image 62 generated by the CCD camera 24 and the virtual space image 64 showing the state in which the game space 50 is viewed from the virtual camera 58 whose position and line-of-sight direction are determined in S7. Is displayed on the game screen 60 (S8). By determining the position and line-of-sight direction of the virtual camera 58 in S7, the physical space image 62 and the virtual space image 64 are aligned, and the display position of the tank 52 on the game screen 60 is determined. .

  On the other hand, when it is not determined that the marker 40 is included (S3; N), or when it is not determined that the predetermined marker 40 is detected (S4; N), the control unit 12 deletes the virtual space image 64 from the game screen 60. , A predetermined message such as “the marker cannot be detected” is displayed on the display unit 18 (S9). In this case, the user changes the position and orientation of the mobile terminal 10 so that the marker 40 is included in the field of view of the CCD camera 24 and the marker 40 is displayed at a detectable position.

  As shown in FIG. 13, the control unit 12 determines whether or not the shooting range indicated by the real space image 62 has changed (S10). For example, the process of S <b> 10 is executed by pattern matching between the most recently generated real space image 62 and the real space image 62 generated before a predetermined period of the real space image 62. Further, for example, it may be determined whether or not the change in the display position of the marker 40 in the real space image 62 during a predetermined period is a predetermined distance or more.

  When it is not determined that the shooting range indicated by the real space image 62 has changed (S10; N), the control unit 12 determines whether or not a direction instruction operation for the tank 52 has been input (S11). When it is determined that the direction instruction operation is input (S11; Y), the control unit 12 moves the tank 52 based on the basic direction corresponding to the input direction instruction operation (S12). That is, the basic direction corresponding to the direction indicating operation input by the user is converted into a vector in the virtual space coordinate system, and the situation of the game space 50 is updated so that the tank 52 moves in this converted direction.

  On the other hand, when it is determined that the shooting range indicated by the real space image 62 has changed (S10; Y), the control unit 12 determines whether or not a direction instruction operation for the tank 52 has been input within a predetermined period (S13). ). When it is determined that the direction instruction operation is input within the predetermined period (S13; Y), the control unit 12 changes the association between the direction instruction operation and the basic direction (S14).

  In S14, for example, the direction indication operation and the basic direction are all associated with the direction in which the change direction of the imaging range represented in the virtual space coordinate system is converted into the local coordinate system. Is changed. The association changed in S14 is temporarily stored in the main storage unit 16, for example.

  The control unit 12 moves the tank 52 based on the association changed in S14 (S15). In S15, the position of the tank 52 is controlled so that the tank 52 moves in the direction of change of the shooting range indicated by the real space image 62. For example, the tank 52 moves in the direction in which the field of view of the virtual camera 58 has changed.

  When the control unit 12 moves the tank 52 in S14, the association changed in S13 is restored (S16). The control unit 12 determines whether or not the end condition is satisfied (S17). The end condition is a predetermined condition for ending this process. For example, the end condition is a game over condition, whether or not a game end instruction is input, and the like.

  If it is determined that the end condition is satisfied (S17; Y), the process ends. If it is not determined that the end condition is satisfied (S17; N), the process returns to S3.

  According to the mobile terminal 10 described above, when the shooting range indicated by the real space image 62 changes and the direction instruction operation is performed within a predetermined period, the association between the direction instruction operation and the basic direction is changed. Thus, the tank 52 can be moved in the direction in which the real space image 62 spreads. That is, it is possible to prevent the tank 52 from coming off the game screen 60 and assist the user in operation.

  For example, when the tank 52 is moved in the direction in which the mobile terminal 10 is tilted based on the detection signal of the gyro sensor or the acceleration sensor, the tank 52 is removed from the visual field of the virtual camera 58, so the tank 52 becomes invisible. There is a possibility that. However, since the mobile terminal 10 of the embodiment moves the tank 52 in the direction in which the real space image 62 spreads, the user has made an erroneous operation that would cause the tank 52 to protrude from the display area of the game screen 60. However, the tank 52 can be moved to the expanded space and the movement control can be performed so as to be within the game screen 60, and the user's operation assistance can be effectively performed.

[2. Second Embodiment]
Next, another embodiment of the mobile terminal 10 according to the present invention will be described. In the first embodiment, the case where the association between the direction instruction operation and the basic direction is changed based on the change direction of the shooting range indicated by the real space image 62 has been described. In the second embodiment, it is determined whether or not the association needs to be changed based on the display position of the tank 52. That is, in a situation where it is not necessary to change the association between the direction instruction operation and the basic direction, the user performs the direction instruction operation with the original association without changing the association.

  The hardware configuration and functional blocks of the mobile terminal 10 of the second embodiment are the same as those of the first embodiment.

  The movement control unit 78 according to the second embodiment, when a change in the positional relationship between the shooting range indicated by the real space image 62 in the real space and the position of the operation target (for example, the tank 52) in the virtual space satisfies a predetermined condition. The association between the direction instruction operation and the basic direction is changed. The predetermined condition is a condition relating to a change in the positional relationship between the shooting range indicated by the real space image 62 and the position of the tank 52.

  In the portable terminal 10, the shooting range of the CCD camera 22 and the viewpoint of the virtual camera 58 correspond to each other, so that the positional relationship between the shooting range indicated by the real space image 62 and the position of the tank 52 is the tank on the game screen 60. 52 indicates the display position, for example, the positional relationship and agreement between the center point of the game screen 60 and the display position of the tank 52.

  For example, depending on whether or not the change in the display position of the tank 52 on the game screen 60 satisfies a predetermined condition, whether or not the association between the direction instruction operation and the basic direction is changed is determined.

  FIG. 14 is a diagram for explaining the display position of the tank 52 on the game screen 60. For example, the tank 52 is displayed in any of the areas (the center area 60a, the left end area 60b, the upper end area 60c, the right end area 60d, and the lower end area 60e) obtained by dividing the display area of the game screen 60 into a plurality of areas. The correspondence between the direction indicating operation and the basic direction is changed.

Whether or not the association between the direction instruction operation and the basic direction is changed is determined depending on how the display positions of the tanks 52 have changed across these five areas. Incidentally, _{X S} axis and _{Y S} axis shown in FIG. 14 is a coordinate axis of the screen coordinate system. Each of the above areas is defined by a screen coordinate system.

  FIG. 15 is a diagram illustrating a condition for changing the association between the direction instruction operation and the basic direction. As shown in FIG. 15, for example, before and after the real space image 62 changes, the direction instruction operation and the basic direction are associated with each other according to the condition that the change of the area where the tank 52 is displayed on the game screen 60 is satisfied. It is determined whether or not is changed.

  For example, when the display position of the tank 52 moves from the central area 60a to any one of the left end area 60b to the lower end area 60e due to the change in the real space image 62, the tank 52 is likely to be detached from the game screen 60. Therefore, the user's operation assistance is performed by changing the association between the direction instruction operation and the basic direction.

  Further, for example, when the real space image 62 has changed, but the display position of the tank 52 remains in the central area 60a, the tank 52 is unlikely to be removed from the game screen 60, so the direction indication operation and the basic direction The association is not changed.

  Further, for example, when the display position of the tank 52 moves from any one of the left end region 60b to the lower end region 60e to the central region 60a due to the change in the real space image 62, the tank 52 may be removed from the game screen 60. Therefore, the association between the direction instruction operation and the basic direction is not changed.

  Further, for example, when the real space image 62 has changed, but the display position of the tank 52 remains in any one of the left end region 60b to the lower end region 60e, the shooting range has not changed so much, so the real space image Since there is little possibility that the 62 tank 52 will come off the game screen 60, the association between the direction instruction operation and the basic direction is not changed.

  Further, for example, when the display position of the tank 52 is moved from any one of the left end region 60b to the lower end region 60e to the other left end region 60b to the lower end region 60e due to the change in the real space image 62, the imaging range Moves so much that the tank 52 is likely to be detached from the game screen 60, so that the user's operation assistance is performed by changing the association between the direction instruction operation and the basic direction.

  In the second embodiment, the same processing as in FIG. 12 and FIG. 13 is executed, but when the processing of S14 is executed, the shooting range indicated by the real space image 62 in the real space and the position of the tank 52 in the virtual space It is determined whether or not the change in the positional relationship satisfies a predetermined condition.

  For example, when the condition that the association is not changed is satisfied, the processing of S12 is executed instead of the processing of S14 to S16, and the association between the direction instruction operation and the basic direction is not changed. The position of the tank 52 is controlled. On the other hand, for example, when the condition for changing the association is satisfied, the processes of S14 to S16 are executed.

  According to the mobile terminal 10 of the second embodiment, whether to change the association between the direction instruction operation and the basic direction is determined based on the condition that the change in the display position of the tank 52 satisfies. Therefore, for example, in a situation where the tank 52 is unlikely to protrude from the game screen 60 or a situation where the user is unlikely to make an erroneous input, the user may be confused if the association is changed. In such a situation, the association can be prevented from being changed.

[3. Embodiment 3]
Next, another embodiment of the mobile terminal 10 according to the present invention will be described. In the first embodiment and the second embodiment, the case has been described in which the basic direction associated with all the direction instruction operations is changed to the same direction. That is, the case where the tank 52 moves in the direction in which the real space image 62 spreads regardless of which button 66 the user presses has been described. The association with the basic direction may be changed not only for all direction indicating operations but only for some direction indicating operations.

  The hardware configuration and functional blocks of the mobile terminal 10 of the third embodiment are the same as those of the first embodiment.

  The movement control unit 78 of the third embodiment includes the moving direction of the operation target (for example, the tank 52) determined based on the basic direction corresponding to the direction instruction operation performed within a predetermined period, and the shooting range indicated by the real space image 62. The change direction and the deviation are determined. For example, the basic direction associated with the direction indicating operation performed within a predetermined period is converted into the vector direction of the virtual space coordinate system, and the moving direction of the tank 52 is specified. Note that the tank 52 does not necessarily move in this movement direction, and the movement direction specified here can be said to be the direction in which the tank 52 is about to move.

  The moving direction of the tank 52 is compared with the changing direction of the shooting range indicated by the real space image 62 (that is, the same as the changing direction of the visual field of the virtual camera 58). Here, the cosine (cos θ) of the two vectors is calculated as the deviation between the two directions. Whether or not to change the association between the direction instruction operation and the basic direction is determined according to the cosine.

FIG. 16 is a diagram for explaining the processing of the third embodiment. FIG. 16 shows a case where the real space image 62 spreads leftward and the button 66a is pressed as a direction instruction operation. If the cosines of the vectors of the movement direction V _{LW of the} tank 52 determined based on the association before the change (that is, the front direction of the tank 52) and the shooting direction change direction V _RC are relatively small, These vectors are directed in opposite directions.

  That is, the user is trying to move the tank 52 in the direction in which the real space image 62 is narrowed, and the user may have made an incorrect input. In this case, the correspondence between the direction instruction operation and the basic direction The attachment is changed.

  On the other hand, if the cosines of the vectors of the movement direction of the tank 52 determined based on the association before the change and the change direction of the shooting range are relatively large, the vectors must be in the same direction. Is shown. That is, since the user tries to move the tank 52 in the direction in which the real space image 62 spreads, the association between the direction instruction operation and the basic direction is not changed.

  When the deviation is in the reference range, the movement control unit 78 of the third embodiment moves the operation target (for example, the tank 52) based on the basic direction corresponding to the direction instruction operation performed by the user. For example, when the cosine is in the reference range (for example, cos θ> −0.5), the process for changing the association between the direction instruction operation and the basic direction is restricted, and the tank 52 is moved with the original association. In this case, since the tank 52 is not moved in the direction in which the real space image 62 is narrowed, it is not necessary to change the association, so the association is not changed as described above.

  In addition, when the deviation is not the reference range, the movement control unit 78 of the third embodiment changes the association between the direction instruction operation and the basic direction based on the change content of the shooting range indicated by the real space image 62, and after the change. The operation target (for example, the tank 52) is moved based on the association and the direction instruction operation performed within a predetermined period. For example, when the cosine is not in the reference range (for example, cos θ ≦ −0.5), a process for changing the association between the direction instruction operation and the basic direction is performed. In this case, since the tank 52 is to be moved in the direction in which the real space image 62 is narrowed, the association is changed and the tank 52 is moved in the direction in which the real space image 62 is widened.

  In the third embodiment, the same processing as in FIG. 12 and FIG. 13 is executed, but the tank determined based on the basic direction corresponding to the direction indicating operation performed within a predetermined period when the processing of S14 is executed. A shift between the moving direction of 52 and the changing direction of the shooting range indicated by the real space image 62 is determined. If the deviation is within the predetermined range, the process proceeds to S12. If the deviation is not within the predetermined range, the process proceeds to S14.

  According to the mobile terminal 10 of the third embodiment, the moving direction of the tank 52 determined based on the basic direction corresponding to the direction indicating operation performed within a predetermined period, the changing direction of the shooting range indicated by the real space image 62, Whether or not to change the association between the direction instruction operation and the basic direction is determined according to the deviation. For example, since the user may be confused if the operation assistance is excessively performed, the association should not be changed unless the tank 52 is moved in the direction in which the real space image 62 is narrowed. By doing so, moderate operation assistance can be provided without being confused by the user.

[4. Modified example]
The present invention is not limited to the embodiment described above. Modifications can be made as appropriate without departing from the spirit of the present invention.

  (1) For example, in the above-described embodiment, the case where the change direction of the shooting range indicated by the real space image 62 is represented by a three-dimensional vector in the real space coordinate system has been described. 60 may be represented by a two-dimensional vector (a vector in the screen coordinate system) set on 60. In this case, for example, the direction in which the pixels in which the real space image 62 is spread is acquired as the change direction of the shooting range.

  In this case, the two-dimensional vector indicating the change direction of the shooting range is inversely transformed with the coordinate transformation matrix calculated in S7, thereby moving the tank 52 in the game space 50 (three-dimensional in the virtual space coordinate system). Vector) is specified. The association between the direction instruction operation and the basic direction may be changed so that the tank 52 moves in this direction.

  (2) For example, the case where the basic direction corresponding to the direction instruction operation is expressed in the character coordinate system based on the tank 52 has been described. However, the basic direction is expressed in the virtual space coordinate system. Also good. The basic direction may be defined in a coordinate system different from the virtual viewpoint coordinate system. Note that the basic direction may be expressed in a virtual viewpoint coordinate system.

  (3) For example, in the embodiment, the case where the tank 52 moves on the horizontal plane has been described. However, the range in which the operation target of the user moves is not limited to the example of the embodiment. For example, the operation target may be a ball or a bird, and the virtual three-dimensional space may be moved in the vertical direction by a user operation.

  (4) Moreover, for example, although the direction instruction operation and the basic direction are described as four, the number of the direction instruction operation and the basic direction is not limited to this. For example, the user may be allowed to input eight direction instruction operations.

  (5) For example, in the embodiment, the case where the association is changed based on the change direction of the real space image 62 has been described. However, the association is changed based on the change amount of the real space image 62. It may be. For example, the direction associated with the direction instruction operation may be different based on whether or not the change amount of the real space image 62 is within a predetermined range.

  (6) Further, for example, the basic direction associated with the direction instruction operation may be different depending on the state of the game being executed. For example, when the remaining physical strength of the tank 52 is large, the enemy 56 in the direction of change of the shooting range indicated by the real space image 62 may be set as the moving direction, and when the remaining physical strength of the tank is small, the real space image 62 indicates. The direction of the empty space in the direction of change of the shooting range may be the moving direction, and the tank 52 may be retracted.

  (7) Further, for example, the moving speed of the tank 52, the moving route, the direction of the tank 52, or the posture of the tank 52 may be determined based on the direction of change of the shooting range indicated by the real space image 62.

  For example, the moving speed of the tank 52 is determined based on the positional relationship between the current position of the tank 52 and the position of the object placed in the game space 50 and the direction of change of the shooting range indicated by the real space image 62. . For example, the moving speed of the tank 52 increases as the current position of the tank 52 and the position of the enemy 56 in the direction of change of the shooting range indicated by the real space image 62 increase. Further, for example, the moving speed of the tank 52 may be increased as the amount of change in the shooting range indicated by the real space image 62 is larger.

  Further, for example, the movement route of the tank 52 is determined based on the positional relationship between the current position of the tank 52 and the position of the object placed in the game space 50 and the change direction of the shooting range indicated by the real space image 62. The For example, the movement route is determined so that the tank 52 does not collide with the enemy 56 in the direction of change of the shooting range indicated by the real space image 62.

  Further, for example, the direction or posture of the tank 52 is determined based on the positional relationship between the current position of the tank 52 and the position of the object placed in the game space 50 and the direction of change of the shooting range indicated by the real space image 62. Is done. For example, the orientation or posture of the tank 52 is determined so that the front of the tank 52 faces the enemy 56 in the direction of change of the shooting range indicated by the real space image 62.

  (8) For example, as a method of providing augmented reality to the user, the method using the marker 40 has been described as an example, but the method of providing augmented reality is not limited to the above example. Various other known methods can be applied to the method of providing augmented reality to the user. For example, the following method can be applied.

  (8-1) For example, an object (for example, LED) that emits visible light instead of the marker 40 may be used as a detection target. That is, augmented reality may be provided by photographing visible light using an LED or the like with the CCD camera 24. As a method for providing augmented reality using visible light, for example, there are the following two methods.

  First, as a first method, there is a method of calculating a coordinate transformation matrix by a method similar to the method using the marker 40 as in the embodiment by arranging LEDs based on a predetermined rule. In this case, for example, LEDs are arranged at the four corner positions of the frame 42 of the marker 40. For example, these four LEDs are photographed by the CCD camera 24, and a positional relationship between the CCD camera 24 and the LEDs is acquired by detecting a deviation (distortion) in the light emission position of the LEDs. That is, a coordinate transformation matrix is acquired. Subsequent processing is the same as that of the embodiment, and thus description thereof is omitted.

  Further, as a second method, there is a method of representing predetermined digital data in a pseudo manner by repeatedly turning on and off the LED. For example, “1” is set when the LED is turned on, and “0” is set when the LED is turned off. A coordinate transformation matrix similar to that of the embodiment is achieved by controlling the lighting and extinguishing of the LED so that the digital data obtained based on the real space image obtained by continuously photographing the LED indicates the position and light emission direction of the LED. May be acquired. Subsequent processing is the same as that of the embodiment, and thus description thereof is omitted.

  (8-2) Further, for example, augmented reality may be provided by extracting feature points from an object included in the real space image 62 without arranging the marker 40 in the real space. That is, a so-called markerless augmented reality may be adopted. In this case, an object (or a feature point thereof) arranged in the real space is a detection target.

FIG. 17 is a diagram for explaining the markerless augmented reality. As shown in FIG. 17, for example, feature points of an object arranged in the real space (for example, the contour of a desk or the edge of a cup) are extracted from a known feature point extraction method (for example, a corner detection method using image matching). ). For example, the contour of a desk placed in the real space is set as feature points Q _{1 to} Q _4, and the ground point of the desk is set as feature points Q _{5 to} Q ₈ .

When the position or line-of-sight direction of the CCD camera 24 changes, the position and line-of-sight direction of the CCD camera 24 are calculated by tracking the position change (change amount and change direction) of the feature points Q _{1 to} Q ₈ . After the position and line-of-sight direction of the CCD camera 24 are calculated, augmented reality is provided by calculating a coordinate transformation matrix in the same manner as in the embodiment.

  (8-3) For example, by using the sensor unit 26, augmented reality may be provided without using the marker 40. In the modified example (8-3), the device detection unit that detects the position, orientation, and orientation of the photographing unit 72 (for example, the CCD camera 24) in the real space is realized mainly by the control unit 12 and the sensor unit 26. .

  For example, when the sensor unit 26 includes a GPS sensor, an electronic compass (geomagnetic sensor), and a gyro sensor, for example, the position of the CCD camera 24 in the real space is acquired based on a signal obtained from the GPS sensor. The Further, the orientation of the CCD camera 24 is acquired based on a signal obtained from the electronic compass. Further, the attitude of the CCD camera 24 is acquired based on a signal obtained by the gyro sensor.

  The display control unit 74 according to the modified example (8-3) uses the real space image 62 and the virtual space image 64 based on the detection result (for example, the position, orientation, and orientation of the mobile terminal 10 in the real space). And alignment with.

For example, the display control unit 74 calculates the position and line-of-sight direction of the CCD camera 24 in the real space based on the position, orientation, and orientation of the CCD camera 24 in the real space. That is, the origin O _VC of the real viewpoint coordinate system and the direction of the three axes are specified. Based on the difference between the real space coordinate system (there is no marker 40 here, for example, the coordinate system set at an arbitrary position in the real space (for example, the origin of the GPS sensor)) and the real viewpoint coordinate system. Thus, a coordinate transformation matrix is calculated as in the embodiment.

  Further, in the modified example (8-3), the same processing as that in FIG. 11 is executed. However, instead of the processing of S3 to S6 as a method of providing augmented reality to the user, the mobile terminal 10 The position, orientation, and orientation of the are detected. Then, based on the detection result of the sensor unit 26 (that is, the position, orientation, and orientation of the mobile terminal 10), the position and the line-of-sight direction of the CCD camera 24 in the real space are specified. That is, similarly to S7, augmented reality is provided to the user by specifying the relationship between the real space coordinate system and the real viewpoint coordinate system.

  Various known methods can be applied to the method of detecting the position, orientation, and orientation of the CCD camera 24 in the real space. In addition to the above, for example, the position may be measured using radio waves transmitted from a wireless LAN access point. In this case, the location where each access point is installed is registered as a database. Further, for example, the position of the mobile terminal 10 may be acquired using a so-called IMES (Indoor Messaging System) in which a small base station that transmits GPS radio waves is installed indoors.

  (9) Further, for example, the present invention can be applied to augmented reality other than the game described in the embodiments and modifications. For example, what is to be operated by the user is not limited to a tank, and a car or a character may be placed in the virtual space as an operation target. That is, for example, an augmented reality may be provided in which a race course in which a car runs or a character sings or dances on a desk in a real space. In this case, the position of the car or character may be controlled based on the direction of change of the shooting range indicated by the real space image using the same method as in the embodiment and the modification.

  10 mobile terminals, 12 control units, 14 storage units, 16 main storage units, 18 display units, 20 operation key units, 22 touch panels, 24 CCD cameras, 26 sensor units, 28 communication units, 30 buses, 40 markers, 42 frames, 44 patterns, 50 game spaces, 52 tanks, 54 forts, 56 enemies, 58 virtual cameras, 60 game screens, 62 real space images, 64 virtual space images, 66a forward buttons, 66b reverse buttons, 66c left rotate buttons, 66d right rotate Button, 70 Augmented reality data storage unit, 72 photographing unit, 74 display control unit, 76 operation unit, 78 movement control unit, 80 determination unit.

Claims (6)

Photographing means whose photographing range is changed by a user operation;
A virtual image showing a state where a real space image generated by photographing the real space by the photographing unit and a virtual space in which the operation target of the user is viewed from a viewpoint in which a visual field corresponding to the photographing range is set Display control means for aligning and synthesizing the spatial image and displaying it on the display screen;
Operation means for the user to perform a direction instruction operation of the operation target;
Movement control means for moving the operation object based on a basic direction corresponding to the direction instruction operation performed by the user;
Determination means for determining whether or not the shooting range indicated by the real space image has changed;
Including
The movement control means includes
When it is determined that the shooting range indicated by the real space image has changed and the direction indication operation is performed within a predetermined period, the direction indication operation is performed based on the change content of the shooting range indicated by the real space image. Changing the association with the basic direction, and moving the operation target based on the association after the change and the direction instruction operation performed within the predetermined period;
An image processing apparatus. The movement control means includes
Correspondence between the direction indicating operation and the basic direction when a change in the positional relationship between the shooting range indicated by the real space image in the real space and the position of the operation target in the virtual space satisfies a predetermined condition Change the attachment,
The image processing apparatus according to claim 1. The movement control means includes
Changing the correspondence between the direction indicating operation and the basic direction indicated by a coordinate system different from the viewpoint coordinate system of the viewpoint in the virtual space;
The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. The movement control means includes
Means for determining a deviation between a moving direction of the operation target determined based on the basic direction corresponding to the direction indicating operation performed within the predetermined period and a change direction of the shooting range indicated by the real space image. When,
Means for moving the operation target based on a basic direction corresponding to the direction indicating operation performed by the user when the deviation is in a reference range;
When the deviation is not a reference range, the association between the direction instruction operation and the basic direction is changed based on the change content of the shooting range indicated by the real space image, and the association after the change is within the predetermined period. Means for moving the operation object based on the direction instruction operation performed on
The image processing apparatus according to claim 1, wherein the image processing apparatus includes: A virtual space in which the operation target of the user is arranged from a viewpoint in which a real space image generated by photographing means that changes the photographing range by a user operation and photographing a real space and a field of view corresponding to the photographing range is set. A virtual space image showing a state of seeing the space, and a display control step for performing alignment and composition on the display screen,
A movement control step of moving the operation object based on a basic direction corresponding to the direction instruction operation performed by the user from an operation means for the user to perform a direction instruction operation of the operation object;
A determination step of determining whether or not the shooting range indicated by the real space image has changed;
Including
In the movement control step,
When it is determined that the shooting range indicated by the real space image has changed and the direction indication operation is performed within a predetermined period, the direction indication operation is performed based on the change content of the shooting range indicated by the real space image. Changing the association with the basic direction, and moving the operation target based on the association after the change and the direction instruction operation performed within the predetermined period;
And a control method for the image processing apparatus. Photographing means whose photographing range is changed by a user operation,
A virtual image showing a state where a real space image generated by photographing the real space by the photographing unit and a virtual space in which the operation target of the user is viewed from a viewpoint in which a visual field corresponding to the photographing range is set Display control means for aligning and synthesizing the spatial image and displaying on the display screen;
Operation means for the user to perform a direction instruction operation of the operation target;
A movement control means for moving the operation target based on a basic direction corresponding to the direction instruction operation performed by the user;
Determination means for determining whether or not the shooting range indicated by the real space image has changed;
Including
The movement control means includes
When it is determined that the shooting range indicated by the real space image has changed and the direction indication operation is performed within a predetermined period, the direction indication operation is performed based on the change content of the shooting range indicated by the real space image. Changing the association with the basic direction, and moving the operation target based on the association after the change and the direction instruction operation performed within the predetermined period;
A program for causing a computer to function as an image processing apparatus.

Images