JP3841806B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing technique, and more particularly to a technique for displaying a real object and a virtual object in association with each other.

2. Description of the Related Art In recent years, a technique has been widely used in which a two-dimensional code is captured and recognized by a camera and a predetermined process based on a two-dimensional code pattern is executed. As one of such techniques, an image processing technique has been proposed in which a two-dimensional barcode is imaged by a video camera and a three-dimensional image corresponding to the two-dimensional barcode is displayed on a display device (for example, Patent Documents). 1). In Patent Document 1, coordinates on a space for displaying a three-dimensional object are determined based on the coordinates of the captured two-dimensional barcode and the focal length of the CCD video camera, and the three-dimensional object is superimposed on the two-dimensional barcode. To display.
JP 2000-322602 A

  The technique disclosed in Patent Literature 1 can realize an excellent visual effect by displaying a combination of a real-world object and a virtual object. However, Patent Document 1 only discloses a display technique of a virtual object in a state where the two-dimensional barcode is stationary, and is not conscious of display processing of the virtual object when the two-dimensional barcode is moved. The present inventor pays attention to display processing when moving a two-dimensional barcode, and applies it to the field of processing moving images such as in the game field by elaborating the display processing of the virtual object at this time. And found that it has the potential to achieve even better visual effects.

  An object of the present invention is to provide a technique for displaying a real object and a virtual object in association with each other, and in particular, in a field such as a game for displaying a moving image, a display mode of the virtual object in relation to the movement of the real object. An object is to provide a technique for controlling the above.

  In order to solve the above problems, an aspect of the present invention relates to an image processing apparatus. An image processing apparatus according to this aspect includes a storage unit that stores identification information for identifying a real object and three-dimensional image data of the virtual object in association with each other, and a real object included in a frame image captured by the imaging apparatus A readout unit that reads out the 3D image data of the virtual object associated with the image identification information from the storage unit, and displays the virtual object in association with the display position of the real object using the read out 3D image data. A display control unit to be displayed on the device, and a change detection unit that detects a temporal state change of the real object image captured by the imaging device. In the image processing apparatus of this mode, the display control unit controls the display mode of the virtual object based on the state change detected by the change detection unit. Here, the real object is a real object that exists as a tangible object in the real space, and the virtual object does not exist in the real space, but is expressed by data in the virtual space.

  According to this image processing apparatus, since the display mode of the virtual object is controlled based on the change in the state of the captured real object image, in other words, based on the actual movement of the real object, a new virtual object corresponding to the movement of the real object can be controlled. Visual expression can be realized.

  Another aspect of the present invention relates to a game device. The game device according to this aspect includes a storage unit that stores identification information for identifying a real object and three-dimensional image data of the game character in association with each other, and a real object image included in a frame image captured by the imaging device A reading unit that reads out the three-dimensional image data of the game character associated with the identification information from the storage unit, and a display device that associates the game character with the display position of the real object using the read-out three-dimensional image data A display control unit to be displayed, and a change detection unit that detects a temporal state change of the real object image captured by the imaging device. In the game device of this mode, the display control unit controls the display mode of the game character based on the state change detected by the change detection unit.

  According to this game apparatus, in order to control the display mode of the game character based on the state change of the captured real object image, in other words, based on the movement of the real object, a new visual expression of the game character corresponding to the movement of the real object is generated. realizable.

  Still another aspect of the present invention provides a real object included in a frame image captured by an imaging device from a storage unit that stores identification information for identifying a real object and three-dimensional image data of the virtual object in association with each other. Reading three-dimensional image data of the virtual object associated with the identification information of the image, and displaying the virtual object on the display device in association with the display position of the real object using the read three-dimensional image data; An image processing method comprising: detecting a temporal state change of a real object image captured by an imaging device; and controlling a display mode of a virtual object based on the detected state change.

  Still another aspect of the present invention includes a frame image captured by an imaging device from a storage unit that stores identification information for identifying a real object and three-dimensional image data of a virtual object in association with each other. The virtual object is associated with the display position of the real object using the function of reading the three-dimensional image data of the virtual object associated with the identification information of the real object image and the read three-dimensional image data. A function to be displayed on the display device, a function to detect a temporal state change of the real object image captured by the imaging device, and a function to control the display mode of the virtual object based on the detected state change are executed. A program is provided.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  ADVANTAGE OF THE INVENTION According to this invention, the technique which controls the display mode of a virtual object in relation to the motion of a real object can be provided.

  The present invention provides a technology for detecting a temporal state change of an image of a real object imaged by an imaging device and controlling a display mode of a virtual object based on the detected state change. The real object may be an existing one-dimensional object, two-dimensional object, or three-dimensional object, and is preferably configured to have a characteristic portion for identifying the real object itself. For example, the real object may be a two-dimensional object such as a card having code information expressed in two dimensions as a characteristic part, or may be a three-dimensional object having a unique three-dimensional shape as a characteristic part. Note that the two-dimensional shape of the two-dimensional object may constitute a unique feature portion, and code information serving as the feature portion may be attached to the three-dimensional object. The virtual object may be a so-called character such as a person, an animal, or an object that is three-dimensionally expressed in the virtual space. The following embodiment relates to an image processing technique in a game application, and employs a game character appearing in the game application as a virtual object.

  FIG. 1 shows a configuration of a game system 1 according to an embodiment of the present invention. The game system 1 includes an imaging device 2, an image processing device 10, and an output device 6. The image processing device 10 includes an image analysis device 20 and a game device 30. The image analysis device 20 and the game device 30 may be configured as separate devices, but may be configured as a single unit. The imaging device 2 is a video camera that includes a CCD imaging device or a MOS imaging device. The imaging device 2 images the real space at a predetermined cycle and generates a frame image for each cycle. The imaging region 5 is a range where the imaging device 2 captures an image, and the position and size of the imaging region 5 are adjusted by adjusting the height and direction of the imaging device 2. The game player moves the game card 4 that is a real object with a finger in the imaging area 5. The game card 4 has a characteristic part for uniquely identifying itself.

  The output device 6 includes a display 7 that is a display device. The output device 6 may further include a speaker (not shown). Basically, the image processing apparatus 10 displays the frame image captured by the imaging apparatus 2 on the display 7, and controls display so that a character that is a virtual object is superimposed on the game card 4. The player can easily check whether or not the game card 4 is in the imaging area 5 by looking at the display 7. If not, the player can shift the position of the game card 4 or adjust the orientation of the imaging device 2. Then, the imaging device 2 is caused to image the game card 4.

  In the game application of the embodiment, the player operates the character by performing a predetermined operation on the game card 4. Due to the nature of the game application, it is preferable that the player recognize the unity between the game card 4 and the character, and therefore, the character image is displayed superimposed on the game card 4. When the game card 4 is slowly moved within the imaging area 5, the character moves together following the movement of the game card 4 while maintaining the state of riding on the game card 4.

  The movement of the above characters is controlled by the image processing apparatus 10. First, the image analysis device 20 extracts image information of the game card 4 from the frame image acquired by the imaging device 2. Further, the image analysis device 20 extracts a unique feature portion for identifying the game card 4 from the image information of the game card 4. At this time, the image analysis device 20 determines posture information, direction information, distance information, and the like in the space of the game card 4 from the image information of the game card 4.

  FIG. 2 shows an example of the surface of the game card 4. On the surface of the game card 4, a direction instruction unit 11 and an identification unit 12 are printed. The direction instruction unit 11 is provided to indicate the front part of the game card 4, and the identification unit 12 is provided to express a characteristic portion for uniquely identifying the game card 4. The identification unit 12 is code information created by printing a plurality of blocks in a predetermined section. Among the plurality of blocks, a square block is commonly given to the plurality of game cards 4, and a characteristic portion is substantially formed by blocks other than the four corners. The quadrangular block is used to measure the distance from the imaging device 2.

  Returning to FIG. 1, when the image analysis device 20 recognizes the game card 4 in the frame image, the image analysis device 20, based on the length between the four blocks in the image of the game card 4, Calculate the distance. Further, the image analysis device 20 detects the direction instruction unit 11 of the game card 4 and obtains the direction in which the game card 4 is facing in the frame image. In this case, the direction in which the direction instructing unit 11 is present is the front, and the character is display-controlled so as to face the front on the game card 4. Further, the image analysis device 20 acquires the identification information of the game card 4 from the blocks other than the four corners of the identification unit 12.

  FIG. 1 shows a state in which the game card 4 is simply placed on the table 3, but the game card 4 is tilted with respect to the table 3, for example, or lifted above the table 3. May be. The image analysis device 20 has a function of recognizing a state where the game card 4 is tilted or a state where the height from the table 3 is changed by image analysis. The result of the image analysis performed by the image analysis device 20 is sent to the game device 30. Note that the frame image captured by the imaging device 2 may be sent to the game device 30 and image analysis performed on the game device 30 side. In this case, the image processing apparatus 10 is configured by only the game apparatus 30.

  The game apparatus 30 performs display control so that the character is positioned on the game card 4 on the display screen of the display 7 based on the image analysis result in the image analysis apparatus 20. Characters may be appropriately assigned to the game card 4 for each game scene. In this case, when the game scene is switched, the displayed character is also switched. In the present embodiment, the game apparatus 30 detects a temporal change in the imaging state of the game card 4 based on the image analysis result, and controls the character display mode based on this state change.

  FIG. 3 shows the configuration of the image analysis apparatus 20. The image analysis apparatus 20 includes a frame image acquisition unit 40, a real object extraction unit 42, a state determination unit 44, an identification information acquisition unit 46, an identification information storage unit 48, and a transmission unit 50. The identification information storage unit 48 stores the information of the characteristic part for identifying the real object and the identification information for identifying the real object in association with each other. Specifically, the pattern information of the identification unit 12 of the game card 4 in FIG. 2 and the identification information are stored in a one-to-one correspondence. The identification information is used for character assignment in the game apparatus 30. In particular, in a game application that permits the presence of a plurality of game cards 4, the game cards 4 can be recognized by associating each of the game cards 4 with identification information. The state determination unit 44 determines the state of the real object in the set coordinate system. Specifically, the state determination unit 44 determines the focal length from the posture determination unit 52 that determines the posture, the direction determination unit 54 that determines the direction, and the imaging device 2. A distance determining unit 56 for determining is provided.

  The frame image acquisition unit 40 acquires a real space frame image captured by the imaging device 2. Here, as shown in FIG. 1, it is assumed that one game card 4 is arranged in the imaging region 5. The imaging device 2 periodically acquires frame images, and preferably generates frame images at 1/60 second intervals.

  The real object extraction unit 42 extracts a real object image, that is, an image of the game card 4 from the frame image. This process is performed by converting the image information into a binary bit representation and extracting the image of the game card 4 from the bit representation. This is so-called dot processing, and image extraction processing may be performed by detecting on and off of bits. Further, this processing may be performed by a known image matching technique. In that case, the real object extraction unit 42 registers image information of a real object to be used in a memory (not shown) in advance for the matching process. By matching the pre-registered image information with the captured image information, the image of the game card 4 can be cut out from the frame image.

  The posture determination unit 52 determines the posture of the real object image. Specifically, the coordinates of the center point of the real object image in the frame image, the inclination of the real object image from the stage 3, the height of the real object image from the stage 3 are determined. For this purpose, the state determination unit 44 detects in advance the geometry information of the table 3 for placing and moving the game card 4, and further places the game card 4 on the reference surface using the surface of the table 3 as a reference surface. The geometry information of the posture is recorded as the initial state of the posture of the game card 4. This geometry information may be generated based on the imaging device 2. The state determination unit 44 grasps the position, posture, and the like of the table 3 as coordinate data in the imaging space from the geometry information acquired for the table 3. The posture determination unit 52 calculates the difference in relative state quantity from the posture of the game card 4 recorded as the initial state, ie, the coordinate value in the imaging space, with the inclination and height of the game card 4 with respect to the reference plane of the game card 4 being recorded as the initial state. Calculate as a difference. When the player lifts or tilts the game card 4 by hand, a change in the state amount occurs with respect to the initial posture of the game card 4, and the height and inclination of the real object image with respect to the table 3 change.

  The direction determination unit 54 determines the direction of the real object image. The direction determination unit 54 may determine the direction of the real object by detecting the direction instruction unit 11 shown in FIG. 2 from the real object image. In addition, when the posture determination unit 52 recognizes that the real object is inclined, the direction determination unit 54 may determine the inclined direction as the direction of the real object.

  The distance determination unit 56 determines the distance between the imaging device 2 and the game card 4 based on the length between the four corners of the identification unit 12 in the image of the game card 4.

  The identification information acquisition unit 46 extracts a feature portion from the real object image and acquires corresponding identification information from the identification information storage unit 48. Although only one game card 4 is shown in the example of FIG. 1, the game system 1 of the present embodiment can also support a plurality of different game cards 4. For example, when allowing simultaneous use of five game cards 4, identification information 1 to 5 may be associated with each game card 4.

  The posture information, direction information, distance information, and identification information acquired by the identification information acquisition unit 46 determined by the state determination unit 44 are associated with each other and sent from the transmission unit 50 to the game apparatus 30. When a plurality of game cards 4 are present in the imaging area 5, posture information, direction information, distance information, and identification information are associated with each game card 4 and transmitted from the sending unit 50 to the game device 30. It is done. In the present embodiment, the frame image itself is also sent from the sending unit 50 to the game device 30 in order to display the frame image taken by the imaging device 2 on the display 7.

  FIG. 4 shows the configuration of the game apparatus 30. The game apparatus 30 includes an analysis information acquisition unit 100, a game progress processing unit 102, a character determination unit 104, a character storage unit 106, a change detection unit 110, a display control unit 120, and an action pattern storage unit 122. The change detection unit 110 includes a movement amount monitoring unit 112, a rotation detection unit 114, and an existence confirmation unit 116, and detects a temporal state change of the real object image captured by the imaging device 2.

  The processing function of the game apparatus 30 in the present embodiment is realized by a CPU, a memory, a program loaded in the memory, and the like, and here, a configuration realized by cooperation thereof is illustrated. The program may be built in the game apparatus 30 or may be supplied from the outside in a form stored in a recording medium. Accordingly, those skilled in the art will understand that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof. In the illustrated example, the CPU of the game apparatus 30 has functions as an analysis information acquisition unit 100, a game progress processing unit 102, a character determination unit 104, a change detection unit 110, and a display control unit 120.

  The analysis information acquisition unit 100 receives an analysis result from the image analysis device 20. This analysis result includes posture information, direction information, distance information, and identification information of the game card 4 that is a real object. The analysis information acquisition unit 100 passes the received analysis result to the game progress processing unit 102. The analysis information acquisition unit 100 may directly acquire frame image data from the imaging device 2. In this case, the game apparatus 30 has a function in the image analysis apparatus 20 and may perform the same process as the process described for the image analysis apparatus 20.

  The game progress processing unit 102 controls processing of the entire game application. In the game application in this embodiment, the game progress is composed of a plurality of stages, and different game scenes are set in each stage. The player clears the end condition of each stage step by step, and the game ends when the final stage is cleared. The game progress processing unit 102 manages the progress of the game, and immediately after the start of the game or at the time of switching the game stage, the character determination unit 104 sends the name of the game stage to be started next and the analysis information acquisition unit 100. Communicate identification information sent to you.

  The character storage unit 106 stores the identification information of the game card 4 and the three-dimensional image data of the character in association with each game stage. Based on the game stage name and the identification information, the character determination unit 104 reads the three-dimensional image data of the character associated with the identification information from the character storage unit 106 and supplies it to the game progress processing unit 102. Note that the read three-dimensional image data may be directly supplied to the display control unit 120. The game progress processing unit 102 supplies the display control unit 120 with three-dimensional image data, posture information, direction information, and distance information of the game card 4. The display control unit 120 causes the display 7 to display the character in association with the display position of the game card 4 using the three-dimensional image data.

  Specifically, the display control unit 120 receives a frame image sent from the image analysis device 20 and causes the display 7 to display the frame image. Further, the display control unit 120 recognizes the attitude, orientation, and distance of the game card 4 based on the attitude information, direction information, and distance information of the game card 4, and displays the character displayed on the display 7 using the three-dimensional image data. Determine the posture, orientation, and size of the. For example, if the game card 4 is tilted with respect to the table 3, the character may be tilted and displayed along the normal vector of the game card 4. The display control unit 120 may place the character at any position as long as it overlaps the game card 4. However, in the normal display mode, the display control unit 120 displays the character so as to be positioned above the center point of the game card 4. Determine the location. The character may have internal parameters that express emotions, tone, and the like based on the operation history of the player.

  The motion pattern storage unit 122 stores a character motion pattern in a normal operation state. Specifically, the action pattern storage unit 122 sets action patterns in association with characters, game stages, and internal parameters. Accordingly, the display control unit 120 selects a motion pattern from the motion pattern storage unit 122 based on the character name, the currently played game stage, and the internal parameters of the character, and controls display of the character on the display 7.

  When the player slowly moves the game card 4, the analysis result of the frame image is sequentially sent from the image analysis device 20 to the analysis information acquisition unit 100. An operation of slowly moving or not moving at all in response to the state change of the game card 4 shown below can be called a normal operation state. The display control unit 120 receives the three-dimensional image data of the character and the posture information, direction information, and distance information of the game card 4 from the game progress processing unit 102, and the character image overlaps the display position of the game card 4 on the display 7. Thus, the character is caused to follow the movement of the game card 4. Thereby, the display 7 becomes a display mode in which the character is always on the game card 4, and the player can recognize the sense of unity between the character and the game card 4. As described above, the display control unit 120 displays a character superimposed on the game card 4 in a normal operation state of the game card 4.

  In the following, when the game card 4 takes a predetermined imaging state by the operation of the player, the display control unit 120 does not simply cause the character to follow the movement of the game card 4, but controls the display mode of the character, Change the movement pattern of the character. By using the action to the game card 4 by the player as a trigger for changing the movement pattern of the character, it is possible to give the player a pleasure different from the operation by the normal game controller or the like. The game progress processing unit 102 delivers the posture information, the direction information, and the distance information of the game card 4 to the change detection unit 110 in order to detect whether or not the normal operation state. The change detection unit 110 detects a temporal state change of the image of the game card 4 in the frame image.

  The movement amount monitoring unit 112 monitors the movement amount of the game card 4 imaged by the imaging device 2. Specifically, the movement amount monitoring unit 112 measures the movement speed of the game card 4 based on the center coordinates and distance information included in the posture information of the game card 4. The movement amount monitoring unit 112 stores the center coordinates and distance information of the game card 4 in a memory (not shown) for each frame, and calculates a movement vector from the distance change and the difference of the center coordinates of the game card 4 in a predetermined number of frames. Calculate Thereby, the moving speed is measured. In the case where the center coordinates are provided as the coordinate values in the three-dimensional space, the movement vector can be calculated by simply taking the difference between the center coordinate values. The movement amount monitoring unit 112 may monitor the movement amount of the game card 4 imaged by the imaging device 2 and may monitor the movement amount in the set virtual space, or may monitor the actual movement amount.

  When determining that the moving speed of the game card 4 exceeds a predetermined reference speed, the moving amount monitoring unit 112 notifies the game progress processing unit 102 of the determination result. The moving speed of the game card 4 may be a moving speed in the virtual space of the imaged game card 4 or may be an actual moving speed of the game card 4. Upon receiving the determination result, the game progress processing unit 102 recognizes that the game card 4 has been moved quickly by the player. When this event is called a “slide event”, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of the character name and the slide event. Upon receiving the slide event occurrence notification, the display control unit 120 searches the motion pattern storage unit 122 for a motion pattern corresponding to the slide event set for the character.

  The motion pattern storage unit 122 stores not only the motion pattern in the normal operation state described above but also the motion pattern of the character when the event occurs. The action pattern storage unit 122 sets action patterns in association with characters, game stages, and internal parameters, and also in association with event names. Therefore, the display control unit 120 selects an action pattern from the action pattern storage unit 122 based on the character name, the currently playing game stage, the internal parameters of the character, and the event name, and displays the character on the display 7. Control.

  When notified of the occurrence of the slide event, the display control unit 120 reads from the operation pattern storage unit 122 and executes an action pattern that allows the character to roll on the spot without following the movement of the game card 4. When the game card 4 is moved at a high speed, the player intuitively has an image that is left behind without being able to keep up with the movement. The player selects an action pattern that embodies the image and displays it. By expressing it in the screen 7, image processing in accordance with the player's consciousness is realized.

  FIG. 5 shows the action of the character represented on the display when a slide event occurs. FIG. 5A shows a state where characters are superimposed on the game card 4 and displayed. This state corresponds to a normal operation state, and the character is display-controlled with an operation pattern according to an internal parameter or the like.

  In FIG. 5B, the game card 4 is moved leftward by the player in the screen. In addition, although the finger of the player who operates the game card 4 is displayed on the display 7, the illustration is omitted here. When the movement amount monitoring unit 112 determines that the moving speed of the game card 4 exceeds a predetermined reference speed, the game progress processing unit 102 notifies the display control unit 120 that a slide event has occurred. The display control unit 120 moves the game card 4 in the left direction on the display 7 based on the frame images periodically sent from the image analysis device 20.

  At this time, as shown in FIG. 5C, the display control unit 120 does not cause the character to follow the movement of the game card 4 but turns the character when the occurrence of the slide event is detected. That is, when a slide event occurs, the display control unit 120 stops the movement of the character on the spot so that the character does not overlap the display position of the game card 4. As a result, the display position of the game card 4 and the display position of the character are instantaneously separated.

  As shown in FIG. 5D, the display control unit 120 raises the character at a predetermined timing and moves the character to the display position of the game card 4. This timing may be, for example, the timing after the high-speed movement of the game card 4 ends, or may be the time when a predetermined time has elapsed after the occurrence of the slide event. At this time, the display control unit 120 reproduces the movement motion of the character on the display 7 using the center coordinates of the game card 4 as the movement target point. A series of character actions shown in FIGS. 5A to 5D are determined in the action pattern selected by the display control unit 120. FIG. 5E shows a state where the character has returned to the display position of the game card 4. According to the game application of the present embodiment, the player can enjoy the above-described series of three-dimensional character movements by moving the game card 4. In the game system 1, when the player tries various operation methods of the game card 4, the character can make a new movement on the display, and the interest of the game application can be enhanced.

  The movement amount monitoring unit 112 monitors not only the movement speed but also the movement direction of the game card 4 imaged by the imaging device 2. Specifically, the movement amount monitoring unit 112 stores the center coordinates of the game card 4 in a memory (not shown) for each frame, and calculates a movement vector from the difference in the center coordinates of the game card 4 between frames. Thereby, the direction of the movement vector can be detected.

  When the movement amount monitoring unit 112 compares the direction of the movement vector before and after in time and detects a state where the angle between the movement vectors is substantially 180 degrees a plurality of times within a predetermined time, the movement amount monitoring unit 112 displays the detection result as a game. The progress processing unit 102 is notified. For example, the notification condition may be that the direction of the movement vector is reversed three times in 2 seconds. Upon receiving the detection result, the game progress processing unit 102 recognizes that the game card 4 is reciprocating. When this event is called a “round trip event”, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of the character name and the round trip event. When the display control unit 120 receives the notification of the occurrence of the round-trip event, the display control unit 120 searches the motion pattern storage unit 122 for a motion pattern corresponding to the round-trip event set for the character.

  FIG. 6 shows the action of the character represented on the display when a reciprocal event occurs. FIG. 6A shows a state where the character is hit by reciprocating the game card 4. When the movement amount monitoring unit 112 determines that the game card 4 is reciprocating that satisfies a predetermined standard, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of a reciprocating event. The display control unit 120 displays the reciprocating motion of the game card 4 on the display 7 and changes the character's motion pattern based on the motion pattern retrieved from the motion pattern storage unit 122. In this example, the reciprocating motion of the game card 4 serves as a trigger for executing an action pattern of hitting the character, and the character shows a shocked expression by being hit. During the reciprocating motion of the game card 4, the character may fix the display position as long as it is within the range of the game card 4 without following the reciprocating motion of the game card 4. In addition, when the amplitude of the reciprocating motion is large and the character is out of the range of the game card 4, it is preferable to cause the character to follow the game card 4.

  FIG. 6B shows a state where the character is enlarged by reciprocating the game card 4. As described above, the motion pattern storage unit 122 may store a plurality of types of motion patterns in association with the reciprocating motion of the game card 4. As described above, the motion pattern storage unit 122 may store the motion pattern in association with the game stage, and may further store the motion pattern in association with the internal parameters of the character.

  Returning to FIG. 4, the rotation detection unit 114 detects the rotation of the game card 4. Specifically, the rotation detection unit 114 detects the rotation operation of the game card 4 based on the center coordinates and direction information included in the posture information of the game card 4. The rotation detection unit 114 stores the center coordinates and direction information of the game card 4 in a memory (not shown) for each frame. When the orientation of the game card 4 specified by the direction information changes with time on a substantial plane, and the position of the center coordinates of the game card 4 hardly shifts during the change, the rotation detection unit 114 displays the game card 4. 4 is detected to be rotated. At this time, the rotation detection condition may be that the direction of the game card 4 changes in the same rotation direction by 360 degrees or more.

  When the rotation detection unit 114 determines that the game card 4 is rotating, the rotation detection unit 114 notifies the game progress processing unit 102 of the determination result. When receiving the determination result, the game progress processing unit 102 recognizes that the game card 4 is rotated. When this event is called a “rotation event”, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of the character name and the rotation event. Upon receiving the notification of the occurrence of the rotation event, the display control unit 120 searches the operation pattern storage unit 122 for an operation pattern corresponding to the rotation event set for the character.

  The display control unit 120 selects an action pattern set for the character for the game stage currently being played. The display control unit 120 changes the character's motion pattern using the selected motion pattern, and specifically reads and executes the motion pattern that turns the character's eyes.

  FIG. 7 shows a state where the character is turning his eyes by rotating the game card. The state of turning the eyes returns to the normal state after a predetermined time has elapsed, and returns to the normal state. When the game card 4 is rotated, turning the character's eyes is the same in terms of rotation, and the player can easily understand intuitively. In the game application of the present embodiment, since the game controller is not used, it is preferable that the character motion pattern corresponding to the card operation is sensuously linked to the card operation act itself. In this way, the operability by the player can be facilitated by associating the card operation with the action pattern of the character. By determining the motion pattern of the three-dimensional character by card operation, a game application that has never existed can be realized, and a new experience can be given to the player.

  Returning to FIG. 4, the existence confirmation unit 116 confirms whether or not the game card 4 is present in the imaging region 5. The presence of the game card 4 in the imaging area 5 is determined based on whether or not the information on the game card 4 is analyzed from the image analysis device 20. When the game card 4 is hidden by the player, the image analysis device 20 cannot recognize the image of the game card 4 and the image analysis result of the game card 4 is not sent to the game device 30.

  The presence confirmation unit 116 determines that the real object is not captured by the imaging device 2 when the real object is not recognized in a predetermined number of consecutive frame images. On the other hand, the presence confirmation unit 116 determines that the real object is captured by the imaging device 2 if the frame images in which the real object is not recognized do not continue for a predetermined number or more. Here, the reason that the real object is not recognized in the predetermined number of continuous frame images is to ignore the situation where the game card 4 cannot be detected by accident due to the influence of lighting or the like.

  When the presence confirmation unit 116 determines that the game card 4 is not captured, the presence confirmation unit 116 notifies the game progress processing unit 102 of the determination result. When receiving the determination result, the game progress processing unit 102 recognizes that the game card 4 does not exist in the imaging area 5. When this event is called a “hidden event”, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of the character name and the hidden event. Note that the player can generate a hidden event by hiding the game card 4 by hand or moving the game card 4 out of the imaging area 5. Upon receiving the notification of the occurrence of the hidden event, the display control unit 120 searches the motion pattern storage unit 122 for a motion pattern corresponding to the hidden event set for the character.

  Here, the display control unit 120 erases the character from the screen of the display 7 using the selected action pattern. This is also an easy-to-understand character motion pattern for the player, and the player can intuitively operate the character without fully understanding how to play the game.

  Note that the presence confirmation unit 116 may determine that the state in which the game card 4 is captured and the state in which the game card 4 is not captured are repeated. The player creates a non-imaging state of the game card 4 by holding the hand over the game card 4 and creates an imaging state by removing the hand. When the non-imaging state and the imaging state are repeated a predetermined number of times within a predetermined time, the presence confirmation unit 116 detects a change in the imaging state and notifies the game progress processing unit 102 of the detection result. Upon receiving the determination result, the game progress processing unit 102 recognizes that the state where the game card 4 is captured by the imaging device 2 and the state where the game card 4 is not captured are alternately generated. When this event is called a “switch event”, the game progress processing unit 102 notifies the display control unit 120 of the occurrence of the character name and the switch event. When the display control unit 120 receives the notification of the occurrence of the switch event, the display control unit 120 searches the operation pattern storage unit 122 for an operation pattern corresponding to the switch event set for the character.

  Here, the display control unit 120 causes the display 7 to display a new virtual object using the selected operation pattern. This new virtual object is not displayed in a normal operation state, and the entire virtual object is newly displayed with the occurrence of a switch event as a trigger. This corresponds to the appearance of a “hidden character” in the game industry, and by introducing a new character, it is possible to change the game progress.

  In the game system 1, the player does not necessarily have to remember the motion pattern assigned to the card operation, and may move the character by operating the game card 4 in various ways. The game application of the present embodiment can provide the player with a new way of enjoying the game.

  FIG. 8 shows an image processing flow of the embodiment. In the game apparatus 30, the analysis information acquisition unit 100 acquires the identification information of the game card 4 from the image analysis apparatus 20 (S10). When the character determination unit 104 receives this identification information, the character determination unit 104 reads out the three-dimensional image data of the character associated with the current stage and the identification information from the character storage unit 106 (S12). The display control unit 120 displays the three-dimensional image data of the read character superimposed on the display position of the game card 4 on the display 7 (S14).

  The change detection unit 110 monitors the temporal state change of the game card 4 (S16). When a predetermined state change is detected (Y in S16), the display control unit 120 reads out an operation pattern associated with the state change from the operation pattern storage unit 122 (S18), and displays a character according to the operation pattern. Control (S20). When the stage continues (N in S22), the display control unit 120 restores the character display mode and causes the character to be superimposed on the game card 4. If a predetermined state change is not detected (N in S16), the superimposed display state is maintained unless the stage is switched. When the stage is switched (Y in S22), this flow ends. When the next stage starts, the three-dimensional image data of the character corresponding to the stage is read, and the above-described flow is executed.

  In the above, this invention was demonstrated based on the Example. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to the combination of each component and each processing process, and such modifications are also within the scope of the present invention. . In the embodiment, the example in which the action pattern of the character is changed has been described. For example, based on the change in the imaging state of the game card 4, a new virtual object different from the main character is generated, and the direction away from the main character It is also possible to perform display control for moving the new virtual object.

  As a modified example, the display control unit 120 displays a virtual object different from the character together with the character, and the direction of the image analysis device 20 is triggered by the change detection unit 110 detecting a change in the state of the game card 4. Another virtual object may be moved away from the character in the direction determined by the determination unit 54. Another virtual object may be an item used for game progress, for example, a virtual object such as a “ball” thrown by a character.

  FIG. 9A shows a display example in which the character throws the ball and performs bowling. As described above, the direction determination unit 54 determines the direction of the game card 4 according to the position of the direction instruction unit 11 of the game card 4 in the real space. If the display position of the virtual bowling pin on the display 7 is fixed, the player moves the game card 4 while looking at the display 7 to adjust the position and direction in which the character throws the ball. The bowling pin may be a virtual object displayed on another game card 4. The character determination unit 104 reads the three-dimensional image data of the ball from the character storage unit 106 and supplies it to the game progress processing unit 102 on the condition that the bowling pin is displayed on another game card 4. The display control unit 120 receives the three-dimensional image data of the ball from the game progress processing unit 102 and performs display control so that the character has the ball. When the character is displayed at a desired position by the card movement by the player, the player operates the game card 4 to generate an event set as a trigger for throwing the ball. This event is preferably notified to the player on the screen or through a speaker. Upon receiving the event occurrence notification, the display control unit 120 rolls the ball in the direction determined by the direction determination unit 54 and calculates the number of boring pins to fall according to the direction by a predetermined calculation. At this time, the display control unit 120 unifies the coordinate system of the boring pin and the character into the same coordinate system, and performs the contact determination between the ball that is the moving object and the boring pin, thereby enabling this display processing. Bowling is an example, and a new game story using an object other than the character can be developed by firing a virtual object from the character.

  The direction determination unit 54 can also determine the direction from the direction instruction unit 11 printed on the game card 4, but in the tilted state, a vector along the tilted inclined surface is used as the direction of the game card 4. Can also be determined.

  FIG. 9B shows another display example in which the character throws the ball and performs bowling. The direction determining unit 54 determines an inclined direction of the game card 4 in the real space. This inclination direction is determined as a direction perpendicular to one side of the game card 4 in contact with the table 3 in the table 3. Unlike the example of FIG. 9A, in this case, the direction in which the ball is thrown is determined according to the tangent line between the game card 4 and the table 3. The player places the game card 4 at a desired position and tilts it. At this time, tilting the game card 4 may be set as a trigger for throwing the ball. When the game progress processing unit 102 detects that the game card 4 is tilted from the posture information, the game progress processing unit 102 notifies the display control unit 120 to that effect, and the display control unit 120 reads out the operation pattern, and the direction determining unit 54 Roll the ball in the specified direction.

  In the above-described embodiment and modification, the character display mode is controlled based on the state change of the game card 4. In order to enhance the interest of the game application, not only the display mode of the character but also an effect such as voice may be performed. In this case, when a change in the state of the game card 4 is detected by the change detection unit 110, the game progress processing unit 102 notifies the voice control unit (not shown) to that effect, and the voice control unit transmits the auditory character of the character through the speaker. You may make a special performance. At this time, the game apparatus 30 not only functions as an image processing apparatus but also functions as an audio processing apparatus. Therefore, the game apparatus 30 can also be referred to as a processing apparatus capable of controlling both images and sounds. Note that the game apparatus 30 may perform only voice control based on the state change of the game card 4.

It is a figure which shows the structure of the game system concerning an Example. It is a figure which shows an example of the surface of a game card. It is a figure which shows the structure of an image analyzer. It is a figure which shows the structure of a game device. It is a figure which shows the operation | movement of the character expressed on a display when a slide event generate | occur | produces. It is a figure which shows the operation | movement of the character expressed on a display when a reciprocation event generate | occur | produces. It is a figure which shows the state in which the character is turning eyes. It is a flowchart of an image process. It is a figure which shows the example of a display which a character bowls.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Game system, 2 ... Imaging device, 4 ... Game card, 5 ... Imaging region, 7 ... Display, 10 ... Image processing device, 20 ... Image analysis device, DESCRIPTION OF SYMBOLS 30 ... Game device, 40 ... Frame image acquisition part, 42 ... Real object extraction part, 44 ... State determination part, 46 ... Identification information acquisition part, 48 ... Identification information storage part , 50 ... sending unit, 52 ... posture determining unit, 54 ... direction determining unit, 56 ... distance determining unit, 100 ... analysis information acquiring unit, 102 ... game progress processing unit, 104: Character determination unit, 106: Character storage unit, 110: Change detection unit, 112 ... Movement amount monitoring unit, 114 ... Rotation detection unit, 116 ... Presence check unit, 120 ... Display control unit, 122 ... Operation pattern storage unit.

Claims (7)

A storage unit that stores identification information for identifying a real object and three-dimensional image data of the virtual object in association with each other;
A reading unit that reads out three-dimensional image data of the virtual object associated with the identification information of the real object image included in the frame image captured by the imaging device from the storage unit;
A display control unit for displaying the virtual object on the display device in association with the display position of the real object using the read three-dimensional image data;
A change detection unit that detects a temporal state change of a real object image imaged by the imaging device;
With
The change detection unit includes a movement amount monitoring unit that monitors a movement amount of the real object imaged in the imaging device,
The display control unit is configured to prevent the virtual object from following the movement of the real object when the movement amount monitoring unit determines that the movement speed of the real object exceeds a predetermined reference speed. An image processing apparatus that controls a display mode of a virtual object.   The display control unit controls the display mode of the virtual object so as to stop the movement of the virtual object so that the virtual object does not overlap the display position of the real object. The image processing apparatus described.   The display control unit controls a display mode of the virtual object so that the virtual object is moved to the display position of the real object at a predetermined timing when the display position of the real object is separated from the display position of the virtual object. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image processing apparatus. A storage unit that stores identification information for identifying a real object and three-dimensional image data of the virtual object in association with each other;
A reading unit that reads out three-dimensional image data of the virtual object associated with the identification information of the real object image included in the frame image captured by the imaging device from the storage unit;
A display control unit for displaying the virtual object on the display device in association with the display position of the real object using the read three-dimensional image data;
A change detection unit that detects a temporal state change of a real object image imaged by the imaging device;
With
The change detection unit includes a presence check unit that determines that the real object is not captured by the imaging device when the real object is not recognized in a predetermined number of consecutive frame images.
The display control unit adds a new virtual object to the display device when the presence check unit determines that the state where the real object is captured and the state where the real object is not captured are alternately repeated. An image processing apparatus that controls a display mode of the virtual object so as to be displayed. Corresponding to the identification information of the real object image included in the frame image captured by the imaging device from the storage unit storing the identification information for identifying the real object and the three-dimensional image data of the virtual object in association with each other Reading the three-dimensional image data of the virtual object;
Using the read three-dimensional image data, causing the display unit to display the virtual object in association with the display position of the real object;
Detecting a temporal state change of a real object image captured by the imaging device;
With
The step of detecting the state change monitors a moving amount of the real object imaged in the imaging device,
The step of displaying on the display device controls the display mode so that the virtual object does not follow the movement of the real object when it is determined that the movement speed of the real object exceeds a predetermined reference speed. An image processing method. Corresponding to the identification information of the real object image included in the frame image captured by the imaging device from the storage unit storing the identification information for identifying the real object and the three-dimensional image data of the virtual object in association with each other A function for reading the three-dimensional image data of the virtual object;
A function of displaying the virtual object on the display device in association with the display position of the real object using the read three-dimensional image data;
A function of detecting a temporal state change of a real object image captured by the imaging device;
To the computer,
The function of detecting the state change monitors the movement amount of the real object imaged in the imaging device,
The function displayed on the display device controls the display mode so that the virtual object does not follow the movement of the real object when it is determined that the movement speed of the real object exceeds a predetermined reference speed. A program characterized by that. A computer-readable recording medium on which the program according to claim 6 is recorded.

Images