JP2011096227A - Program, device, system and method of image recognition - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To shorten a time required for detection while maintaining the detection accuracy of an object in image recognition processing. <P>SOLUTION: A game device 1 acquires a captured image captured by a camera. The game device 1 first detects an object area including a predetermined imaging object in the captured image on the basis of pixel values obtained at a first interval in the captured image. The game device 1 then detects the predetermined imaging object from the image in the object area on the basis of pixel values obtained at a second interval smaller than the first interval in the captured image within the object area. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image recognition program, an image recognition apparatus, an image recognition system, and an image recognition method for recognizing a predetermined imaging target from an image captured by an imaging unit.

  Conventionally, there is a technique for detecting a predetermined imaging target from an image (captured image) captured by an imaging unit such as a camera. For example, Non-Patent Document 1 describes performing an image recognition process on a marker included in an image captured by a camera in augmented reality technology. In Non-Patent Document 1, binarization is performed on a captured image using a fixed threshold value to extract a connected region, and an appropriate size and shape are selected from the extracted connected regions and marker candidates are selected. And detecting a marker by performing pattern matching on the marker candidate.

Hirokazu Kato, MarkBillinghurst, Koichi Asano, Keihachiro Tachibana, “Augmented Reality System Based on Marker Tracking and Its Calibration”, Transactions of the Virtual Reality Society of Japan, Vol. 4 No. 4, 1999

  In the recognition method described in Non-Patent Document 1, image recognition processing is performed in units of pixels of an input captured image as they are. Therefore, when the input image becomes high resolution (high accuracy), there is a problem that the time required for the image recognition processing becomes long. On the other hand, if image recognition processing is simply performed with coarser accuracy (in coarser pixel units) than the pixel unit of the input captured image, there is a problem that the marker detection accuracy is lowered.

  Therefore, an object of the present invention is to provide an image recognition program, an image recognition apparatus, an image recognition system, and an image recognition method capable of reducing the time required for detection while maintaining the detection accuracy of an object. .

  The present invention employs the following configurations (1) to (14) in order to solve the above problems.

(1)
The present invention is an image recognition program that is executed in a computer of an information processing apparatus that detects a predetermined imaging target included in a captured image captured by an imaging unit. The image recognition program causes the computer to function as image acquisition means, area detection means, and object detection means. The image acquisition unit acquires a captured image captured by the imaging unit. The area detecting unit detects a target area including a predetermined imaging target in the captured image based on the pixel values obtained at the first interval in the captured image. The target detection unit detects a predetermined imaging target from the image of the target region based on pixel values obtained at a second interval smaller than the first interval in the captured image in the target region.

The “imaging means” may be provided in advance in the information processing apparatus, or may be configured separately from the information processing apparatus. Further, when the information processing means includes the imaging means in advance, the imaging means may be fixedly attached to the information processing apparatus or may be detachable from the treasure processing apparatus. The imaging unit may operate in conjunction with the operation of the information processing apparatus, or may operate independently of the operation of the information processing apparatus.
In the embodiment described later, the “predetermined imaging target” is the first graphic 55 included in the marker 51. However, any object can be used as long as it can be recognized by the recognition process. Good. For example, the “predetermined imaging target” may be a part of the face or body of the user (player).
The “information processing apparatus” is a concept including an arbitrary computer that performs information processing by executing a computer program, in addition to the game apparatus described in the embodiments described later. Further, it does not matter whether the “information processing apparatus” is portable.
The “image recognition program” is, for example, a game program described in an embodiment described later, but includes a concept including an application program executed on a personal computer or a portable terminal.
The “image acquisition unit” may acquire a captured image from an imaging unit provided in the information processing apparatus via an internal bus or the like, or acquire a captured image by communication from an imaging unit outside the information processing apparatus. Alternatively, the captured image may be acquired from the storage medium by attaching the storage medium storing the captured image captured by the captured image to the information processing apparatus.
The “pixel value” may be any information as long as it is a value indicating the color, brightness, or the like set for the pixel.
The “region detection means” may be any device that detects a target region including a predetermined imaging target, and any specific method for detecting the target region may be used.
The “second interval” may be any length as long as it is smaller than the “first interval”. Further, as in the embodiment described later, when a plurality of types of processing are executed in the processing for detecting the imaging target, at least one of the processing is performed based on the pixel value obtained at the second interval. That's fine.
The “target detection means” may be any means that detects a predetermined imaging target from the target region, and any specific method for detecting the predetermined imaging target may be used.
The above-mentioned “detecting a predetermined imaging target” refers to imaging in addition to calculating the position (and / or orientation) of the predetermined imaging target in the captured image as in the embodiments described later and (2) below. This is a concept including determining whether or not a predetermined imaging target exists in an image (detecting the presence of a predetermined imaging target). That is, the detection result of the target detection unit may be information on the position and / or orientation of a predetermined imaging target in the captured image, or information indicating whether or not the imaging target exists in the captured image. May be.

  According to the configuration of (1) above, after the target region is first detected with low accuracy by the region detection means, the process of detecting a predetermined imaging target from the target region is performed with high accuracy. According to this, the detection process of the target area is performed with low accuracy, and the processing range of the process for detecting the predetermined imaging target is limited to the target area, so that the recognition process can be performed in a short time. In addition, since the process for detecting the predetermined imaging target is performed with high accuracy, the predetermined imaging target can be detected with high accuracy. Therefore, according to the configuration of (1) above, it is possible to provide a recognition process that can reduce the time required for detection while maintaining the detection accuracy of the target.

(2)
The target detection unit may detect the imaging target by calculating a position of a predetermined imaging target in the captured image based on pixel values obtained at the second interval.

  According to the configuration of (2) above, the position of the predetermined imaging target in the captured image can be accurately calculated with high accuracy. As in the configuration (2), when the position is calculated as the detection result of the imaging target, the accuracy of the calculated position and the time required for the calculation are important. Therefore, when the position of a predetermined imaging target is calculated as in the configuration (2) above, the present invention that can reduce the time required for detection while maintaining the detection accuracy is particularly effective.

(3)
The predetermined imaging target may include a figure in which a plurality of internal figures are drawn. At this time, the target detection means detects a plurality of internal figures from the image of the target area, and calculates the center position of each internal figure based on the pixel values obtained at the second interval, thereby obtaining a predetermined imaging target. The position of is detected.

The “internal figure” may be a figure drawn inside a predetermined imaging target, and in the embodiment described later, it is a plurality of white circles (FIG. 6), but its shape, number, size, color May be anything.
The “center position of the internal figure” does not have to be the exact center position of the internal figure, and may be any position near the center of the internal figure that can be calculated by various calculations. . For example, the “center position of the internal graphic” may be calculated as an average value of pixels representing the outline of the internal graphic, or when some weight is given to each pixel of the captured image, May be calculated as the barycentric position of each pixel.

  According to the configuration of (3) above, the position of the predetermined imaging target is detected not by the contour of the predetermined imaging target but by the center position of the internal figure. Here, when the captured image is blurred, it is difficult to accurately detect the contour of the predetermined imaging target. Therefore, the position of the predetermined imaging target cannot be accurately detected from the contour. On the other hand, the center position is accurately calculated even when the captured image is blurred. Therefore, according to the configuration of (3) above, even when the captured image is blurred, a predetermined imaging is performed. The position of the target can be calculated accurately.

(4)
The image recognition program may further cause the computer to function as a contour extracting unit that extracts a contour of a predetermined imaging target in the target region detected by the region detecting unit. At this time, the object detection unit detects a plurality of internal figures from the region in the contour extracted by the contour extraction unit.

  According to the configuration of (4) above, since the internal graphic is detected from the area within the contour extracted by the contour extracting means, the range for detecting the internal graphic can be appropriately determined. According to this, since it is not necessary to perform the internal graphic detection process on the useless area, the recognition process can be performed more efficiently.

(5)
The contour extracting unit may extract a contour of a predetermined imaging target based on pixel values obtained at a third interval that is smaller than the first interval and larger than the second interval in the captured image.

  According to the configuration of (5) above, the contour extraction process is performed with lower accuracy than the internal graphic detection process, and therefore the contour extraction process can be performed in a shorter time. As a result, the recognition process can be performed in a shorter time.

(6)
The image recognition program may cause the computer to function as an object determination unit that determines whether the contour extracted by the contour extraction unit represents a predetermined imaging target. At this time, the target detection unit detects a plurality of internal figures from the area corresponding to the contour determined to represent the predetermined imaging target by the target determination unit.

  According to the configuration of (6) above, it is possible to determine whether or not a predetermined imaging target is included in the target region based on the extracted contour. When a predetermined imaging target is not included in the target area, the internal graphic detection process is not performed for the target area. Therefore, according to the configuration of (6) above, since the target area that does not include the predetermined imaging target can be excluded from the target of the internal graphic detection process, the internal graphic detection process is prevented from being wasted. can do. According to this, the recognition process can be performed efficiently, and the recognition process can be performed in a shorter time. In particular, when the configurations of (5) and (6) above are combined, the contour extraction processing performed at a relatively high speed (because it is performed at a low accuracy) causes a relative (because it is performed at a high accuracy). Therefore, it is possible to omit the time-consuming process of detecting the internal figure, so that the recognition process can be performed more efficiently.

(7)
The object detection unit may include an internal region detection unit and a position calculation unit. The internal area detection means detects a plurality of internal graphic areas in the target area. The position calculation means calculates the center position of each area detected by the internal area detection means.

  With configuration (7) above, the center position of the internal graphic can be easily calculated by detecting the area of the internal graphic and calculating the center position of the detected area.

(8)
The internal region detection means may detect a plurality of internal graphic regions based on pixel values obtained at a fourth interval smaller than the first interval and larger than the second interval in the captured image.

  According to the configuration of (8), the internal graphic area detection process is performed with a lower accuracy than the center position calculation process, and therefore the internal graphic area detection process can be performed in a shorter time. As a result, the recognition process can be performed in a shorter time.

(9)
The object detection unit may further include a number determination unit that determines whether or not the number of regions detected by the internal region detection unit is a number within a predetermined range. At this time, the position calculation means calculates the center position of each area detected by the internal area detection means for the target area where the number of areas within the predetermined range is detected.

  With configuration (9) above, it is possible to determine whether or not the number of regions detected by the internal region detection means is an appropriate number. If the number of regions is not appropriate, it is determined that the target region does not include a predetermined imaging target, and the center position calculation process is not performed. Therefore, according to the configuration of (9) above, it is possible to exclude a target area that does not include a predetermined imaging target from the center position calculation processing target, thereby preventing the calculation processing from being performed wastefully. Can do. According to this, the recognition process can be performed efficiently, and the recognition process can be performed in a shorter time. In particular, in the case of combining the configurations of (8) and (9) above (because it is performed with low accuracy), it is performed with high accuracy (by being performed with high accuracy) by the detection processing of the area of the internal figure performed at a relatively high speed. ) Since the calculation process of the center position which requires a relatively long time can be omitted, the recognition process can be performed more efficiently.

(10)
The area detecting means may detect the target area by extracting a predetermined contour of the imaging target based on a pixel value between adjacent pixels at the first interval.

  According to the configuration of (10) above, the target region can be easily detected by extracting the contour of the predetermined imaging target.

(11)
The region detection unit may detect the target region based on pixel values relating to pixels arranged at the first interval among the pixels of the captured image acquired by the image acquisition unit. At this time, the target detection unit detects a predetermined imaging target based on pixel values relating to pixels arranged at the second interval among the pixels of the captured image acquired by the image acquisition unit.

  According to the configuration of (11) above, the same captured image can be used in the target region detection processing and the predetermined imaging target detection processing. According to this, since it is not necessary to generate a low-resolution image, the recognition process can be performed more easily and in a shorter time.

(12)
The computer may further function as a pattern acquisition unit that acquires a pattern image representing a predetermined imaging target from a storage unit accessible by the information processing apparatus. At this time, the target detection unit determines whether or not a predetermined imaging target is included in the target region by comparing the image of the target region with the pattern image.

  With configuration (12) above, it is possible to easily detect a predetermined imaging target by a so-called pattern matching technique.

(13)
The target detection means extracts a contour of a predetermined imaging target from the image of the target region based on the pixel value obtained at the second interval, and based on the extracted contour, the position of the predetermined imaging target in the captured image May be calculated.

  With configuration (13) above, it is possible to easily calculate the position of the predetermined imaging target in the captured image based on the contour of the predetermined imaging target.

(14)
The image recognition program may further cause the computer to function as a positional relationship calculation unit and a display control unit. The positional relationship calculation unit calculates the positional relationship between the predetermined imaging target and the imaging unit based on the detection result of the target detection unit when a predetermined imaging target is detected by the target detection unit. The display control unit generates a virtual image representing a virtual object arranged in the virtual space based on the positional relationship, and causes the display unit to display an image obtained by synthesizing the virtual image with the captured image.

  With configuration (14) above, high-speed and accurate recognition processing can be used in augmented reality technology that displays a composite image obtained by combining a captured image and a virtual image. In augmented reality technology, in order to accurately synthesize a captured image and a virtual image without shifting the position, it is required to accurately detect a predetermined imaging target in the recognition process. Therefore, a high-speed and accurate recognition process with the configuration (1) is particularly effective.

  Further, the present invention may be implemented in the form of an image recognition apparatus that includes means equivalent to the above-described means. In this image recognition apparatus, each means described above may be realized by a computer that executes an image recognition program, or a part or all of each means described above may be realized by a dedicated circuit. Further, the present invention may be implemented in the form of an image recognition system including one or more information processing apparatuses including the above-described units. At this time, the one or more information processing apparatuses may perform direct communication via wired or wireless communication, or may perform communication via a network. Furthermore, the present invention may be implemented in the form of an image recognition method performed by each of the above means.

  According to the present invention, first, a target region is detected with low accuracy by the region detection means, and then processing for detecting a predetermined imaging target from the target region is performed with high accuracy, thereby detecting the target while maintaining the detection accuracy. Can be shortened.

External view of game device according to this embodiment Block diagram showing an example of the internal configuration of the game device The figure which shows the example which uses a game device The figure which shows an example of the game image displayed on a game device The figure which shows the object area | region used by the image recognition process in this embodiment. The figure which shows the marker used in this embodiment The figure which shows the image of the 1st figure with which the right-and-left outline was blurred The figure which shows the center position of the internal figure in a part of 1st figure The figure which shows the center point of each internal figure in the 1st figure The figure which shows the various data which are used in the processing with the game program Main flowchart showing the flow of game processing executed in the game device The flowchart which shows the flow of the recognition process (step S2) shown in FIG. The figure which shows the pixel to which a pixel value is referred in the process of step S11 in a captured image. The flowchart which shows the flow of the marker detection process (step S15) shown in FIG. The figure which shows an example of the area | region used as the process target of step S21. The figure which shows the pixel to which a pixel value is referred in the process of step S21 in a captured image. The figure which shows an example of the captured image showing a 1st figure The figure which shows the four vertexes P11-P14 calculated by each straight line L1-L4 shown in FIG. The figure which shows an example of the predetermined area | region set by step S27 The figure which shows the marker used by the modification in this embodiment The flowchart which shows the flow of the marker detection process in the modification of this embodiment The figure which shows the marker in the modification of this embodiment The figure which shows the marker in the other modification of this embodiment

[Hardware configuration of game device]
Hereinafter, an image recognition program and an image recognition apparatus according to an embodiment of the present invention will be described with reference to the drawings. The present invention can be realized by executing an image recognition program in an arbitrary information processing apparatus (computer) that displays an image on a display device. In the present embodiment, FIG. The case where the game apparatus 1 shown is used will be described.

  FIG. 1 is an external view of a game apparatus 1 according to the present embodiment. Here, a portable game device is shown as an example of the game device. Note that the game apparatus 1 has a built-in camera, and functions as an imaging apparatus that captures an image with the camera, displays the captured image on a screen, and stores data of the captured image.

  In FIG. 1, a game apparatus 1 is a foldable portable game apparatus, and shows the game apparatus 1 in an open state (open state). The game apparatus 1 is configured in such a size that the user can hold it with both hands or one hand even in an open state.

  The game apparatus 1 includes a lower housing 11 and an upper housing 21. The lower housing 11 and the upper housing 21 are connected so as to be openable and closable (foldable). In the example of FIG. 1, the lower housing 11 and the upper housing 21 are each formed in a horizontally-long rectangular plate shape, and are coupled so as to be rotatable at their long side portions. Normally, the user uses the game apparatus 1 in the open state. Further, when the user does not use the game apparatus 1, the user stores the game apparatus 1 in a closed state. In the example shown in FIG. 1, the game apparatus 1 is not limited to the closed state and the open state, and the angle formed by the lower housing 11 and the upper housing 21 is an arbitrary angle between the closed state and the open state. The opening / closing angle can be maintained by a frictional force generated in the connecting portion. That is, the upper housing 21 can be made stationary with respect to the lower housing 11 at an arbitrary angle.

  The lower housing 11 is provided with a lower LCD (Liquid Crystal Display) 12. The lower LCD 12 has a horizontally long shape, and is arranged such that the long side direction coincides with the long side direction of the lower housing 11. In the present embodiment, an LCD is used as the display device built in the game apparatus 1, but other arbitrary display devices such as a display device using EL (Electro Luminescence) are used. May be. The game apparatus 1 can use a display device having an arbitrary resolution. Note that an image captured by the inner camera 23 or the outer camera 25 is displayed on the lower LCD 12 in real time.

  The lower housing 11 is provided with operation buttons 14A to 14K and a touch panel 13 as input devices. As shown in FIG. 1, among the operation buttons 14A to 14K, a direction input button 14A, an operation button 14B, an operation button 14C, an operation button 14D, an operation button 14E, a power button 14F, a start button 14G, and a select button 14H. Is provided on the inner main surface of the lower housing 11. The inner main surface is a surface that becomes the inner side when the upper housing 21 and the lower housing 11 are folded. In the example shown in FIG. 1, the direction input button 14 </ b> A and the power button 14 </ b> F are on the left and right sides (left side in FIG. 1) of the main LCD 14 with respect to the lower LCD 12 provided near the center of the inner main surface of the lower housing 11. Provided on the surface. In addition, the operation buttons 14B to 14E, the start button 14G, and the select button 14H are provided on the inner main surface of the lower housing 11 that is on the other side of the lower LCD 12 (right side in FIG. 1). The direction input button 14A, the operation buttons 14B to 14E, the start button 14G, and the select button 14H are used for performing various operations on the game apparatus 1. The direction input button 14A is used for, for example, a selection operation. The operation buttons 14B to 14E are used for, for example, a determination operation or a cancel operation. The power button 14F is used for turning on / off the power of the game apparatus 1.

  In FIG. 1, the operation buttons 14I to 14K are not shown. For example, the L button 14I is provided at the left end portion of the upper side surface of the lower housing 11, and the R button 14J is provided at the right end portion of the upper side surface of the lower housing 11. The L button 14I and the R button 14J are used, for example, to perform a shooting instruction operation (shutter operation) on the game apparatus 1 having a shooting function. Furthermore, the volume button 14K is provided on the left side surface of the lower housing 11. The volume button 14K is used to adjust the volume of the speaker provided in the game apparatus 1.

  In addition, the game apparatus 1 includes a touch panel 13 as an example of a pointing device that is an input device that can specify an arbitrary position on the screen as an input device different from the operation buttons 14A to 14K. The touch panel 13 is mounted so as to cover the screen of the lower LCD 12. In the present embodiment, for example, a resistive film type touch panel is used as the touch panel 13. However, the touch panel 13 is not limited to the resistive film type, and any type of touch panel can be used. In the present embodiment, the touch panel 13 having the same resolution (detection accuracy) as that of the lower LCD 12 is used, for example. However, the resolution of the touch panel 13 and the resolution of the lower LCD 12 are not necessarily matched. An insertion port (shown by a broken line in FIG. 1) is provided on the right side surface of the lower housing 11. The insertion opening can accommodate a touch pen 27 used for performing an operation on the touch panel 13. In addition, although the input (touch input) with respect to the touch panel 13 is normally performed using the touch pen 27, not only the touch pen 27 but the touch panel 13 can also be operated with a user's finger | toe.

  Further, on the right side surface of the lower housing 11, an insertion slot (indicated by a two-dot chain line in FIG. 1) for storing the memory card 28 is provided. A connector (not shown) for electrically connecting the game apparatus 1 and the memory card 28 is provided inside the insertion slot. The memory card 28 is, for example, an SD (Secure Digital) memory card, and is detachably attached to the connector. The memory card 28 is used, for example, for storing (saving) an image captured by the game apparatus 1 or reading an image generated by another apparatus into the game apparatus 1.

  Further, an insertion slot (indicated by a one-dot chain line in FIG. 1) for accommodating the memory card 29 is provided on the upper side surface of the lower housing 11. A connector (not shown) for electrically connecting the game apparatus 1 and the memory card 29 is also provided inside the insertion slot. The memory card 29 is a recording medium on which an information processing program such as a game program is recorded, and is detachably attached to an insertion port provided in the lower housing 11.

  Three LEDs 15 </ b> A to 15 </ b> C are attached to the left portion of the connecting portion between the lower housing 11 and the upper housing 21. Here, the game apparatus 1 can perform wireless communication with other devices, and the first LED 15A is lit when the power of the game apparatus 1 is on. The second LED 15B is lit while the game apparatus 1 is being charged. The third LED 15C is lit when wireless communication is established. Therefore, the three LEDs 15A to 15C can notify the user of the power on / off status of the game apparatus 1, the charging status, and the communication establishment status.

  On the other hand, the upper housing 21 is provided with an upper LCD 22. The upper LCD 22 has a horizontally long shape and is arranged such that the long side direction coincides with the long side direction of the upper housing 21. As in the case of the lower LCD 12, instead of the upper LCD 22, a display device having any other method and any resolution may be used. A touch panel may be provided so as to cover the upper LCD 22. On the upper LCD 22, for example, an operation explanation screen for teaching the user the roles of the operation buttons 14A to 14K and the touch panel 13 is displayed.

  The upper housing 21 is provided with two cameras (an inner camera 23 and an outer camera 25). As shown in FIG. 1, the inner camera 23 is attached to the inner main surface near the connecting portion of the upper housing 21. On the other hand, the outer camera 25 is a surface on the opposite side of the inner main surface to which the inner camera 23 is attached, that is, the outer main surface of the upper housing 21 (the outer surface when the game apparatus 1 is in a closed state, It is attached to the rear surface of the upper housing 21 shown in FIG. In FIG. 1, the outer camera 25 is indicated by a broken line. As a result, the inner camera 23 can capture an image in the direction in which the inner main surface of the upper housing 21 faces, and the outer camera 25 can rotate in the direction opposite to the image capturing direction of the inner camera 23, that is, the outer main surface of the upper housing 21. It is possible to image the direction in which the surface faces. Thus, in the present embodiment, the two inner cameras 23 and the outer cameras 25 are provided so that the imaging directions are opposite to each other. For example, the user can take an image of a scene viewed from the game apparatus 1 with the inner camera 23 and can also capture an image of the scene viewed from the game apparatus 1 on the opposite side of the user with the outer camera 25. Can do.

  Note that a microphone (a microphone 43 shown in FIG. 2) is accommodated as an audio input device on the inner main surface near the connecting portion. A microphone hole 16 is formed on the inner main surface near the connecting portion so that the microphone 43 can detect the sound outside the game apparatus 1. The position where the microphone 43 is housed and the position of the microphone hole 16 do not necessarily have to be the connecting portion. For example, the microphone 43 is housed in the lower housing 11, and A microphone hole 16 may be provided.

  A fourth LED 26 (shown by a broken line in FIG. 1) is attached to the outer main surface of the upper housing 21. The fourth LED 26 is turned on at the time when photographing is performed by the inner camera 23 or the outer camera 25 (the shutter button is pressed). Further, it is lit while a moving image is shot by the inner camera 23 or the outer camera 25. The fourth LED 26 can notify the subject to be photographed and the surroundings that photographing by the game apparatus 1 has been performed (performed).

  In addition, with respect to the upper LCD 22 provided near the center of the inner main surface of the upper housing 21, sound release holes 24 are formed in the main surfaces on both the left and right sides. A speaker is housed in the upper housing 21 behind the sound release hole 24. The sound release hole 24 is a hole for releasing sound from the speaker to the outside of the game apparatus 1.

  As described above, the upper housing 21 includes the inner camera 23 and the outer camera 25 that are imaging means for capturing an image, and the upper LCD 22 that is a display means for mainly displaying an operation explanation screen. Provided. On the other hand, the lower housing 11 has an input device (touch panel 13 and operation buttons 14A to 14K) for performing an operation input to the game apparatus 1 and a lower LCD 12 which is a display means for displaying a captured image. And are provided. Therefore, when using the game apparatus 1, the user grasps the lower housing 11 and performs input to the input device while viewing the captured image (image captured by the camera) displayed on the lower LCD 12. be able to.

  Next, the internal configuration of the game apparatus 1 will be described with reference to FIG. FIG. 2 is a block diagram illustrating an example of the internal configuration of the game apparatus 1.

  2, the game apparatus 1 includes a CPU 31, a main memory 32, a memory control circuit 33, a storage data memory 34, a preset data memory 35, a memory card interface (memory card I / F) 36, a memory card I / F 37, Wireless communication module 38, local communication module 39, real-time clock (RTC) 40, power supply circuit 41, interface circuit (I / F circuit) 42, first GPU (Graphics Processing Unit) 45, second GPU 46, first VRAM (Video RAM) 47 The second VRAM 48 and the LCD controller 49 are provided. These electronic components are mounted on an electronic circuit board and housed in the lower housing 11 (or the upper housing 21).

  The CPU 31 is information processing means for executing a predetermined program (here, the image display program according to the present embodiment). In the present embodiment, as an example of the image display program, a game program is stored in a memory (for example, the storage data memory 34) or the memory card 28 and / or 29 in the game apparatus 1, and the CPU 31 executes the game program. By doing so, a game process to be described later is executed. Note that the program executed by the CPU 31 may be stored in advance in a memory in the game apparatus 1, may be acquired from the memory card 28 and / or 29, or may be obtained by communication with other devices. It may be acquired from the device.

  A main memory 32, a memory control circuit 33, and a preset data memory 35 are connected to the CPU 31. A storage data memory 34 is connected to the memory control circuit 33. The main memory 32 is a storage means used as a work area or buffer area for the CPU 31. That is, the main memory 32 stores various data used for the game processing, and stores programs acquired from the outside (memory cards 28 and 29, other devices, etc.). In the present embodiment, for example, a PSRAM (Pseudo-SRAM) is used as the main memory 32. The storage data memory 34 is a storage unit for storing a program executed by the CPU 31, data of images taken by the inner camera 23 and the outer camera 25, and the like. The storage data memory 34 is configured by a nonvolatile storage medium, and is configured by, for example, a NAND flash memory in the present embodiment. The memory control circuit 33 is a circuit that controls reading and writing of data with respect to the storage data memory 34 in accordance with instructions from the CPU 31. The preset data memory 35 is storage means for storing data (preset data) such as various parameters set in advance in the game apparatus 1. As the preset data memory 35, a flash memory connected to the CPU 31 via an SPI (Serial Peripheral Interface) bus can be used.

  Memory card I / Fs 36 and 37 are each connected to CPU 31. The memory card I / F 36 reads and writes data to and from the memory card 28 attached to the connector according to instructions from the CPU 31. The memory card I / F 37 reads and writes data to and from the memory card 29 attached to the connector in accordance with instructions from the CPU 31. In the present embodiment, image data captured by the inner camera 23 and the outer camera 25 and image data received from other devices are written to the memory card 28, or image data stored in the memory card 28 is stored in the memory card 28. And stored in the storage data memory 34, or the read image data is read from the memory card 28 and transmitted to another device. Various programs stored in the memory card 29 are read out and executed by the CPU 31.

  Note that the game program may be supplied not only to the computer system through an external storage medium such as the memory card 29 but also to the computer system through a wired or wireless communication line. The game program may be recorded in advance in a nonvolatile storage device inside the computer system. The information storage medium for storing the game program is not limited to the above-described nonvolatile storage device, and may be a CD-ROM, a DVD, or an optical disk storage medium similar to them.

  The wireless communication module 38 is, for example, IEEE 802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. Further, the local communication module 39 has a function of performing wireless communication with the same type of game device by a predetermined communication method. The wireless communication module 38 and the local communication module 39 are connected to the CPU 31. The CPU 31 transmits / receives data to / from other devices via the Internet using the wireless communication module 38, and transmits / receives data to / from other game devices of the same type using the local communication module 39. be able to.

  Further, the RTC 40 and the power supply circuit 41 are connected to the CPU 31. The RTC 40 counts the time and outputs it to the CPU 31. For example, the CPU 31 can calculate the current time (date) based on the time counted by the RTC 40. The power supply circuit 41 controls power supplied from a power supply (typically a battery, which is stored in the lower housing 11) of the game apparatus 1, and supplies power to each component of the game apparatus 1.

  Further, the game apparatus 1 includes a microphone 43 and an amplifier 44. The microphone 43 and the amplifier 44 are each connected to the I / F circuit 42. The microphone 43 detects the voice of the user uttered toward the game apparatus 1 and outputs a voice signal indicating the voice to the I / F circuit 42. The amplifier 44 amplifies the audio signal from the I / F circuit 42 and outputs it from a speaker (not shown). The I / F circuit 42 is connected to the CPU 31.

  The touch panel 13 is connected to the I / F circuit 42. The I / F circuit 42 includes a sound control circuit that controls the microphone 43 and the amplifier 44 (speaker), and a touch panel control circuit that controls the touch panel 13. The voice control circuit performs A / D conversion and D / A conversion on the voice signal, or converts the voice signal into voice data of a predetermined format. The touch panel control circuit generates touch position data (detected coordinate data described later) in a predetermined format based on a signal from the touch panel 13 and outputs it to the CPU 31. The touch position data is data indicating the coordinates of the position detected by the touch panel 13 as the position where the input is performed on the input surface of the touch panel 13. The touch panel control circuit repeatedly reads signals from the touch panel 13 and generates detected coordinate data at a rate of once per predetermined time.

  The operation button 14 includes the operation buttons 14A to 14K and is connected to the CPU 31. From the operation button 14 to the CPU 31, operation data indicating an input status (whether or not the button is pressed) for each of the operation buttons 14A to 14K is output. The CPU 31 obtains operation data from the operation button 14 to execute processing corresponding to the input to the operation button 14.

  The inner camera 23 and the outer camera 25 are each connected to the CPU 31. The inner camera 23 and the outer camera 25 capture an image in accordance with an instruction from the CPU 31 and output the captured image data to the CPU 31. In the present embodiment, the CPU 31 issues an imaging instruction to either the inner camera 23 or the outer camera 25, and the camera that has received the imaging instruction captures an image and sends the image data to the CPU 31. Note that the inner camera 23 and the outer camera 25 can also shoot moving images. That is, the inner camera 23 and the outer camera 25 can repeatedly perform imaging and repeatedly transmit imaging data to the CPU 31.

  A first VRAM 47 is connected to the first GPU 45, and a second VRAM 48 is connected to the second GPU 46. In response to an instruction from the CPU 31, the first GPU 45 generates a first display image based on data for generating a display image stored in the main memory 32 and draws the first display image in the first VRAM 47. Similar to the first GPU 45, the second GPU 46 generates a second display image in accordance with an instruction from the CPU 31 and draws it in the second VRAM 48. The first VRAM 47 and the second VRAM 48 are connected to the LCD controller 49.

  The LCD controller 49 includes a register 491. The register 491 stores a value of 0 or 1 according to an instruction from the CPU 31. When the value of the register 491 is 0, the LCD controller 49 outputs the first display image drawn in the first VRAM 47 to the lower LCD 12 and outputs the second display image drawn in the second VRAM 48 to the upper LCD 22. To do. If the value of the register 491 is 1, the first display image drawn in the first VRAM 47 is output to the upper LCD 22, and the second display image drawn in the second VRAM 48 is output to the lower LCD 12. For example, the CPU 31 can cause the lower LCD 12 to display an image acquired from either the inner camera 23 or the outer camera 25 and cause the upper LCD 22 to display an operation explanation screen generated by a predetermined process.

[Overview of game processing]
Next, with reference to FIG. 3 to FIG. 9, the game process performed during the game process executed by the game program will be described. The game program performs image recognition processing on the real space image (captured image) captured by the camera, and synthesizes the virtual space image (virtual image) with the captured image based on the result of the image recognition processing. The game image is displayed and the player is allowed to play the game.

  FIG. 3 is a diagram illustrating an example of using a game device. In the present embodiment, as shown in FIG. 3, the player (user) installs the marker 51 as a predetermined imaging target at an arbitrary location (on the desk 50 in FIG. 3), and the game apparatus 1 The marker 51 and its surroundings are imaged using it. The camera used for imaging may be either the inner camera 23 or the outer camera 25. Here, the case where the outer camera 25 is used will be described as an example. In this embodiment, the marker 51 is a thin plate on which a predetermined pattern (see FIG. 6) is drawn.

  FIG. 4 is a diagram illustrating an example of a game image displayed on the game device. FIG. 4 shows a game image displayed on the upper LCD 22 of the game apparatus 1 when the game apparatus 1 images the marker 51 placed on the desk 50 as shown in FIG. In this embodiment, the game image is displayed on the upper LCD 22, but the game image may be displayed on the lower LCD 12. As shown in FIG. 4, an image of the cannon 52 is combined with a captured image and displayed on the upper LCD 22 as a virtual object image (virtual image) in the virtual space. Thus, an image as if the cannon 52 is present on the actual desk 50 can be displayed. In FIG. 4, an image of the cannon 52 is displayed as an example of the virtual object, but the virtual object may be anything. Thus, in this embodiment, a game image as if a virtual object exists in the real space is displayed using augmented reality technology.

  Note that an image obtained by combining a virtual image with a captured image (synthesized image) can be generated by, for example, the following process. That is, the game apparatus 1 first executes (image) recognition processing for recognizing the marker 51 included therein from the image captured by the camera. Here, information indicating the position and orientation of the marker 51 in the captured image is calculated as the processing result of the recognition process. When the marker 51 is recognized, the game apparatus 1 calculates the positional relationship between the game apparatus 1 (camera) and the marker 51 from the position and orientation of the marker 51 that is the processing result of the recognition process. Furthermore, the game apparatus 1 calculates the position and orientation of the virtual camera in the virtual space based on the positional relationship. The position and orientation of the virtual camera are calculated so that the positional relationship between the virtual camera and the virtual object in the virtual space matches the positional relationship between the game apparatus 1 and the marker 51 in the real space. When the position of the virtual camera is determined, the game apparatus 1 generates a virtual image in which the virtual object is viewed from the position of the virtual camera, and synthesizes the virtual image with the captured image. Through the above processing, the game apparatus 1 can generate and display a composite image. Note that the process of calculating the position of the virtual camera from the above positional relationship may be the same as the process in the conventional augmented reality technology.

[Overview of image recognition processing]
As described above, in the present embodiment, the game apparatus 1 detects the marker 51 from the captured image by the recognition process. If the marker 51 cannot be accurately detected in the recognition process, the virtual image cannot be generated accurately, and the virtual image cannot be synthesized at the correct position on the captured image. Reality will fade. Therefore, in the present embodiment, the game apparatus 1 executes the following recognition process in order to detect the marker 51 accurately and efficiently.

(Recognition processing using multiple types of accuracy)
FIG. 5 is a diagram showing a target area used in the image recognition processing in the present embodiment. In the present embodiment, when a captured image to be recognized is acquired, the game apparatus 1 first detects a region (referred to as “target region”) 54 including the marker 51 in the captured image (region detection). Process) (see the upper column of the table shown in FIG. 5). This region detection process is performed on the region 53 of the entire captured image, and is performed with lower accuracy than the marker detection process described later. Here, “low accuracy” means relatively increasing the interval between pixels that refer to pixel values in the processing. For example, in the present embodiment, the game apparatus 1 performs region detection processing with reference to pixel values at intervals of three pixels of the captured image (that is, with reference to pixel values by skipping two pixels). Thus, since the detection process of the target area 54 is performed with low accuracy, the detection process is performed at a relatively high speed.

  When the target area 54 is detected, the game apparatus 1 performs a process of detecting the marker 51 from the image of the target area 54 (marker detection process). The marker detection process is performed only for the target area 54 (not the area 53 of the entire captured image), and is performed with higher accuracy than the area detection process (see the lower column in FIG. 5). That is, in the marker detection process, the referenced pixel is finer than the area detection process, in other words, the interval between the pixels that refer to the pixel value is smaller than the area detection process. Specifically, in the present embodiment, the game apparatus 1 performs region detection processing with reference to pixel values at intervals of one pixel of the captured image (that is, with reference to pixel values of all pixels). . Thus, since the marker detection process is performed with high accuracy, the position and orientation of the marker 51 obtained as a detection result are calculated with high accuracy. In addition, since the marker detection process is performed only on the target area 54, the game apparatus 1 can perform the process at a higher speed than when the process is performed on the area 53 of the entire captured image.

  As described above, in the present embodiment, the game apparatus 1 first performs the process of detecting the target area 54 and performs the process of detecting the marker 51 for the target area 54. Therefore, the marker 51 detection process can be performed only on the necessary area (that is, the target area), and the marker 51 detection process can be performed in a short time. Further, since the detection process of the target area 54 is performed with low accuracy, this detection process can be performed in a short time, and the entire recognition process can be performed in a short time. Furthermore, since the detection process of the marker 51 is performed with high accuracy, the position and orientation of the marker 51 obtained as a recognition result can be calculated with high accuracy. That is, according to the present embodiment, recognition processing for detecting the marker 51 from the captured image can be performed at high speed and with high accuracy.

  In the present embodiment, processing is performed with different accuracy even in the marker detection processing. That is, the marker detection process includes a plurality of processes, and each process is executed with different accuracy. Details of these processes will be described later.

(Recognition process to detect a marker by the center position of internal figure)
Further, in the recognition processing in the present embodiment, the marker 51 is detected by calculating the center position of the internal figure using the marker 51 including the figure in which a plurality of internal figures are drawn. FIG. 6 is a diagram showing markers used in the present embodiment. As shown in FIG. 6, the marker 51 includes a first graphic 55 and a second graphic 57. The first graphic 55 has a plurality (eight in FIG. 6) of white circular graphics 56 drawn therein. In the present embodiment, the white circular graphic 56 is an internal graphic. In the present embodiment, the eight internal figures 56 are arranged at the positions of the four vertices of a quadrangle (square) and the midpoints of the four sides. The first figure may be any figure as long as a plurality of figures (internal figures) are drawn inside, and the position, shape, size, and color of the internal figure are any. But you can. The second graphic 57 is a graphic representing an arrow in FIG. 6, but the second graphic 57 may be any graphic.

  In the present embodiment, since the marker 51 shown in FIG. 6 is used, in the region detection process described above, the game apparatus 1 detects the region of the first graphic 55 of the marker 51 as the target region. In the marker detection process described above, the position and orientation of the marker 51 are detected by detecting the first graphic 55 with high accuracy.

  Here, in order to detect the position of the marker 51, a method of detecting the contour of the first graphic 55 of the marker 51 is considered. FIG. 7 is a diagram illustrating an image of the first graphic 55 in which the left and right contours are blurred. The captured image captured by the camera may be blurred due to a so-called camera shake or the like, resulting in an unclear outline. For example, if the camera (game device 1) moves left and right when an image is captured, the outline is blurred left and right and becomes unclear as shown in FIG. In this way, when the captured image is blurred and the contour becomes unclear, it is difficult to accurately detect the contour by the above-described method of detecting the contour, and therefore it is possible to accurately calculate the position of the figure. Can not. The contour can be detected based on, for example, a difference in pixel values (luminance value, color value, etc.) between pixels, but when the captured image is blurred, the position of the detected contour is the content of the image, the threshold value, etc. It will change greatly according to. For example, in the example of the image shown in FIG. 7, the dotted line 58 shown in FIG. 7 may be detected as a contour, or the dotted line 59 may be detected as a contour. In these cases, the length of the first graphic 55 in the horizontal direction is not correctly calculated, and the position of the first graphic 55 cannot be accurately calculated. As described above, in the method of detecting the contour and calculating the position of the marker from the contour, the position of the marker may not be accurately calculated when the captured image is blurred.

  Therefore, in the recognition processing in the present embodiment, the game apparatus 1 detects the internal graphic 56 of the first graphic 55 and detects the marker 51 by calculating the center position of each internal graphic 56. FIG. 8 is a diagram showing the center position of the internal graphic 56 in a part of the first graphic 55. In the marker detection process described above, the game apparatus 1 detects the area (contour) of the internal graphic 56 and calculates the center point P of the detected area (see FIG. 8). Here, in FIG. 8, even if the white dotted line shown in FIG. 8 is detected as a contour due to blurring in the captured image, the center is the same regardless of whether the black dotted line shown in FIG. 8 is detected as a contour. Point P is at the same position.

  FIG. 9 is a diagram showing the center point of each internal graphic 56 in the first graphic 55. In the present embodiment, the center points P1 to P8 of each internal graphic 56 are calculated. Here, the length in the left-right direction of the first graphic 55 is represented by the length from the point P1 to the point P3, and the like. This length is the same whether the white dotted line shown in FIG. 8 is detected as a contour or the black dotted line is detected as a contour. That is, this length can be accurately calculated even when the captured image is blurred.

  Although details will be described later, in the present embodiment, the game apparatus 1 specifies the positions of the four points of the marker 51 from the center points P1 to P8 of each internal graphic 56. The positions of the four points are the positions of four vertices of a quadrangle (square) represented by the eight internal figures 56. The position and orientation of the marker 51 are specified by the positions of these four points. However, since the positions of the four points are the vertices of the square, the marker 51 has the same value when rotated by 90 °, 180 °, or 270 ° and when it does not rotate. The direction of the marker 51 with respect to the four left and right directions cannot be specified, and the direction of the marker 51 cannot be uniquely specified. Therefore, the game apparatus 1 uniquely identifies the direction of the marker 51 by detecting the second graphic 57. A method for specifying the orientation of the marker 51 using the second graphic 57 will be described later.

  As described above, in the present embodiment, in the marker detection process, the game apparatus 1 calculates the center points P <b> 1 to P <b> 8 of each internal graphic 56. And the position of the marker 51 is calculated by specifying the positions of the four points of the marker 51 from the center points P1 to P8. Here, since the center points P1 to P8 are accurately calculated even when the captured image is blurred, the game apparatus 1 uses the center points P1 to P8 even when the captured image is blurred. By using it, the position of the marker 51 can be accurately calculated.

[Details of game processing]
Next, the details of the game process executed by the game program will be described with reference to FIGS. First, various data used in the game process will be described. FIG. 10 is a diagram showing various data used in the processing by the game program. In FIG. 10, a game program 61, captured image data 62, and game process data 63 are stored in the main memory 32 of the game apparatus 1.

  The game program 61 is a program for causing the CPU 31 of the game apparatus 1 to execute a game process (FIG. 11) described later. The game program 61 is stored in the main memory 32 by being partially or entirely read from the memory card 29 at an appropriate timing. The game program 61 includes a program for executing the above-described image recognition processing in addition to a program for progressing the game.

  The captured image data 62 is data indicating an image (captured image) captured by the camera (the inner camera 23 or the outer camera 25). As the captured image data 62, only the latest image data acquired from the camera may be stored, or a predetermined number of image data may be stored in order from the newest.

  The game process data 63 is various data used in the game process. The game processing data 63 includes target area data 64, contour data 65, internal area data 66, center position data 67, vertex data 68, and recognition result data 69. In addition to the above, the game processing data 63 stores various data necessary for the game, such as data related to various objects appearing in the game (the cannon 52 and the like) and sound data such as BGM.

  The target area data 64 is data indicating the above-described target area, that is, the area of the first graphic 55 in the captured image. The target area data 64 is obtained by the area detection process described above. In the present embodiment, since the detection process is performed with low accuracy in the area detection process, an area that is not the first graphic 55 may actually be detected as an area of the first graphic 55. Therefore, in the area detection process, a plurality of areas may be detected as target areas. In this case, target area data 64 for each detected area is stored in the main memory 32 (that is, a plurality of target area data is stored). Note that the target area data 64 is not limited as long as it indicates an area, may indicate only the outline of the area, or may indicate each pixel corresponding to the area.

  The contour data 65 is data indicating the contour of the first graphic 55 in the captured image. The contour data 65 is data indicating the region of the first graphic 55 as with the target region data 64, but differs in that it is detected with higher accuracy than the target region data 64.

  The internal area data 66 is data indicating the area of the internal graphic 56 in the first graphic 55. As described above, the first graphic 55 includes a plurality (eight) internal figures 56. Therefore, if eight internal figures 56 are correctly detected in the marker detection process, eight internal area data 66 are stored in the main memory 32. However, in fact, all eight internal figures 56 may not be detected. In this case, the same number of internal area data 66 as the number of detected internal figures 56 is stored in the main memory 32. It should be noted that the internal region data 66 only needs to indicate the region, and may indicate only the outline of the region, or may indicate each pixel corresponding to the region.

  The center position data 67 is data indicating the center position of each internal graphic 56. The center position data 67 indicates the same number of center positions as the number of internal figures 56 detected in the marker detection process.

  The vertex data 68 is data indicating the positions of four vertices of a quadrangle (square) represented by the eight internal figures 56. Although details will be described later, the four vertices are calculated based on the center positions of the eight internal figures 56.

  The recognition result data 69 is data indicating the position and orientation of the marker 51 output as a result of the recognition process. Although details will be described later, the position and orientation of the marker 51 are calculated based on the four vertices indicated by the vertex data 68 and the orientation of the second graphic 57 with respect to the first graphic 55 in the captured image.

  Next, details of the game process performed in the game apparatus 1 will be described with reference to FIGS. FIG. 11 is a main flowchart showing a flow of game processing executed in the game apparatus 1. When the power of the game apparatus 1 is turned on by pressing the power button 14F, the CPU 31 of the game apparatus 1 displays a menu screen (for example, an image (icon) representing each application) for instructing activation of various applications. Image). The user gives an instruction to start the game program 61 on the menu screen, for example, an instruction to select an icon of the game program 61. When the activation of the game program 61 is instructed on the menu screen, the CPU 31 initializes the main memory 32 and the like, and then starts executing the game program 61 for performing the processing shown in FIG. In the present embodiment, execution of the game program 61 causes the CPU 31 to function as each unit recited in the claims. That is, the game program 61 causes the CPU 31 to function as each unit recited in the claims.

  First, in step S1, the CPU 31 determines whether a captured image has been acquired. That is, it is determined whether captured image data is sent from the camera (outer camera 25 in the present embodiment). When captured image data is sent from the camera, the CPU 31 stores the data as captured image data 62 in the main memory 32. In the present embodiment, the processing loop of steps S1 to S7 is executed at a rate of once per frame time (1/60 seconds), whereas the camera has a lower frequency (several frames). Assume that the captured image is sent once a hour). However, in other embodiments, the captured image may be acquired once per frame time (or shorter than one frame time). If the determination result of step S1 is affirmative, the process of step S2 is executed. On the other hand, when the determination result of step S1 is negative, the process of steps S2 to S4 is skipped and the process of step S5 described later is executed.

  In step S2, the CPU 31 executes a recognition process for the captured image acquired in step S1. In the recognition process, it is determined whether or not the image of the marker 51 is included in the captured image. If the image of the marker 51 is included, the direction and position of the marker 51 in the captured image are calculated. The details of the recognition process will be described below with reference to FIG.

  FIG. 12 is a flowchart showing the flow of the recognition process (step S2) shown in FIG. In the recognition process, first, in step S <b> 11, the CPU 31 executes the above-described area detection process, that is, detects an area (target area) including the first graphic 55. As described above, the region detection process in step S11 is performed with low accuracy. Specifically, the CPU 31 detects a target area from the captured image based on pixel values obtained at intervals of three pixels in the captured image. More specifically, the CPU 31 reads the captured image data 62 from the main memory 32, and extracts pixel values at intervals of three pixels from each pixel value of each pixel indicated by the image data. Then, the target area is detected based on the extracted pixel value. FIG. 13 is a diagram illustrating pixels whose pixel values are referred to in the process of step S11 in the captured image. In FIG. 13, pixels indicated by diagonal lines are pixels whose pixel values are referred to.

  In the present embodiment, the target area is detected using a difference between pixel values of adjacent pixels (in this embodiment, a luminance value, but may be a color value). That is, in this embodiment, since a white frame is formed around the black first graphic 55 (see FIG. 6), the first pixel 55 is detected by detecting a pixel whose luminance value is lower than a predetermined value by a predetermined value. Pixels in the outline portion of one figure 55 can be detected. The “adjacent pixels” are pixels adjacent to each other whose pixel values are referred to. In the example of FIG. 13, a certain pixel indicated by hatching and a pixel (indicated by hatching) at a position separated by three pixels are the “adjacent pixels”. After detecting the pixels in the contour portion, the CPU 31 specifies one target region by setting adjacent pixels among the detected pixels as one set. Data indicating each pixel forming the specified target area is stored in the main memory 32 as one target area data 64 for each target area. If the specified target area is too large or too small, the CPU 31 may exclude the target area and not store the target area data 64 indicating the target area. Following step S11, the process of step S12 is executed.

  In addition, what kind of method may be sufficient as the detection method of the object area | region in said step S11. For example, in another embodiment, the CPU 31 may detect the contour of the first graphic 55 by binarizing the pixel value with a predetermined threshold, and may set the region within the detected contour as the target region. In another embodiment, the CPU 31 may detect the target region by a pattern matching method. That is, the pattern image of the first graphic 55 may be stored in the main memory 32 in advance, and the target region may be detected by detecting a region that matches (similar) the pattern image from the region of the captured image. . In this case, the pattern matching process may be the same as the process in the conventional augmented reality technology.

  In this embodiment, the CPU 31 detects the area of the first graphic 55 as the target area in step S11. However, in other embodiments, the area of the marker 51 including the first graphic 55 and the second graphic 57 is detected. You may make it detect as an object area | region.

  In the present embodiment, the target area is detected with low accuracy as described above. In the target area detection processing, the luminance difference between the first graphic 55 of the marker 51 and the surrounding white frame is used. . Therefore, if the white frame of the marker 51 is too thin, the white frame does not appear in the captured image with low accuracy, and the target area may not be accurately detected. Therefore, the white frame of the marker 51 is preferably set to a certain thickness so that it appears on the captured image even with low accuracy. The thickness of the white frame is appropriately determined according to the range of distance assumed as the distance from the camera to the marker 51 when the game apparatus 1 is used, the resolution of the captured image, the size of the marker 51, and the like. .

  In step S12, the CPU 31 determines whether or not the target area has been detected in step S11. If the target area is not detected in step S11, the process of step S13 is executed. On the other hand, when one or more target areas are detected in step S11, the process of step S14 is executed.

  In step S13, the CPU 31 determines that recognition of the marker 51 has failed. That is, the CPU 31 stores data indicating recognition failure in the main memory 32 as recognition result data 69. After step S13, the CPU 31 ends the recognition process.

  In step S14, the CPU 31 selects one of the target areas detected in step S11. When a plurality of target areas are detected in step S11, the processing steps in steps S14 to S16 are repeatedly executed. In this case, in step S14, a target area that has not yet been selected in the processing step is selected. Following step S14, the process of step S15 is executed.

  In step S15, the CPU 31 executes a marker detection process on the target area selected in step S14. The marker detection process is a process for detecting the marker 51 from the target region and calculating the position and orientation of the marker 51. Details of the marker detection process will be described below with reference to FIG.

  FIG. 14 is a flowchart showing the flow of the marker detection process (step S15) shown in FIG. In the marker detection process, first, in step S21, the CPU 31 extracts the contour of the first graphic 55 with medium accuracy. Here, “medium accuracy” means higher than the accuracy in the processing of step S11 and lower than the accuracy in the processing of steps S23 and S25 described later. That is, in this embodiment, in the marker detection process, the process is performed with two levels of accuracy, and the process of step S21 is performed with the lower accuracy. Details of the processing in step S21 will be described below.

  The process of step S21 is performed in the target area detected in step S11, but the accuracy of the process is different between the process of step S11 and the process of step S21. Therefore, in step S21, the CPU 31 first determines a region to be processed in step S21 corresponding to the target region detected in step S11. That is, the region to be processed in step S21 is determined from each pixel indicated by the target region data 64 obtained in step S11. More specifically, the CPU 31 refers to pixels (hatched pixels shown in FIG. 13) that are referred to in the low-precision processing (step S11) and eight surrounding pixels (pixels surrounded by a one-dot chain line shown in FIG. 13). Are associated with each other. Then, an area composed of the pixels in the target area detected in step S11 and the surrounding eight pixels corresponding to the pixel is set as a process target area in step S21. FIG. 15 is a diagram illustrating an example of a region to be processed in step S21. In FIG. 15, it is assumed that the pixels Q1 to Q3 are detected in step S11, and the region on the right side of the pixels Q1 to Q3 is detected as the target region. In this case, the area A1 to the right of the alternate long and short dash line shown in FIG. 15 is the process target area in step S21.

  When the area to be processed is determined, the CPU 31 extracts the contour of the first graphic 55 with medium accuracy. In the present embodiment, this contour extraction process is performed based on pixel values obtained at intervals of two pixels among pixel values of each pixel in the captured image. FIG. 16 is a diagram illustrating pixels whose pixel values are referred to in the process of step S21 in the captured image. For example, when the area A1 shown in FIG. 15 is the area to be processed in step S21, the pixels indicated by diagonal lines in FIG. 16 are pixels whose pixel values are referred to in the contour extraction process.

  Note that the contour extraction process in step S21 may be performed by a method using a difference in luminance values as in the method of detecting the target region in step S11, or may be performed by other methods. Data of each pixel forming the contour extracted in step S21 is stored in the main memory 32 as contour data 65. Following step S21, the process of step S22 is executed.

  In step S <b> 22, the CPU 31 determines whether or not the contour extracted in step S <b> 21 represents the first graphic 55. In the present embodiment, the determination in step S22 is performed by the CPU 31 depending on whether the outline represents a quadrangle. In other words, the CPU 31 reads out the contour data 65 from the main memory 32 and approximates each pixel forming the contour with a straight line, thereby calculating a polygon formed by each pixel. Then, whether or not the contour represents the first graphic 55 is determined based on whether or not the calculated polygon is a quadrangle. If the determination result of step S22 is affirmative, the process of step S23 is executed. On the other hand, when the determination result of step S22 is negative, a process of step S30 described later is executed.

  In step S22, whether or not the extracted contour represents the first graphic 55 is determined based on whether or not the contour is a quadrangle, but may be performed by other methods. For example, in another embodiment, when the size of the marker 51 is known, the above determination is made as to whether or not the size (area or circumference) of the extracted contour is a predetermined value or more. Etc. may be performed.

  In step S23, the CPU 31 detects the area of the internal graphic 56 with high accuracy. Here, “high accuracy” means higher than the accuracy in the process of step S11 and the process of step S21. In the present embodiment, the CPU 31 refers to the pixel values at intervals of one pixel, that is, refers to the pixel values of all the pixels in the region to be processed in step S23, and determines the region of the internal graphic 56. To detect.

  The region to be processed in step S23 is performed within the region represented by the contour extracted in step S21, but the processing accuracy is different between the processing in step S21 and the processing in step S23. Therefore, in step S23, as in step S21, it is necessary to determine a region to be processed. The area to be processed can be determined by the same method as in step S21. That is, the CPU 31 associates the reference pixel that can be referred to in the medium-precision processing (step S21) and the corresponding three pixels (for example, the right, lower, and lower right pixels of the reference pixel). Then, the reference pixel included in the contour extracted in step S21 and the three pixels corresponding to the reference pixel are set as processing targets in step S23.

  When the area to be processed is determined, the CPU 31 detects the area of the internal graphic 56 with high accuracy. Here, the detection of the area of the internal graphic 56 may be performed by a method using a difference in luminance value as in the method of detecting the target area in step S11, or may be performed by other methods. . Since the internal figure 56 to be detected in step S23 is white and the surrounding area is black, when adopting a method using a difference in luminance value, the CPU 31 has a luminance value equal to or greater than a predetermined value compared to adjacent pixels. It is preferable to detect high pixels. In the present embodiment, the CPU 31 detects (a plurality of) pixels representing the contour of the internal graphic 56 as information indicating the area of the internal graphic 56. Data of each pixel indicating the area of the internal graphic 56 detected in step S23 is stored in the main memory 32 as internal area data 66. When a plurality of internal figures 56 are detected in step S23, one internal area data 66 is stored in the main memory 32 for each internal figure. Following step S23, the process of step S24 is executed.

  In another embodiment, in step S23, the CPU 31 may set conditions regarding the shape and / or size of the detected area, and exclude areas that do not satisfy the condition. That is, the CPU 31 may not store the internal region data 66 for regions that are not circular in shape and regions that are not within a predetermined range among regions detected using the luminance difference.

  In step S24, the CPU 31 determines whether or not a predetermined number of areas have been detected in step S23. Here, the predetermined number is the number of internal figures 56 drawn in the first figure 55, that is, eight. Specifically, the CPU 31 determines whether or not the number of internal area data 66 stored in the main memory 32 is eight. If the determination result of step S24 is affirmative, the process of step S25 is executed. On the other hand, when the determination result of step S24 is negative, a process of step S30 described later is executed.

  In step S25, the CPU 31 calculates the center position of the internal figure 56 with high accuracy. The center position of the internal graphic 56 is calculated based on the area detected in step S23, that is, each pixel representing the outline of the internal graphic 56. The specific method for calculating the center position may be any method. For example, the center position can be obtained by calculating the average value of the position coordinates of each pixel representing the contour of the internal graphic 56. In the present embodiment, the CPU 31 reads the internal area data 66 from the main memory 32 and calculates the average value of each pixel indicated by the internal area data 66. Since eight internal area data 66 are stored, an average value is calculated for each data. Data indicating the calculated eight kinds of average values is stored in the main memory 32 as the center position data 67. Following step S25, the process of step S26 is executed.

  In step S26, the CPU 31 calculates the positions of the four vertices representing the position of the marker 51 from the center position calculated in step S25. Here, the four vertices are quadrangular vertices in which the eight internal figures 56 are the vertices and the midpoints of the sides. Hereinafter, a method for calculating the four vertices will be described with reference to FIGS. 17 and 18.

  FIG. 17 is a diagram illustrating an example of a captured image representing the first graphic. In FIG. 17, points P <b> 1 to P <b> 8 are points representing the positions calculated in step S <b> 26 as the center positions of the eight internal figures 56. In step S26, the CPU 31 first calculates four straight lines L1 to L4 passing through three of the eight central positions. Specifically, the CPU 31 passes a straight line L1 passing through points P1 to P3, a straight line L2 passing through points P1, P4, and P6, a straight line L3 passing through points P6 to P8, and points P3, P5, and P8. A straight line L4 is calculated. Each of the straight lines L1 to L4 can be calculated as an approximate straight line passing through three points using a least square method or the like.

  FIG. 18 is a diagram showing four vertices P11 to P14 calculated by the straight lines L1 to L4 shown in FIG. When the straight lines L1 to L4 are calculated, the CPU 31 calculates square vertices P11 to P14 formed by the straight lines L1 to L4. That is, the CPU 31 calculates an intersection point P11 between the straight line L1 and the straight line L2, an intersection point P12 between the straight line L1 and the straight line L4, an intersection point P13 between the straight line L2 and the straight line L3, and an intersection point P14 between the straight line L3 and the straight line L4. . As described above, the four vertices P11 to P14 can be calculated. Data indicating the calculated positions of the four vertices is stored in the main memory 32 as vertex data 68. Following step S26, the process of step S27 is executed.

  By calculating the four vertices in step S26, the positions of the four points in the marker 51 can be calculated. Here, the position and orientation of the marker 51 can be calculated from the positions of the four points in the marker 51. However, the positions of the four points are the vertices of the square, and the same value is obtained when the marker 51 is rotated by 90 °, 180 °, or 270 ° and when it is not rotated. For this reason, at the positions of the four points, it is impossible to specify the orientation of the marker 51 in the four directions of up, down, left, and right, and the orientation of the marker 51 cannot be uniquely specified. Therefore, the CPU 31 specifies the orientation of the marker 51 with respect to the four directions of up, down, left, and right by the processing of the following steps S27 to S29, and uniquely specifies the orientation of the marker 51. It should be noted that from which one of the four directions of the upper, lower, left, and right sides of the marker 51 is determined in advance (known), or when the direction of the marker 51 in the four directions of the upper, lower, left, and right is not conscious (the marker 51 is 90 In the case where the state rotated by °, 180 ° or 270 ° and the state not rotated are regarded as the same state), the CPU 31 omits the processing of the following steps S27 to S29 and uses the positions of the four points as the recognition result. Also good.

  In step S <b> 27, the CPU 31 detects the second graphic 57 from the area around the first graphic 55. In the present embodiment, the CPU 31 detects the second graphic 57 from four predetermined areas located in the four directions, up, down, left, and right, of the quadrangle formed by the four vertices calculated in step S26. To do. The second graphic 57 is located next to the first graphic 55 (see FIG. 6), and the position of the first graphic 55 is specified by the four vertices. This is because the second figure 57 can be detected.

  FIG. 19 is a diagram illustrating an example of the predetermined area set in step S27. In FIG. 19, an area α surrounded by straight lines L5 to L8 is the predetermined area. The predetermined area α is a predetermined area set on the side P11-P12 among the four sides of the quadrangle formed by the four vertices P11 to P14. Further, the straight lines L5 to L8 are respectively the lower side, the upper side, the right side, and the left side of the predetermined region α when the side where the predetermined region α is located is upward with respect to the quadrangle formed by the four vertices P11 to P14. It is. The straight lines L5 to L8 that are the four sides of the predetermined region α can be set as follows, for example. The straight line L5 that is the lower side is set at a position parallel to the straight line L1 and separated from the straight line L1 by a predetermined first distance in a direction perpendicular to the straight line L1. The straight line L6 that is the upper side is set at a position that is parallel to the straight line L1 and that is separated from the straight line L1 by a predetermined second distance (greater than the first distance) in a direction perpendicular to the straight line L1. Further, the straight line L7 that is the right side is parallel to the straight line L4 that is the right side of the quadrangle formed by the vertices P11 to P14, and is set at a position separated from the straight line L4 by a predetermined third distance in the direction of the straight line L1. The straight line L8 that is the left side is set to a position that is parallel to the straight line L2 that is the left side of the quadrangle formed by the vertices P11 to P14 and that is separated from the straight line L2 by a predetermined fourth distance in the direction of the straight line L1. As described above, the predetermined area α can be calculated. Further, the other three predetermined areas can be calculated in the same manner as the predetermined area α.

  Any method may be used to detect the second graphic 57 from within the four predetermined areas. In the present embodiment, the second graphic 57 is detected by a pattern matching technique. That is, the pattern image of the second graphic 57 is stored in the main memory 32 in advance, and the CPU 31 detects the second graphic 57 by detecting an area that matches (similar to) the pattern image from within each predetermined area. . Further, the accuracy of the process of detecting the second graphic 57 may be performed with any accuracy. Note that this processing can be performed by narrowing the range to be processed to the predetermined region, and thus does not necessarily have to be performed with high accuracy.

  As a specific process of step S27, the CPU 31 first reads the vertex data 68 from the main memory 32 and specifies the four predetermined areas based on the four vertices. Next, the pattern image of the second graphic 57 is read from the main memory 32, and a portion that matches the pattern image is detected from the four specified areas. Following step S27, step S28 is executed.

  In step S28, the CPU 31 determines whether or not the second graphic 57 has been detected in step S27. That is, it is determined whether or not there is an area where the second graphic 57 is detected among the four predetermined areas. If the determination result of step S28 is affirmative, the process of step S29 is executed. On the other hand, when the determination result of step S28 is negative, a process of step S30 described later is executed.

  In step S <b> 29, the CPU 31 specifies the orientation of the marker 51 (related to the four directions of up, down, left, and right). Here, if the direction in which the predetermined area in which the second graphic 57 is detected is found as seen from the quadrangle formed by the four vertices (which direction the predetermined area is from when viewed from the square) is known, imaging is performed. Since the direction in which the second graphic 57 is arranged with respect to the first graphic 55 in the image is known, the direction of the marker 51 can be specified. That is, in step S29, the orientation of the marker 51 is specified according to the direction in which the second graphic 57 is arranged with respect to the first graphic 55 in the captured image.

  In the present embodiment, the position and orientation of the marker 51 are represented by the two vectors V1 and V2 connecting the four vertices. Specifically, when the direction in which the second graphic 57 is arranged with respect to the first graphic 55 is the upward direction, the vector V1 is a vector having the upper left vertex as the start point and the upper right vertex as the end point. . In the above case, the vector V2 is a vector having a lower left vertex as a start point and a lower right vertex as an end point. Assuming the vectors V1 and V2 as described above, when the predetermined area where the second graphic 57 is detected is located on the side of the straight line L1 shown in FIG. 19, for example, the straight line L1 shown in FIG. This is the upper side of the quadrangle formed by the vertices. Therefore, in this case, the vector V1 is calculated as a vector having the point P11 as a start point and the point P12 as an end point, and the vector V2 is calculated as a vector V2 having the point P13 as a start point and the point P14 as an end point. The position and orientation of the marker 51 can be represented by these vectors V1 and V2.

  In step S29, the CPU 31 uses the four vertices to generate a vector V1 and a vector V1 so that the second figure 57 with respect to the quadrangle formed by the four vertices is oriented according to the direction of the predetermined area. V2 is calculated. Data indicating the calculated vectors V1 and V2 is stored in the main memory 32. After step S29 described above, the CPU 31 ends the marker detection process.

  On the other hand, in step S <b> 30, the CPU 31 determines that there is no first graphic 55 in the target region that is the processing target of the current marker detection process. In this case, the various data 65 to 68 calculated in steps S <b> 21 to S <b> 28 in the current marker detection process are deleted from the main memory 32. After step S30, the CPU 31 ends the marker detection process.

  According to the marker detection process, the CPU 31 detects a plurality of internal figures from the image of the target area (step S23), and detects the marker 51 by calculating the center position of each internal figure (steps S25 and S25). S26). Thus, by using the center position of the internal figure, the position of the marker 51 can be accurately calculated even when the captured image is blurred.

  Further, according to the series of processes in steps S21 to S23, the contour of the first graphic 55 is extracted from the target area (step S21), and it is determined whether or not the extracted contour represents the first graphic 55. (Step S22). When the extracted contour represents the first graphic 55, a plurality of internal graphics 56 are detected from the area corresponding to the contour (step S23). That is, the internal graphic 56 is detected from the area corresponding to the contour determined to represent the first graphic 55. According to this, since the target area that does not include the first graphic 55 can be removed before the process of detecting the internal graphic 56, it is possible to prevent the process of detecting the internal graphic 56 from being performed wastefully. The recognition process can be performed efficiently. Furthermore, according to the present embodiment, the contour extraction process can be performed at a higher speed than the process of detecting the internal graphic 56, so that the game apparatus 1 can perform the recognition process in a shorter time.

  Further, according to the series of processing of steps S23 to S25, a plurality of internal figures 56 are respectively detected in the target area (step S23), and it is determined whether or not the number of detected areas is a predetermined number. (Step S24). If the number of detected areas is a predetermined number, the center position of the area is calculated (step S25). That is, the center position of the internal graphic 56 is calculated for the target area where a predetermined number of areas are detected. According to this, since the target area that does not include the first graphic 55 can be removed before the process of calculating the center position of the internal graphic 56, it is possible to prevent the process of calculating the center position from being performed wastefully. Recognition processing can be performed efficiently.

  In the marker detection process, the accuracy of performing the internal graphic 56 detection process (step S23) is the same as the accuracy of performing the central position calculation process (step S25) of the internal graphic 56. Here, in another embodiment, the CPU 31 may perform the detection process of the internal graphic 56 with a lower accuracy than the accuracy of performing the calculation process of the center position of the internal graphic 56. That is, the interval between the pixel values used for the internal graphic 56 detection process may be larger than the interval between the pixel values used in the center position calculation process. According to this, the detection process of the internal figure 56 can be performed at higher speed, and the recognition process can be performed at higher speed. At this time, the detection processing of the internal graphic 56 may be performed with the same accuracy as the contour extraction processing (step S21) of the first graphic 55, or may be performed with higher accuracy than the extraction processing. .

  Returning to the description of FIG. 12, the process of step S16 is executed after the marker detection process of step S15. In step S16, the CPU 31 determines whether marker detection processing has been performed for all target regions detected in step S11. If the determination result of step S16 is affirmative, the process of step S17 is executed. On the other hand, if the determination result of step S16 is negative, the process of step S14 is executed again. Thereafter, the processes in steps S14 to S16 are repeated until it is determined in step S16 that the marker detection process has been performed for all target regions. Thereby, marker detection processing is performed for each target region detected in step S11.

  In step S17, the CPU 31 determines whether or not the marker 51 has been detected in the marker detection process in step S15. Specifically, the CPU 31 determines whether or not the marker 51 is detected based on whether or not the recognition result data 69 is stored in the main memory 32. If the determination result of step S17 is affirmative, the process of step S18 is executed. On the other hand, if the determination result of step S17 is negative, the process of step S13 is executed again. That is, when the determination result of step S17 is negative, it is determined that the recognition of the marker 51 has failed, and the recognition process ends.

  In step S18, the CPU 31 stores information indicating the position and orientation of the marker 51 obtained by the marker detection process in step S15 as a recognition result. That is, data indicating the vectors V1 and V2 calculated in step S29 of the marker detection process is stored in the main memory 32 as the recognition result data 69. After step S18, the CPU 31 ends the recognition process.

  Note that the vectors V1 and V2 may be calculated by marker detection processing for a plurality of target areas detected in step S11 from one captured image. That is, a plurality of sets of vectors V 1 and V 2 may be stored in the main memory 32 in one recognition process. In this case, one set represents the correct marker 51, and the other set is calculated by erroneous detection. Therefore, in the above case, the CPU 31 determines the vector V1 representing the correct marker 51 based on the shape and arrangement of each figure (first figure 55, internal figure 56, second figure 57) obtained in the marker detection process. You may make it select V2. For example, based on the outline of the first graphic 55 extracted in step S21, the arrangement of each internal graphic 56 detected in step S23, the similarity of the second graphic 57 detected in step S27 to the pattern image, and the like. Thus, the correct vectors V1 and V2 can be selected. The CPU 31 stores data indicating the selected vectors V1 and V2 as recognition result data 69 in the main memory. Thereby, even when a plurality of sets of vectors V1 and V2 are obtained (by erroneous detection) in the recognition process, a correct recognition result can be obtained. In another embodiment, when a plurality of sets of vectors V1 and V2 are obtained in one recognition process, the CPU 31 may determine that the recognition has failed.

  According to the recognition process, a process of detecting a target area is performed based on pixel values obtained at a first interval (an interval corresponding to three pixels) in a captured image (step S11). Then, based on pixel values obtained at a second interval (an interval of one or two pixels) smaller than the first interval in the captured image in the target region, the first graphic 55 is obtained from the image of the target region. A process of detecting is performed. According to this, the recognition process can be performed at high speed by performing the process of detecting the target region in a short time. In addition, since the first graphic 55 is detected with high accuracy, the first graphic 55 can be accurately detected with high accuracy. That is, according to the recognition process, the process of detecting the first graphic 55 from the captured image can be performed at high speed and with high accuracy.

  In this embodiment, the recognition process in step S2 is executed within one frame time, and the recognition process is completed in the process loop of steps S1 to S7. Here, depending on the resolution of the captured image and the processing capability of the CPU 31, etc., it may be difficult to complete one recognition process in one frame time. Therefore, in other embodiments, the recognition process does not necessarily have to be executed in the same cycle as the display process (step S5) or the like executed with one frame time as a cycle. That is, in one process of step S2, only a part of the entire recognition process may be executed. In this case, the processing executed in one step S2 is adjusted to a processing amount such that the series of processing in steps S1 to S7 can be completed within one frame time. In other words, the recognition process may be executed in parallel separately from the processing loop of steps S1 to S7 (except S2), and may be executed when the CPU 31 is in an idle state.

  Returning to the description of FIG. 11, the process of step S3 is executed after the recognition process of step S2. In step S3, the CPU 31 determines whether or not the recognition process in step S2 has succeeded. The determination in step S3 can be made based on whether or not the recognition result data 69 stored in the main memory 32 indicates the position and orientation of the marker 51. If the determination result of step S3 is affirmative, the process of step S4 is executed. On the other hand, when the determination result of step S3 is negative, the process of step S4 is skipped and the process of step S5 described later is executed.

  In step S4, the CPU 31 calculates the position and orientation of the virtual camera based on the recognition result in step S2. The virtual camera is set in the virtual space in order to determine a viewpoint, a line-of-sight direction, and a viewing angle when generating an image of the virtual space. In step S4, the CPU 31 first calculates the positional relationship between the game apparatus 1 (camera) and the marker 51 from the position and orientation of the marker 51 that is the recognition result in step S2. This positional relationship is represented, for example, as the other three-dimensional position and orientation when one of the game apparatus 1 and the marker 51 is used as a reference. Next, the CPU 31 calculates the position and orientation of the virtual camera in the virtual space based on the positional relationship. The position and orientation of the virtual camera are calculated so that the positional relationship between the virtual camera in the virtual space and the position corresponding to the marker 51 in the virtual space matches the positional relationship between the game apparatus 1 and the marker 51 in the real space. The The process in step S4 may be the same method as the process used in the conventional augmented reality technology as described in Non-Patent Document 1 described above, for example. If the recognition process in step S2 fails, the CPU 31 does not calculate the position and orientation of the virtual camera, and uses the position and orientation of the virtual camera calculated when the recognition process was last successful. Following step S4, the process of step S5 is executed.

  In step S5, the CPU 31 executes a game control process. The game control process is a process for advancing the game by operating an object in the virtual space according to a user (player) input or the like. Specifically, in the game control process, the movement of the player character is controlled according to the user input, or the movement of the object (cannon 52 shown in FIG. 4) is controlled according to the control rules defined in the game program 61. Processing is included. Further, the CPU 31 may use the recognition result data 69 as a game input in addition to the operation results for various input devices (the touch panel 13 and the operation buttons 14A to 14L) provided in the game apparatus 1. For example, the position of the virtual camera calculated from the recognition result data 69 may be used as the position of the player character. Following step S5, the process of step S6 is executed.

  In step S <b> 6, the CPU 31 generates a composite image obtained by combining the captured image and the virtual object image (virtual image) in the virtual space, and displays the composite image on the display device (upper LCD 22). That is, the CPU 31 first generates a virtual image based on the position and orientation of the virtual camera set in step S4. Thereby, an image of the virtual object viewed from the position of the virtual camera is generated. Next, the CPU 31 combines the generated virtual object image with the captured image, and generates and displays a combined image. The captured image used for generating the composite image may be the latest acquired captured image or the latest captured image for which the recognition process has been successful. Note that the composite image generation process may be the same process as in the conventional augmented reality technology. Following step S6, the process of step S7 is executed.

  In step S7, the CPU 31 determines whether or not to end the game. The determination in step S7 is made based on, for example, whether or not the game has been cleared, whether or not the game is over, and whether or not the player has given an instruction to stop the game. If the determination result of step S7 is negative, the process of step S1 is executed again. Thereafter, the processing loop of steps S1 to S7 is repeatedly executed until it is determined in step S7 that the game is to be ended. On the other hand, when the determination result of step S7 is affirmative, the CPU 31 ends the game process shown in FIG. This is the end of the description of the game process.

  According to the above game process, the target detection process (step S11) for detecting the area of the first graphic 55 from the captured image, and the marker detection process (step S15) for detecting the marker 51 including the first graphic 55 from the detected area. Thus, recognition processing can be performed at high speed and with high accuracy by changing the processing accuracy. Further, by calculating the central position of the internal graphic 56 in the first graphic 55 and calculating the position (four vertices) on the marker 51 based on the central position, even when the captured image is blurred, The position of the marker 51 can be accurately calculated.

  Note that, according to the series of processing in steps S3 to S6, when the marker 51 is detected (Yes in step S3), the positional relationship between the marker 51 and the outer camera 25 is the processing result in the recognition processing (marker 51) (step S4). Then, a virtual image representing a virtual object placed in the virtual space is generated based on the positional relationship, and an image obtained by combining the captured image with the virtual image is displayed on the display device (step S5). Thus, in the present embodiment, the above recognition process can be used in so-called augmented reality technology. Here, when using augmented reality technology as in this embodiment, it is preferable to accurately calculate the position and orientation of the marker 51 in the recognition process. Further, in the game processing as in the present embodiment, it is not preferable that the response to the game operation is deteriorated or the game progress is delayed due to the delay of the processing, and the recognition processing is preferably performed at high speed. Therefore, the recognition process according to the present embodiment, which can be performed at high speed and with high accuracy, is particularly effective for game processing using augmented reality technology.

[Modification]
The above-described embodiment is an example for carrying out the present invention. In other embodiments, the present invention can be implemented with, for example, the configuration described below.

(Modifications using other recognition methods)
In the above embodiment, the recognition method is used in which the first graphic 55 having the internal graphic 56 is taken as an imaging target, and the first graphic 55 is detected by calculating the center position of the internal graphic 56. Here, the method of recognizing the imaging target is not limited to the method of calculating the center position of the internal graphic 56 described above, and may be another method. Hereinafter, a modified example using a recognition method based on a pattern matching technique will be described as a modified example using another recognition method.

  FIG. 20 is a diagram illustrating markers used in a modification example of the present embodiment. In this modification, a marker 71 shown in FIG. 20 is used instead of the marker 51 shown in FIG. The marker 71 does not have a plurality of internal figures, but includes a third figure 72 in which a predetermined figure (arrow in FIG. 20) is drawn. In this modification, the third graphic 72 is a target to be recognized (detected) by the recognition process. The predetermined figure may be any shape, but is preferably a figure that does not have the same shape as before rotation when rotated by a predetermined angle so that the orientation of the marker 71 can be specified. .

  FIG. 21 is a flowchart showing the flow of marker detection processing in a modification of the present embodiment. Note that the recognition process in the present modification is the same as the recognition process shown in FIG. 12 except for the marker detection process. Therefore, also in this modification, the area detection process (step S11) is executed with low accuracy.

  In the marker detection process of this modification, first, in step S41, the CPU 31 extracts the contour of the third graphic 72 of the marker 71 with high accuracy. That is, the CPU 31 refers to the pixel value at an interval of one pixel in the target area and extracts the outline of the third graphic 72. Note that the contour extraction method is the same as that in step S21. Further, data of each pixel forming the extracted contour is stored in the main memory 32. Following step S41, the process of step S42 is executed.

  In step S42, the CPU 31 determines whether or not the contour extracted in step S21 represents a marker. The determination in step S42 can be performed in the same manner as the determination process in step S22 described above, although the contour accuracy is different from the determination process in step S22 described above. If the determination result of step S42 is affirmative, the process of step S43 is executed. On the other hand, when the determination result of step S42 is negative, a process of step S47 described later is executed.

  In step S <b> 43, the CPU 31 reads a pattern image representing the third graphic 72 from the main memory 32. Note that the pattern image representing the third graphic 72 may be one type, or a total of four types of pattern images are prepared: a pattern image and three images rotated by 90 °, 180 °, and 270 °. May be. Following step S43, the process of step S44 is executed.

  In step S44, the CPU 31 performs pattern matching on the third graphic 72 in the target area. That is, an area that matches (similar) the pattern image read in step S43 is detected from the target area. Note that the pattern matching in step S44 may be performed with any accuracy, and is not necessarily performed with high accuracy. Following step S44, the process of step S45 is executed.

  In step S45, the CPU 31 determines whether or not the third graphic 72 is detected in the target area by the pattern matching in step S44. If the determination result of step S45 is affirmative, the process of step S46 is executed. On the other hand, if the determination result of step S45 is negative, the process of step S47 is executed.

  In step S46, the CPU 31 specifies the position and orientation of the marker 71 based on the result of pattern matching in step S44. That is, the CPU 31 calculates the positions of the four vertices from the outline of the third graphic 72 extracted in step S41, and based on the orientation of the pattern image when matched with the captured image, the third in the four directions, up, down, left, and right. The direction of the figure 72 is calculated. Further, the above-described vectors V1 and V2 are calculated for the third graphic 72 from the calculated positions of the four vertices and the orientation of the third graphic 72 in the four directions. As a result, the position and orientation of the marker 71 can be specified. After step S46, the CPU 31 ends the marker detection process.

  On the other hand, in step S47, the CPU 31 determines that there is no marker in the target area that is the target of the current marker detection process. The process in step S47 is the same as that in step S40. After step S47, the CPU 31 ends the marker detection process.

  As described above, according to the present modification, in the marker detection process shown in FIG. 21, the position of the marker 71 (third graphic 72) can be calculated with high accuracy by the processes of steps S41 and S46. Similarly to the above embodiment, the area detection process (step S11) is performed with low accuracy, and the marker detection process is performed only on the target area. Therefore, as in the above embodiment, the recognition process can be performed in a short time. Therefore, also in this modification, the recognition process can be performed at high speed and with high accuracy as in the above embodiment.

  In addition, the recognition process only needs to detect the target (first graphic 55) by the low-precision area detection process and the high-precision marker detection process. Specific detection methods in these two types of processes are as follows. Any method may be used. For example, the game apparatus 1 may perform detection using a pattern matching technique in both of the two types of processes. That is, the game device 1 detects a target region that matches (similar) the pattern image from the captured image by performing pattern matching based on the pixel values obtained at intervals of three pixels in the region detection process. Then, in the marker detection process, whether or not the first image 55 exists in the target area by performing pattern matching based on the pixel value obtained at intervals of one pixel in the detected target area. Determine whether. Also by this, the game apparatus 1 can perform a recognition process at high speed and with high precision similarly to the said embodiment.

(Modification regarding the method of calculating four vertices from the center position)
In step S24, the CPU 31 determines whether the number of areas of the internal graphic 56 detected in step S23 is a predetermined number. Here, in the present embodiment, four vertices are specified by calculating four straight lines from the center positions of the eight internal figures 56, so even if all eight internal figures 56 are not detected. It is possible to calculate four vertices. For example, four straight lines L1 to L4 can be calculated without any one of the center points P1 to P8 in FIG. Therefore, in another embodiment, in step S24, the CPU 31 may determine whether or not the number of areas of the internal graphic 56 detected in step S23 is a predetermined range. Here, the predetermined range is a range including the number of internal figures (eight), for example, a range of 7 to 8.

In the above embodiment, when four straight lines are calculated from the seven central positions, it is necessary to specify one straight line from the two central positions. In this case, it is necessary to prevent an incorrect straight line from being calculated from two center positions of an incorrect combination (for example, a straight line passing through the central point P3 and the central point P4). Therefore, for example, the CPU 31 may calculate four straight lines from the seven center positions so as to satisfy the following condition.
(Condition 1) Two or three center positions out of the seven center positions are passed.
(Condition 2) It is substantially parallel to any of the four straight lines representing the contour of the first graphic 55.
(Condition 3) It is within a predetermined distance from any of the four straight lines representing the outline of the first graphic 55 (does not pass near the center of the first graphic 55).
In the above (Condition 2) and (Condition 3), the four straight lines representing the outline of the first graphic 55 may be those calculated in Step S22. By calculating straight lines that satisfy the above conditions 1 to 3, it is possible to calculate four straight lines from the seven central positions.

  As described above, while the first graphic 55 on which three or more internal figures arranged on a straight line are drawn is taken as an imaging target, an approximate straight line is calculated from the center positions of the two or more internal figures, and this approximation is performed. You may make it calculate the position (position of four vertices) of the marker 51 with a straight line. According to this, by providing redundancy to the number of internal figures, the positions of the four vertices can be calculated even when all the internal figures cannot be detected.

(Modifications related to markers)
In the above embodiment, the case where the marker 51 including the first graphic 55 in which the eight internal graphics 56 are drawn is described as an example. However, the marker that can be used in the present invention is not limited thereto. For example, in other embodiments, the number of internal figures 56 is not limited to eight. FIG. 22 is a diagram illustrating a marker in a modified example of the present embodiment. The marker 73 shown in FIG. 22 includes a first graphic 74 having 12 internal graphics 56 and a second graphic 57 similar to FIG. Further, the twelve internal figures 56 in the first figure 74 are arranged at the positions of four vertices and four sides of a square (square). In another embodiment, a marker 73 shown in FIG. 22 may be used instead of the marker 51 shown in FIG. When the marker 73 is used, in step S26 described above, the CPU 31 may calculate the straight lines L1 to L4 from three or four center positions. According to this, since the number of internal figures can be made redundant, the CPU 31 can calculate the positions of the four vertices even when all the internal figures cannot be detected.

  In the above embodiment, all the internal figures 56 have the same color (white). However, in other embodiments, the color of the internal figure 56 is made different depending on the position in the first figure 55. Also good. FIG. 23 is a diagram illustrating a marker in another modification example of the present embodiment. The marker 75 shown in FIG. 23 includes a first graphic 76 on which 12 internal figures 56 are drawn, similarly to the first graphic 74 shown in FIG. Here, in the first graphic 76, the colors of the internal figures 56 arranged at the positions of the four sides of the quadrangle (square) are different for each side (in FIG. Expressed in intervals). In another embodiment, a marker 75 shown in FIG. 23 may be used instead of the marker 51 shown in FIG. In this case, the CPU 31 can calculate the orientation of the marker 75 (orientation in the four directions, up, down, left, and right) without using the second graphic. That is, the CPU 31 can identify the orientation of the marker 75 in the four directions, up, down, left, and right, by detecting the color of the internal graphic 56.

  In the above embodiment, the marker 51 is provided with a white frame around the black first graphic 55 for the purpose of easily detecting the first graphic 55. Therefore, the detection target by the recognition process is the first graphic 55 in the marker 51. Here, in other embodiments, there may be no white frame, and the entire marker 51 may be a detection target by the recognition process.

  In the above embodiment, the marker 51 includes the second graphic 57 at a position different from the first graphic 55. The CPU 31 specifies the orientation of the first graphic 55 (orientation in the four directions, up, down, left, and right) based on the direction in which the second graphic 57 is arranged with respect to the first graphic 55 (S29). Here, in another embodiment, a marker in which a second graphic is drawn inside the first graphic may be used. For example, in the marker 51 shown in FIG. 6, the second graphic 57 may be formed inside a square represented by eight internal graphics 56 inside the first graphic 55. The second graphic may be any graphic as long as the direction can be specified, such as the arrow graphic in the above embodiment. At this time, after calculating four vertices in step S26, the CPU 31 detects the second graphic from the area inside the quadrangle formed by the four vertices. And the direction of the 1st figure about four directions of up and down, right and left is specified from the direction of the detected 2nd figure. As described above, as in the above embodiment, the orientation of the first graphic in the four directions, top, bottom, left, and right can be specified. However, when a marker in which the second graphic is drawn inside the first graphic is used, both the inner graphic and the second graphic are included in the first graphic. Therefore, there is a high possibility of erroneous detection when detecting the inner figure, and the accuracy of the recognition process may be reduced (compared to the above embodiment). In other words, in the above embodiment, since the marker in which only the inner graphic is formed in the first graphic is used, the inner graphic can be detected more accurately, and the accuracy of the recognition process can be improved.

  In the above embodiment, the marker 51 is a thin plate-like member on which a predetermined graphic is drawn. In other embodiments, the marker 51 is realized by displaying the predetermined graphic on a display device. May be. For example, in another embodiment, another portable game device in which the marker 51 is displayed on the screen may be used as the marker 51.

(Modified example of captured image used for recognition processing)
In the above-described embodiment, the same captured image is used in the region detection process (step S11) performed with low accuracy and the marker detection process (step S15) performed with high accuracy. Here, in other embodiments, images having different resolutions may be used in the above two processes. Specifically, in the region detection process, an image having a lower resolution than the captured image used in the marker detection process may be used. For example, when the game apparatus 1 generates an image with a low resolution from an image captured by the camera because it is necessary for a process other than the recognition process, the image with the low resolution is used in the area detection process. It may be.

(Modification regarding marker position calculation)
In the above embodiment, the CPU 31 calculates the center position of the internal graphic 56, and calculates the positions of the markers 51 in the captured image (positions of the four vertices P11 to P14) based on the center position. Here, in another embodiment, the CPU 31 may use the center position of the internal graphic 56 as the position of the marker 51 as it is. For example, the CPU 31 may use the points P1, P3, P6 and the point P8 shown in FIG. 9 and FIG. Moreover, in the said embodiment, although the position of four points (position of the four vertexes P11-P14) was used as information showing the position and direction of the marker 51, the position of the three or more points of the marker 51 is used. If it is known, the position and orientation of the marker 51 can be specified. Therefore, in other embodiments, the position of three or five or more points may be used as information indicating the position and orientation of the marker 51. Therefore, the CPU 31 can detect the position and orientation of the marker 51 based on the center position of three or more internal figures.

(Other variations)
In the embodiment described above, the game apparatus 1 performs the recognition process on the captured image acquired in real time by the outer camera 25 included in the game apparatus 1. Here, in other embodiments, the captured image that is the target of the recognition process may be a captured image in the past, or may be a captured image acquired by the game apparatus 1 from an external device. . In the above embodiment, the outer camera 25 is mounted on the game apparatus 1 in advance. However, in another embodiment, an external camera that can be attached to and detached from the game apparatus 1 is used instead of the outer camera 25. Also good.

  In the above embodiment, the case where the recognition process is used in the augmented reality technology has been described as an example. In addition, the object to be recognized in the recognition process is a marker used in augmented reality technology. Here, the present invention can be applied to arbitrary image recognition processing for recognizing an arbitrary imaging target from a captured image. For example, in another embodiment, the recognition process is used for a technique for recognizing a user's face from a captured image and performing a predetermined process (for example, a process of adding an image such as a graffiti) on the recognized face. May be.

  In the above embodiment, the recognition process is executed by only one information processing device (game device 1). In other embodiments, the image recognition system includes a plurality of information processing devices that can communicate with each other. The plurality of information processing apparatuses may share and execute the recognition process.

  As described above, the present invention can be used for, for example, a game program or a game device for the purpose of shortening the time required for detection while maintaining the detection accuracy of an object.

1 Game device 13 Touch panel 22 Upper LCD
25 Outside camera 31 CPU
32 Main memory 51 Marker 54 Target area 55 First figure 56 Internal figure 57 Second figure

Claims (17)

An image recognition program executed in a computer of an information processing apparatus for detecting a predetermined imaging target included in a captured image captured by an imaging unit,
Image acquisition means for acquiring a captured image captured by the imaging means;
Based on pixel values obtained at first intervals in the captured image, area detection means for detecting a target area including the predetermined imaging target in the captured image;
Based on pixel values obtained at a second interval smaller than the first interval in a captured image in the target region, the computer is used as target detection means for detecting the predetermined imaging target from the image of the target region. An image recognition program that works.   The image according to claim 1, wherein the target detection unit detects the imaging target by calculating a position of the predetermined imaging target in the captured image based on a pixel value obtained at the second interval. Recognition program. The predetermined imaging target includes a figure in which a plurality of internal figures are drawn,
The target detection unit detects the plurality of internal figures from the image of the target area, and calculates the center position of each internal figure based on the pixel values obtained at the second interval. The image recognition program according to claim 1, wherein the image recognition program detects a position of an imaging target. The image recognition program further causes the computer to function as a contour extracting unit that extracts a contour of the predetermined imaging target in the target region detected by the region detecting unit,
The image recognition program according to claim 3, wherein the target detection unit detects each of the plurality of internal figures from a region in the contour extracted by the contour extraction unit.   The contour extracting unit extracts a contour of the predetermined imaging target based on pixel values obtained at a third interval that is smaller than the first interval and larger than the second interval in the captured image. Item 5. The image recognition program according to Item 4. The image recognition program causes the computer to function as an object determination unit that determines whether the contour extracted by the contour extraction unit represents the predetermined imaging target,
6. The image according to claim 4, wherein the target detection unit detects the plurality of internal figures from within an area corresponding to an outline determined to represent the predetermined imaging target by the target determination unit. 7. Recognition program. The object detection means includes
An internal area detecting means for detecting areas of a plurality of internal figures in the target area,
The image recognition program according to any one of claims 3 to 6, further comprising a position calculation unit that calculates a center position of each region detected by the internal region detection unit.   The internal area detection means detects the areas of the plurality of internal figures based on pixel values obtained at a fourth interval that is smaller than the first interval and larger than the second interval in the captured image. The image recognition program according to claim 7. The target detection means further includes a number determination means for determining whether or not the number of areas detected by the internal area detection means is a number within a predetermined range,
The said position calculation means calculates the center position of each area | region detected by the said internal area | region detection means about the said object area | region where the number of area | regions within the said predetermined range were detected. Image recognition program.   The said area | region detection means detects an object area | region by extracting the outline of the said predetermined imaging target based on the pixel value between the adjacent pixels in the said 1st space | interval. The image recognition program according to item 1. The region detection unit detects the target region based on pixel values related to pixels arranged at the first interval among the pixels of the captured image acquired by the image acquisition unit,
The said object detection means detects the said predetermined imaging target based on the pixel value regarding the pixel arrange | positioned by the said 2nd space | interval among each pixel of the captured image acquired by the said image acquisition means. The image recognition program according to claim 10. Causing the computer to further function as pattern acquisition means for acquiring a pattern image representing the predetermined imaging target from storage means accessible by the information processing apparatus;
The target detection unit determines whether or not the predetermined imaging target is included in the target region by comparing the image of the target region with the pattern image. Image recognition program.   The target detection means extracts a contour of the predetermined imaging target from the image of the target region based on the pixel value obtained at the second interval, and based on the extracted contour, the target image in the captured image The image recognition program according to claim 1 or 12, wherein a position of a predetermined imaging target is calculated. A positional relationship calculation unit that calculates a positional relationship between the predetermined imaging target and the imaging unit based on a detection result of the target detection unit when the predetermined imaging target is detected by the target detection unit;
The computer further functions as a display control unit that generates a virtual image representing a virtual object arranged in the virtual space based on the positional relationship, and causes the display unit to display an image obtained by synthesizing the virtual image with the captured image. The image recognition program according to claim 1, wherein the image recognition program is executed. An image recognition device for detecting a predetermined imaging target included in a captured image captured by an imaging means,
Image acquisition means for acquiring a captured image captured by the imaging means;
Based on pixel values obtained at first intervals in the captured image, area detection means for detecting a target area including the predetermined imaging target in the captured image;
Object detection means for detecting the predetermined imaging object from the image of the target area based on pixel values obtained at a second interval smaller than the first interval in the captured image in the target area; Image recognition device. An image recognition system for detecting a predetermined imaging target included in a captured image captured by an imaging means,
Image acquisition means for acquiring a captured image captured by the imaging means;
Based on pixel values obtained at first intervals in the captured image, area detection means for detecting a target area including the predetermined imaging target in the captured image;
Object detection means for detecting the predetermined imaging object from the image of the target area based on pixel values obtained at a second interval smaller than the first interval in the captured image in the target area; Image recognition system. An image recognition method for detecting a predetermined imaging target included in a captured image captured by an imaging means,
An image acquisition step of acquiring a captured image captured by the imaging means;
An area detecting step of detecting a target area including the predetermined imaging target in the captured image based on pixel values obtained at first intervals in the captured image;
A target detection step of detecting the predetermined imaging target from the image of the target region based on pixel values obtained at a second interval smaller than the first interval in the captured image in the target region, Image recognition method.