JP2010142553A - Game program and game apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a novel game program and a game apparatus. <P>SOLUTION: The game apparatus 10 includes an inward camera 32, and a skin color is extracted from an imaged image imaged by the inward camera 32. During execution of the game, it is determined whether or not a skin color region being above a predetermined ratio is detected at a specific region (110, 110'). If the skin color region is above the predetermined ratio, it is determined the face of a user is correctly recognized, and feature points are detected from the imaged image. By detecting whether or not the entire face of the user is recognized, by detecting a part of the face or its movement, on the basis of the feature points, the game is progressed depending on whether or not a predetermined condition is satisfied. However, if the skin color is equal to or less than the predetermined ratio, it is determined that the face of the user is not correctly recognized, and a determination based on the feature points is not performed. An operation with the imaged user's image is reflected on the game processing just as the user intended. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to a game program and a game apparatus, and more particularly to a game program and a game apparatus that recognize all or part of a captured user's face and execute a game process based on the recognition result.

An example of background art is disclosed in Patent Document 1. The image output device disclosed in Patent Document 1 captures a user's face with two cameras, identifies eye movements by detecting eyeballs and irises from the captured face image, and identifies the identified eyes. The display area of the output image is changed according to the movement.
JP 2006-202181 A [G06F 3/033, A63F 13/00, G06T 1/00, G06T 7/60]

  However, since the image output device of Patent Document 1 is based on the premise that the user's face has been photographed, it is determined that the movement of the eyes has been identified by detecting something other than the user's eyes, and the output image May change the display area. Therefore, when this background technology is applied to a game in which an operation is performed using a captured user image, the operation based on the captured user image cannot be reflected in the game processing as intended by the user. There is.

  Therefore, a main object of the present invention is to provide a novel game program and game apparatus.

  Another object of the present invention is to provide a game program and a game apparatus capable of reflecting an operation by a captured user image in a game process as intended by the user.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate correspondence relationships with embodiments described later to help understanding of the present invention, and do not limit the present invention in any way.

  A first invention is a game program to be executed by a computer of a device that executes a game process based on image data indicating an image corresponding to all or part of a user's face, an imaging step for imaging a user, and imaging The face recognition step for recognizing all or part of the user's face based on the image data indicating the image captured in the step, and the user's skin color is detected based on the image data indicating the image captured in the imaging step Based on information indicating all or part of the user's face recognized by the face recognition step when it is determined that the user's skin color is detected by the skin color detection determination step and the skin color detection determination step. A game program for executing a game process step for executing a game process.

  In the first invention, the game program causes the computer of the device (10) that executes the game process based on image data indicating an image corresponding to all or part of the user's face to execute the following steps. The imaging step (50, S3) images the user. In the face recognition step (50, S7), all or part of the user's face is detected based on the image data indicating the image captured in the imaging step. In the skin color detection step (50, S3, S5), it is determined whether or not the user's skin color is detected based on the image data indicating the image captured in the imaging step. In the game processing step (50, S9, S15, S17, S19), when it is determined that the user's skin color is detected by the skin color detection determination step ("YES" in S5), the user recognized by the face recognition step The game process is executed based on information indicating all or part of the face.

  According to the first aspect of the invention, the game process is executed based on information indicating all or part of the user's face only when the user's skin color is detected from the captured user image. Can be reflected in the game process as intended by the user.

  The second invention is dependent on the first invention, and further executes a skin color area calculating step for calculating the size of the skin color area based on the image data indicating the image captured in the imaging step, and the skin color detection determining step Determines that the user's skin color has been detected when the size of the skin color region calculated in the skin color region calculation step is equal to or greater than a predetermined value.

  In the second invention, the game program further causes the skin color area calculating step (50, S5) to be executed. The skin color detection determination step determines that the user's skin color has been detected when the size of the skin color region calculated by the skin color region calculation step is equal to or greater than a predetermined value.

  According to the second invention, since it is determined that the skin color is detected only when the size of the skin color region is equal to or larger than the predetermined value, for example, the average skin color in the play state intended by the game programmer or developer By setting the size of the area to a predetermined value, the game can be played in the intended play state.

  A third invention is according to the second invention, wherein the skin color area calculating step calculates a ratio of the skin color area in the image captured by the imaging step, and the skin color detecting step is calculated by the skin color area calculating step. When the ratio is equal to or greater than the predetermined value, it is determined that the user's skin color has been detected.

  In the third invention, the skin color area calculating step calculates the ratio of the skin color area in the captured image. The skin color detection step determines that the user's skin color has been detected when the ratio calculated by the skin color region calculation step is equal to or greater than a predetermined value.

  In the third invention, similarly to the second invention, the game can be played in the intended play state.

  A fourth invention is according to the second invention, and further executes an area setting step for setting a specific area in the image captured by the imaging step, and the skin color area calculating step is set by the area setting step The size of the skin color area in the specific area is calculated.

  In the fourth invention, the region setting step (50) sets a specific region (110, 110 ') in the image photographed by the photographing step. Therefore, for example, the skin color area calculating step calculates the size of the skin color area in the specific area set by the area setting step.

  According to the fourth invention, since a specific area is set, whether the skin color is detected by a relatively simple process if a place where a user's face or hand is thought to be present is set as the specific area It can be judged.

  The fifth invention is dependent on the first to fourth inventions, and the face recognition step recognizes the user's face and detects the skin color of the user based on the image data indicating the image captured by the imaging step. When it is determined that the skin color of the user is detected by the skin color detection determination step, the game processing step indicates face information indicating all or part of the user's face recognized by the face recognition step and the user's skin color. Game processing is executed based on the skin color information.

  In the fifth invention, the face recognition step recognizes the user's face and detects the skin color of the user based on the image data indicating the image captured by the imaging step. The game processing step executes game processing based on information indicating all or part of the user's face detected by the skin color detection determination step and the user's skin color information.

  According to the fifth aspect of the invention, the game process is executed based on the skin color and the information indicating the face. For example, only the skin color is detected and the face that does not detect the face information is hidden. The game can be progressed based on the state of detecting the skin color and showing the face as if the information indicating the face is detected.

  The sixth invention is dependent on the first invention, and the skin color feature data calculation for calculating the skin color feature data indicating the skin color feature of the user based on the image data indicating the image obtained by imaging the user in advance. Whether or not the skin color detection determination step detects the skin color of the user based on the image data indicating the image captured in the imaging step and the skin color feature data calculated in the skin color feature data calculation step. Determine.

  In the sixth invention, the game program further executes a skin color feature data calculation step (50). The skin color feature data calculation step calculates skin color feature data (hue, saturation, and luminance threshold values) indicating the skin color features of the user based on image data indicating an image obtained by imaging the user in advance. In the skin color detection determination step, it is determined whether or not the user's skin color is detected based on the image data indicating the image captured in the imaging step and the skin color feature data calculated in the skin color feature data calculation step.

  According to the sixth aspect, since it is determined whether or not the user's skin color is detected based on the user's skin color feature obtained from the pre-captured image, the skin color feature is obtained prior to the start of the game. By doing so, it is possible to eliminate errors due to skin color characteristics that are different for each user and errors due to differences in time and place for playing the game.

  A seventh invention is dependent on the first invention, and a feature point data acquiring step for acquiring all or a part of feature point data of a user's face based on image data indicating an image captured by the imaging step; And a feature point data determination step for determining whether or not the feature point data acquired by the feature point data acquisition step satisfies a predetermined condition, and the game processing step detects the skin color of the user by the skin color detection determination step. Based on information indicating all or part of the face of the user recognized by the face recognition step when it is determined that the feature point data satisfies the predetermined condition by the feature point data determination step. Execute game processing.

  In the seventh invention, the feature point data acquisition step (50) acquires all or part of the feature point data of the user's face based on the image data indicating the image captured by the imaging step. The feature point data determination step (50, S9) determines whether or not the feature point data acquired by the feature point data acquisition step satisfies a predetermined condition. In the game processing step, it is determined that the user's skin color is detected in the skin color detection determination step (“YES” in S5), and the feature point data determination step determines that the feature point data satisfies a predetermined condition. (“YES” in S9), the game process is executed based on information indicating all or part of the user's face recognized in the face recognition step.

  According to the seventh aspect, since the game process is executed when the skin color of the user is detected and it is determined that the feature point data satisfies the predetermined condition, the game process is executed more accurately based on the image of the user's face. The game process can be executed.

  An eighth invention is a game device that executes a game process based on image data indicating an image corresponding to the whole or a part of a user's face, and is an image picked up by the image pickup means and the image pickup means. Whether or not the user's skin color has been detected is determined based on image data indicating an image captured by the image recognition unit and a face recognition unit that recognizes all or part of the user's face. A game that executes a game process based on information indicating all or part of the user's face recognized by the face recognition means when the skin color detection determination means and the skin color detection determination means determine that the user's skin color has been detected. It is a game device provided with a processing means.

  In the eighth invention as well, similar to the first invention, the operation by the captured user image can be reflected in the game process as intended by the user.

  According to the present invention, the game process is executed based on information indicating all or part of the user's face only when the user's skin color is detected from the captured user's image. Can be reflected in the game process as intended by the user.

  The above object, other objects, features and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

  Referring to FIG. 1, a game apparatus 10 according to an embodiment of the present invention includes an upper housing 12 and a lower housing 14, and the upper housing 12 and the lower housing 14 are connected to each other so as to be openable and closable (foldable). Has been. In the example of FIG. 1, the upper housing 12 and the lower housing 14 are each formed in a horizontally-long rectangular plate shape, and are connected so as to be rotatable at the long side portions of each other. That is, the game apparatus 10 of this embodiment is a foldable portable game apparatus, and FIG. 1 shows the game apparatus 10 in an opened state (open state). The game apparatus 10 is configured in such a size that the user can hold it with both hands or one hand even in an open state.

  Normally, the user uses the game apparatus 10 in the open state. Further, the user stores the game apparatus 10 in a closed state when not using the game apparatus 10. However, the game apparatus 10 is not limited to the closed state and the open state, and the friction generated in the connection portion is not limited to the angle formed by the upper housing 12 and the lower housing 14 between the closed state and the open state. The opening / closing angle can be maintained by force or the like. That is, the upper housing 12 can be made stationary with respect to the lower housing 14 at an arbitrary angle.

  The game apparatus 10 is equipped with cameras (32, 34) to be described later. Images are captured by the cameras (32, 34), the captured images are displayed on the screen, and data of the captured images is displayed. It also functions as an imaging device that stores data.

  As shown in FIG. 1, the upper housing 12 is provided with a first LCD 16, and the lower housing 14 is provided with a second LCD 18. The first LCD 16 and the second LCD 18 have a horizontally long shape, and are arranged such that the long side directions thereof coincide with the long side directions of the upper housing 12 and the lower housing 14. For example, the resolution of the first LCD 16 and the second LCD 18 is set to 256 (horizontal) × 192 (vertical) pixels (dots).

  In this embodiment, an LCD is used as the display, but an EL (Electronic Luminescence) display or a plasma display may be used instead of the LCD. In addition, the game apparatus 10 can use a display device having an arbitrary resolution. Although the details will be described later, the second LCD 18 is mainly used for displaying an image (still image or moving image) captured by the inner camera 32 or the outer camera 34 in real time or displaying a game screen. It is done.

  The lower housing 14 is provided with operation buttons 20a-20k as input devices. As shown in FIG. 1, among the operation buttons 20a-20k, a direction input button 20a, an operation button 20b, an operation button 20c, an operation button 20d, an operation button 20e, a power button 20f, a start button 20g, and a select button 20h. Are arranged on the surface (inner surface) of the lower housing 14 on the side where the second LCD 18 is provided. Specifically, the direction input button 20a and the power button 20f are arranged on the left side of the second LCD 18, and the operation buttons 20b-20e, 20g, and 20h are arranged on the right side of the second LCD 18. When the upper housing 12 and the lower housing 14 are folded, the operation buttons 20a-20h are housed inside the game apparatus 10.

  The direction input button (cross key) 20a functions as a digital joystick, and is used for instructing the moving direction of the player object, moving the cursor, and the like. Each operation button 20b-20e is a push button, and is used, for example, to cause the player object to perform an arbitrary action, execute determination or cancellation, and the like. The power button 20f is a push button and is used to turn on / off the main power of the game apparatus 10. The start button 20g is a push button, and is used to interrupt (pause) or start (resume) the game. The select button 20h is a push button, and is used for selecting a game mode or a menu.

  Although the operation buttons 20i-20k are omitted in FIG. 1, as shown in FIG. 2A, the operation button (L button) 20i is provided at the left end portion of the upper side surface of the lower housing 14, and is operated. The (R button) 20j is provided at the right end of the upper side surface of the lower housing 14. As shown in FIG. 2B, the volume button 20k is provided on the left side surface of the lower housing 14.

  2A is a view as seen from the upper surface (upper housing 12) side in a state where the game apparatus 10 is folded, and FIG. 2B is a left side view when the game apparatus 10 is also folded. It is the figure seen from the surface.

  The L button 20i and the R button 20j are push buttons, and can be used for the same operation as the operation buttons 20b-20e or can be used for an auxiliary operation of the operation buttons 20b-20e. In this embodiment, the L button 20i and the R button 20j can also be used to perform a shooting instruction operation (shutter operation). The volume button 20k is configured by using two push buttons, and is used to adjust the volume of sound output from two speakers (not shown) (right speaker and left speaker). In this embodiment, the volume button 20k is provided with an operation part including two pressing parts, and the push button is provided corresponding to each pressing part. Accordingly, when one of the pressing parts is pressed, the volume is increased, and when the other pressing part is pressed, the volume is decreased. For example, when the pressing unit is continuously pressed, the volume is gradually increased or gradually decreased.

  Returning to FIG. 1, the game apparatus 10 further includes a touch panel 22 as an input device different from the operation buttons 20 a to 20 k. The touch panel 22 is attached so as to cover the screen of the second LCD 18. In this embodiment, the touch panel 22 is, for example, a resistive film type touch panel. However, the touch panel 22 is not limited to the resistive film type, and any pressing type touch panel can be used. In this embodiment, the touch panel 22 having, for example, the same resolution (detection accuracy) as that of the second LCD 18 is used. However, the resolution of the touch panel 22 and the resolution of the second LCD 18 do not necessarily match.

  Further, an insertion port (broken line shown in FIG. 1) is provided on the right side surface of the lower housing 14. The insertion slot can accommodate a touch pen 24 that is used to perform an operation on the touch panel 22. Normally, input to the touch panel 22 is performed using the touch pen 24, but the touch panel 22 is not limited to the touch pen 24 and can be operated with a user's finger. Therefore, for example, when the touch pen 24 is not used, it is not necessary to provide an insertion port and a storage unit for the touch pen 24.

  Further, an insertion slot (indicated by a two-dot chain line in FIG. 1) for storing the memory card 26 is provided on the right side surface of the lower housing 14. A connector (not shown) for electrically connecting the game apparatus 10 and the memory card 26 is provided inside the insertion slot. The memory card 26 is an SD card, for example, and is detachably attached to the connector. The memory card 26 is used, for example, for storing (saving) an image captured by the game apparatus 10 or reading an image generated (captured) or stored by another apparatus into the game apparatus 10.

  Further, an insertion port (indicated by a one-dot chain line in FIG. 1) for storing the memory card 28 is provided on the upper side surface of the lower housing 14. A connector (not shown) for electrically connecting the game apparatus 10 and the memory card 28 is also provided inside the insertion slot. The memory card 28 is a recording medium on which a game program, other application programs, and necessary data are recorded, and is detachably attached to an insertion port provided in the lower housing 14.

  An indicator 30 is provided at the left end portion of the connecting portion (hinge) between the upper housing 12 and the lower housing 14. The indicator 30 includes three LEDs 30a, 30b, and 30c. Here, the game apparatus 10 can perform wireless communication with other devices, and the first LED 30a lights up when wireless communication is established. The second LED 30b lights up while the game apparatus 10 is being charged. The third LED 30c is lit when the main power supply of the game apparatus 10 is on. Therefore, the indicator 30 (LEDs 30a to 30c) can notify the user of the communication establishment status, the charging status, and the main power on / off status of the game apparatus 10.

  As described above, the upper housing 12 is provided with the first LCD 16. In this embodiment, the touch panel 22 is provided so as to cover the second LCD 18, but the touch panel 22 may be provided so as to cover the first LCD 16. Alternatively, two touch panels 22 may be provided so as to cover each of the first LCD 16 and the second LCD 18. For example, the second LCD 18 displays an operation explanation screen or a game screen for teaching the user the role or operation method of the operation buttons 20a-20k and the touch panel 22.

  The upper housing 12 is provided with two cameras (an inner camera 32 and an outer camera 34). As shown in FIG. 1, the inner camera 32 is in the vicinity of the connecting portion between the upper housing 12 and the lower housing 14, and the display surface of the first LCD 16 and its photographing surface are provided on the surface on which the first LCD 16 is provided. Mounted parallel or flush. On the other hand, as shown in FIG. 2A, the outer camera 34 is a surface opposite to the surface to which the inner camera 32 is attached, that is, the outer surface of the upper housing 12 (the game apparatus 10 is in a closed state). In this case, the outer surface is attached to the rear surface of the upper housing 12 shown in FIG. However, in FIG. 1, the outer camera 34 is indicated by a broken line.

  Therefore, the inner camera 32 can take an image in the direction in which the inner surface of the upper housing 12 faces, and the outer camera 34 has a direction opposite to the imaging direction of the inner camera 32, that is, the outer surface of the upper housing 12. It is possible to image the direction in which the Thus, in this embodiment, the two cameras 32 and 34 are provided so that the imaging directions of the inner camera 32 and the outer camera 34 are opposite to each other. For example, a user who holds the game device 10 can take a picture (for example, including the user) of the user looking at the user from the game device 10 side, and can capture images of the user from the game device 10 side. A scene viewed from the opposite direction can be captured by the outer camera 34.

  A microphone 84 (see FIG. 3) is housed as an audio input device on the inner surface near the connecting portion. A hole 36 for the microphone 84 is formed on the inner surface near the connecting portion so that the microphone 84 can detect the sound outside the game apparatus 10. The position where the microphone 84 is stored and the position of the hole 36 for the microphone 84 do not necessarily have to be the connecting portion. For example, the microphone 84 is stored in the lower housing 14, and the lower housing corresponds to the storage position of the microphone 84. 14 may be provided with a hole 36 for the microphone 84.

  A fourth LED 38 (shown by a broken line in FIG. 1) is attached to the outer surface of the upper housing 12 in the vicinity of the outer camera 34. The fourth LED 38 is lit when a picture is taken by the inner camera 32 or the outer camera 34 (the shutter button is pressed). In addition, when a moving image is shot by the inner camera 32 or the outer camera 34, the fourth LED 38 continues to be lit during shooting. That is, by turning on the fourth LED 38, it is possible to notify the person to be photographed and the surroundings that photographing by the game apparatus 10 has been performed (performed).

  In addition, sound release holes 40 are formed in the upper housing 12 on both sides of the first LCD 16. The speaker described above is accommodated in the upper housing 12 at a position corresponding to the sound release hole 40. The sound release hole 40 is a hole for releasing sound from the speaker to the outside of the game apparatus 10.

  As described above, the upper housing 12 is a display means for displaying the inner camera 32 and the outer camera 34, which are components for capturing images, and mainly the captured images and the game screen. 1LCD16 is provided. On the other hand, the lower housing 14 includes an input device (operation buttons 20 (20a-20k) and a touch panel 22) for performing an operation input to the game apparatus 10, and a display means for displaying an operation explanation screen and a game screen. A second LCD 18 is provided. Therefore, the game apparatus 10 includes two screens (16, 18) and two operation units (20, 22).

  FIG. 3 is a block diagram showing an electrical configuration of the game apparatus 10 of this embodiment. As shown in FIG. 3, the game apparatus 10 includes a CPU 50, a main memory 52, a memory control circuit 54, a storage data memory 56, a preset data memory 58, a memory card interface (memory card I / F) 60, a memory card I. / F62, wireless communication module 64, local communication module 66, real-time clock (RTC) 68, power supply circuit 70, interface circuit (I / F circuit) 72, first GPU (Graphics Processing Unit) 74, second GPU 76, first VRAM ( (Video RAM) 78, second VRAM 80, and LCD controller 82. These electronic components (circuit components) are mounted on an electronic circuit board and stored in the lower housing 14 (or the upper housing 12).

  The CPU 50 is information processing means for executing a predetermined program. In this embodiment, the predetermined program is stored in a memory (for example, the storage data memory 56) or the memory card 26 and / or 28 in the game apparatus 10, and the CPU 50 executes the predetermined program by executing the predetermined program. A game process to be described later is executed.

  The program executed by the CPU 50 may be stored in advance in a memory in the game apparatus 10, may be acquired from the memory card 26 and / or 28, or may be obtained by communicating with other devices. You may acquire from another apparatus.

  A main memory 52, a memory control circuit 54, and a preset data memory 58 are connected to the CPU 50. In addition, a storage data memory 56 is connected to the memory control circuit 54. The main memory 52 is a storage means used as a work area or buffer area for the CPU 50. That is, the main memory 52 stores (temporarily stores) various data used for the game processing, and stores programs acquired from the outside (such as the memory cards 26 and 28 and other devices). In this embodiment, for example, a PSRAM (Pseudo-SRAM) is used as the main memory 52. The storage data memory 56 is a storage unit for storing (saving) a program executed by the CPU 50, data of images taken by the inner camera 32 and the outer camera 34, and the like. The storage data memory 56 is configured by a nonvolatile storage medium, and for example, a NAND flash memory can be used. The memory control circuit 54 controls reading and writing of data with respect to the storage data memory 56 in accordance with instructions from the CPU 50. The preset data memory 58 is a storage means for storing data (preset data) such as various parameters set in advance in the game apparatus 10. As the preset data memory 58, a flash memory connected to the CPU 50 by an SPI (Serial Peripheral Interface) bus can be used.

  Memory card I / Fs 60 and 62 are each connected to CPU 50. The memory card I / F 60 reads and writes data to and from the memory card 26 attached to the connector according to instructions from the CPU 50. Further, the memory card I / F 62 reads and writes data from and to the memory card 28 attached to the connector according to instructions from the CPU 50. In this embodiment, image data corresponding to images taken by the inner camera 32 and the outer camera 34 and image data received from another device are written to the memory card 26 or image data stored in the memory card 26. Is read from the memory card 26 and stored in the storage data memory 56 or transmitted to another device. Various programs stored in the memory card 28 are read out and executed by the CPU 50.

  Note that the game program may be supplied not only to the game apparatus 10 through an external storage medium such as the memory card 28 but also to the game apparatus 10 through a wired or wireless communication line. Further, the game program may be recorded in advance in a nonvolatile storage device inside the game apparatus 10. Furthermore, the information storage medium for storing the game program is not limited to the nonvolatile storage device, but may be a CD-ROM, a DVD, or an optical disk storage medium similar to them.

  The wireless communication module 64 is, for example, IEEE802.11. It has a function of connecting to a wireless LAN by a method compliant with the b / g standard. Further, the local communication module 66 has a function of performing wireless communication with the same type of game device by a predetermined communication method. The wireless communication module 64 and the local communication module 66 are connected to the CPU 50. The CPU 50 transmits / receives data to / from other devices via the Internet using the wireless communication module 64, and transmits / receives data to / from other game devices of the same type using the local communication module 66. be able to.

  Further, the RTC 68 and the power supply circuit 70 are connected to the CPU 50. The RTC 68 counts the time and outputs it to the CPU 50. For example, the CPU 50 can calculate the current time (date) based on the time counted by the RTC 68. The power supply circuit 70 controls power supplied from a power supply (typically a battery and is housed in the lower housing 14) of the game apparatus 10, and supplies power to each circuit component of the game apparatus 10. .

  The game apparatus 10 includes a microphone 84 and an amplifier 86. The microphone 84 and the amplifier 86 are connected to the I / F circuit 72, respectively. The microphone 84 detects the user's voice or sound (applause, hand clapping, etc.) uttered or generated toward the game apparatus 10, and outputs an audio signal indicating the sound or sound to the I / F circuit 72. The amplifier 86 amplifies the audio signal given from the I / F circuit 72 and gives it to a speaker (not shown). The I / F circuit 72 is connected to the CPU 50.

  The touch panel 22 is connected to the I / F circuit 72. The I / F circuit 72 includes a sound control circuit that controls the microphone 84 and the amplifier 86 (speaker), and a touch panel control circuit that controls the touch panel 22. The voice control circuit performs A / D conversion and D / A conversion on the voice signal, or converts the voice signal into voice data of a predetermined format. The touch panel control circuit generates touch position data in a predetermined format based on a signal from the touch panel 22 and outputs it to the CPU 50. For example, the touch position data is data indicating coordinates of a position where an input is performed on the input surface of the touch panel 22.

  The touch panel control circuit reads signals from the touch panel 22 and generates touch position data at a rate of once per predetermined time. The CPU 50 can know the position where the input is performed on the touch panel 22 by acquiring the touch position data via the I / F circuit 72.

  The operation button 20 includes the operation buttons 20a-20k described above and is connected to the CPU 50. From the operation button 20 to the CPU 50, operation data indicating an input state (whether or not the button is pressed) for each operation button 20a-20k is output. CPU50 acquires the operation data from the operation button 20, and performs the process according to the acquired operation data.

  The inner camera 32 and the outer camera 34 are each connected to the CPU 50. The inner camera 32 and the outer camera 34 capture an image in accordance with an instruction from the CPU 50, and output image data corresponding to the captured image to the CPU 50. In this embodiment, the CPU 50 issues an imaging instruction to one of the inner camera 32 and the outer camera 34, and the camera (32, 34) receiving the imaging instruction captures an image and sends the image data to the CPU 50. .

  A first VRAM 78 is connected to the first GPU 74, and a second VRAM 80 is connected to the second GPU 76. The first GPU 74 generates a first display image based on data for generating a display image stored in the main memory 52 in accordance with an instruction from the CPU 50, and draws the first display image in the first VRAM 78. Similarly, the second GPU 76 generates a second display image in accordance with an instruction from the CPU 50 and draws it in the second VRAM 80. The first VRAM 78 and the second VRAM 80 are connected to the LCD controller 82.

  The LCD controller 82 includes a register 82a. The register 82a stores a value “0” or “1” in accordance with an instruction from the CPU 50. When the value of the register 82 a is “0”, the LCD controller 82 outputs the first display image drawn in the first VRAM 78 to the second LCD 18 and the second display image drawn in the second VRAM 80 as the second display image. 1 Outputs to the LCD 16. When the value of the register 82 a is “1”, the first display image drawn in the first VRAM 78 is output to the first LCD 16, and the second display image drawn in the second VRAM 80 is output to the second LCD 18. To do. In this embodiment, the CPU 50 causes the first LCD 16 to display the silhouette of the face (skin color area) and the game screen corresponding to the images taken by the inner camera 32 and the outer camera 34, and displays an operation explanation screen generated by a predetermined process. A game screen or the like is displayed on the second LCD 18.

  Although illustration is omitted, for example, a shooting instruction button explanation image, a camera switching button explanation image, a zoom instruction button explanation image, a display image selection button explanation image, and the like are displayed on the second LCD 18 as the operation explanation screen. The shooting instruction button explanation image is an image showing the position of the operation button 20 for giving a shooting instruction. The camera switching button explanation image is an image indicating the position of the operation button 20 for instructing camera switching. Here, the camera switching instruction is an instruction to switch a camera to be imaged among the inner camera 32 and the outer camera 34. The zoom instruction button explanation image is an image indicating the position of the operation button 20 for performing a zoom change instruction. Here, the zoom change instruction is an instruction to enlarge / reduce the image displayed on the first LCD 16. The display image selection button explanation image is an image indicating the position of the operation button 20 for performing a display change instruction. Here, the display image selection instruction is an instruction to select a stored image to be displayed on the first LCD 16 when the stored image stored in the game apparatus 10 is displayed on the first LCD 16.

  FIG. 4 shows an example of a game screen 100 displayed on the first LCD 16 when playing a virtual game (action game) using the game apparatus 10 of this embodiment. In the game screen 100 shown in FIGS. 4A and 4B, an object (lip object) 102 that imitates a human lip is displayed at the approximate center thereof. In addition, an object (pan object) 104 imitating a pan is displayed below the lip object 102.

  In this virtual game, when the user or player opens and closes his mouth so as to eat bread, it detects it based on the image taken by the inner camera 32, opens and closes the lip object 102 displayed on the game screen 100, A state of eating the bread object 104 is expressed. For example, if the bread object 104 is eaten up within the time limit, the game is cleared, but if the bread object 104 is not eaten up within the time limit, the game is over.

  As shown in FIGS. 4A and 4B, the user's face, that is, the skin color, is extracted from the photographed image, and the extracted skin color region is displayed on the game screen 100 as a silhouette 106. In this embodiment, whether or not the user's face is recognized, and if the user's face is recognized, the size (distance between the game screen 100 (inner camera 32) and the user's face) is determined. Whether or not it is appropriate is notified to the user by the shape and size of the silhouette 106.

  As shown in FIGS. 4 and 5, the recognized user's face is a posture or an intention intended by the developer or programmer as long as it is substantially elliptical and fits in the game screen (100, 200). It can be said that it is in a state.

  However, in this embodiment, as shown in FIG. 5, a specific area (specific area) 110 is set in the display area of the first LCD 16, and the specific area 110 is flesh colored in units of 4 pixels (2 × 2 grids). In the total number of grids (24 × 56 ÷ 4 = 338 in this embodiment), whether the ratio between the skin color and the number of detected grids is a predetermined value (for example, 35%) or more. No, it is determined whether the user's face is correctly recognized (skin color determination process). That is, whether the position of the user's face (head) with respect to the game screen (first LCD 16) or the inner camera 32 (position in the front-rear direction and the left-right direction) is an appropriate position for the play state intended by the programmer or developer. Is judged.

  Note that a method for extracting skin color from a captured image (skin color extraction processing) will be described in detail later. Similarly, threshold value adjustment processing used for skin color extraction processing will be described in detail later.

  As shown in FIG. 5, the coordinate system of the first LCD 16 has the upper left vertex of the display area as the origin O, the right direction of the drawing is the positive direction of the X axis, and the downward direction of the drawing is the positive direction of the Y axis. Direction. Hereinafter, this is the same in this embodiment.

  That is, it is determined whether or not the user's face is near the center of the screen. If the user's face does not exist near the center of the screen, it is determined that the user's mouth is not near the pan object 104, and the opening / closing of the mouth is not detected. On the other hand, when the user's face exists near the center of the screen, it is determined that the user's mouth is also near the pan object 104, and the opening / closing of the mouth is detected.

  The specific area 110 is defined by a rectangle having a length of 12 pixels (dots) on the left and right and a length of 28 pixels on the top and bottom with respect to the center of the screen (display area) of the first LCD 16. However, this is only an example, and the position and size (shape) of the specific area 110 can be arbitrarily set depending on the type of game.

  Further, during the game, a feature point of the user is detected from the image taken by the inner camera 32, and based on the feature point, the user's face is recognized or a specific facial expression (face movement) of the user is detected. I try to do it. However, the reason why the user's face is recognized not only based on the skin color area but also based on the feature point is to prevent a malfunction that erroneously recognizes the skin color area other than the face as the face.

  Here, a specific determination process for opening and closing the mouth will be described. FIG. 6 is an illustrative view showing a user's face image photographed by the inner camera 32 and its feature points Pi (i = 1, 2,..., 54, 55). However, as shown in FIG. 7, the image developed in the main memory 52 is shown in the camera coordinate system, and the image development region (development region) 150 is a size corresponding to the size of the display region of the first LCD 16. Now, it is secured in the main memory 52. The top left vertex of the development area 150 is the origin O of the camera coordinates. In the camera coordinate system, the right direction of the drawing is the positive direction of the X axis, and the downward direction of the drawing is the positive direction of the Y axis. The face images shown in FIGS. 6 and 7 are user face images taken by the inner camera 32, and a mirror image of the face image is developed in the development area 150.

  In FIG. 7, for convenience of drawing, the feature point Pi shown in FIG. 6 is omitted, and only the face image is shown.

  First, among the feature points Pi, the midpoint between the coordinates (X1, Y1) of the feature point P31 at the upper center of the upper lip and the coordinates (X2, Y2) of the feature point P35 at the center of the lower end of the upper lip (middle point of the upper lip) LipU coordinates (LipU_X, LipU_Y) are calculated, and the coordinates of the feature point P38 (X3, Y3) at the center of the upper end of the lower lip and the coordinates (X4, Y4) of the feature point P41 of the lower end center of the lower lip are calculated. The coordinates (LipD_X, LipD_Y) of the point (midpoint of the lower lip) LipD are calculated. Further, a difference (lip difference) (Lip_dX, Lip_dY) in the respective axial directions of these two middle points LipU and LipD is calculated. However, the lip difference (Lip_dX, Lip_dY) is an absolute value and is calculated according to Equation 1.

[Equation 1]
Lip_dX = | LipU_X-LipD_X |
Lip_dY = | LipU_Y-LipD_Y |
Of the feature points Pi, the coordinates (X5, Y5) of the feature point P17 at the center of the left eye (center of the pupil) and the coordinates (X6, Y6) of the feature point P13 of the center of the upper end of the left eye (center of the upper eyelid). The coordinates (EyeL_X, EyeL_Y) of the midpoint (midpoint of the left eye) EyeL are calculated. Similarly, among the feature points Pi, the coordinates (X7, Y7) of the feature point P24 at the center of the right eye (pupil center) and the coordinates (X8, Y8) of the feature point P20 at the center of the upper end of the right eye (center of the upper eyelid) Calculate the coordinates (EyeR_X, EyeR_Y) of the middle point (the middle point of the right eye) EyeR. Also, the difference (eye difference) (Eye_dX, Eye_dY) of each of these two midpoints EyeL and EyeR in each axial direction is calculated. However, the eye difference (Eye_dX, Eye_dY) is an absolute value and is calculated according to Equation 2.

[Equation 2]
Eye_dX = | EyeL_X-EyeR_X |
Eye_dY = | EyeL_Y-EyeR_Y |
Subsequently, it is determined whether or not Expression 3 to Expression 7 is satisfied. This is to eliminate the case where the game is played in a mode different from the play state intended by the developer or programmer. However, “||” means or (∪).

[Equation 3]
(LipU_X <16 || LipU_X> 240 || LipU_Y <16 || LipU_Y> 176)
(LipD_X <16 || LipD_X> 240 || LipD_Y <16 || LipD_Y> 176)
(EyeL_X <16 || EyeL_X> 240 || EyeL_Y <16 || EyeL_Y> 176)
(EyeR_X <16 || EyeR_X> 240 || EyeR_Y <16 || EyeR_Y> 176)
[Equation 4]
Lip_dX> 64
[Equation 5]
Lip_dY> 96
[Equation 6]
Eye_dX> 96
[Equation 7]
Eye_dY> 24
If any one of these equations 3 to 7 is satisfied, it is determined that the play state assumed by the developer or the developer is not assumed, and the current mouth open / close determination processing is invalidated. Specifically, when any one of Equation 3 is satisfied, it is determined that the feature point Pi is too close to the edge of the screen. That is, the user's lips and eyes are too close to the edge of the screen. In such a case, it is determined that there is no false detection or a play state assumed by a developer or the like.

  When Expression 4 is satisfied, it is determined that the upper and lower lips are shifted too much to the left and right. In such a case, it is determined that the play state is not assumed by the developer or the like, such as erroneous detection or the user's face being too close to the screen (first LCD 16). Further, when Expression 5 is satisfied, it is determined that the upper and lower lips are too open up and down. In such a case, it is determined that the play state is not assumed by the developer, such as erroneous detection or the user's face being too close to the screen (first LCD 16).

  When Expression 6 is satisfied, it is determined that the left and right eyes are too far apart. In such a case, it is determined that the play state is not assumed by the developer or the like, such as erroneous detection or the user's face being too close to the screen (first LCD 16). Further, when Expression 7 is satisfied, it is determined that the left and right eyes are shifted too much up and down. In such a case, it is determined that the play state is not assumed by the developer or the like, such as erroneous detection or the user's face being too close to the screen (first LCD 16).

  If the user's face (skin color area) is correctly recognized by the skin color determination process and none of the above formulas 3 to 7 is satisfied, it is determined that the play state is intended by the developer and the like. Judge whether to open or close. When Expression 8 is satisfied, it is determined that the user is closing his mouth, while when Expression 9 is satisfied, it is determined that the user is opening his mouth. However, the bread object 104 is eaten once by the lip object 102 when it is determined that the user has closed the mouth after opening the mouth.

[Equation 8]
Lip_dY <16
[Equation 9]
Lip_dY ≧ 16
In this way, in the game shown in FIGS. 4A and 4B, if it is determined by the skin color determination process that the silhouette 106 of the user's face (skin color region) is near the pan object 104, Based on the point Pi, it is determined whether or not the play state is intended by the developer or the like. Here, if the play state is intended, it is further determined whether or not the user has opened and closed the mouth based on the feature point Pi. When it is determined that the user has opened and closed his / her mouth, a game screen 100 (not shown) of eating bread is displayed on the first LCD 16 accordingly. Therefore, for example, in the state shown in FIG. 4A, the lip object 102 opens and closes. At this time, the pan object 104 is gnawed by the lip object 102, and as shown in FIG. A game screen 100 is displayed as if the game has been lost.

  However, in the game screen 100 shown in FIG. 4B, after the pan object 104 is gnawed, it is further detected that the user's mouth has been opened and closed. In response, the lip object 102 is opened and the pan object is opened. A state of trying to bite 104 is shown.

  In this embodiment, the bread is divided into a plurality of times (for example, two times) and eaten, so that if the user's mouth opening / closing is detected a plurality of times within the time limit, all the bread objects 104 are lost. That is, when the predetermined goal is achieved, the game is cleared. However, if the opening / closing of the user's mouth is not detected a plurality of times within the time limit, the pan object 104 is not lost, the predetermined goal cannot be achieved, and the game is over.

  8A and 8B show examples of game screens 200 of other virtual games (action games). As shown in FIG. 8 (A), an object imitating a baby (baby object) 202 is displayed in the approximate center of the game screen 200, and an object imitating a stuffed toy puppy (stuffed object) 204 is on the left side. Is displayed. In addition, an instruction image display area 206 for instructing the user to perform an action to be performed (gesturing hand gesture) is provided at the upper left portion of the game screen 200. The game screen 200 displays a silhouette 208 of the user's face (skin color area).

  In the virtual game using the game screen 200 shown in FIG. 8A and FIG. 8B, if the user succeeds in the so-called “not-a-life” operation within the time limit, the baby object 202 is displayed. The game can be completed by accomplishing a predetermined purpose. However, if the user fails in the “not in case” operation and exceeds the time limit, that is, if the baby object 202 cannot be lifted within the time limit, the game is over.

  Even in such a virtual game, whether or not the user's face is correctly recognized is determined by the skin color determination process, and if the user's face is correctly recognized, the user is further determined based on the feature point Pi of the user's face. By recognizing a part of the face of the person, the operation of “I don't know” is detected.

  First, in this virtual game, skin color determination processing is performed as described above, but a specific area 110 ′ having a size (range) different from that of the specific area 110 illustrated in FIG. 5 is set. Specifically, as shown in FIG. 9, with the center of the screen (display end area) of the first LCD 16 as a reference, it has 28 pixels on the left and right, 28 pixels on the top, and 12 pixels on the bottom. The specific area 110 ′ is set to have. This is because when a user covers his / her face with a hand, the operation differs for each user, such as when only the eyes are hidden and when the eyes, nose and mouth are hidden.

  In addition, the skin color area is determined in this way because the skin color should be somewhat distributed near the center of the screen of the first LCD 16 when the user's face is visible. This is because the skin color is considered to be distributed even when the face is covered with bare hands.

  As described above, the skin color area is determined by recognizing the face only by the feature point Pi so that the face is moved out of the screen (where the inner camera 32 cannot shoot) and the face is not reflected. This is because the same situation as the situation where the face is covered with the hand (that is, the situation where the feature point Pi is not detected) can be created, and such a play state is not an aspect intended by the developer or the like.

  The skin color determination process is the same as that described with reference to FIG. 5, and it is detected whether the specific area 110 ′ is skin color in units of 2 × 2 pixels, and the total number of grids (in this embodiment, 56 × 40 ÷ 4 = 560), it is determined whether or not the user's face is correctly recognized depending on whether or not the ratio of the number of grids in which the skin color is detected is a predetermined value (for example, 35%) or more. It is.

  When the skin color determination process detects that the user is covering the face, next, based on the feature point Pi, as described above, the middle points EyeL and EyeR are calculated for the left and right eyes, respectively. The difference (Eye_dX, Eye_dY) is calculated. However, since the feature points Pi and the mathematical formulas (Equations 1 and 2) used for these calculations are as described above, a duplicate description is omitted.

  Next, it is determined whether or not Expression 10 is satisfied. When three or more of the four conditions of Formula 10 are satisfied, it is determined that the user's face can be detected with the size and position intended by the programmer or the like. However, “&&” means AND.

[Equation 10]
(EyeL_X> 16 && EyeL_X <240)
(EyeL_Y> 16 && EyeL_Y <176)
(EyeR_X> 16 && EyeR_X <240)
(EyeR_Y> 16 && EyeR_Y <176)
Based on the state in which the user's face can be detected and the state in which the user's face cannot be detected, it is determined whether or not the user is covering (hiding) the face with a hand. In this embodiment, the evaluation value (variable EV) is used to make the determination. However, the variable EV is an integer that varies between 0 and 100. Further, the variable EV is changed so that it is evaluated that the user's face is hidden as the variable EV is closer to 0 and the user's face is hidden as the variable EV is closer to 100. is there.

  At the start of the evaluation of whether the user is hiding his face, the variable EV is set to zero. Next, when a state in which a face can be detected is detected from the start of evaluation until it is determined that the user has hidden the face, the variable EV is subtracted according to Equation 11 to detect a state in which the face cannot be detected. Then, the variable EV is added according to Equation 12. However, as described above, the variable EV is changed so as not to exceed the lower limit value 0 and the upper limit value 100. Further, the value to be subtracted and the value to be added are empirically obtained values, and need not be limited to these.

[Equation 11]
EV = EV-12
[Equation 12]
EV = EV + 16
When the variable EV is changed according to Equations 11 and 12, and the variable EV, that is, the evaluation value exceeds a predetermined value (70 in this embodiment), it is determined that the user has hidden his / her face. Subsequently, when a state in which the face can be detected is detected from when the user hides the face until the next time the face is shown, the variable EV is subtracted according to Equation 13, and a state in which the face cannot be detected is detected. Then, the variable EV is added according to Equation 14. However, as described above, the variable EV is changed so as not to exceed the lower limit value 0 and the upper limit value 100. Further, the value to be subtracted and the value to be added are empirically obtained values, and need not be limited to these.

[Equation 13]
EV = EV-16
[Equation 14]
EV = EV + 14
When the variable EV is changed according to the equations 13 and 14, and the variable EV, that is, the evaluation value becomes equal to or less than a predetermined value (30 in this embodiment), it is determined that the user has shown his face.

  In this way, when the user hides his face and then shows his / her face, it is determined that the operation of “I'm not there” has succeeded. As described above, if the user succeeds in the operation of “if not,” the baby object 202 can be relaxed and the game is cleared.

  Here, in the virtual games shown in FIG. 4 and FIG. 8, the skin color determination process is executed in both cases. However, since the skin color varies depending on each user, and also changes depending on the environment such as the time and place where the game is played, the threshold value used for the process of extracting the skin color from the photographed image (hue lower limit, hue) Upper limit, saturation lower limit, saturation upper limit, luminance lower limit, luminance upper limit) are adjusted in advance. This adjustment process is performed using an image obtained by photographing the user prior to starting the main part of the game. However, the adjustment process described below may be executed not only before the start of the main part of the game but also at a critical point such as when the content of the game (surface, scene, etc.) changes.

  For example, prior to the start of the main game, an adjustment screen 300 as shown in FIG. 10A is displayed on the first LCD 16. The adjustment screen 300 includes a face-type index (face-type index) 302 for instructing the position and size for displaying an image (face image) obtained by capturing the user's face, and an image (the image of the user's hand). A hand-shaped index (hand-shaped index) 304 for indicating the position and size for displaying the hand image is displayed.

  When playing the virtual game of this embodiment, as shown in FIG. 11, the user places the game apparatus 10 on a table such as a desk and is inclined with respect to the horizontal plane (lower housing 14). In addition, the upper housing 12 is inclined. Therefore, the user can set the distance (front and back) between the user's face (head) or the entire body and the game apparatus 12 (inner camera 32) so that his / her face image captured by the inner camera 32 fits in the face type index 302. ) And the horizontal (horizontal) position, and the vertical position of the face (head) and the tilt of the face (head) are also adjusted. Although illustration is omitted, similarly, the user has the right hand of the user and the game device 12 (inner camera 32) so that the image of his / her right hand (palm) captured by the inner camera 32 fits in the hand index 304. The distance (front-rear position) and the horizontal position are adjusted, and the vertical position of the right hand and the tilt of the palm of the right hand are also adjusted.

  Using such an adjustment screen 300, the game apparatus 10 performs a commemorative photo of the user and acquires a plurality (3 to 4 in this example) of captured images. In this embodiment, the shooting timing is instructed by the voice guide from the game apparatus 10, and the shooting timing and shot images every few frames before about 1 second are acquired. Specifically, a message indicating that a commemorative photo is to be taken is output from the speaker of the game apparatus 10, and then a sound of “3, 2, 1, pasha” that counts down and a shutter sound are output from the speaker. However, in this embodiment, in order to obtain a plurality of photographed images, for example, 4 frames (frame: screen update unit (1/60 seconds) after counting “1” and before the shutter operation is executed. ), The captured image from the inner camera 32 is stored in the storage data memory 56. That is, since 1 second corresponds to about 16 frames, 3 to 4 captured images are acquired. The threshold value adjustment described above is executed using the data of the plurality of photographed images (hereinafter referred to as “evaluation image data”), but not only the skin color determination process but also the threshold value adjustment process and the later-described process. The CPU 50 also executes the skin color extraction process.

  In the threshold adjustment process, first, an initial value (static) is given to each threshold. For the first evaluation image data, an initial value according to the threshold table for skin color determination shown in FIG. 12 is given, but for the evaluation image data after the next time, the threshold (optimum value) converged by the previous adjustment processing. Is given as the initial value. Further, as can be seen from FIG. 12, the maximum and minimum threshold values (both static) that can be changed by the adjustment process are set in association with the respective initial values.

  However, the hue is processed with a value obtained by converting 360 angles into 256 (0 or more and less than 256) levels. Since hue circulates, 0 and 256 are equivalent. Therefore, in the case of the initial value of the hue shown in FIG. 12, the hue lower limit is 220 and the hue upper limit is 25. Therefore, the extraction ranges as skin colors are 220, 221, 222,..., 254, 255, 0, 1 , ..., 24, 25.

  Actually, it is necessary to satisfy not only the hue but also the saturation and luminance. Therefore, when all of the hue, saturation and luminance belong to the extraction range, the skin color is determined.

  Further, a predetermined pose, in this embodiment, corresponds to the appearance in which the palm of the face and the right hand fits in each of the face type index 302 and the hand type index 304 displayed on the adjustment screen 300 shown in FIG. Static data (hereinafter referred to as “determination data”). For example, the determination data is loaded from the memory card 28 attached to the game apparatus 10 and stored in the main memory 52. Specifically, the data for determination is data indicating whether each pixel of the first LCD 16 has the attribute A, B, or C (data with accuracy of every two dots). Here, the attribute A means that it should be skin color, the attribute B means that it should not be skin color, and the attribute C means that it may be either (regardless of evaluation).

  Returning to FIG. 10B, the distribution of pixels distinguished by attribute A, attribute B, and attribute C when the captured image is displayed on the adjustment screen 300 shown in FIG. The pixel indicated by attribute A is indicated by area 310. The pixel indicated by attribute B is indicated by area 312. The pixel indicated by the attribute C is indicated by an area 314 other than the area 310 and the area 312 in the display area of the first LCD 16.

The following two values Q and R are calculated for each 2 × 2 pixel grid of the evaluation image data while referring to the corresponding pixel determination data, and the evaluation value X _crt is calculated from these values Q and R. To do. However, the value Q is the number of grids in the case where the pixel of the target image data for evaluation is skin color and the determination data of the corresponding pixel indicates the attribute A. In other words, the value Q is the number of grids for which a good evaluation that the skin color is correctly extracted is obtained. On the other hand, the value R is the number of grids in the case where the pixel of the target image data for evaluation has a flesh color, but the determination data of the corresponding pixel indicates the attribute B. In other words, it is the number of grids that have been poorly evaluated as noise or incorrect extraction. The evaluation value X _crt is a value obtained by subtracting the value R from the value Q. It can be said that the larger the value, the more correctly the skin color can be extracted.

Such evaluation is performed by setting each threshold value as a reference value for the first evaluation image data. Then, the evaluation value X _crt corresponding to each threshold value is stored (temporarily stored) in the main memory 52 so as to be identifiable. Next, evaluation is performed for each threshold when each threshold is set to a minimum value and a maximum value.

For example, the hue lower limit threshold is set to the minimum value shown in FIG. 12, and the evaluation value X _min is calculated as described above. It is determined whether or not the evaluation value X _min exceeds the previous evaluation value X _pre (here, the evaluation value X _crt ). If the evaluation value X _min exceeds the previous evaluation value X _pre , it is determined that the adjustment direction is correct, and an intermediate value between the current threshold value and the current optimum value (here, the reference value) is set. A new threshold value is set, the above-described evaluation is performed again, and the threshold value with the higher evaluation is adopted as the optimum value. That is, the threshold value is set to a minimum value or an intermediate value between the minimum value and the current optimum value (here, the reference value).

On the other hand, if the evaluation value X _min is lower than the previous evaluation value X _pre , it is determined that the adjustment direction is reversed, the hue lower limit threshold is set to the maximum value shown in FIG. Thus, the evaluation value X _max is calculated. It is determined whether or not the evaluation value X _max exceeds the previous evaluation value X _pre (here, the evaluation value X _crt ). When the evaluation value X _max exceeds the previous evaluation value X _pre , an intermediate value between the current threshold value and the current optimum value is set as a new threshold value, and the above-described evaluation is performed again. The higher threshold is adopted as the optimum value. However, if the evaluation value _{Xmax is} also lower than the previous evaluation value _Xpre , it is determined that the original threshold value is the optimum value, and the threshold value is not changed.

  For other hue upper limit, saturation lower limit, saturation upper limit, luminance lower limit, and luminance upper limit, the threshold values are adjusted (selected) according to the above procedure. For the evaluation image data after the next time, a threshold value (optimum value) selected (converged) by the adjustment processing up to the previous time is given as an initial value. Then, the adjustment process is executed for all the evaluation image data, and the finally selected threshold value (optimum value) is used in the skin color extraction process described below.

  As described above, in this embodiment, the threshold value for determining the skin color of the player in the current environment (such as the lighting condition) is adjusted using a photographic image obtained by photographing the player before the start of the game. This makes it possible to accurately detect the skin color during the game.

  In this embodiment, threshold adjustment processing is performed using a plurality of photographic images. Thus, the threshold can be accurately adjusted, and the evaluation process for the (n + 1) th photographic image is efficiently performed based on the threshold optimized by the nth photographic image.

  Furthermore, in this embodiment, since the threshold value is adjusted for each of hue, saturation, and luminance, the threshold value in the environment at that time can be set accurately. Note that only the hue and saturation may be adjusted, or only the hue and luminance may be adjusted.

  Furthermore, since not only the area that should be the skin color but also the area that should be other than the skin color is determined, the threshold value can be accurately set.

  As described above, when the skin color is extracted, first, the data for the skin color extraction process is constructed from the image (captured image) captured by the inner camera 32. Here, the captured image is acquired by API in the format of RGB (each component is 5 bits). However, the photographed image is composed of 256 × 192 dots that is the same as the display area of the first LCD 16.

  The data for skin color extraction processing (extraction processing data) is generated corresponding to a grid area of 8 × 8 pixels, 4 × 4 pixels, and 2 × 2 pixels. In this embodiment, since the display area of the first LCD 16 is 256 × 192 dots, 32 × 24 grids are generated when the display area is divided into 8 × 8 pixel grid areas. An example in the case of dividing into 8 × 8 pixel grid is shown in FIG. In FIG. 13A (FIG. 13B and FIG. 14 are also the same), a grid of 8 × 8 pixels is indicated by a thick line frame. However, in FIG. 13A (FIGS. 13B and 14 are the same), the display area of the first LCD 16 is shown by 24 × 16 pixels for the sake of simplicity. In FIGS. 13 and 14, the smallest lattice (square frame) corresponds to one pixel (dot). Similarly, when the display area is divided into a 4 × 4 pixel grid area, 64 × 48 grids are generated. An example in the case of dividing into a 4 × 4 pixel grid is shown in FIG. Further, when the display area is divided into a 2 × 2 pixel grid area, 128 × 96 grids are generated. An example in the case of dividing into a 2 × 2 pixel grid is shown in FIG.

  The data for extraction processing shows information on evaluation values (different from the above-described evaluation values), state values, and color values. The evaluation value is a value indicating the likelihood of skin color, and is indicated by 8-bit data. In this embodiment, the evaluation values are 0 to 64 and 255. However, skin-likeness is indicated by 0 to 64, and 255 means invalid. In addition, the larger the evaluation value, the more likely it is that it looks like a skin color.

  The state value is data for holding the content (state) determined by each processing described later, and is represented by 16-bit data. Specifically, 1-bit information indicating whether it is skin color, 8-bit information indicating whether it is skin color for each of the eight surrounding grids, a 1-bit flag for forcibly considering itself as skin color, It is indicated by a 1-bit flag for regarding itself as not having skin color, and an extra 5 bits are added.

  The color value is 16-bit data represented in the RGB555 format, and RGB is data in which each of the RGB is expressed with a gradation of 5 bits and an extra 1 bit is added. However, when evaluating whether a captured image is a skin color in units of grids, color values (RGB555 format) are converted into luminance, hue, and saturation and used.

  Next, generation of data for extraction processing will be described. First, color values are prepared by processing 1 for a 2 × 2 pixel grid. In this embodiment, the process 1 calculates an average value of the color values of four grids (small) included in the grid (large). Here, since the grid is a 2 × 2 pixel, each of the four grids (small) included in the 2 × 2 pixel grid (large) corresponds to one pixel.

  In this embodiment, in process 1, the average value of the color values of the four grids (small) included in the grid (large) is calculated, but the upper left of the four grids (small). You may make it employ | adopt the color value of a grating | lattice.

  Next, color values are prepared by processing 1 for a 4 × 4 pixel grid. Further, color values are prepared by processing 1 for an 8 × 8 pixel grid. However, when divided into 4 × 4 pixel grids, each of the four grids (small) included in the 4 × 4 grid (large) means a 2 × 2 pixel grid. Further, when divided into 8 × 8 pixel grids, each of the four grids (small) included in the 8 × 8 pixel grid (large) means a 4 × 4 pixel grid.

  Subsequently, the color value is corrected by processing 2 for the 8 × 8 pixel grid. In this embodiment, the process 2 calculates the median luminance value based on the color value for 9 grids including 8 surrounding grids centered on its own grid, and calculates the color value of the grid having the median value. Change to the color value of your own grid.

  However, color value correction is invalidated for the outermost grid. Further, in this embodiment, the process 2 obtains the median luminance based on the color value for a total of nine grids including its own grid and eight surrounding grids including itself, and the color of the grid having the median value. Although the value is changed (corrected) to the color value of its own grid, the color value may not be corrected.

  Next, the state value is initialized by processing 3 for the 8 × 8 pixel grid. In this embodiment, the process 3 determines whether or not each grid is a skin color based on the threshold value adjusted by the threshold value adjustment process described above, and reflects it in the state value. Specifically, based on the color values prepared and corrected as described above, it is selected (determined) whether each of luminance, hue, and saturation is within or outside the threshold range.

  Subsequently, the evaluation value is initialized by processing 4 for the 8 × 8 pixel grid. In this embodiment, the process 4 is weighted from the state values of the surrounding grids and reflected in the evaluation value. Specifically, the evaluation value of the central lattice is “1” for a lattice whose determination result by the threshold is within the threshold, and a lattice whose determination result by the threshold is outside the threshold with respect to the corresponding positional relationship lattice. “0” is calculated as the sum of weighted values. When the weight value of each grid is Wi (i is the number assigned to the grid: 1-9) and the value of the determination result based on the threshold value of each grid is Hi, the evaluation value Y of the center grid is obtained according to Equation 15. It is done.

[Equation 15]
Y = W1 * H1 + W2 * H2 + W3 * H3 + W4 * H4 + W5 * H5 + W6 * H6 + W7 * H7 + W8 * H8 + W9 * H9
In this embodiment, as shown in FIG. 15A, weighting is performed on the own lattice and the surrounding eight lattices with the own lattice as the center. However, the sum of the weight values is “64”. Specifically, the weight “0” is set for the grids (1), (3), (7), and (9), and the grids (2), (4), (6), and (8) are set. A weight “10” is set for, and a weight “24” is set for the grid of (5), that is, its own grid.

  FIG. 15B shows an example of a determination result (result of process 3) based on the threshold value of each lattice. In FIG. 15B, the lattices (3), (5), (6), and (8) that are within the threshold range are marked with “◯”, and the lattices that are outside the threshold range ( 1), (2), (4), (7), (9) are marked with “x”.

  Accordingly, in the example shown in FIGS. 15A and 15B, the evaluation value Y of the central lattice is calculated as shown in Equation 16 according to Equation 15.

[Equation 16]
Y = 0 * 0 + 10 * 0 + 0 * 1 + 10 * 0 + 24 * 1 + 10 * 1 + 0 * 0 + 10 * 1 + 0 * 0 = 44
In this way, the evaluation value Y is initialized. When the evaluation value Y is initialized, the evaluation value is corrected by processing 5 for the 8 × 8 pixel grid. In this embodiment, the process 5 corrects the state value of the central lattice based on the evaluation value Y of the adjacent lattice and reflects it in the evaluation value.

  Specifically, if the evaluation value Y of each of the eight surrounding grids is 20 or more, the surrounding grid (periphery) is considered to be filled with skin color, and the evaluation value Y initialized with respect to its own grid Regardless of this, the evaluation value Y of its own grid is also set to the skin color. That is, in the state value, a flag for forcibly considering its own grid as a skin color is turned on.

  On the other hand, if the evaluation value Y of each of the eight surrounding grids is less than 20, the surrounding grid (periphery) is considered not to be a skin color, and regardless of the evaluation value Y initialized for the own grid, its own grid Assume that the evaluation value Y is not skin color. In other words, in the state value, a flag for forcibly considering that the grid is not a skin color is turned on.

  As will be described later, the process 5 is also performed on a 4 × 4 pixel grid or a 2 × 2 pixel grid. However, in a higher layer (larger grid), the process 5 is forcibly regarded as a skin color or forced. If it is considered that the skin color is not, the result in the upper layer is given priority.

  When the process 5 is executed for the 8 × 8 pixel grid and the state value is corrected, the process 6 is executed to initialize the state value of the 4 × 4 pixel grid. That is, the state value of the 8 × 8 pixel grid is set as the initial value of each state value of the 4 × 4 pixel grid included therein.

  Next, an evaluation value is initialized by processing 4 for a 4 × 4 pixel grid. Since the process 4 is as described above, a duplicate description is omitted. The same applies hereinafter. Further, the state value of the 4 × 4 pixel grid is corrected by the process 5 and reflected in the evaluation value Y, and the process 6 initializes the state value of the 2 × 2 pixel grid. That is, the state value of the 4 × 4 pixel grid is set as the initial value of each state value of the 2 × 2 pixel grid included therein.

  Subsequently, an evaluation value is initialized by processing 4 for a 2 × 2 pixel grid. Then, the state value is corrected by processing 5 for the 2 × 2 pixel grid and reflected in the evaluation value Y. Therefore, it is possible to detect whether or not the skin color is obtained by referring to the state value in units of 2 × 2 pixels.

  As described above, in this embodiment, first, with respect to an image obtained by imaging, whether or not each divided image (each grid) having a relatively large size A (8 × 8 in the above example) is flesh-colored. Determination processing, specifically, processing for determining that the luminance, hue, and saturation of the color value are within a certain range (hereinafter the same), and the determination result for each grid of size A (the above example) Then, “state value”) is generated. Thereafter, processing is performed for each lattice having a relatively small size B (4 × 4 in the above example). Specifically, first, the determination result of each grid of size B is set using the determination result of each grid of size A (process 6 described above). For this reason, it is not necessary to perform a process for determining whether or not the lattice has a small size. Then, the determination result of each grid of size B is corrected using the determination result of the surrounding grid (the above-described processing 4 and processing 5). As a result, the determination result varies in units of smaller size B, and a determination result with high resolution is generated.

  In this embodiment, in a relatively large size A grid, a grid set with a high possibility of skin color, in the above-mentioned example, “a 1-bit flag for forcibly considering itself as a skin color” is set. The set (on) grid is determined to be flesh-colored for the entire area of the size A grid even if it is determined that it is not flesh-colored in a relatively small size B grid included in the size A grid. The Thereby, noise can be reduced. Similarly, in a relatively large size A grid, a grid set with a low possibility of skin color, in the above example, “1 bit flag to force itself to be non-skin color” is set. Regarding (ON), even if it is determined that the skin color is relatively small in the relatively small size B lattice in the lattice, it is determined that the entire region of the size A lattice is not skin color. Thereby, noise can be reduced.

  FIG. 16 is an illustrative view showing one example of a memory map of the main memory 52. As shown in FIG. 16, the main memory 52 includes a game program storage area 90 and a data storage area 92. The game program storage area 90 stores a game program. The game program includes a main processing program 90a, an image generation program 90b, an image display program 90c, an image update program 90d, a shooting program 90e, a threshold adjustment program 90f, and a skin color extraction program 90g. , A first face recognition program 90h, a feature point acquisition program 90i, a second face recognition program 90j, and the like.

  The main process program 90a is a program for executing the main process of the virtual game of this embodiment. The image generation program 90b uses the image data 92a described later to generate a game image for displaying a screen necessary for a game such as a game screen (100, 200, etc.) or an operation screen on the first LCD 16 or the second LCD 18. It is a program. The image display program 90c is a program for displaying the game image generated according to the image generation program 90b on the first LCD 16 or the second LCD 18. The image update program 90d is a program for updating the game screen (game image) displayed on the first LCD 16 or the second LCD 18. The shooting program 90e is a program for shooting a subject with the inner camera 32 or the outer camera 34 in accordance with a user instruction, a game event, or the like. Since it is the same as a normal digital camera, detailed description is omitted. However, when the camera (shooting) mode is set, a real-time moving image (through image) is acquired and used as a viewfinder or the like. The through image is displayed on the second LCD 18. Also, a captured image of a still image is acquired by a user's shutter operation or a game event, and the corresponding captured image data is stored in the storage data memory 56.

  Prior to the start of the main part of the virtual game, the threshold adjustment program 90f is a program for adjusting the hue, saturation, and luminance thresholds for skin color determination in the skin color extraction process to optimum values based on the photographed image. . The skin color extraction program 90g is a program for extracting the skin color from the captured image in units of 2 × 2 pixels. In the first face recognition program 90h, the ratio of the skin color area in the specific area (110, 110 ') set in advance exceeds a predetermined value (35%) based on the skin color information extracted according to the skin color extraction program 90g. This is a program for determining whether or not the user's face is correctly recognized when the predetermined value is exceeded.

  The feature point acquisition program 90i performs image processing such as edge detection on the captured image captured by the inner camera 32 at regular time intervals (for example, 10 frames), so that 55 pieces of the facial image of the user are recorded. This is a program for acquiring feature points Pi. The second face recognition program 90j recognizes the entire face of the user based on the feature points Pi acquired according to the feature point detection program 90i, or a part of the user's face (in this embodiment, mouth and eyes) or It is a program for recognizing the movement (in this embodiment, opening and closing of the mouth).

  Although illustration is omitted, the game program storage area 90 also stores a sound output program, a backup program, and the like. The sound output program is a program for generating and outputting a game character's voice, onomatopoeic sound, sound effect, and sounds necessary for the game such as BGM. The backup program is a program for storing (saving) game data (intermediate data, result data) stored in the main memory 52 in the memory card 26.

  FIG. 17 is an illustrative view showing detailed contents of the data storage area 92 shown in FIG. As shown in FIG. 17, the data storage area 92 includes image data 92a, shooting data 92b, feature point data 92c, upper lip extraction coordinate data 92d, lower lip extraction coordinate data 92e, left eye extraction coordinate data 92f, and right eye extraction coordinates. Data 92g, lip difference data 92h, eye difference data 92i, setting threshold data 92j, determination data 92k, hue threshold data 92m, saturation threshold data 92n, luminance threshold data 92p, and the like are stored.

  The image data 92a is polygon data or texture data for generating a game image. The photographic data 92b is data of a photographic image (photographed image data) taken by the inner camera 32 or the outer camera 34, and is read out from the storage data memory 56 and stored (temporarily stored) in the main memory 52. In this embodiment, a plurality of (for example, four) photographed image data can be stored. As described above, the feature point data 92c is coordinate data corresponding to each of the feature points Pi acquired according to the feature point detection program 90i, and is updated every predetermined time (10 frames).

  The upper lip extraction coordinate data 92d is the coordinate data of the middle point LipU of the upper lip calculated based on the feature point Pi as described above. The lower lip extraction coordinate data 92e is the coordinate data of the lower lip midpoint LipD calculated based on the feature point Pi as described above. As described above, the left-eye extraction coordinate data 92f is the coordinate data of the middle point EyeL of the left eye calculated based on the feature point Pi. As described above, the right-eye extraction coordinate data 92g is the coordinate data of the middle point EyeR of the right eye calculated based on the feature point Pi.

  The lip difference data 92h is data on the lip difference (Lip_dX, Lip_dY) in each axial direction between the middle point LipU of the upper lip and the middle point LipD of the lower lip, as described above. As described above, the eye difference data 92i is data regarding eye differences (Eye_dX, Eye_dY) in the respective axial directions between the middle point EyeL of the left eye and the middle point EyeR of the right eye.

  The setting threshold data 92j is data regarding the skin color determination threshold table shown in FIG. 12, and is data generated in advance by a programmer or the like. The determination data 92k is data for defining areas 310, 312, and 314 corresponding to the attributes A, B, and C described with reference to FIG.

  The hue threshold data 92m is numerical data of an optimum value for the hue threshold adjusted according to the above-described threshold adjustment program 90f. Similarly, the saturation threshold data 92n is numerical data of an optimum value for the saturation threshold adjusted according to the above-described threshold adjustment program 90f. Similarly, the brightness threshold data 92p is numerical data of an optimum value for the brightness threshold adjusted according to the above-described threshold adjustment program 90f.

  Although not shown, the data storage area 92 stores other data such as sound data, and is provided with counters (timers) and flags necessary for game processing.

  Specifically, the CPU 50 shown in FIG. 3 executes the game process shown in FIG. Although illustration is omitted, the threshold value adjustment process described above is executed prior to the start of the game process (main part) shown in FIG. Although illustration is omitted, photographing processing by the inner camera 32 is started at the same time or almost simultaneously with the start of the adjustment processing. Furthermore, although illustration is omitted, when the game process is started, the feature point Pi acquisition process is executed at regular time intervals (10 frames). These processes are executed by multitasking.

  As shown in FIG. 18, when starting the game process, the CPU 50 displays a game initial screen in step S1. For example, the game screen 100 shown in FIG. 4A and the game screen 300 shown in FIG. 8A are displayed on the first LCD 16. These differ depending on the game to be played. In the subsequent step S3, the skin color is extracted from the photographed image. Here, as described above, the skin color is extracted from the captured image corresponding to the latest captured image data included in the captured data 92b.

  In the next step S5, it is determined whether or not the user's face is correctly recognized. Specifically, it is determined whether the skin color area in the specific area (110, 110 ′) exceeds a predetermined ratio (35%) using the result extracted in step S3. If “NO” in the step S5, that is, if the skin color area in the specific area (110, 110 ′) is equal to or less than a predetermined ratio, it is determined that the user's face is not correctly recognized, and the process directly proceeds to the step S13. .

  On the other hand, if “YES” in the step S5, that is, if the skin color area in the specific area (110, 110 ′) exceeds a predetermined ratio, it is determined that the user's face is correctly recognized, and the step S7 is performed. Thus, the feature point Pi is detected. Here, the CPU 50 detects the feature point data 92 c stored in the data area 92. Subsequently, in step S9, it is determined whether or not a predetermined condition is satisfied. Here, the CPU 50 determines whether the user's mouth has been opened or closed based on the feature point Pi, or changes from a state in which the user's face (eyes) is not recognized to a state in which the face (eyes) is recognized. To determine whether or not These judgments differ depending on the type of game. Moreover, in the example of the virtual game described above, a virtual game that recognizes the entire face is not described, but whether or not a predetermined condition is satisfied is determined based on all feature points Pi (entire face). It is also possible to execute such game processing. That is, the CPU 50 determines whether or not a predetermined condition is satisfied by recognizing the entire user's face or by recognizing a part of the user's face or its movement.

  If “NO” in the step S9, that is, if the predetermined condition is not satisfied, the process proceeds to a step S13 as it is. On the other hand, if “YES” in the step S9, that is, if a predetermined condition is satisfied, it is determined whether or not the game is cleared in a step S11. Here, the CPU 50 determines whether or not a game clear condition is satisfied. For example, in the virtual game described above, the game clear condition corresponds to the fact that all bread has been eaten or “I have no money” within the time limit.

  If “YES” in the step S11, that is, if the game is cleared, a game clear process is executed in a step S17, and the game process is ended. For example, in step S17, various effects such as a game screen representing that the game is cleared, a sound effect is output, a score is added, and a level is raised are executed.

  On the other hand, if “NO” in the step S11, that is, if the game is not cleared, it is determined whether or not the game is over in a step S13. Here, the CPU 50 determines whether or not the game over condition is satisfied. For example, in the above-described virtual game, the game over condition corresponds to the fact that the bread was not eaten within the time limit and the “not good” was not successful.

  If “NO” in the step S13, that is, if the game is not over, the game screen (100, 200) is updated in a step S15 so as to continue the game, and then the process returns to the step S3. Although a detailed description is omitted, if it is determined in step S9 that the predetermined condition is satisfied, for example, a game screen 100 in which the lip object 102 bites the pan object 104 is displayed, or the user moves his / her face. When it is detected that the cover is covered with a hand, in the instruction image display area 206, an operation to be performed next by the user (here, removing the hand covering the face) is displayed, and the progress of the game As you can see, the game screen is updated. On the other hand, if “YES” in the step S13, that is, if the game is over, a game over process is executed in a step S19, and the game process is ended. For example, in step S19, effects such as displaying a game screen representing game over, outputting sound effects, reducing the life of the player or player character, and the like are executed.

  According to this embodiment, the skin color is extracted from the captured image, it is determined whether the user's face is recognized based on the extracted skin color, and only when the user's face is recognized, Since it is determined whether or not a predetermined condition is satisfied based on the feature point, the game can be played in the play state intended by the programmer or developer, and the operation intended by the player can be executed. It is possible to correctly determine whether or not a predetermined condition is satisfied.

  In this embodiment, it is determined whether or not a predetermined condition is satisfied by determining whether or not the user's face is recognized based on the facial feature points and whether or not the user has opened and closed the mouth. Although virtual games have been described, it is not necessary to be limited to these. For example, based on the eyebrow feature points and eye feature points, it is determined whether or not a predetermined facial expression such as a smile or an angry face has been detected, and whether or not a predetermined condition is satisfied is determined. It is also possible to run a game.

  Also, in this embodiment, in order to request higher accuracy, when a specific area where the user's face is predicted to be set is set and the ratio of the skin color area in the specific area exceeds a predetermined value, the user However, the user's face may be recognized based on the ratio of the skin color area in the entire display area of the LCD.

  Further, in this embodiment, the silhouette is displayed on the game screen to inform the user of the skin color area recognized by the game apparatus, but the present invention is not limited to this. In the game screen, an index indicating whether or not the face (skin color area) is recognized and the appropriateness of the size (distance between the face and the game screen (internal camera)) is displayed on the game screen. A bar graph or the like may be displayed to inform the user.

  Furthermore, the configuration of the game device need not be limited to that of the embodiment. For example, there may be one display device (LCD) and one camera. Further, the touch panel may not be provided. Furthermore, a touch panel may be provided on two LCDs.

FIG. 1 is an illustrative view showing one embodiment of an external configuration of a game apparatus according to the present invention. FIG. 2 is an illustrative view showing a top view and a left side view of the state in which the game apparatus shown in FIG. 1 is folded. FIG. 3 is a block diagram showing an electrical configuration of the game apparatus shown in FIGS. FIG. 4 is an illustrative view showing one example of a game screen displayed on the first LCD shown in FIG. FIG. 5 is an illustrative view showing a skin color determination area set for a display area when a virtual game using the game screen shown in FIG. 4 is played. FIG. 6 is an illustrative view showing feature points detected from a user's face image. FIG. 7 is an illustrative view showing a developed area of a photographed image photographed by the camera shown in FIG. FIG. 8 is an illustrative view showing another example of a game screen displayed on the first LCD of FIG. FIG. 9 is an illustrative view showing a skin color determination area set in the display area when the virtual game using the game screen shown in FIG. 8 is played. FIG. 10 is an illustrative view showing an example of an adjustment screen displayed on the first LCD and an example of determination data when this adjustment screen is used. FIG. 11 is an illustrative view showing the positional relationship between the user's head (face) and the game device when playing the virtual game of this embodiment. FIG. 12 is a chart showing threshold values for skin color determination. FIG. 13 is an illustrative view showing a grid obtained by dividing the display area of the first LCD into 8 × 8 pixels and 4 × 4 pixels. FIG. 14 is an illustrative view showing a grid obtained by dividing the display area of the first LCD into 2 × 2 pixels. FIG. 15 is an illustrative view showing a weighting value for calculating an evaluation value of a certain grid when skin color is extracted, and an illustrative view showing an example of a determination result by a threshold value. FIG. 16 is an illustrative view showing a memory map of the main memory shown in FIG. FIG. 17 is an illustrative view showing specific contents of the data storage area shown in FIG. FIG. 18 is a flowchart showing the game processing of the CPU shown in FIG.

Explanation of symbols

10 ... Game device 12, 14 ... Housing 16, 18 ... LCD
20 ... Operation buttons 22 ... Touch panel 24 ... Touch pen 26, 28 ... Memory card 32, 34 ... Camera 50 ... CPU
52 ... Main memory 54 ... Memory control circuit 56 ... Storage data memory 58 ... Preset data memory 60, 62 ... Memory card I / F
64 ... Wireless communication module 66 ... Local communication module 68 ... RTC
70 ... Power supply circuit 72 ... I / F circuit 74, 76 ... GPU
78, 80 ... VRAM
82 ... LCD controller 84 ... Microphone

Claims (8)

A game program to be executed by a computer of an apparatus that executes game processing based on image data indicating an image corresponding to all or part of a user's face,
An imaging step of imaging the user;
A face recognition step for recognizing all or a part of the user's face based on image data indicating the image captured by the imaging step;
It is determined that the skin color of the user is detected by the skin color detection determination step for determining whether or not the user's skin color is detected based on the image data indicating the image captured by the imaging step, and the skin color detection determination step. A game program for executing a game process step for executing a game process based on information indicating all or part of the face of the user recognized by the face recognition step. Based on the image data indicating the image captured by the imaging step, the skin color region calculating step for calculating the size of the skin color region is further executed,
The game program according to claim 1, wherein the skin color detection determination step determines that the user's skin color has been detected when the size of the skin color region calculated by the skin color region calculation step is equal to or greater than a predetermined value. The skin color region calculating step calculates a ratio of a skin color region in the image captured by the imaging step,
The game program according to claim 2, wherein the skin color detecting step determines that the user's skin color is detected when the ratio calculated by the skin color region calculating step is equal to or greater than a predetermined value. In the image captured by the imaging step, further execute an area setting step for setting a specific area,
The game program according to claim 2, wherein the skin color area calculating step calculates a size of a skin color area within the specific area set by the area setting step. The face recognition step recognizes the user's face based on the image data indicating the image captured by the imaging step, and detects the user's skin color;
When it is determined that the skin color of the user is detected by the skin color detection determination step, the game processing step includes face information indicating all or a part of the user's face recognized by the face recognition step, and the user's skin color The game program according to claim 1, wherein the game process is executed based on skin color information indicating a skin color. Based on image data indicating an image obtained by imaging the user in advance, a skin color feature data calculating step for calculating skin color feature data indicating a skin color feature of the user is further executed.
In the skin color detection determination step, it is determined whether or not the user's skin color is detected based on the image data indicating the image captured in the imaging step and the skin color feature data calculated in the skin color feature data calculation step. The game program according to claim 1. A feature point data acquisition step that acquires all or part of feature point data of a user's face based on image data indicating an image captured by the imaging step, and a feature point acquired by the feature point data acquisition step Further executing a feature point data determination step for determining whether the data satisfies a predetermined condition,
In the game processing step, when it is determined that the skin color of the user is detected by the skin color detection determination step, and when the feature point data is determined to satisfy a predetermined condition by the feature point data determination step, The game program according to claim 1, wherein game processing is executed based on information indicating all or part of the face of the user recognized by the face recognition step. A game device that executes game processing based on image data indicating an image corresponding to all or part of a user's face,
Imaging means for imaging the user;
Face recognition means for recognizing all or part of the user's face based on image data indicating an image picked up by the image pickup means;
Based on the image data indicating the image captured by the imaging unit, it is determined that the user's skin color is detected by the skin color detection determination unit that determines whether the user's skin color is detected, and the skin color detection determination unit. A game apparatus comprising game processing means for executing game processing based on information indicating all or part of the face of the user recognized by the face recognition means.

Images