JP5804781B2 - GAME SYSTEM, INFORMATION PROCESSING METHOD, GAME DEVICE, AND INFORMATION PROCESSING PROGRAM - Google Patents

Description

  The present invention relates to a game system, an information processing method, a game device, and an information processing program using a display device such as a television.

  Conventionally, there is a game system that uses a television as a display device, and in such a game system, it is considered to execute various applications in addition to the game. For example, in the game system described in Patent Document 1, it is described that, in addition to the game device executing the game program, an application for displaying a TV program guide on the TV is executed (FIG. 10 of Patent Document 1). reference). As described above, it is considered to use the game system for various purposes other than the game purpose.

JP 2009-200553 A

  Here, a television used also as a display device in a game system displays an image (game image, image of the above-mentioned program guide, etc.) by the game system, as well as a use other than the game system (displays a television broadcast video). Use). In addition, when the game system is used for various purposes as described above, since the opportunity to use the game system itself increases, the game system is used when the user is watching TV (for example, using the game system). It is also conceivable to use the above-mentioned program table). However, conventionally, there has been a problem in operability when switching between a state in which the television is used in the game system and a state in which the television is used for purposes other than the game system.

  That is, in the game system described in Patent Document 1, an operation on the television cannot be performed while the application is being executed. Therefore, when an application for displaying a TV program guide is being executed, when the user wants to watch a desired program in the program guide, the user switches the input by holding the TV remote control separately from the operation device of the game system. Operation and channel change operation must be performed. As described above, conventionally, it is complicated and troublesome for a user to perform an operation on a display device (such as a television) during execution of an application.

  Therefore, an object of the present invention is to provide a game system, an information processing method, a game apparatus, and an information processing program capable of easily performing operations on a display device such as a television during execution of an application.

  The present invention employs the following configurations (1) to (12) in order to solve the above problems.

(1)
An example of the present invention is a game system including a game device and an operation device.
The operating device includes an operating unit, a receiving unit, and a display unit. The operation unit outputs operation data representing an operation on the operation device. The receiving unit receives an image output from the game device. The display unit displays the received image.
The game apparatus includes an acquisition unit, and an application execution unit. The game system also includes an operation image generation unit, an instruction determination unit, and a device control unit. The acquisition unit acquires operation data. The application execution unit executes the application and outputs an image generated by the execution to at least one of the operation device and a predetermined display device separate from the operation device. The operation image generation unit determines whether or not a predetermined operation has been performed on the controller device during execution of the application based on the operation data, and when the predetermined operation is performed , the predetermined operation image is displayed. Generate and output to the display unit of the controller device. The instruction determination unit determines, based on the operation data, whether or not there is a control instruction for performing control on a predetermined display device in a state where the operation image is displayed on the display unit. When there is a control instruction, the apparatus control unit performs control according to the control instruction on a predetermined display device.

  According to the configuration of (1) above, when a predetermined operation is performed on the operating device during execution of the application, an operation image is displayed on the screen of the operating device. In a state where the operation image is displayed, when there is a control instruction for a predetermined display device, control according to the control instruction is performed on the predetermined display device. Therefore, according to the configuration of (1) above, when an application is used in the game system, an operation on a predetermined display device can be performed without ending the execution of the application. In addition, since the user can perform the predetermined operation and the control instruction with one operating device, the user can easily operate the display device even while the application is being executed. For example, the user can easily perform an operation of watching a television program during the execution of the application or restarting the application thereafter using the operation device.

  Note that the game device may include a first image generation unit that generates an application selection image and outputs the image to the operation device. The application selection image includes an image representing an application executable by the game device. At this time, the application execution unit executes the selected application when an application to be executed is selected in a state where the application selection image is displayed.

(2)
The instruction determination unit may determine, based on the operation data, whether or not there has been an end instruction to end the running application in a state where the operation image is displayed on the display unit. At this time, the application execution unit ends the execution of the application when there is an end instruction.

  According to the configuration of (2) above, in the state where the operation image is displayed, if there is an end instruction, the execution of the application is ended. Therefore, the user can perform an operation for the display device and an operation for terminating the application in a state where the operation image is displayed.

(3)
The application execution unit may temporarily stop the execution of the application when a predetermined operation is performed on the controller device during the execution of the application. At this time, the instruction determination unit determines, based on the operation data, whether or not there is a restart instruction for restarting the execution of the application in a state where the operation image is displayed on the display unit. The application execution unit resumes execution of the application when there is a resume instruction.

  According to the configuration of (3) above, in the state where the operation image is displayed, if there is a restart instruction, the execution of the application is restarted. Therefore, the user can easily restart the application after performing an operation on the display device in a state where the operation image is displayed. According to this, for example, a series of operations can be easily performed using the operation device in the case where the user watches the television for a while while using the application and then resumes using the application.

(4)
The predetermined display device may be capable of receiving a television broadcast and displaying a program. At this time, when the operation image is displayed on the display unit, the operation image generation unit generates an image including a television program guide and outputs the image to a predetermined display device.

  According to the configuration of (4) above, when the operation image is displayed on the operation device, the TV program guide is displayed on the predetermined display device. According to this, the user can perform an operation on the television (predetermined display device) while watching the program guide, and can more easily perform a channel selection operation on the television.

(5)
The operation image generation unit may generate an operation image including a control instruction image representing a control instruction as the operation image. At this time, the instruction determination unit determines that there is a control instruction when a control instruction image is selected.

  According to the configuration of (5) above, a control instruction image representing a control instruction is displayed on the operation device as the operation image. According to this, the user can give a control instruction more easily.

(6)
The predetermined display device may be capable of receiving a television broadcast and displaying a program. At this time, the operation image generation unit generates an image including a TV program guide as the operation image and outputs the image to the operation device.

  According to the configuration of (6) above, a television program guide is displayed on the operation device as an operation image. According to this, the user can perform an operation on the television (predetermined display device) while watching the program guide, and can more easily perform a channel selection operation on the television.

(7)
The predetermined display device may be switchable between a first mode for displaying an image input from the game device and a second mode for displaying an image of a television broadcast. At this time, the device control unit performs control to switch the mode of the predetermined display device to the second mode in response to the operation image being displayed on the operation device.

  According to the configuration of (7) above, when the display is switched to the operation image, the user does not need to perform an input switching operation on a predetermined display device, so that the user's operation can be simplified.

(8)
The predetermined display device may be switchable between a first mode for displaying an image input from the game device and a second mode for displaying an image of a television broadcast. The device control unit performs control to switch the mode of the predetermined display device to the first mode in response to switching of an image displayed on the operation device from an operation image to an image generated by execution of an application.

  According to the configuration of (8) above, when the display is switched from the operation image to the application image, the user does not need to perform an input switching operation on a predetermined display device, and thus the user's operation is further simplified. Can do.

(9)
The operating device may further include a control signal output unit that outputs a control signal for controlling a predetermined display device. In this case, the apparatus control unit, Ru to send a control signal indicating the control contents corresponding to the control instruction to the control signal output unit.

  With configuration (9) above, it is possible to easily control a predetermined display device from the game device via the operation device.

(10)
The control signal output unit may output an infrared signal as the control signal. At this time, the predetermined display device includes an infrared light receiving unit capable of receiving an infrared signal.

  According to the configuration of (10) above, the operating device can control a predetermined display device by an infrared signal. Note that a general television (the television 2 in the embodiment described later) has a function of receiving an infrared signal and performing an operation designated by the infrared signal. Therefore, according to the configuration of (10) above, a television can be used as the predetermined display device, and a highly versatile game system can be provided.

(11)
The device control unit may output a control signal indicating the control content according to the control instruction to a predetermined display device.

  According to the configuration of (11) above, the information processing apparatus can easily control a predetermined display device by transmitting a control command. With configuration (12) above, for example, a television that complies with the HDMI standard can be used as the predetermined display device, and a highly versatile game system can be provided.

(12)
The operating device may further include a touch panel provided on the screen of the display unit. At this time, the operation image generation unit generates an image including a control instruction image representing a control instruction as the operation image. The instruction determination unit determines that there is a control instruction when a control instruction image is selected on the touch panel.

  With configuration (12) above, the user can issue a control instruction using the touch panel. According to this, the user can give an instruction by an easy and intuitive operation using the touch panel.

  Note that another example of the present invention may be the game device included in the game system of the above (1) to (12), or each unit in the game device (including the acquisition unit and the operation image output unit). It may be a game program (information processing program) that causes the computer of the game device to function as an equivalent means. Furthermore, another example of the present invention may be in the form of an information processing method performed in the game system or game device of the above (1) to (12).

  According to the present invention, an operation image is displayed on the screen of the operation device in response to a predetermined operation being performed on the operation device during execution of the application, and in a state where the operation image is displayed, a control instruction is issued. It is possible to control a predetermined display device. Thus, when an application is used in the game system, an operation on a predetermined display device can be performed without ending the execution of the application.

External view of game system 1 Block diagram showing the internal configuration of the game apparatus 3 The perspective view which shows the external appearance structure of the controller 5 The perspective view which shows the external appearance structure of the controller 5 The figure which shows the internal structure of the controller 5 The figure which shows the internal structure of the controller 5 Block diagram showing the configuration of the controller 5 The figure which shows the external appearance structure of the terminal device 7 The figure which shows the external appearance structure of the terminal device 7 The figure which shows a mode that the user hold | gripped the terminal device 7 sideways The block diagram which shows the internal structure of the terminal device 7 Block diagram showing connection relation between game system 1 and external device State transition diagram showing transition of operation state in game system 1 The figure which shows the memory area in the main memory of the game device 3 The flowchart which shows the flow of the start process performed with the game device 3 The flowchart which shows the flow of the menu process performed with the game device 3 The figure which shows an example of the application selection image displayed on the terminal device 7 The flowchart which shows the flow of the television processing performed with the game device 3 The figure which shows an example of the apparatus control image displayed on the terminal device 7 The flowchart which shows the flow of the browser process performed with the game device 3 The figure which shows an example of the browser image displayed on the terminal device 7 The flowchart which shows the flow of the application interruption process performed with the game device 3 The figure which shows an example of the setting image displayed on the terminal device 7 The figure which shows an example of the remote control image displayed on the terminal device 7

[1. Overall configuration of game system]
Hereinafter, a game system 1 according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is an external view of the game system 1. In FIG. 1, a game system 1 includes a stationary display device (hereinafter referred to as “TV”) 2 typified by a television receiver, a stationary game device 3, an optical disc 4, a controller 5, and a marker device. 6 and the terminal device 7. The game system 1 executes game processing in the game device 3 based on a game operation using the controller 5 and the terminal device 7, and displays a game image obtained by the game processing on the television 2 and / or the terminal device 7. is there.

  An optical disk 4 that is an example of an information storage medium that can be used interchangeably with the game apparatus 3 is detachably inserted into the game apparatus 3. The optical disc 4 stores an information processing program (typically a game program) to be executed in the game apparatus 3. An insertion slot for the optical disk 4 is provided on the front surface of the game apparatus 3. The game apparatus 3 executes the game process by reading and executing the information processing program stored in the optical disc 4 inserted into the insertion slot.

  The game apparatus 3 is connected to the television 2 via a connection cord. The television 2 displays a game image obtained by a game process executed in the game device 3. The television 2 has a speaker 2a (FIG. 2), and the speaker 2a outputs game sound obtained as a result of the game processing. In other embodiments, the game apparatus 3 and the stationary display apparatus may be integrated. The communication between the game apparatus 3 and the television 2 may be wireless communication.

  A marker device 6 is installed around the screen of the television 2 (upper side of the screen in FIG. 1). Although details will be described later, the user (player) can perform a game operation to move the controller 5, and the marker device 6 is used for the game device 3 to calculate the movement, position, posture, and the like of the controller 5. The marker device 6 includes two markers 6R and 6L at both ends thereof. The marker 6 </ b> R (same for the marker 6 </ b> L) is specifically one or more infrared LEDs (Light Emitting Diodes), and outputs infrared light toward the front of the television 2. The marker device 6 is connected to the game device 3 by wire (may be wireless), and the game device 3 can control lighting of each infrared LED provided in the marker device 6. The marker device 6 is portable, and the user can install the marker device 6 at a free position. Although FIG. 1 shows a mode in which the marker device 6 is installed on the television 2, the position and orientation in which the marker device 6 is installed are arbitrary.

  The controller 5 gives operation data representing the content of the operation performed on the own device to the game apparatus 3. The controller 5 and the game apparatus 3 can communicate with each other by wireless communication. In the present embodiment, for example, Bluetooth (registered trademark) technology is used for wireless communication between the controller 5 and the game apparatus 3. In other embodiments, the controller 5 and the game apparatus 3 may be connected by wire. In FIG. 1, the game system 1 includes one controller 5, but the game system 1 may include a plurality of controllers 5. That is, the game apparatus 3 can communicate with a plurality of controllers, and a plurality of people can play a game by using a predetermined number of controllers simultaneously. A detailed configuration of the controller 5 will be described later.

  The terminal device 7 is portable and has a size that can be held by a user. The user can use the terminal device 7 by holding it in his hand or by placing the terminal device 7 in a free position. Although the detailed configuration will be described later, the terminal device 7 includes an LCD (Liquid Crystal Display: liquid crystal display) 51 which is a display means, and input means (a touch panel 52, a gyro sensor 74 and the like which will be described later). The terminal device 7 and the game device 3 can communicate wirelessly (may be wired). The terminal device 7 receives data of an image (for example, a game image) generated by the game device 3 from the game device 3 and displays the image on the LCD 51. In the present embodiment, an LCD is used as the display device. However, the terminal device 7 may have any other display device such as a display device using EL (Electro Luminescence). . In addition, the terminal device 7 transmits operation data representing the content of the operation performed on the own device to the game device 3.

[2. Internal configuration of game device 3]
Next, the internal configuration of the game apparatus 3 will be described with reference to FIG. FIG. 2 is a block diagram showing an internal configuration of the game apparatus 3. The game apparatus 3 includes a CPU (Central Processing Unit) 10, a system LSI 11, an external main memory 12, a ROM / RTC 13, a disk drive 14, an AV-IC 15, and the like.

  The CPU 10 executes a game process by executing a game program stored on the optical disc 4, and functions as a game processor. The CPU 10 is connected to the system LSI 11. In addition to the CPU 10, an external main memory 12, a ROM / RTC 13, a disk drive 14, and an AV-IC 15 are connected to the system LSI 11. The system LSI 11 performs processing such as control of data transfer between components connected thereto, generation of an image to be displayed, and acquisition of data from an external device. The internal configuration of the system LSI 11 will be described later. The volatile external main memory 12 stores a program such as a game program read from the optical disc 4 or a game program read from the flash memory 17, or stores various data. Used as a work area and buffer area. The ROM / RTC 13 includes a ROM (so-called boot ROM) in which a program for starting the game apparatus 3 is incorporated, and a clock circuit (RTC: Real Time Clock) that counts time. The disk drive 14 reads program data, texture data, and the like from the optical disk 4 and writes the read data to an internal main memory 11e or an external main memory 12 described later.

  The system LSI 11 is provided with an input / output processor (I / O processor) 11a, a GPU (Graphics Processor Unit) 11b, a DSP (Digital Signal Processor) 11c, a VRAM (Video RAM) 11d, and an internal main memory 11e. Although not shown, these components 11a to 11e are connected to each other by an internal bus.

  The GPU 11b forms part of a drawing unit and generates an image according to a graphics command (drawing command) from the CPU 10. The VRAM 11d stores data (data such as polygon data and texture data) necessary for the GPU 11b to execute the graphics command. When an image is generated, the GPU 11b creates image data using data stored in the VRAM 11d. In the present embodiment, the game device 3 generates both an image (game image) displayed on the television 2 and an image (game image) displayed on the terminal device 7. Hereinafter, an image displayed on the television 2 may be referred to as a “television image”, and an image displayed on the terminal device 7 may be referred to as a “terminal image”.

  The DSP 11c functions as an audio processor, and generates sound data using sound data and sound waveform (tone color) data stored in the internal main memory 11e and the external main memory 12. In the present embodiment, as for the sound (game sound), both the sound output from the speaker of the television 2 and the sound output from the speaker of the terminal device 7 are generated as in the case of the game image. Hereinafter, the audio output from the television 2 may be referred to as “television audio”, and the audio output from the terminal device 7 may be referred to as “terminal audio”.

  Of the images and sounds generated by the game apparatus 3 as described above, image and sound data output by the television 2 is read by the AV-IC 15. The AV-IC 15 outputs the read image data to the television 2 via the AV connector 16, and outputs the read audio data to the speaker 2 a built in the television 2. Thus, an image is displayed on the television 2 and a sound is output from the speaker 2a. The game apparatus 3 and the television 2 may be connected by any method, but the game apparatus 3 transmits a control command for controlling the television 2 to the television 2 by wire or wirelessly. Also good. For example, an HDMI cable conforming to the HDMI (High-Definition Multimedia Interface) standard may be used. In the HDMI standard, it is possible to control a connected device by a function called CEC (Consumer Electronics Control). Accordingly, when the game apparatus 3 can control the television 2 as in the case where an HDMI cable is used, the game apparatus 3 turns on the television 2 at an appropriate timing or inputs the television 2. Can be switched.

  Of the images and sounds generated by the game apparatus 3, the image and sound data output from the terminal apparatus 7 is transmitted to the terminal apparatus 7 by the input / output processor 11a and the like. Data transmission to the terminal device 7 by the input / output processor 11a and the like will be described later.

  The input / output processor 11a performs transmission / reception of data to / from components connected to the input / output processor 11a and downloads data from an external device. The input / output processor 11a is connected to the flash memory 17, the network communication module 18, the controller communication module 19, the expansion connector 20, the memory card connector 21, and the codec LSI 27. An antenna 22 is connected to the network communication module 18. An antenna 23 is connected to the controller communication module 19. The codec LSI 27 is connected to a terminal communication module 28, and an antenna 29 is connected to the terminal communication module 28.

  The game device 3 can connect to a network such as the Internet and communicate with an external information processing device (for example, another game device or various servers). In other words, the input / output processor 11a can be connected to a network such as the Internet via the network communication module 18 and the antenna 22, and can communicate with other information processing apparatuses connected to the network. The input / output processor 11a periodically accesses the flash memory 17 to detect the presence / absence of data that needs to be transmitted to the network. If there is such data, the input / output processor 11a communicates with the network via the network communication module 18 and the antenna 22. Send. Further, the input / output processor 11a receives data transmitted from the external information processing apparatus or data downloaded from the download server via the network, the antenna 22 and the network communication module 18, and receives the received data in the flash memory 17. Remember. By executing the game program, the CPU 10 reads out the data stored in the flash memory 17 and uses it in the game program. In addition to data transmitted and received between the game apparatus 3 and the external information processing apparatus, the flash memory 17 stores game save data (game result data or intermediate data) played using the game apparatus 3. May be. The flash memory 17 may store a game program.

  The game apparatus 3 can receive operation data from the controller 5. That is, the input / output processor 11a receives the operation data transmitted from the controller 5 via the antenna 23 and the controller communication module 19, and stores (temporarily stores) it in the buffer area of the internal main memory 11e or the external main memory 12.

  Further, the game apparatus 3 can transmit and receive data such as images and sounds to and from the terminal device 7. When transmitting a game image (terminal game image) to the terminal device 7, the input / output processor 11 a outputs the game image data generated by the GPU 11 b to the codec LSI 27. The codec LSI 27 performs predetermined compression processing on the image data from the input / output processor 11a. The terminal communication module 28 performs wireless communication with the terminal device 7. Therefore, the image data compressed by the codec LSI 27 is transmitted to the terminal device 7 via the antenna 29 by the terminal communication module 28. In the present embodiment, the image data transmitted from the game apparatus 3 to the terminal apparatus 7 is used for the game, and if the displayed image is delayed in the game, the operability of the game is adversely affected. For this reason, it is preferable that the transmission of image data from the game apparatus 3 to the terminal device 7 is as little as possible. Therefore, in this embodiment, the codec LSI 27 is, for example, H.264. The image data is compressed using a highly efficient compression technique such as H.264 standard. Other compression techniques may be used, and when the communication speed is sufficient, the image data may be transmitted without compression. The terminal communication module 28 is a communication module that has received, for example, Wi-Fi authentication. For example, the terminal communication module 28 uses a MIMO (Multiple Input Multiple Output) technique adopted in the IEEE802.11n standard. Wireless communication may be performed at high speed, or another communication method may be used.

  In addition to the image data, the game apparatus 3 transmits audio data to the terminal device 7. That is, the input / output processor 11 a outputs the audio data generated by the DSP 11 c to the terminal communication module 28 via the codec LSI 27. The codec LSI 27 performs compression processing on the audio data in the same manner as the image data. The compression method for the audio data may be any method, but a method with a high compression rate and less deterioration of the sound is preferable. In other embodiments, audio data may be transmitted without being compressed. The terminal communication module 28 transmits the compressed image data and audio data to the terminal device 7 via the antenna 29.

  Further, the game apparatus 3 transmits various control data to the terminal apparatus 7 as necessary in addition to the image data and the sound data. The control data is data representing a control instruction for a component included in the terminal device 7, and for example, an instruction to control lighting of the marker unit (marker unit 55 shown in FIG. 11) or a camera (camera 56 shown in FIG. 11). Indicates an instruction to control imaging. The input / output processor 11 a transmits control data to the terminal device 7 in accordance with an instruction from the CPU 10. With respect to this control data, the codec LSI 27 does not perform data compression processing in the present embodiment, but may perform compression processing in other embodiments. Note that the above-described data transmitted from the game device 3 to the terminal device 7 may or may not be encrypted as necessary.

  The game apparatus 3 can receive various data from the terminal device 7. Although details will be described later, in the present embodiment, the terminal device 7 transmits operation data, image data, and audio data. Each data transmitted from the terminal device 7 is received by the terminal communication module 28 via the antenna 29. Here, the image data and audio data from the terminal device 7 are subjected to the same compression processing as the image data and audio data from the game device 3 to the terminal device 7. Therefore, these image data and audio data are sent from the terminal communication module 28 to the codec LSI 27, subjected to expansion processing by the codec LSI 27, and output to the input / output processor 11a. On the other hand, the operation data from the terminal device 7 has a smaller amount of data than images and sounds, and therefore may not be subjected to compression processing. Further, encryption may or may not be performed as necessary. Accordingly, the operation data is received by the terminal communication module 28 and then output to the input / output processor 11 a via the codec LSI 27. The input / output processor 11a stores (temporarily stores) the data received from the terminal device 7 in the buffer area of the internal main memory 11e or the external main memory 12.

  Further, the game apparatus 3 can be connected to another device or an external storage medium. That is, the expansion connector 20 and the memory card connector 21 are connected to the input / output processor 11a. The expansion connector 20 is a connector for an interface such as USB or SCSI. A network such as an external storage medium is connected to the expansion connector 20, a peripheral device such as another controller is connected, or a wired communication connector is connected to replace the network communication module 18 with a network. You can communicate with. The memory card connector 21 is a connector for connecting an external storage medium such as a memory card. For example, the input / output processor 11a can access an external storage medium via the expansion connector 20 or the memory card connector 21 to store data in the external storage medium or read data from the external storage medium.

  The game apparatus 3 is provided with a power button 24, a reset button 25, and an eject button 26. The power button 24 and the reset button 25 are connected to the system LSI 11. When the power button 24 is turned on, power is supplied to each component of the game apparatus 3 from an external power source by an AC adapter (not shown). When the reset button 25 is pressed, the system LSI 11 restarts the boot program for the game apparatus 3. The eject button 26 is connected to the disk drive 14. When the eject button 26 is pressed, the optical disk 4 is ejected from the disk drive 14.

  In other embodiments, some of the components included in the game apparatus 3 may be configured as expansion devices that are separate from the game apparatus 3. At this time, the expansion device may be connected to the game apparatus 3 via the expansion connector 20, for example. Specifically, the extension device includes, for example, each component of the codec LSI 27, the terminal communication module 28, and the antenna 29, and may be detachable from the extension connector 20. According to this, the said game device can be set as the structure which can communicate with the terminal device 7 by connecting the said expansion apparatus with respect to the game device which is not provided with said each component.

[3. Configuration of controller 5]
Next, the controller 5 will be described with reference to FIGS. FIG. 3 is a perspective view showing an external configuration of the controller 5. FIG. 4 is a perspective view showing an external configuration of the controller 5. 3 is a perspective view of the controller 5 as seen from the upper rear side, and FIG. 4 is a perspective view of the controller 5 as seen from the lower front side.

  3 and 4, the controller 5 includes a housing 31 formed by plastic molding, for example. The housing 31 has a substantially rectangular parallelepiped shape whose longitudinal direction is the front-rear direction (the Z-axis direction shown in FIG. 3), and is a size that can be gripped with one hand of an adult or a child as a whole. The user can perform a game operation by pressing a button provided on the controller 5 and moving the controller 5 itself to change its position and posture (tilt).

  The housing 31 is provided with a plurality of operation buttons. As shown in FIG. 3, a cross button 32a, a first button 32b, a second button 32c, an A button 32d, a minus button 32e, a home button 32f, a plus button 32g, and a power button 32h are provided on the upper surface of the housing 31. It is done. In the present specification, the upper surface of the housing 31 on which these buttons 32a to 32h are provided may be referred to as a “button surface”. On the other hand, as shown in FIG. 4, a recess is formed on the lower surface of the housing 31, and a B button 32i is provided on the rear inclined surface of the recess. A function corresponding to the information processing program executed by the game apparatus 3 is appropriately assigned to each of the operation buttons 32a to 32i. The power button 32h is for remotely turning on / off the main body of the game apparatus 3. The home button 32 f and the power button 32 h are embedded in the upper surface of the housing 31. Thereby, it is possible to prevent the user from pressing the home button 32f or the power button 32h by mistake.

  A connector 33 is provided on the rear surface of the housing 31. The connector 33 is used to connect another device (for example, another sensor unit or controller) to the controller 5. Further, locking holes 33a are provided on both sides of the connector 33 on the rear surface of the housing 31 in order to prevent the other devices from being easily detached.

  A plurality (four in FIG. 3) of LEDs 34 a to 34 d are provided behind the upper surface of the housing 31. Here, the controller type (number) is assigned to the controller 5 to distinguish it from other controllers. The LEDs 34a to 34d are used for the purpose of notifying the user of the controller type currently set in the controller 5 and notifying the user of the remaining battery level of the controller 5. Specifically, when a game operation is performed using the controller 5, any one of the plurality of LEDs 34a to 34d is turned on according to the controller type.

  Further, the controller 5 has an imaging information calculation unit 35 (FIG. 6), and a light incident surface 35a of the imaging information calculation unit 35 is provided on the front surface of the housing 31, as shown in FIG. The light incident surface 35a is made of a material that transmits at least infrared light from the markers 6R and 6L.

  A sound release hole 31a is formed between the first button 32b and the home button 32f on the upper surface of the housing 31 for emitting sound from the speaker 47 (FIG. 5) built in the controller 5 to the outside.

  Next, the internal structure of the controller 5 will be described with reference to FIGS. 5 and 6 are diagrams showing the internal structure of the controller 5. FIG. FIG. 5 is a perspective view showing a state in which the upper housing (a part of the housing 31) of the controller 5 is removed. FIG. 6 is a perspective view showing a state in which the lower casing (a part of the housing 31) of the controller 5 is removed. The perspective view shown in FIG. 6 is a perspective view of the substrate 30 shown in FIG.

  In FIG. 5, a substrate 30 is fixed inside the housing 31, and operation buttons 32 a to 32 h, LEDs 34 a to 34 d, an acceleration sensor 37, an antenna 45, and a speaker 47 are provided on the upper main surface of the substrate 30. Etc. are provided. These are connected to a microcomputer (microcomputer) 42 (see FIG. 6) by wiring (not shown) formed on the substrate 30 and the like. In the present embodiment, the acceleration sensor 37 is disposed at a position shifted from the center of the controller 5 with respect to the X-axis direction. This makes it easier to calculate the movement of the controller 5 when the controller 5 is rotated about the Z axis. The acceleration sensor 37 is disposed in front of the center of the controller 5 in the longitudinal direction (Z-axis direction). Further, the controller 5 functions as a wireless controller by the wireless module 44 (FIG. 6) and the antenna 45.

  On the other hand, in FIG. 6, an imaging information calculation unit 35 is provided at the front edge on the lower main surface of the substrate 30. The imaging information calculation unit 35 includes an infrared filter 38, a lens 39, an imaging element 40, and an image processing circuit 41 in order from the front of the controller 5. These members 38 to 41 are respectively attached to the lower main surface of the substrate 30.

  Further, the microcomputer 42 and the vibrator 46 are provided on the lower main surface of the substrate 30. The vibrator 46 is, for example, a vibration motor or a solenoid, and is connected to the microcomputer 42 by wiring formed on the substrate 30 or the like. When the vibrator 46 is actuated by an instruction from the microcomputer 42, vibration is generated in the controller 5. As a result, a so-called vibration-compatible game in which the vibration is transmitted to the user's hand holding the controller 5 can be realized. In the present embodiment, the vibrator 46 is disposed slightly forward of the housing 31. That is, by arranging the vibrator 46 on the end side of the center of the controller 5, the entire controller 5 can be vibrated greatly by the vibration of the vibrator 46. The connector 33 is attached to the rear edge on the lower main surface of the substrate 30. 5 and 6, the controller 5 includes a crystal resonator that generates a basic clock of the microcomputer 42, an amplifier that outputs an audio signal to the speaker 47, and the like.

  The shape of the controller 5, the shape of each operation button, the number of acceleration sensors and vibrators, and the installation positions shown in FIGS. 3 to 6 are merely examples, and other shapes, numbers, and installation positions may be used. May be. In the present embodiment, the imaging direction by the imaging unit is the positive Z-axis direction, but the imaging direction may be any direction. That is, the position of the imaging information calculation unit 35 in the controller 5 (the light incident surface 35a of the imaging information calculation unit 35) does not have to be the front surface of the housing 31, and other surfaces can be used as long as light can be taken in from the outside of the housing 31. May be provided.

  FIG. 7 is a block diagram showing the configuration of the controller 5. The controller 5 includes an operation unit 32 (operation buttons 32a to 32i), an imaging information calculation unit 35, a communication unit 36, an acceleration sensor 37, and a gyro sensor 48. The controller 5 transmits data representing the content of the operation performed on the own device to the game apparatus 3 as operation data. Hereinafter, the operation data transmitted from the controller 5 may be referred to as “controller operation data”, and the operation data transmitted from the terminal device 7 may be referred to as “terminal operation data”.

  The operation unit 32 includes the operation buttons 32a to 32i described above, and the operation button data indicating the input state (whether or not each operation button 32a to 32i is pressed) to each operation button 32a to 32i is transmitted to the microcomputer of the communication unit 36. Output to 42.

  The imaging information calculation unit 35 is a system for analyzing the image data captured by the imaging unit, discriminating a region having a high luminance in the image data, and calculating a center of gravity position, a size, and the like of the region. Since the imaging information calculation unit 35 has a sampling period of, for example, about 200 frames / second at the maximum, it can track and analyze even a relatively fast movement of the controller 5.

  The imaging information calculation unit 35 includes an infrared filter 38, a lens 39, an imaging element 40, and an image processing circuit 41. The infrared filter 38 passes only infrared rays from the light incident from the front of the controller 5. The lens 39 collects the infrared light transmitted through the infrared filter 38 and makes it incident on the image sensor 40. The image sensor 40 is a solid-state image sensor such as a CMOS sensor or a CCD sensor, for example, and receives the infrared light collected by the lens 39 and outputs an image signal. Here, the marker unit 55 and the marker device 6 of the terminal device 7 to be imaged are configured by a marker that outputs infrared light. Therefore, by providing the infrared filter 38, the image sensor 40 receives only the infrared light that has passed through the infrared filter 38 and generates image data. Therefore, the image of the imaging target (the marker unit 55 and / or the marker device 6) is captured. More accurate imaging can be performed. Hereinafter, an image captured by the image sensor 40 is referred to as a captured image. Image data generated by the image sensor 40 is processed by the image processing circuit 41. The image processing circuit 41 calculates the position of the imaging target in the captured image. The image processing circuit 41 outputs coordinates indicating the calculated position to the microcomputer 42 of the communication unit 36. The coordinate data is transmitted to the game apparatus 3 as operation data by the microcomputer 42. Hereinafter, the coordinates are referred to as “marker coordinates”. Since the marker coordinates change corresponding to the direction (tilt angle) and position of the controller 5 itself, the game apparatus 3 can calculate the direction and position of the controller 5 using the marker coordinates.

  In other embodiments, the controller 5 may not include the image processing circuit 41, and the captured image itself may be transmitted from the controller 5 to the game apparatus 3. At this time, the game apparatus 3 may have a circuit or a program having the same function as the image processing circuit 41, and may calculate the marker coordinates.

  The acceleration sensor 37 detects the acceleration (including gravity acceleration) of the controller 5, that is, detects the force (including gravity) applied to the controller 5. The acceleration sensor 37 detects the value of the acceleration (linear acceleration) in the linear direction along the sensing axis direction among the accelerations applied to the detection unit of the acceleration sensor 37. For example, in the case of a multi-axis acceleration sensor having two or more axes, the component acceleration along each axis is detected as the acceleration applied to the detection unit of the acceleration sensor. The acceleration sensor 37 is, for example, a capacitive MEMS (Micro Electro Mechanical System) type acceleration sensor, but other types of acceleration sensors may be used.

  In the present embodiment, the acceleration sensor 37 has a vertical direction (Y-axis direction shown in FIG. 3), a horizontal direction (X-axis direction shown in FIG. 3), and a front-back direction (Z-axis direction shown in FIG. 3) with reference to the controller 5. ) Linear acceleration is detected in each of the three axis directions. Since the acceleration sensor 37 detects acceleration in the linear direction along each axis, the output from the acceleration sensor 37 represents the linear acceleration value of each of the three axes. That is, the detected acceleration is represented as a three-dimensional vector in an XYZ coordinate system (controller coordinate system) set with the controller 5 as a reference.

  Data representing the acceleration detected by the acceleration sensor 37 (acceleration data) is output to the communication unit 36. Note that the acceleration detected by the acceleration sensor 37 changes corresponding to the direction (inclination angle) and movement of the controller 5 itself, so the game apparatus 3 calculates the direction and movement of the controller 5 using the acquired acceleration data. can do. In the present embodiment, the game apparatus 3 calculates the attitude, tilt angle, and the like of the controller 5 based on the acquired acceleration data.

  A computer such as a processor of the game apparatus 3 (for example, the CPU 10) or a processor of the controller 5 (for example, the microcomputer 42) is processed based on an acceleration signal output from the acceleration sensor 37 (the same applies to an acceleration sensor 73 described later). It can be easily understood by those skilled in the art from the description of the present specification that by performing the above, it is possible to estimate or calculate (determine) further information regarding the controller 5. For example, when processing on the computer side is executed on the assumption that the controller 5 on which the acceleration sensor 37 is mounted is stationary (that is, the processing is executed assuming that the acceleration detected by the acceleration sensor is only gravitational acceleration). When the controller 5 is actually stationary, it can be determined whether or not the attitude of the controller 5 is inclined with respect to the direction of gravity based on the detected acceleration. Specifically, whether or not the controller 5 is inclined with respect to the reference depending on whether or not 1G (gravity acceleration) is applied, based on the state in which the detection axis of the acceleration sensor 37 is directed vertically downward. It is possible to know how much it is inclined with respect to the reference according to its size. Further, in the case of the multi-axis acceleration sensor 37, it is possible to know in detail how much the controller 5 is inclined with respect to the direction of gravity by further processing the acceleration signal of each axis. . In this case, the processor may calculate the tilt angle of the controller 5 based on the output from the acceleration sensor 37, or may calculate the tilt direction of the controller 5 without calculating the tilt angle. Good. Thus, by using the acceleration sensor 37 in combination with the processor, the tilt angle or posture of the controller 5 can be determined.

  On the other hand, when it is assumed that the controller 5 is in a dynamic state (a state where the controller 5 is moved), the acceleration sensor 37 detects an acceleration corresponding to the movement of the controller 5 in addition to the gravitational acceleration. Therefore, the moving direction of the controller 5 can be known by removing the gravitational acceleration component from the detected acceleration by a predetermined process. Even if it is assumed that the controller 5 is in a dynamic state, the direction of gravity is obtained by removing the acceleration component corresponding to the movement of the acceleration sensor from the detected acceleration by a predetermined process. It is possible to know the inclination of the controller 5 with respect to. In another embodiment, the acceleration sensor 37 is a built-in process for performing a predetermined process on the acceleration signal before outputting the acceleration signal detected by the built-in acceleration detection means to the microcomputer 42. An apparatus or other type of dedicated processing apparatus may be provided. A built-in or dedicated processing device converts the acceleration signal into a tilt angle (or other preferred parameter) if, for example, the acceleration sensor 37 is used to detect static acceleration (eg, gravitational acceleration). It may be a thing.

  The gyro sensor 48 detects angular velocities about three axes (XYZ axes in the present embodiment). In this specification, with the imaging direction (Z-axis positive direction) of the controller 5 as a reference, the rotation direction around the X axis is the pitch direction, the rotation direction around the Y axis is the yaw direction, and the rotation direction around the Z axis is the roll direction. Call. The gyro sensor 48 only needs to be able to detect angular velocities about three axes, and any number and combination of gyro sensors may be used. For example, the gyro sensor 48 may be a three-axis gyro sensor or a combination of a two-axis gyro sensor and a one-axis gyro sensor to detect an angular velocity around three axes. Data representing the angular velocity detected by the gyro sensor 48 is output to the communication unit 36. Further, the gyro sensor 48 may detect an angular velocity around one axis or two axes.

  The communication unit 36 includes a microcomputer 42, a memory 43, a wireless module 44, and an antenna 45. The microcomputer 42 controls the wireless module 44 that wirelessly transmits data acquired by the microcomputer 42 to the game apparatus 3 while using the memory 43 as a storage area when performing processing.

  Data output from the operation unit 32, the imaging information calculation unit 35, the acceleration sensor 37, and the gyro sensor 48 to the microcomputer 42 is temporarily stored in the memory 43. These data are transmitted to the game apparatus 3 as operation data (controller operation data). That is, the microcomputer 42 outputs the operation data stored in the memory 43 to the wireless module 44 when the transmission timing to the controller communication module 19 of the game apparatus 3 arrives. The wireless module 44 modulates a carrier wave of a predetermined frequency with operation data using, for example, Bluetooth (registered trademark) technology, and radiates a weak radio signal from the antenna 45. That is, the operation data is modulated by the wireless module 44 into a weak radio signal and transmitted from the controller 5. The weak radio signal is received by the controller communication module 19 on the game apparatus 3 side. By demodulating and decoding the received weak radio signal, the game apparatus 3 can acquire operation data. Then, the CPU 10 of the game apparatus 3 performs a game process using the operation data acquired from the controller 5. Note that wireless transmission from the communication unit 36 to the controller communication module 19 is sequentially performed at predetermined intervals, but game processing is generally performed in units of 1/60 seconds (one frame time). Therefore, it is preferable to perform transmission at a period equal to or shorter than this time. The communication unit 36 of the controller 5 outputs the operation data to the controller communication module 19 of the game apparatus 3 at a rate of once every 1/200 seconds, for example.

  As described above, the controller 5 can transmit the marker coordinate data, the acceleration data, the angular velocity data, and the operation button data as the operation data representing the operation on the own device. Further, the game apparatus 3 executes a game process using the operation data as a game input. Therefore, by using the controller 5, the user can perform a game operation for moving the controller 5 itself in addition to the conventional general game operation for pressing each operation button. For example, an operation of tilting the controller 5 to an arbitrary posture, an operation of instructing an arbitrary position on the screen by the controller 5, an operation of moving the controller 5 itself, and the like can be performed.

  In the present embodiment, the controller 5 does not have a display unit that displays a game image, but may include a display unit that displays, for example, an image representing the remaining battery level.

[4. Configuration of Terminal Device 7]
Next, the configuration of the terminal device 7 will be described with reference to FIGS. FIG. 8 is a plan view showing an external configuration of the terminal device 7. 8A is a front view of the terminal device 7, FIG. 8B is a top view, FIG. 8C is a right side view, and FIG. 8D is a bottom view. FIG. 9 is a rear view of the terminal device 7. FIG. 10 is a diagram illustrating a state in which the user holds the terminal device 7 sideways.

  As shown in FIG. 8, the terminal device 7 includes a housing 50 that is generally a horizontally-long rectangular plate shape. That is, it can be said that the terminal device 7 is a tablet-type information processing device. The housing 50 may have a curved surface as long as it has a plate shape as a whole, or may have a protrusion or the like in part. The housing 50 is large enough to be gripped by the user. Therefore, the user can move the terminal apparatus 7 or change the arrangement position of the terminal apparatus 7. The length of the terminal device 7 in the vertical direction (z-axis direction) is preferably 100 to 150 [mm]. In the present embodiment, the length is 133.5 [mm]. The lateral (x-axis direction) length of the terminal device 7 is preferably 200 to 250 [mm], and in this embodiment is 228.26 [mm]. The thickness (length in the y-axis direction) of the terminal device 7 is preferably about 15 to 30 [mm] at the plate-like portion, and about 30 to 50 [mm] including the thickest portion. In this embodiment, 23.6 (the thickest part is 40.26) [mm]. The weight of the terminal device 7 is about 400 to 600 [g], and is 530 [g] in the present embodiment. Although details will be described later, even if the terminal device 7 is a relatively large terminal device (operation device) as described above, the user device is easy to hold and operate.

  The terminal device 7 has an LCD 51 on the surface (front side) of the housing 50. The screen size of the LCD 51 is preferably 5 inches or more, and here is 6.2 inches. The terminal device 7 of the present embodiment is easy to operate even if a large LCD is provided due to the configuration that it is easy to hold and operate. In another embodiment, a smaller LCD 51 may be provided so that the size of the terminal device 7 is relatively small. The LCD 51 is provided near the center of the surface of the housing 50. Therefore, the user can move the terminal device 7 while viewing the screen of the LCD 51 by holding the housings 50 on both sides of the LCD 51 as shown in FIG. FIG. 10 shows an example in which the user holds the terminal device 7 horizontally by holding the housings 50 on both the left and right sides of the LCD 51 (in a long horizontal direction), but the terminal device 7 is held vertically. It can also be held (with a long vertical orientation).

  As illustrated in FIG. 8A, the terminal device 7 includes a touch panel 52 on the screen of the LCD 51 as an operation unit. In the present embodiment, the touch panel 52 is a resistive film type touch panel. However, the touch panel is not limited to the resistive film type, and any type of touch panel such as a capacitance type can be used. The touch panel 52 may be a single touch method or a multi-touch method. In the present embodiment, the touch panel 52 having the same resolution (detection accuracy) as the resolution of the LCD 51 is used. However, the resolution of the touch panel 52 and the resolution of the LCD 51 are not necessarily matched. The input to the touch panel 52 is normally performed using the touch pen 60, but the input to the touch panel 52 is not limited to the touch pen 60 and can be performed by a user's finger. The housing 50 is provided with a storage hole 60a for storing the touch pen 60 used for operating the touch panel 52 (see FIG. 8B). Here, the storage hole 60a is provided on the upper surface of the housing 50 so that the touch pen 60 does not fall, but may be provided on the side surface or the lower surface. Thus, since the terminal device 7 includes the touch panel 52, the user can operate the touch panel 52 while moving the terminal device 7. That is, the user can directly input (by the touch panel 52) to the screen while moving the screen of the LCD 51.

  As shown in FIG. 8, the terminal device 7 includes two analog sticks 53A and 53B and a plurality of buttons (keys) 54A to 54M as operation means (operation unit). Each analog stick 53A and 53B is a device capable of indicating a direction. In each of the analog sticks 53A and 53B, a movable member (stick part) operated by a user's finger can slide in any direction (any angle in the up / down / left / right and diagonal directions) with respect to the surface of the housing 50. It is configured. That is, it is a directional input device sometimes called a slide pad. The movable members of the analog sticks 53A and 53B may be of a type that tilts in an arbitrary direction with respect to the surface of the housing 50. In the present embodiment, an analog stick of a type in which the movable member slides is used. Therefore, the user can operate each of the analog sticks 53A and 53B without moving the thumb greatly, and the housing 50 is held firmly. The operation can be performed. When the analog sticks 53A and 53B are of a type in which the movable member tilts, the degree of input (degree of tilt) is easy for the user to understand, and detailed operations can be performed more easily.

  The left analog stick 53A is provided on the left side of the screen of the LCD 51, and the right analog stick 53B is provided on the right side of the screen of the LCD 51. Therefore, the user can make an input for instructing the direction with the left and right hands using the analog stick. Also, as shown in FIG. 10, each analog stick 53A and 53B is provided at a position where the user can operate with the left and right portions of the terminal device 7 (the left and right side portions of the LCD 51) being gripped. Even when the terminal device 7 is held and moved, the analog sticks 53A and 53B can be easily operated.

  Each button 54A to 54L is an operation means (operation unit) for performing a predetermined input, and is a key that can be pressed. As shown below, each button 54A-54L is provided in the position which a user can operate in the state which hold | gripped the left-right part of the terminal device 7 (refer FIG. 10). Therefore, the user can easily operate these operation means even when the user moves the terminal device 7.

  As shown in FIG. 8A, a cross button (direction input button) 54A and buttons 54B to 54H and 54M among the operation buttons 54A to 54L are provided on the surface of the housing 50. That is, these buttons 54A to 54H and 54M are arranged at positions where they can be operated with the user's thumb (see FIG. 10).

  The cross button 54A is provided on the left side of the LCD 51 and below the left analog stick 53A. That is, the cross button 54A is arranged at a position where it can be operated with the left hand of the user. The cross button 54A has a cross shape, and is a button capable of instructing at least the vertical and horizontal directions.

  The buttons 54B to 54D are provided below the LCD 51. These three buttons 54B to 54D are arranged at positions that can be operated by both the left and right hands. The terminal device 7 has a power button 54M for turning on / off the power of the terminal device 7. It is also possible to remotely turn on / off the game apparatus 3 by operating the power button 54M. The power button 54M is provided on the lower side of the LCD 51 like the buttons 54B to 54D. The power button 54M is provided on the right side of the buttons 54B to 54D. Therefore, the power button 54M is disposed at a position where it can be operated (easy to operate) with the right hand. The four buttons 54E to 54H are provided on the right side of the LCD 51 and below the right analog stick 53B. That is, the four buttons 54E to 54H are arranged at positions that can be operated with the right hand of the user. Further, the four buttons 54E to 54H are arranged so as to have a vertical / left / right positional relationship (relative to the center position of the four buttons 54E to 54H). Therefore, the terminal device 7 can also function the four buttons 54 </ b> E to 54 </ b> H as buttons for instructing the user in the up / down / left / right directions.

  In the present embodiment, the analog sticks 53A and 53B are arranged above the cross button 54A and the buttons 54E to 54H. Here, each of the analog sticks 53A and 53B protrudes in the thickness direction (y-axis direction) from the cross button 54A and the buttons 54E to 54H. Therefore, if the arrangement of the analog stick 53A and the cross button 54A is reversed, when the user operates the cross button 54A with his / her thumb, the thumb may hit the analog stick 53A and the operation may be mistaken. is there. The same problem occurs when the arrangement of the analog stick 53B and the buttons 54E to 54H is reversed. On the other hand, in the present embodiment, the analog sticks 53A and 53B are arranged above the cross button 54A and the buttons 54E to 54H, so that when the user operates the analog sticks 53A and 53B, The possibility that the finger hits each of the buttons 54E to 54H is lower than that in the above case. Thus, in this embodiment, the possibility of erroneous operation can be reduced, and the operability of the terminal device 7 can be improved. However, in another embodiment, the analog stick 53A may be disposed opposite to the cross button 54A, or the analog stick 53B may be disposed opposite to the buttons 54E to 54H as necessary.

  Here, in this embodiment, some operation units (respective analog sticks 53A and 53B, cross button 54A, and three buttons 54E to 54G) are arranged on the upper and lower sides of the housing 50 on both the left and right sides of the display unit (LCD 51). It is provided above the center of the direction (y-axis direction). When operating these operation units, the user mainly holds the upper side of the terminal device 7 from the vertical center. Here, when the user grips the lower side of the housing 50, the gripped terminal device 7 becomes unstable (especially when the terminal device 7 has a relatively large size as in the present embodiment). , It becomes difficult for the user to hold the terminal device 7. On the other hand, in this embodiment, when operating the operation unit, the user mainly holds the upper side of the terminal device 7 from the center in the vertical direction, and the housing 50 is laterally held by the palm. Can support. Therefore, the user can hold the housing 50 in a stable state, and the terminal device 7 can be easily held, so that the operation unit can be easily operated. In other embodiments, at least one operation unit may be provided on each of the left and right sides of the display unit above the center of the housing 50. For example, only the analog sticks 53 </ b> A and 53 </ b> B may be provided above the center of the housing 50. Further, for example, when the cross button 54A is provided above the left analog stick 53A and the four buttons 54E to 54H are provided above the right analog stick 53B, the cross button 54A and the four buttons 54E to 54H are provided. It may be provided above the center of the housing 50.

  In the present embodiment, a protrusion (a flange 59) is provided on the back side of the housing 50 (on the opposite side of the surface on which the LCD 51 is provided) (see FIG. 8C and FIG. 9). As shown in FIG. 8C, the flange portion 59 is a mountain-shaped member provided so as to protrude from the back surface of the substantially plate-shaped housing 50. The protrusion has a height (thickness) that can be hooked on a user's finger that holds the back surface of the housing 50. The height of the protrusion is preferably 10 to 25 [mm], and in this embodiment is 16.66 [mm]. Moreover, it is preferable that the lower surface of the protruding portion has an inclination of 45 ° or more (more preferably 60 ° or more) with respect to the back surface of the housing 50 so that the protruding portion can be easily caught on the user's finger. As shown in FIG. 8C, the lower surface of the protrusion may be formed to have a larger inclination angle than the upper surface. As shown in FIG. 10, the user gets tired even when the terminal device 7 is a relatively large size by holding his / her finger on the buttocks 59 (with the buttocks 59 on the finger). The terminal device 7 can be gripped in a stable state. That is, it can be said that the collar part 59 is a support member for supporting the housing 50 with a finger, and can also be called a finger hook part.

  Further, the flange portion 59 is provided above the center in the vertical direction of the housing 50. The flange portion 59 is provided at a position approximately opposite to the operation portion (each analog stick 53A and 53B) provided on the surface of the housing 50. That is, the protrusion is provided in a region including a position on the opposite side of the operation unit provided on each of the left and right sides of the display unit. Therefore, when operating the said operation part, the user can hold | grip the terminal device 7 so that the collar part 59 may be supported with a middle finger or a ring finger (refer FIG. 10). As a result, the terminal device 7 becomes easier to hold, and the operation unit becomes easier to operate. Further, in the present embodiment, the protrusion has a hook-like shape extending from side to side (the protruding portion), so that the user holds the terminal device 7 with the middle finger or the ring finger along the lower surface of the protrusion. And the terminal device 7 becomes easier to hold. In addition, the collar part 59 should just be formed so that the (protruded part) may be extended in the left-right direction, and is not restricted to the shape extended in a horizontal direction as shown in FIG. In other embodiments, the flange portion 59 may extend in a direction slightly inclined from the horizontal direction. For example, the collar portion 59 may be provided so as to be inclined upward (or downward) from the left and right ends toward the center.

  In the present embodiment, the hook 59 having a hook-like shape is adopted as the protrusion formed on the back surface of the housing because a locking hole described later is provided in the hook 59. The part may have any shape. For example, in another embodiment, on the back side of the housing 50, two protrusions may be provided on both the left and right sides (no protrusion is provided in the center in the left-right direction) (see FIG. 29). . In another embodiment, the cross-sectional shape of the protrusion (the shape in the cross section perpendicular to the x-axis direction) is such that the terminal device 7 can be more firmly supported by the user's finger (the protrusion is It may be saddle-shaped (a shape in which the lower surface is recessed).

  In addition, the width | variety regarding the up-down direction of a projection part (hook part 59) may be what. For example, the protrusion may be formed up to the upper side of the housing 50. That is, the upper surface of the protrusion may be formed at the same position as the upper side surface of the housing 50. At this time, the housing 50 has a two-stage configuration in which the lower side is thin and the upper side is thick. As described above, it is preferable that the housing 50 is formed with surfaces (lower surfaces of the protrusions) facing downward on the left and right sides of the back surface. Accordingly, the user can easily grip the operating device by placing a finger on the surface. The “surface facing downward” may be formed at any position on the back surface of the housing 50, but is preferably positioned above the center of the housing 50.

  Further, as shown in FIGS. 8A, 8B, and 8C, the first L button 54I and the first R button 54J are provided on both the left and right sides of the upper surface of the housing 50, respectively. . In the present embodiment, the first L button 54 </ b> I and the first R button 54 </ b> J are provided in an obliquely upper portion (upper left portion and upper right portion) of the housing 50. Specifically, the first L button 54I is provided at the left end of the upper side surface of the plate-shaped housing 50 and is exposed from the upper left side surface (in other words, exposed from both the upper and left side surfaces). ) The first R button 54J is provided at the right end of the upper side surface of the housing 50, and is exposed from the upper right side surface (in other words, exposed from both the upper and right side surfaces). In this way, the first L button 54I is disposed at a position operable with the user's left index finger, and the first R button 54J is disposed at a position operable with the user's right hand index finger (see FIG. 10). In other embodiments, the operation units provided on the left and right sides of the upper surface of the housing 50 do not need to be provided at the left and right ends, and may be provided at positions other than the ends. In addition, operation portions may be provided on the left and right side surfaces of the housing 50, respectively.

  Further, as shown in FIG. 8 (c) and FIG. 9, the second L button 54K and the second R button 54L are disposed on the protruding portion (the collar portion 59). The second L button 54K is provided in the vicinity of the left end of the collar 59. The second R button 54L is provided in the vicinity of the right end of the collar portion 59. That is, the second L button 54K is provided slightly above the left side (left side when viewed from the front side) of the housing 50, and the second R button 54L is right side (when viewed from the front side) of the housing 50. On the right side). In other words, the second L button 54K is provided at a position (substantially) opposite to the left analog stick 53A provided on the surface, and the second R button 54L is provided (substantially) opposite to the right analog stick 53B provided on the surface. It is provided in the position. As described above, the second L button 54K is disposed at a position operable with the user's left middle finger (or index finger), and the second R button 54L is disposed at a position operable with the user's right middle finger (or index finger). (See FIG. 10). Moreover, the 2nd L button 54K and the 2nd R button 54L are provided in the upper surface of the said collar part 59, as shown to the (c) figure of FIG. Therefore, the second L button 54K and the second R button 54L have button surfaces that face upward (obliquely upward). When the user holds the terminal device 7, the middle finger or index finger is considered to move up and down, so that the user can easily press the second L button 54 </ b> K and the second R button 54 </ b> L by turning the button surface upward.

  As described above, in the present embodiment, the operation units (analog sticks 53A and 53B) are provided on the left and right sides of the display unit (LCD 51) above the center of the housing 50, and the operation is performed on the back side of the housing 50. Separate operation units (second L button 54K and second R button 54L) are provided at positions opposite to the unit. According to this, since the said operation part and another operation part are arrange | positioned in the position where the front side and back side of the housing 50 mutually oppose, when operating these operation parts, the user sets the housing 50 to the front side. It can be gripped so as to be pinched from the back side. Further, when operating these operation units, the user grips the upper side of the housing 50 with respect to the vertical center, so that the terminal device 7 can be gripped on the upper side and the terminal device 7 can be supported by the palm ( (See FIG. 10). As described above, the user can stably hold the housing 50 in a state in which the user can operate at least four operation units, can be easily held by the user, and has high operability. (Terminal device 7) can be provided.

  As described above, in the present embodiment, the user can easily hold the terminal device 7 by holding the terminal device 7 with the finger placed on the lower surface of the protrusion (the collar portion 59). In addition, since the second L button 54K and the second R button 54L are provided on the upper surface of the protrusion, the user can easily operate these buttons in the above state. For example, the user can easily hold the terminal device 7 in the following manner.

  That is, as shown in FIG. 10, the user can hold the terminal device 7 by placing the ring finger on the lower surface of the heel part 59 (the dashed line shown in FIG. 10) (supporting the heel part 59 with the ring finger). It is. At this time, the user can operate the four buttons (first L button 54I, first R button 54J, second L button 54K, and second R button 54L) with the index finger and the middle finger. For example, when a required game operation has many buttons to be used and is relatively complicated, many buttons can be easily operated by gripping as shown in FIG. Since each analog stick 53A and 53B is provided above the cross button 54A and the buttons 54E to 54H, the user can operate the analog sticks 53A and 53B with the thumb when a relatively complicated operation is required. Convenient. In FIG. 10, the user places a thumb on the surface of the housing 50, an index finger on the upper surface of the housing 50, a middle finger on the upper surface of the collar portion 59 on the rear surface of the housing 50, and a ring finger on the lower surface of the collar portion 59. The terminal device 7 is held by placing a little finger on the back surface of 50. Thus, the user can firmly hold the terminal device 7 so as to wrap the housing 50 from all sides.

  In addition, the user can hold the terminal device 7 by placing the middle finger on the lower surface of the collar portion 59. At this time, the user can easily operate the two buttons (the second L button 54K and the second R button 54L) with the index finger. For example, when the requested game operation is relatively simple with few buttons to be used, the terminal device 7 may be gripped by placing the middle finger on the lower surface of the collar portion 59. At this time, since the user can hold the lower side of the housing 50 with two fingers (ring finger and little finger), the user can hold the terminal device 7 firmly.

  In the present embodiment, the lower surface of the collar portion 59 is positioned between the analog sticks 53A and 53B and the cross button 54A and the four buttons 54E to 54H (more than the analog sticks 53A and 53B). And provided above the cross button 54A and the four buttons 54E to 54H). Therefore, when holding the terminal device 7 by placing the ring finger against the buttocks 59 (FIG. 10), the analog sticks 53A and 53B can be easily operated with the thumb, and the terminal device 7 is placed by placing the middle finger against the buttocks 59. When grasping, it is easy to operate the cross button 54A and the four buttons 54E to 54H with the thumb. That is, in any of the above two cases, the user can perform a direction input operation while firmly holding the terminal device 7.

  Further, as described above, the user can also hold the terminal device 7 while holding it vertically. That is, the user can hold the terminal device 7 in the vertical direction by holding the upper side or the lower side of the terminal device 7 with one hand. As described above, since the user can hold the terminal device 7 with one hand, for example, the user can perform an operation of holding the terminal device 7 with one hand and performing an input to the touch panel 52 with the other hand. Is possible.

  In addition, when gripping the upper side of the terminal device 7, the user can grip the terminal device 7 firmly by placing a finger other than the thumb against the lower surface of the collar portion 59. In particular, in the present embodiment, since the collar portion 59 is formed to extend to the left and right, the user places a finger other than the thumb on the collar portion 59 regardless of the position on the upper side of the terminal device 7. And the terminal device 7 can be firmly held. That is, when the terminal device 7 is used in a vertical orientation, the collar portion 59 can be used as a handle. On the other hand, when gripping the lower side of the terminal device 7 with one hand, the user can operate the buttons 54B to 54D with the hand. Therefore, for example, it is possible to perform operations on the buttons 54B to 54D with a hand holding the terminal device 7 while performing an input on the touch panel 52 with one hand, and more operations can be performed.

  In addition, regarding the terminal device 7 in the present embodiment, since the protrusion (the flange portion 59) is provided on the back surface, the terminal device 7 is placed with the screen of the LCD 51 (the surface of the housing 50) facing upward. The screen is slightly tilted. This makes it easier to see the screen when the terminal device 7 is placed. Further, it becomes easy to perform an input operation on the touch panel 52 in a state where the terminal device 7 is placed. In another embodiment, an additional protrusion having the same height as the flange 59 may be formed on the back surface of the housing 50. According to this, in a state where the screen of the LCD 51 is facing upward, the terminal device 7 can be placed so that the screen is horizontal as each projection comes into contact with the floor surface. Further, the additional protrusions may be removable (or foldable). According to this, the terminal device can be placed both in a state where the screen is slightly inclined and in a state where the screen is horizontal. That is, when the terminal device 7 is placed and used, the heel portion 59 can be used as a leg portion.

  Functions corresponding to the game program are appropriately assigned to the buttons 54A to 54L. For example, the cross button 54A and the buttons 54E to 54H may be used for a direction instruction operation or a selection operation, and the buttons 54B to 54E may be used for a determination operation or a cancel operation. Further, the terminal device 7 may have a button for turning on / off the screen display of the LCD 51 and a button for performing connection setting (pairing) with the game device 3.

  As illustrated in FIG. 8A, the terminal device 7 includes a marker portion 55 including a marker 55 </ b> A and a marker 55 </ b> B on the surface of the housing 50. The marker unit 55 is provided on the upper side of the LCD 51. Each of the markers 55A and 55B is composed of one or more infrared LEDs, like the markers 6R and 6L of the marker device 6. Infrared LEDs constituting the markers 55A and 55B are disposed inside a window portion that transmits infrared light. The marker unit 55 is used for the game device 3 to calculate the movement of the controller 5 and the like, similar to the marker device 6 described above. Further, the game apparatus 3 can control lighting of each infrared LED included in the marker unit 55.

  The terminal device 7 includes a camera 56 that is an imaging unit. The camera 56 includes an imaging element (for example, a CCD image sensor or a CMOS image sensor) having a predetermined resolution, and a lens. As shown in FIG. 8, in this embodiment, the camera 56 is provided on the surface of the housing 50. Therefore, the camera 56 can take an image of the face of the user who has the terminal device 7, and can, for example, take an image of the user who is playing the game while looking at the LCD 51. In the present embodiment, the camera 56 is disposed between the two markers 55A and 55B.

  The terminal device 7 includes a microphone 79 that is a voice input unit. A microphone hole 50 c is provided on the surface of the housing 50. The microphone 79 is provided inside the housing 50 behind the microphone hole 50c. The microphone 79 detects sounds around the terminal device 7 such as a user's voice.

  The terminal device 7 includes a speaker 77 which is an audio output unit. As shown in FIG. 8D, a speaker hole 57 is provided below the surface of the housing 50. The output sound of the speaker 77 is output from the speaker hole 57. In the present embodiment, the terminal device 7 includes two speakers, and speaker holes 57 are provided at positions of the left speaker and the right speaker. The terminal device 7 includes a knob 64 for adjusting the volume of the speaker 77. The terminal device 7 includes an audio output terminal 62 for connecting an audio output unit such as an earphone. Here, considering that an additional device is connected to the lower side surface of the housing, the audio output terminal 62 and the knob 64 are provided on the upper side surface of the housing 50. It may be provided on the side surface.

  The housing 50 is provided with a window 63 for emitting an infrared signal from the infrared communication module 82 to the outside of the terminal device 7. Here, the window 63 is provided on the upper side surface of the housing 50 so that an infrared signal is emitted forward of the user when both sides of the LCD 51 are gripped. However, in other embodiments, the window 63 may be provided at any position such as the back surface of the housing 50.

  Further, the terminal device 7 includes an expansion connector 58 for connecting other devices to the terminal device 7. The extension connector 58 is a communication terminal for transmitting / receiving data (information) to / from other devices connected to the terminal device 7. In the present embodiment, the extension connector 58 is provided on the lower side surface of the housing 50 as shown in FIG. Note that any other additional device connected to the expansion connector 58 may be used. For example, a controller (gun-type controller or the like) used for a specific game or an input device such as a keyboard may be used. . If it is not necessary to connect an additional device, the expansion connector 58 may not be provided. The expansion connector 58 may include a terminal for supplying power to the additional device and a terminal for charging.

  Further, the terminal device 7 includes a charging terminal 66 for acquiring power from the additional device, separately from the expansion connector 58. When the charging terminal 66 is connected to a charging stand (not shown), power is supplied from the charging stand to the terminal device 7. In the present embodiment, the charging terminal 66 is provided on the lower side surface of the housing 50. Therefore, when the terminal device 7 and the additional device are connected, in addition to transmitting and receiving information via the extension connector 58, it is also possible to supply power from one to the other. Thus, by providing the charging terminal 66 around the expansion connector 58 (on the left and right sides), when the terminal device 7 and the additional device are connected, it is possible to supply power as well as transmit and receive information. . Further, the terminal device 7 has a charging connector, and the housing 50 has a cover portion 61 for protecting the charging connector. The charging connector can be connected to a charger 86 to be described later. When the charging connector is connected to the charger, power is supplied from the charger 86 to the terminal device 7. In this embodiment, considering that the additional device is connected to the lower side surface of the housing, the charging connector (cover portion 61) is provided on the upper side surface of the housing 50. It may be provided on the side surface.

  As shown in FIGS. 8D and 9, locking holes 59 a and 59 b that can lock the claw portions of the additional device are provided on the lower surface of the protruding portion (the flange portion 59). The locking holes 59 a and 59 b are used when connecting another additional device to the terminal device 7. That is, the additional device has a claw portion that can be locked in the locking holes 59a and 59b. When the additional device is connected to the terminal device 7, the claw portion is locked in the locking holes 59a and 59b. The terminal device 7 and the additional device are fixed. Further, screw holes may be further provided inside the locking holes 59a and 59b, and the additional device may be firmly fixed with screws. Here, the protrusion provided on the back surface of the terminal device 7 is a flange 59 having a hook shape. That is, the collar part 59 is provided extending in the left-right direction. As shown in FIG. 9, the locking holes 59 a and 59 b are provided near the center (with respect to the left-right direction) of the lower surface of the flange portion 59. Note that the number of the locking holes 59a and 59b provided on the lower surface of the flange portion 59 may be any number, but when it is one, it is preferably provided in the center of the flange portion 59, and there are a plurality Are preferably arranged symmetrically. According to this, it is possible to stably connect the additional device while keeping the left and right balance equal. Further, when the locking hole is provided near the center, the size of the additional device can be reduced as compared with the case where the locking hole is provided at both the left and right ends. As described above, the flange portion 59 can be used as a locking member of the additional device.

  In the present embodiment, as shown in FIG. 8D, locking holes 50 a and 50 b are provided on the lower surface of the housing 50. Therefore, when connecting the additional device to the terminal device 7, the terminal device 7 and the additional device are fixed by the four claw portions being respectively locked in the four locking holes. As a result, the additional device can be firmly connected to the terminal device 7. It should be noted that screw holes may be provided inside the locking holes 50a and 50b, and the additional device may be screwed. The screwing position may be anywhere, but, for example, the supporting portion of the additional device that contacts the back surface of the housing 50 and the flange portion 59 may be screwed. In other embodiments, the locking holes provided in the housing may be arranged in any manner.

  Further, the terminal device 7 has a battery lid 67 that can be attached to and detached from the housing 50. A battery (battery 85 shown in FIG. 11) is disposed inside the battery lid 67. In the present embodiment, the battery lid 67 is provided on the back side of the housing 50 and is provided on the lower side of the protrusion (the flange 59).

  Further, the housing 50 of the terminal device 7 is provided with holes 65a and 65b for tying strap strings. As shown in FIG. 8D, in the present embodiment, the holes 65 a and 65 b are provided on the lower surface of the housing 50. In the present embodiment, two holes 65 a and 65 b are provided on each of the left and right sides of the housing 50. That is, the hole 65 a is provided on the left side from the center of the lower surface of the housing 50, and the hole 65 b is provided on the right side of the center of the lower surface of the housing 50. The user may tie a strap to either of the holes 65a and 65b and tie the strap to his / her wrist. Thereby, even if the user drops the terminal device 7 or the terminal device 7 is removed from the hand, the terminal device 7 can be prevented from dropping or colliding with another object. In the present embodiment, since the holes are provided on both the left and right sides, the user can tie the strap to either hand, which is convenient.

  In addition, regarding the terminal device 7 shown in FIGS. 8 to 11, the shapes of the operation buttons and the housing 50, the number of components, the installation positions, and the like are merely examples, and other shapes, numbers, and installation positions are provided. It may be.

  Next, the internal configuration of the terminal device 7 will be described with reference to FIG. FIG. 11 is a block diagram illustrating an internal configuration of the terminal device 7. As shown in FIG. 11, in addition to the configuration shown in FIG. 8, the terminal device 7 includes a touch panel controller 71, a magnetic sensor 72, an acceleration sensor 73, a gyro sensor 74, a user interface controller (UI controller) 75, a codec LSI 76, a speaker. 77, a sound IC 78, a microphone 79, a wireless module 80, an antenna 81, an infrared communication module 82, a flash memory 83, a power supply IC 84, and a battery 85. These electronic components are mounted on an electronic circuit board and stored in the housing 50.

  The UI controller 75 is a circuit for controlling input / output of data to / from various input / output units. The UI controller 75 is connected to the touch panel controller 71, the analog stick 53 (analog sticks 53 </ b> A and 53 </ b> B), the operation buttons 54 (operation buttons 54 </ b> A to 54 </ b> L), the marker unit 55, the magnetic sensor 72, the acceleration sensor 73, and the gyro sensor 74. Is done. The UI controller 75 is connected to the codec LSI 76 and the extension connector 58. A power supply IC 84 is connected to the UI controller 75, and power is supplied to each unit via the UI controller 75. A built-in battery 85 is connected to the power supply IC 84 to supply power. In addition, a charger 86 or a cable that can acquire power from an external power source can be connected to the power supply IC 84 via a charging connector. The terminal device 7 can be connected to the external power source using the charger 86 or the cable. Power supply and charging from can be performed. The terminal device 7 can also be charged by mounting the terminal device 7 on a cradle having a charging function (not shown). That is, although not shown, a cradle capable of acquiring power from an external power source can be connected to the power supply IC 84 via the charging terminal 66, and the terminal device 7 supplies power from the external power source using the cradle. And can be charged.

  The touch panel controller 71 is a circuit that is connected to the touch panel 52 and controls the touch panel 52. The touch panel controller 71 generates touch position data in a predetermined format based on a signal from the touch panel 52 and outputs it to the UI controller 75. The touch position data represents the coordinates of the position where the input has been performed on the input surface of the touch panel 52. The touch panel controller 71 reads a signal from the touch panel 52 and generates touch position data at a rate of once per predetermined time. Various control instructions for the touch panel 52 are output from the UI controller 75 to the touch panel controller 71.

  The analog stick 53 outputs to the UI controller 75 stick data representing the direction and amount in which the stick unit operated by the user's finger has slid (or tilted). In addition, the operation button 54 outputs operation button data representing the input status (whether or not it has been pressed) to each of the operation buttons 54 </ b> A to 54 </ b> L to the UI controller 75.

  The magnetic sensor 72 detects the direction by detecting the magnitude and direction of the magnetic field. The azimuth data indicating the detected azimuth is output to the UI controller 75. In addition, a control instruction for the magnetic sensor 72 is output from the UI controller 75 to the magnetic sensor 72. For the magnetic sensor 72, an MI (magnetic impedance) element, a fluxgate sensor, a Hall element, a GMR (giant magnetoresistance) element, a TMR (tunnel magnetoresistance) element, an AMR (anisotropic magnetoresistance) element, or the like was used. Although there is a sensor, any sensor may be used as long as it can detect the direction. Strictly speaking, in a place where a magnetic field is generated in addition to the geomagnetism, the obtained azimuth data does not indicate the azimuth, but even in such a case, the terminal device 7 moves. Since the orientation data changes, the change in the attitude of the terminal device 7 can be calculated.

  The acceleration sensor 73 is provided inside the housing 50 and detects the magnitude of linear acceleration along the direction of three axes (xyz axes shown in FIG. 8A). Specifically, the acceleration sensor 73 has a linear acceleration of each axis with the long side direction of the housing 50 as the x axis, the direction perpendicular to the surface of the housing 50 as the y axis, and the short side direction of the housing 50 as the z axis. Detect the size of. Acceleration data representing the detected acceleration is output to the UI controller 75. A control instruction for the acceleration sensor 73 is output from the UI controller 75 to the acceleration sensor 73. The acceleration sensor 73 is, for example, a capacitive MEMS acceleration sensor in the present embodiment, but other types of acceleration sensors may be used in other embodiments. Further, the acceleration sensor 73 may be an acceleration sensor that detects a uniaxial or biaxial direction.

  The gyro sensor 74 is provided inside the housing 50 and detects angular velocities around the three axes of the x axis, the y axis, and the z axis. Angular velocity data representing the detected angular velocity is output to the UI controller 75. Further, a control instruction for the gyro sensor 74 is output from the UI controller 75 to the gyro sensor 74. Any number and combination of gyro sensors may be used to detect the three-axis angular velocity, and the gyro sensor 74 is similar to the gyro sensor 48 in that a two-axis gyro sensor, a one-axis gyro sensor, It may be constituted by. Further, the gyro sensor 74 may be a gyro sensor that detects a uniaxial or biaxial direction.

  The UI controller 75 outputs operation data including touch position data, stick data, operation button data, azimuth data, acceleration data, and angular velocity data received from each of the above components to the codec LSI 76. When another device is connected to the terminal device 7 via the extension connector 58, the operation data may further include data representing an operation on the other device.

  The codec LSI 76 is a circuit that performs compression processing on data transmitted to the game apparatus 3 and expansion processing on data transmitted from the game apparatus 3. The codec LSI 76 is connected to the LCD 51, the camera 56, the sound IC 78, the wireless module 80, the flash memory 83, and the infrared communication module 82. The codec LSI 76 includes a CPU 87 and an internal memory 88. Although the terminal device 7 is configured not to perform the game process itself, it is necessary to execute a minimum program for management and communication of the terminal device 7. When the power is turned on, the program stored in the flash memory 83 is read into the internal memory 88 and executed by the CPU 87, whereby the terminal device 7 is activated. A part of the internal memory 88 is used as a VRAM for the LCD 51.

  The camera 56 captures an image in accordance with an instruction from the game apparatus 3 and outputs the captured image data to the codec LSI 76. Control instructions for the camera 56 such as an image capturing instruction are output from the codec LSI 76 to the camera 56. Note that the camera 56 can also capture moving images. That is, the camera 56 can repeatedly capture images and repeatedly output image data to the codec LSI 76.

  The sound IC 78 is a circuit that is connected to the speaker 77 and the microphone 79 and controls input / output of audio data to and from the speaker 77 and the microphone 79. That is, when audio data is received from the codec LSI 76, the sound IC 78 outputs an audio signal obtained by performing D / A conversion on the audio data to the speaker 77, and outputs sound from the speaker 77. The microphone 79 detects a sound (such as a user's voice) transmitted to the terminal device 7 and outputs a sound signal indicating the sound to the sound IC 78. The sound IC 78 performs A / D conversion on the audio signal from the microphone 79 and outputs audio data in a predetermined format to the codec LSI 76.

  The infrared communication module 82 emits an infrared signal and performs infrared communication with other devices. Here, the infrared communication module 82 includes, for example, a function of performing infrared communication according to the IrDA standard and a function of outputting an infrared signal (control signal) for controlling the television 2.

  The codec LSI 76 transmits the image data from the camera 56, the audio data from the microphone 79, and the operation data from the UI controller 75 to the game apparatus 3 via the wireless module 80 as terminal operation data. In the present embodiment, the codec LSI 76 performs the same compression processing as the codec LSI 27 on the image data and the audio data. The terminal operation data and the compressed image data and audio data are output to the wireless module 80 as transmission data. An antenna 81 is connected to the wireless module 80, and the wireless module 80 transmits the transmission data to the game apparatus 3 via the antenna 81. The wireless module 80 has the same function as the terminal communication module 28 of the game apparatus 3. That is, the wireless module 80 has a function of connecting to a wireless LAN by a method compliant with, for example, the IEEE 802.11n standard. The data to be transmitted may or may not be encrypted as necessary.

  As described above, the transmission data transmitted from the terminal device 7 to the game apparatus 3 includes operation data (terminal operation data), image data, and audio data. When another device is connected to the terminal device 7 via the extension connector 58, the data received from the other device may be further included in the transmission data. The codec LSI 76 may transmit the data received by the infrared communication by the infrared communication module 82 to the game apparatus 3 by including the data in the transmission data as necessary.

  Further, as described above, compressed image data and audio data are transmitted from the game apparatus 3 to the terminal apparatus 7. These data are received by the antenna 81 (receiving unit) and sent to the codec LSI 76 via the wireless module 80. The codec LSI 76 decompresses the received image data and audio data. The expanded image data is output to the LCD 51, and the image is displayed on the LCD 51. That is, the codec LSI 76 (CPU 87) displays the received image data on the display unit. The expanded audio data is output to the sound IC 78, and the sound IC 78 outputs sound from the speaker 77.

  When the control data is included in the data received from the game apparatus 3, the codec LSI 76 and the UI controller 75 issue a control instruction to each unit according to the control data. As described above, the control data is a control instruction for each component (in this embodiment, the camera 56, the touch panel controller 71, the marker unit 55, the sensors 62 to 64, and the infrared communication module 82) included in the terminal device 7. It is data to represent. In the present embodiment, as the control instruction represented by the control data, an instruction to operate each of the above components or to stop (stop) the operation can be considered. That is, components that are not used in the game may be paused in order to reduce power consumption. In that case, the transmission data transmitted from the terminal device 7 to the game device 3 includes data from the paused components. Do not let it. In addition, since the marker part 55 is infrared LED, control may just be ON / OFF of supply of electric power.

  The game apparatus 3 can control the operation of the television 2 by controlling the output of the infrared communication module 82. That is, the game apparatus 3 outputs an instruction (the control data) for causing the infrared communication module 82 to output an infrared signal corresponding to a control command for controlling the television 2 to the terminal apparatus 7. In response to this instruction, the codec LSI 76 causes the infrared communication module 82 to output an infrared signal corresponding to the control command. Here, the television 2 includes an infrared light receiving unit capable of receiving an infrared signal. When the infrared signal output from the infrared communication module 82 is received by the infrared light receiving unit, the television 2 performs an operation according to the infrared signal. The instruction from the game apparatus 3 may indicate an infrared signal pattern, or when the terminal device 7 stores an infrared signal pattern, the instruction indicates the pattern. Also good.

  As described above, the terminal device 7 includes operation means such as the touch panel 52, the analog stick 53, and the operation button 54. However, in other embodiments, instead of these operation means or together with these operation means. The configuration may include other operation means.

  In addition, the terminal device 7 includes a magnetic sensor 72, an acceleration sensor 73, and a gyro sensor 74 as sensors for calculating the movement of the terminal device 7 (including changes in position and posture, or position and posture). In other embodiments, the configuration may include only one or two of these sensors. Moreover, in other embodiment, it may replace with these sensors or the structure provided with another sensor with these sensors may be sufficient.

  The terminal device 7 includes the camera 56 and the microphone 79. However, in other embodiments, the terminal device 7 may not include the camera 56 and the microphone 79, or may include only one of them. Good.

  Further, the terminal device 7 is configured to include the marker unit 55 as a configuration for calculating the positional relationship between the terminal device 7 and the controller 5 (the position and / or orientation of the terminal device 7 viewed from the controller 5). In other embodiments, the marker unit 55 may not be provided. In another embodiment, the terminal device 7 may include other means as a configuration for calculating the positional relationship. For example, in another embodiment, the controller 5 may include a marker unit, and the terminal device 7 may include an image sensor. Furthermore, in this case, the marker device 6 may be configured to include an imaging element instead of the infrared LED.

(Connection configuration between game system and external device)
FIG. 12 is a block diagram showing a connection relationship between the game system 1 and an external device. As shown in FIG. 12, the game device 3 in the game system 1 can communicate with an external device 91 via a network 90. The network 90 is an arbitrary communication network such as the Internet. The external device 91 is a Web server or another terminal device that communicates with the game device 3 (for example, a counterpart game device in the case of performing a communication game). Note that the number of external devices 91 is not necessarily one, and the game apparatus 3 may communicate with a plurality of external devices. The game apparatus 3 communicates with the network 90 and acquires web pages, various applications, game data, and the like from the external apparatus 91 via the network 90. The acquired image information or an image generated based on the acquired information is output from the game device 3 to the terminal device 7 and / or the television 2. The game system 1 can display various images on two display devices.

[5. Outline of operation in game system]
Next, an outline of operations executed in the game system 1 will be described. In the present embodiment, the game system 1 can execute various applications including a game program (game application), perform operations on the television 2 (and display a TV program guide), and display a web page. it can. In addition, the game system 1 operates in the state where the television is used in the game system 1 (a state where an image generated in the game system 1 is displayed on the television 2) and the television for other purposes than the game system 1 by the following operations. It is possible to perform a switching operation between the state to be used (a state in which an image different from the image generated in the game system 1 is displayed on the television 2) with good operability.

  FIG. 13 is a state transition diagram showing the transition of the operation state in the game system 1. As shown in FIG. 13, when the game system 1 (game device 3) is activated, first, the game system 1 enters a start state. The start state is a state in which a predetermined operation image is displayed on the terminal device 7 after the game system 1 is activated. Here, the predetermined operation image may be any image as long as the user using the terminal device 7 performs an operation. In the present embodiment, any of an application selection image (hereinafter referred to as “application selection image”), a device control image, and a browser image, which will be described later, is displayed on the terminal device 7 as a predetermined operation image. In another embodiment, a plurality of images of the three types of images may be displayed on the terminal device 7 simultaneously (for example, by dividing the display area into a plurality of images). Hereinafter, the operation image displayed in the start state may be referred to as a “main operation image” for the purpose of distinguishing it from an operation image displayed in the interruption state described later.

  Note that the “start state” is a state in which an application can be selected after the game apparatus 3 is activated, and is not limited to immediately after activation. For example, a predetermined image may be displayed for a predetermined time immediately after activation, and then may be in a start state, or a predetermined application may be executed and then returned to the start state after the end.

  In the start state, a predetermined main operation image is displayed on the terminal device 7, and operations in the game system are performed using the terminal device 7. That is, the terminal device 7 that is an operation device includes an operation unit that outputs operation data (terminal operation data) representing an operation on the operation device. The terminal device 7 also includes a receiving unit (antenna 81 and wireless module 80) that receives an image output from the game device 3, and a display unit (LCD 51) that displays the received image. On the other hand, the game apparatus 3 includes an acquisition unit that acquires the operation data, and generates the main operation image in the start state. Furthermore, the game apparatus 3 outputs the main operation image to the terminal device 7.

  In the present embodiment, as shown in FIG. 13, the start state includes a menu state, a first television operation state, and a browser state. The menu state is a state (mode) in which various applications can be selected and executed. The first television operation state is a state (mode) in which an operation on the television 2 can be performed. The browser state is a state (mode) in which a web page can be displayed. Although details will be described later, the application selection image (FIG. 17) in the menu state, the device control image (FIG. 19) in the first television operation state, and the browser image (FIG. 21) in the browser state are respectively terminal devices. 7 is displayed. The application selection image is an operation image for selecting an application that can be executed by the game apparatus 3. The device control image is an operation image for performing an operation (control) on the television 2. The browser image is an operation image including a web page acquired via the network 90. In the start state, a predetermined operation image (main operation image) may be displayed on the terminal device 7, and in another embodiment, only one type of image is always displayed in the start state. There may be.

  Further, when the game system 1 is activated and enters a start state, the game system 1 is in any one of the above three states (menu state, first television operation state, and browser state). Note that the state immediately after the game system 1 is activated may be determined in any way. In the present embodiment, the state at the time when the power of the game system 1 was turned off last time is set as the state immediately after startup. Although details will be described later, the game system 1 can transition from one of the three states to the other two states according to a predetermined change operation (see FIG. 13). ). That is, the user can easily switch the operation image to be displayed on the terminal device 7 among the application selection image, the device control image, and the browser image by the change operation.

  Here, in the menu state in the start state, the user can select and execute one of various applications including the game program using the terminal device 7. In the first television operation state in the start state, the user can perform an operation on the television 2 using the terminal device 7. That is, the game apparatus 3 determines whether or not there has been a selection instruction and whether or not there has been a control instruction in the start state (steps S14 and S23 to be described later). The selection instruction is an instruction to select one of one or more applications that can be executed by the game apparatus 3 including a game application. The control instruction is an instruction to perform control on the television 2. When there is the selection instruction, the game apparatus 3 executes the selected application and outputs an image generated by executing the application to at least one of the terminal device 7 and the television 2 (step S15 described later). Further, when there is the control instruction, the game apparatus 3 performs control according to the control instruction on the television 2 (step S24 described later).

  As described above, in the present embodiment, since the operation image (main operation image) is displayed on the terminal device 7 in the start state, the user performs an operation on the game system 1 by looking at the screen of the terminal device 7. In addition, the TV 2 can display other images such as a TV program. Therefore, the user can use the game system 1 while watching a television program. For example, the user can select an application to be executed while viewing the operation image of the terminal device 7 while watching the television program on the screen of the television 2. In the start state, the user can operate the television 2 using the terminal device 7. Since the user can perform an operation on the game system 1 and an operation on the television 2 using the terminal device 7, a state where the television is used in the game system 1 and a state where the television is used for purposes other than the game system 1 Switching operation between them can be easily performed. For example, in the present embodiment, a television program guide (electronic program guide) is displayed as a device control image. Therefore, the user activates the game system 1 to display the program guide on the terminal device 7 and displays the displayed program. A series of operations for selecting a television program to be viewed from the table and displaying it on the television 2 can be performed only by the terminal device 7. As described above, in the present embodiment, the terminal device 7 can improve the operability of the switching operation.

  In the present embodiment, the game apparatus 3 switches the application selection image and the device control image to be displayed on the terminal device 7. However, in other embodiments, the game apparatus 3 displays the application selection image and the device control image. The image may be displayed on the terminal device 7 at the same time. According to this, the user performs an operation (the change operation) for changing the state between a state where the selection instruction can be made (menu state) and a state where the control instruction can be made (first television operation state). These two instructions can be performed freely without performing the above. In another embodiment, the game apparatus 3 may allow a user to issue the control instruction in a menu state where the application selection image is displayed, or the apparatus control image is displayed. The user may be able to give the selection instruction in the first television operation state. Also by this, the user can freely give the two instructions without performing the changing operation.

  In addition, the game apparatus 3 further determines whether or not there is a display instruction for displaying a web page in the start state (steps S17 and S26 described later). When there is a display instruction, the game system 1 transitions to a browser state (see FIG. 13). In this case, the game apparatus 3 generates a browser image generated based on data acquired from the network 90 via an external communication unit (network communication module 18) that communicates with the network 90 on the terminal device 7 and the television 2. Output to at least one (steps S31 and S35 described later). That is, the game device 3 displays a browser image including a web page on the terminal device 7 and / or the television 2 in response to the display instruction. Thus, in this embodiment, the user can also browse the web page in the start state.

  In addition, when an operation for selecting an application is performed using the terminal device 7 in the menu state, the game apparatus 3 executes the selected application. That is, the game apparatus 3 generates a main operation image (the application selection image) including an image (an icon or the like) representing an application that can be executed by the game apparatus 3 and outputs the main operation image to the terminal device 7. Then, when an application to be executed is selected in a state where the main operation image is displayed, the selected application is executed. Thereby, as shown in FIG. 13, the game system 1 transits to the application execution state. Furthermore, the game device 3 outputs an image generated by executing the application to at least one of the terminal device 7 and the television 2. It should be noted that which of the two display devices (or both) outputs the image depends on the application to be executed and may be determined in any way.

  When a predetermined interruption operation is performed on the terminal device 7 in the application execution state, the game system 1 transitions to the interruption state (see FIG. 13). The suspended state is a state in which use of a running application is suspended. When a predetermined interruption operation is performed on the terminal device 7 during the execution of the application, the game device 3 generates a predetermined operation image and outputs it to the terminal device 7 (step S44 described later). The predetermined operation image may be any image as long as the user using the terminal device 7 performs an operation. In the present embodiment, a setting image (FIG. 23) or a remote control image (FIG. 24) described later is displayed on the terminal device 7 as a predetermined operation image. In another embodiment, only one type of operation image may be displayed on the terminal device 7 or the two types of images may be displayed on the terminal device 7 at the same time. Hereinafter, the operation image displayed in the suspended state is referred to as a “sub-operation image”.

  The interruption state includes a setting state and a second television operation state. In the present embodiment, when the application execution state transitions to the suspended state, the game system 1 first transitions to the set state. The setting state is a state in which the setting image is displayed. Although the details will be described later, the setting image is an operation image for performing various setting operations related to the game system 1. In the interrupted state, it is only necessary to display a predetermined sub-operation image on the terminal device 7. In other embodiments, only one type of the two types of images is always displayed in the interrupted state. It may be a configuration.

  In the present embodiment, when a predetermined operation is performed in the set state, the game system 1 transitions to the second television operation state. The second television operation state is a state in which the remote control image is displayed. Although details will be described later, the remote control image is an operation image for performing an operation on the television 2, and is an operation image including a button image representing an operation on the television 2. The second television operation state is a state in which an operation on the television 2 can be performed using the terminal device 7. Note that when a predetermined operation is performed in the second television operation state, the game system 1 transitions from the second television operation state to the setting state. That is, the user can freely transition between the setting state and the second television operation state by a predetermined operation (see FIG. 13).

  As described above, in the present embodiment, the user can perform an operation on the television 2 in the interrupted state. That is, the game apparatus 3 determines whether there is a control instruction for controlling the television 2 in a state where the sub operation image is displayed on the terminal apparatus 7 based on the terminal operation data. When there is a control instruction, the television 2 is controlled according to the control instruction. Therefore, in the present embodiment, when the application is used in the game system 1, the user can perform an operation on the television 2 without ending the execution of the application. According to this, the switching operation between the state in which the television is used in the game system 1 and the state in which the television is used for purposes other than the game system 1 becomes easier. For example, the user can easily perform an operation of watching a television program during the execution of the application or restarting the application thereafter using the terminal device 7.

  In the present embodiment, in the interrupted state, an end instruction for ending an application being executed and a restart instruction for resuming the execution of the application can be performed. That is, when there is a resume instruction in the suspended state, the game apparatus 3 resumes execution of the application. Thereby, the game system 1 transits to the application execution state (see FIG. 13). If there is an end instruction in the interrupted state, the game apparatus 3 ends the execution of the application. As a result, the game system 1 transitions to the start state (see FIG. 13). In the present embodiment, the game system transitions to the menu state as shown in FIG. 13 in the above case. However, in other embodiments, the game system 1 is in the first television operation state or the browser state. You may transition to. In another embodiment, when the application is terminated in the suspended state, the user may be able to select which of the three states included in the start state is to be changed. As described above, in the present embodiment, the user can resume or end the execution of the application from the suspended state.

[6. Details of processing in game system]
Next, details of information processing executed in the game system will be described. First, various data used in information processing will be described. FIG. 14 is a diagram showing a storage area in the main memory (the external main memory 12 or the internal main memory 11e) of the game apparatus 3. As shown in FIG. 14, the main memory of the game apparatus 3 is provided with a main program area 100, an application program area 101, an acquisition data area 103, and processing data 107 area. The main memory stores data necessary for processing such as image data and audio data used for information processing in addition to the data shown in FIG. A part of the storage area shown in FIG. 14 may be provided in another storage device (flash memory 17 or the like) accessible by the game apparatus 3.

  The main program area 100 stores a main program for managing the transition of the various states described above in the game system 1. In the present embodiment, each step of the flowchart shown in FIG. 15 and the like is executed by the CPU 10 executing the main program. The main program is a program for executing each processing of menu processing, television processing, browser processing, and interruption processing, which will be described later. In other embodiments, each program for executing these processings. May be stored separately (as a separate program) in the main memory. A part or all of the main program is read from a predetermined storage device and stored in the main memory at an appropriate timing after the game apparatus 3 is powered on. The predetermined storage device may be any storage means that can be accessed by the game apparatus 3, but is assumed to be the flash memory 17 in the present embodiment. The main program may be acquired from the optical disk 4 or another device outside the game apparatus 3 (for example, via the Internet) instead of the flash memory 17.

  The application program area 101 stores various applications (programs) that can be executed by the game apparatus 3. One or more applications are stored in the application program area 101. In the present embodiment, the one or more applications include a game program 102. A part or all of the game program 102 is read from the optical disc 4 and stored in the main memory at an appropriate timing after the game apparatus 3 is turned on. Note that the game program 102 may be obtained from the flash memory 17 or an external device of the game apparatus 3 (for example, via the Internet) instead of the optical disc 4. Note that other applications included in the application program area 101 may also be acquired from any storage means accessible by the game apparatus 3, as with the game program 102.

  In the acquisition data area 103, various data acquired by the game apparatus 3 from the terminal apparatus 7 and the like are stored. In the present embodiment, terminal operation data 104, camera image data 105, and microphone sound data 106 are stored in the acquisition data area 103. In addition to the data shown in FIG. 14, controller operation data acquired from the controller 5 may be stored in the acquisition data area 103.

  The terminal operation data 104 is data representing a player's operation on the terminal device 7. The terminal operation data 104 is transmitted from the terminal device 7, acquired by the game device 3, and stored in the main memory. Note that the game apparatus 3 can communicate with a plurality of terminal devices, and can acquire operation data from each terminal device. When there are a plurality of terminal devices, each terminal operation data respectively transmitted from each terminal device is stored in the main memory. A predetermined number of terminal operation data may be stored in the main memory in order from the latest (last acquired) for each terminal device.

  The terminal operation data 104 includes angular velocity data, acceleration data, operation button data, stick data, touch position data, and azimuth data. The angular velocity data is data representing the angular velocity detected by the gyro sensor 74. In the present embodiment, the angular velocity data represents the angular velocities around the three axes xyz shown in FIG. 8, but in other embodiments, the angular velocity data may represent any one or more axial angular velocities. That's fine. The acceleration data is data representing the acceleration (acceleration vector) detected by the acceleration sensor 73. In the present embodiment, the acceleration data represents a three-dimensional acceleration whose components are accelerations in the xyz three-axis directions shown in FIG. 9, but in other embodiments, any one or more directions are used. Any acceleration can be used. The gyro sensor 74 and the acceleration sensor 73 are examples of inertial sensors. The game apparatus 3 can calculate the attitude of the terminal device 7 based on the detection results (angular velocity data and acceleration data) of the inertia sensor.

  The operation button data is data representing an input state for each of the operation buttons 54 </ b> A to 54 </ b> L provided on the terminal device 7. Specifically, the operation button data represents whether or not each of the operation buttons 54A to 54L is pressed. The stick data is data representing the direction and amount in which the stick portion of the analog stick 53 (analog sticks 53A and 53B) has slid (or tilted). The direction and amount may be expressed as, for example, a two-dimensional coordinate or a two-dimensional vector. The touch position data is data representing a position (touch position) where an input is performed on the input surface of the touch panel 52. In the present embodiment, the touch position data represents a coordinate value of a two-dimensional coordinate system for indicating a position on the input surface. When the touch panel 52 is a multi-touch method, the touch position data may represent a plurality of touch positions. The azimuth data is data representing the azimuth detected by the magnetic sensor 72. In the present embodiment, the azimuth data represents the direction of a predetermined azimuth (for example, north) with reference to the terminal device 7. When a magnetic field other than geomagnetism is generated, the azimuth data does not strictly indicate an absolute azimuth (north, etc.). However, since the relative direction of the terminal device 7 with respect to the direction of the magnetic field at the place is indicated by the orientation data, even in such a case, the orientation of the terminal device 7 or a change in the orientation is calculated based on the orientation data. It is possible.

  Note that the terminal operation data 104 only needs to represent an operation on the terminal device 7 and may include only one of the above-described data included in the terminal operation data 104 in the present embodiment. Further, when the terminal device 7 has other input means (for example, a touch pad, an imaging means of the controller 5, etc.), the terminal operation data 104 includes data representing an operation on the other input means. Also good. When the movement of the terminal device 7 itself is used as a game operation as in the present embodiment, the terminal operation data 104 includes information related to the movement of the terminal device 7 such as acceleration data, angular velocity data, or azimuth data. It is preferable to include data to be expressed (data whose value changes according to movement). Note that the information related to the movement of the terminal device 7 is, for example, information such as the position, posture, position change (velocity), posture change (angular velocity), acceleration, and angular acceleration of the terminal device 7.

  The camera image data 105 is data representing an image (captured image) captured by the camera 56 of the terminal device 7. The camera image data 105 is image data obtained by decompressing compressed image data transmitted from the terminal device 7 by the codec LSI 27, and is stored in the main memory by the input / output processor 11a. The microphone sound data 106 is data representing the sound (microphone sound) detected by the microphone 79 of the terminal device 7. The microphone sound data 106 is sound data obtained by decompressing the compressed sound data transmitted from the terminal device 7 by the codec LSI 27, and is stored in the main memory by the input / output processor 11a.

  The processing data area 107 stores various data used in information processing (FIG. 15 and the like) in the game apparatus 3. In the present embodiment, the processing data area 107 stores initial state data 108, program guide data 109, home page data 110, save data 111, and the like. In addition to the data shown in FIG. 14, the processing data area 107 stores various data used when executing each application.

  The initial state data 108 is data representing an initial state that is transitioned immediately after the game system 1 is activated among the three types of states included in the start state. The initial state data 108 may be set in any way, but in the present embodiment, when the game system 1 is terminated (when the power is turned off or put into the sleep state), the state at that time To be determined.

  The program guide data 109 is data for generating a TV program guide displayed on the terminal device 7 at the time of transition to the first television operation state. In the present embodiment, program guide data is acquired from a predetermined server via the network 90 at an appropriate timing, and the data is stored in the main memory as program guide data 109. In the present embodiment, when the game system 1 transitions to the first television operation state, a program guide generated based on the program guide data 109 is first displayed on the terminal device 7.

  The homepage data 110 is data for generating a homepage (web page displayed at the start of the browser function) displayed on the terminal device 7 at the time of transition to the browser state described above. In the present embodiment, preset homepage data is acquired from a predetermined server via the network 90 at an appropriate timing, and the data is stored in the main memory as homepage data 110. In the present embodiment, when the game system 1 transitions to the browser state, an image of a web page generated based on the home page data 110 is first displayed on the terminal device 7.

  The save data 111 is save data of an application such as the game program 102. Although details will be described later, when the execution of the application is terminated, data representing the contents of the game and representing the game parameters necessary for restarting the game is stored as save data 111 in the main memory.

  Next, the flow of information processing executed in the game system 1 (game device 3) will be described in detail with reference to FIGS. FIG. 15 is a flowchart showing a flow of start processing executed by the game apparatus 3. The start process is a process executed in response to the activation of the game system 1 (game device 3). That is, when the power of the game apparatus 3 is turned on, the CPU 10 of the game apparatus 3 executes a startup program stored in a boot ROM (not shown), thereby initializing each unit such as the main memory. Thereafter, the main program stored in the flash memory 17 is read into the main memory, and the CPU 10 starts executing the main program. The flowchart shown in FIG. 15 is a flowchart showing a process performed after the above process is completed.

  In addition, the process of each step in each flowchart shown to drawing is only an example, and if the same result is obtained, you may replace the process order of each step. Moreover, the value of the variable and the threshold value used in the determination step are merely examples, and other values may be adopted as necessary. Further, in this specification, the processing of each step of each flowchart is described as being executed by the CPU 10, but the processing of some steps in each flowchart may be executed by a processor or a dedicated circuit other than the CPU 10. Good.

  First, in step S1, the CPU 10 determines an initial state to be changed first among the three states included in the start state (menu state, first television operation state, and browser state). Here, the initial state may be determined in any way, and in the present embodiment, the initial state at the next activation is determined based on the state at the end of the previous game system 1. That is, when the game system 1 is terminated (when the power is turned off or in the sleep state), data representing the state at that time is stored in the flash memory 17. In step S1, the CPU 10 reads the data from the flash memory 17, and determines an initial state based on the state represented by the data. Specifically, when the data represents one of three states included in the start state, the state represented by the data is determined as the initial state. If the data represents an application execution state or an interruption state, the menu state is determined as the initial state. Data representing the initial state determined as described above is stored in the main memory as initial state data. As described above, in the present embodiment, since the state at the end of the previous game system 1 is the initial state, the user who used the previous game system 1 starts using the game system 1 in an appropriate state. be able to. Following step S1, the process of step S2 is executed.

  In other embodiments, the initial state may be determined by other methods. For example, the initial state may be determined as a state specified in advance by the user. Also by this, like the present embodiment, the use of the game system 1 can be started in a state appropriate for the user. Further, for example, when the game apparatus 3 can identify the user, the initial state may be determined so as to be in a state corresponding to each user who uses the game system 1. According to this, even when a plurality of users use one game system 1, each user can start using the game system 1 in a state suitable for him / herself. The “state corresponding to each user” may be a state determined based on a state at the end when the user used the game system 1 last time, or a state designated by the user. There may be. In addition, any method may be used for identifying the user. For example, the CPU 10 may identify the user by causing the user to input an ID and / or password at startup. For example, CPU10 may image a user's face with the camera 56 of the terminal device 7, and may identify a user using a captured image. Specifically, the user can be identified by registering in advance an image of the user's face and comparing the image captured by the camera 56 with the registered image.

  In steps S2 and S3, program guide and homepage data are acquired in advance. Here, these data may be acquired at any timing. In the present embodiment, these data are acquired when the game system 1 is activated. In other embodiments, these data may be acquired at the time of transition to a state in which the program guide or homepage is to be displayed (that is, the first television operation state or the browser state). As for homepage data, data acquired when the game system 1 was previously used may be stored using a cache function in a general browser.

  In step S <b> 2, the CPU 10 acquires a program guide from a predetermined server (external device 91) via the network 90. That is, the CPU 10 first generates a program guide acquisition request and transmits the acquisition request to the server via the network 90. This server may be any server that can provide program guide data, such as a web server that stores program guide data. In response to the acquisition request, the server transmits program guide data to the game apparatus 3 via the network 90. Program guide data transmitted from the server is stored in the flash memory 17 (or main memory). In another embodiment, instead of the data acquired from the server, program guide image data generated from the data may be stored as the program guide data 109. Following step S2, the process of step S3 is executed.

  In step S3, the CPU 10 acquires data of a web page (homepage) accessed at the start of the browser function. That is, the CPU 10 first generates an acquisition request for the web page and transmits the acquisition request to a predetermined server via the network 90. This server may be any information processing device that provides the preset web page. In response to the acquisition request, the server transmits web page data to the game apparatus 3 via the network 90. Web page data transmitted from the server is stored in the flash memory 17 (or main memory). In other embodiments, instead of the data acquired from the server, image data of a web page generated from the data may be stored as the home page data 110. After step S3, the CPU 10 ends the start process.

  After the start process is completed, a process corresponding to the initial state determined in step S1 is executed. That is, when the initial state is the menu state, menu processing described later is executed, when the initial state is the first television operation state, TV processing described later is executed, and when the initial state is the browser state. The browser process described later is executed. Hereinafter, details of the menu process, the television process, and the browser process will be described.

(Menu processing)
FIG. 16 is a flowchart showing the flow of menu processing executed by the game apparatus 3. First, in step S <b> 11, the CPU 10 acquires various data from the terminal device 7. Here, at least terminal operation data is acquired from the terminal device 7, and in this embodiment, camera image data and microphone sound data are further acquired. Since the terminal device 7 repeatedly transmits the above data (terminal operation data, camera image data, and microphone sound data) to the game device 3, the game device 3 sequentially receives the data. In the game apparatus 3, the terminal communication module 28 sequentially receives each data, and the camera image data and the microphone sound data are sequentially decompressed by the codec LSI 27. Then, the input / output processor 11a sequentially stores terminal operation data, camera image data, and microphone sound data in the main memory. The CPU 10 reads the data from the main memory at an appropriate timing. Following step S11, the process of step S12 is executed.

  In step S <b> 12, the CPU 10 displays an application selection image on the terminal device 7. That is, the CPU 10 generates an application selection image and outputs (transmits) the generated application selection image to the terminal device 7. The application selection image may be any image as long as the user can select an application executable by the game apparatus 3. Hereinafter, with reference to FIG. 17, an application selection image according to the present embodiment will be described.

  FIG. 17 is a diagram illustrating an example of an application selection image displayed on the terminal device 7. As illustrated in FIG. 17, the application selection image includes an application area 121, a notification area 123, a television icon 125, and a browser icon 126. The application selection image shown in FIG. 17 is a menu image for selecting an application to be executed on the game apparatus 3.

  In FIG. 17, an application area 121 is an area for displaying one or more application icons 122. The application icon 122 is an image (icon) representing an application executable by the game apparatus 3. In the present embodiment, an application represented by the application icon 122 is executed by an operation of selecting the application icon 122. In FIG. 17, the application area 121 can be scrolled left and right in accordance with a predetermined scroll operation, and the displayed application icon 122 is changed by scrolling left and right. The notification area 123 is an area for displaying notification items for the user. Any information may be displayed in the notification area 123. For example, introduction of a new application, notifications regarding various services using the game system 1, and the like are displayed. Note that the game apparatus 3 may update the content of the notification item by accessing a predetermined server via the network 90 and acquiring the notification item information at an appropriate timing.

  The television icon 125 is an image representing an instruction to transition to the first television operation state. Although details will be described later, in the present embodiment, the game system 1 transitions to the first television operation state by an operation of selecting the television icon 125 (the change operation described above). The browser icon 126 is an image representing an instruction to transition to the browser state. Although details will be described later, in the present embodiment, the game system 1 transitions to the browser state by an operation of selecting the browser icon 126 (the change operation described above). That is, in the menu state, by performing an operation of selecting the television icon 125 or the browser icon 126, the user can easily transition the game system 1 to the first television operation state or the browser state.

  In step S12, the CPU 10 and the GPU 11b read data necessary for generating an application selection image such as image data of various icons from the VRAM 11d, and generate an application selection image using the read data. The generated application selection image is stored in the VRAM 11d. Further, the CPU 10 outputs an application selection image to the terminal device 7. Specifically, the image data of the application selection image stored in the VRAM 11d is sent to the codec LSI 27 by the CPU 10, and a predetermined compression process is performed by the codec LSI 27. The compressed image data is transmitted to the terminal device 7 through the antenna 29 by the terminal communication module 28. The terminal device 7 receives the image data transmitted from the game device 3 by the wireless module 80, and a predetermined decompression process is performed on the received image data by the codec LSI 76. The image data that has undergone the decompression process is output to the LCD 51. As a result, the application selection image is displayed on the LCD 51. In step S <b> 12, sound data may be transmitted to the terminal device 7 together with the image data, and the sound may be output from the speaker 77 of the terminal device 7. Following step S12, the process of step S13 is executed.

  In step S13, the television image is displayed on the television 2. That is, the CPU 10 generates a predetermined television image and outputs the generated television image to the television 2. The television image displayed in the menu state may have any content. The television image may be, for example, an image related to an application that can be executed by the game apparatus 3 (for example, a title image or demo video of a game application) or an image displayed in the notification area 123. . In the menu state, it is assumed that the television 2 is used for purposes other than the game system 1 (for example, for displaying television programs), so the television image has the auxiliary content as described above. It is preferable.

  In step S13, the CPU 10 and the GPU 11b read data necessary for generating a television image from the VRAM 11d, and generate a television image. The generated television image is stored in the VRAM 11d. Further, the CPU 10 outputs a game image to the television 2. Specifically, the CPU 10 sends TV image data stored in the VRAM 11 d to the AV-IC 15. In response, the AV-IC 15 outputs television image data to the television 2 via the AV connector 16. As a result, the television image is displayed on the television 2. In step S13, sound data may be output to the television 2 together with image data, and sound may be output from the speaker 2a of the television 2. Following step S13, the process of step S14 is executed.

  In step S14, the CPU 10 determines whether or not there has been a selection instruction. The selection instruction is an instruction for selecting an application to be executed. The operation for performing the selection instruction may be any operation. In this embodiment, the selection instruction is performed by an operation of touching the application icon 122 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether any of the application icons 122 displayed on the terminal device 7 is touched based on the touch data. In other embodiments, for example, a selection instruction may be performed using a cursor displayed on the terminal device 7. That is, the CPU 10 moves the cursor according to a direction input operation on the terminal device 7 (for example, a direction input operation on the analog stick 53), and when a predetermined operation is performed in a state where the cursor points to the application icon 122, You may make it determine with the application icon 122 having been designated. If the determination result of step S14 is affirmative, the process of step S15 is executed. On the other hand, if the determination result of step S14 is negative, the process of step S16 is executed.

  In step S15, the CPU 10 executes the application selected by the selection instruction. That is, the CPU 10 reads the application from a storage unit (such as the optical disk 4 or the flash memory 17) in which the selected application is stored, and stores the application in the main memory. Then, execution of the application stored in the main memory is started. After step S15, the CPU 10 ends the menu process and executes an application execution process. As a result, the game system 1 transitions to the application execution state.

  Although not shown, in the application execution process, information processing according to the application is executed. The application to be executed may have any content, and the information processing by executing the application may be any process. The CPU 10 outputs an image generated by executing the application to at least one of the terminal device 7 and the television 2. Note that the determination of whether the generated image is displayed on the terminal device 7, the television 2, or the terminal device 7 and the television 2 may be performed in any manner. The above determination may be made by, for example, an application or a user. When the generated image is displayed only on the terminal device 7, the television program can be displayed on the television 2, and the user can use the application of the game system 1 while viewing the television program. .

  On the other hand, in step S16, the CPU 10 determines whether or not to transit to the first television operation state. That is, the CPU 10 determines whether or not a change operation for changing from the menu state to the first television operation state has been performed. This change operation may be any operation, and in the present embodiment, is an operation of touching the television icon 125 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the television icon 125 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S16 is affirmative, the CPU 10 ends the menu process and executes a TV process described later. On the other hand, when the determination result of step S16 is negative, the process of step S17 is executed.

  In this embodiment, various instructions (operations) in steps S14, S16, S17, S23, S25, S26, S33, S34, S43, S45, S47, S50, and S52 are performed using the touch panel 52. Is called. That is, the CPU 10 (and the GPU 11b) can generate an image representing various instructions (the selection instruction and the control instruction, the change operation instruction, etc.), and the image is output to the terminal device 7 for display. Then, when the image is selected (touched) on the touch panel 52, it is determined that the instruction has been given. According to this, the user can give various instructions by an easy and intuitive operation using the touch panel.

  In other embodiments, various instructions may be performed by other operations. For example, all or some of the instructions may be performed by an operation using a cursor displayed on the terminal device 7. That is, the predetermined instruction may be performed by moving the cursor according to a direction input operation on the terminal device 7 and performing a predetermined operation in a state where the cursor indicates an image (icon or the like) representing the predetermined instruction. . Further, for example, all or some of the instructions may be performed by an operation of pressing any of the buttons 54 </ b> A to 54 </ b> M of the terminal device 7. More specifically, the instruction to change the three states in the start state (the change operation) is performed by an operation of pressing a predetermined button (for example, the second L button 54K and the second R button 54L) of the terminal device 7. Also good. That is, the application selection image, the device control image, and the browser image may be sequentially switched in response to pressing a predetermined button on the terminal device 7. For example, all or some of the instructions may be performed by an operation of moving the terminal device 7. The movement of the terminal device 7 can be calculated (estimated) using the detection results of the inertial sensor (the acceleration sensor 73 and the gyro sensor 74) and the magnetic sensor 72.

  In step S17, the CPU 10 determines whether or not to transition to the browser state. Specifically, the CPU 10 determines whether or not there is a display instruction to display a web page, in other words, whether or not a change operation for changing from the menu state to the browser state is performed. This change operation may be any operation, and in this embodiment, is an operation of touching the browser icon 126 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the browser icon 126 displayed on the terminal device 7 is touched based on the touch data. If the determination result in step S17 is affirmative, the CPU 10 ends the menu process and executes a browser process described later. On the other hand, when the determination result of step S17 is negative, the process of step S11 is executed again. Thereafter, the menu process is repeatedly executed until the determination result in any one of steps S14, S16, and S17 becomes affirmative.

(TV processing)
FIG. 18 is a flowchart showing a flow of television processing executed by the game apparatus 3. The television processing is executed when the initial state is determined to be the first television operation state, or when a change operation for changing from the menu state or the browser state to the first television operation state is performed. First, in step S <b> 21, the CPU 10 acquires various data from the terminal device 7. The process of step S21 is the same as that of the above-mentioned step S11. Following step S21, the process of step S22 is executed.

  In step S <b> 22, the CPU 10 displays a device control image on the terminal device 7. That is, the CPU 10 generates a device control image and outputs (transmits) the generated device control image to the terminal device 7. The device control image may be any operation image that represents an operation on the television 2. In the present embodiment, an image of a television program guide acquired from the external device 91 via the network 90 is displayed as a device control image. Hereinafter, with reference to FIG. 19, an apparatus control image in the present embodiment will be described.

  FIG. 19 is a diagram illustrating an example of a device control image displayed on the terminal device 7. As shown in FIG. 19, the device control image includes a program guide area 130, various operation button images 136 to 138, a menu icon 124, and a browser icon 126. The device control image shown in FIG. 19 is a program guide image including an image representing a control instruction to the television 2.

  In FIG. 19, a program guide area 130 is an area for displaying a program guide generated based on data acquired from a predetermined server via the network 90. Here, the program guide area 130 includes a channel image 131, a program information area 132, a channel selection instruction button 133, and a scroll bar 134. The channel image 131 is an image representing a television channel and a program name. In FIG. 19, channel images 131 are arranged vertically, and the user can scroll and display the channel image 131 by moving the knob 135 of the scroll bar 134. The program information area 132 is an area where program information (program contents, performers, etc.) of a television program represented by a predetermined channel image 131 among the displayed channel images 131 is displayed. Here, the predetermined channel image 131 is the channel image 131 touched by the user.

  The channel selection instruction button 133 is an image representing a control instruction for the television 2, and specifically represents a control instruction for selecting a channel represented by the predetermined channel image. Note that the image indicating the control instruction is not limited to the channel selection instruction button 133, for example, a button indicating a control instruction for turning on or off the TV 2, a button indicating a control instruction for changing the volume of the TV 2, or a TV 2 may be included in the device control image.

  The date operation button 136 is a button for giving an instruction to change the date (and time) of the program guide to be displayed. The detail display button 137 is a button for instructing to display more detailed information than the content displayed in the program information area 132. The setting button 138 is a button for giving an instruction to display a setting screen related to the program guide and the terminal device 7. In addition to the above, for example, an image for searching a program guide, an image for displaying a favorite program list, and the like may be displayed on the terminal device 7. The user can browse the contents of the television program or operate the television 2 so as to select a desired channel using the image of the above program guide. In the present embodiment, since the TV program guide is displayed in the first TV operation state, the user can conveniently operate the channel of the TV 2 by viewing the program guide.

  The menu icon 124 is an image representing an instruction to transition to the menu state. Although details will be described later, in the present embodiment, the game system 1 transitions to the menu state by an operation of selecting the menu icon 124 (the change operation described above). The browser icon 126 is the same as that shown in FIG. Therefore, in the first television operation state, by performing an operation of selecting the menu icon 124 or the browser icon 126, the user can easily transition the game system 1 to the menu state or the browser state.

  In step S22, the CPU 10 and the GPU 11b read data necessary for generating a device control image from the main memory and the VRAM 11d, and generate a device control image using the read data. In the present embodiment, the image displayed in the program guide area 130 is generated using the program guide data 109 stored in the main memory. Therefore, in this embodiment, the device control image can be quickly displayed on the terminal device 7 even immediately after the transition to the first television operation state. The generated device control image is stored in the VRAM 11d. Further, the CPU 10 outputs a device control image to the terminal device 7. The process of displaying the image on the terminal device 7 by outputting the image to the terminal device 7 is the same as in step S12. As a result, the device control image is displayed on the LCD 51. In step S <b> 22, sound data may be transmitted to the terminal device 7 together with the image data, and the sound may be output from the speaker 77 of the terminal device 7. Following step S22, the process of step S23 is executed.

  In step S23, it is determined whether or not there is a control instruction. The control instruction is an instruction to perform control on the television 2. The operation for performing the control instruction may be any operation. In the present embodiment, the operation is performed by touching a button (tuning instruction button 133) representing the control instruction displayed on the terminal device 7. Instruction is given. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the button displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S23 is affirmative, the process of step S24 is executed. On the other hand, if the determination result of step S23 is negative, the process of step S24 is skipped and the process of step S25 is executed.

  In step S <b> 24, the CPU 10 performs control on the television 2 in accordance with the control instruction. Here, the method by which the game system 1 controls the television 2 may be any method. In the present embodiment, a television is obtained by a first method in which the game device 3 outputs a control signal to the television 2 via the terminal device 7 and a second method in which the game device 3 directly outputs a control signal to the television 2. 2 is controlled.

  In the first method, the CPU 10 instructs the terminal device 7 to transmit a control signal indicating the control content according to the control instruction. In accordance with this instruction, the terminal device 7 outputs a control signal for controlling the television 2. Specifically, the CPU 10 outputs control data representing the above instruction to the terminal device 7. In the present embodiment, data representing various control signals for causing the television 2 to perform various operations is stored in the flash memory 17 or the main memory. CPU10 selects the data showing the control signal according to the said control instruction from the said various control signals, and outputs it to the terminal device 7 as control data. The terminal device 7 that has received the control data outputs a control signal represented by the control data. In the present embodiment, an infrared signal is output as the control signal. That is, the codec LSI 76 of the terminal device 7 causes the infrared communication module 82 to output an infrared signal corresponding to the control signal represented by the control data. The infrared signal is received by the infrared light receiving unit of the television 2, so that the television 2 performs an operation according to the control instruction.

  In the second method, the CPU 10 outputs a control signal indicating the control content according to the control instruction to the television 2. That is, the CPU 10 selects a control signal (for example, a signal conforming to the HDMI standard) according to the control instruction from various control signals for causing the television 2 to perform various operations. Then, the selected control signal is output to the television 2 via the AV connector 16. Also by this, the television 2 performs an operation according to the control instruction.

  The format of the control signal (infrared signal) for controlling the television 2 may differ depending on the model of the television 2. Therefore, the game apparatus 3 may store in advance control signals of various types corresponding to a plurality of models. At this time, the game apparatus 3 may select a format corresponding to the television 2 at a predetermined timing (for example, at the time of initial setting), and thereafter, a control signal of the selected format may be used. As a result, the game apparatus 3 can support a plurality of types of televisions.

  In the present embodiment, the game apparatus 3 may control the television 2 using at least one of the above two methods, or may control the television 2 using both methods. After step S24, the process of step S25 is executed.

  In step S25, the CPU 10 determines whether or not to transit to the menu state. That is, the CPU 10 determines whether or not a change operation for changing from the first television operation state to the menu state has been performed. This change operation may be any operation, and in this embodiment, is an operation of touching the menu icon 124 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the menu icon 124 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S25 is affirmative, the CPU 10 ends the television process and executes the menu process described above. On the other hand, if the determination result of step S25 is negative, the process of step S26 is executed.

  In step S26, the CPU 10 determines whether or not to transition to the browser state. The process in step S26 is the same as the process in step S17 described above. When the determination result of step S26 is affirmative, the CPU 10 ends the television process and executes a browser process described later. On the other hand, if the determination result of step S26 is negative, the process of step S27 is executed.

  In step S27, the CPU 10 performs other television processing. The other television processing is processing executed in television processing other than the processing in steps S21 to S26. For example, the CPU 10 changes the display content of the program guide area 130 in accordance with the selection operation on the operation button images 136 to 138, or the process of scrolling and displaying the channel image 131 in accordance with the scroll operation on the scroll bar 134. Etc. In the present embodiment, assuming that the television 2 is used for displaying a television program in the first television operation state, the CPU 10 does not generate or output a television image in step S27. However, in other embodiments, television images may be generated and output. For example, the device control image representing the program guide may be output to the television 2 and displayed as a television image. According to this, the user can browse the program guide on a larger screen.

  Following step S27, the process of step S21 is executed again. Thereafter, the television processing is repeatedly executed until the determination result in any one of steps S25 and S26 becomes affirmative.

  According to the above television processing, the user can browse the program guide by the terminal device 7 (step S22) and can operate the television 2 (steps S23 and S24). In addition, the user can easily change from the first television operation state to the menu state or the browser state by a change operation (steps S25 and S26).

  In other embodiments, the input of the television 2 may be switched at the transition from the menu state or the browser state to the first television operation state (so as to display a television broadcast). That is, when the television 2 can switch between the first mode in which an image input from the game device 3 is displayed and the second mode in which an image of television broadcasting is displayed, the CPU 10 selects an application in the terminal device 7. Control of switching the mode of the television 2 to the second mode may be performed in response to switching of the display from the image (or browser image) to the device control image. Specifically, at the start of the television processing, the CPU 10 may control the input switching for the television 2 so as to be in the second mode for displaying the television broadcast. According to this, since the user does not need to perform the input switching operation of the television 2 at the time of transition to the first television operation state, the user's operation can be simplified. The input switching control may be performed by the first method or the second method in step S24.

  In another embodiment, the input of the television 2 may be switched (so as to display an image from the game apparatus 3) at the transition from the first television operation state to the menu state or the browser state. That is, when the television 2 can switch between the first mode for displaying an image input from the game apparatus 3 and the second mode for displaying an image of the television broadcast, the CPU 10 controls the terminal device 7 to perform device control. Control may be performed to switch the mode of the television 2 to the first mode in response to switching of the display from the image to the application selection image (or browser image). Specifically, if the determination result in step S16 is affirmative, the CPU 10 may control input switching for the television 2 so as to be in the first mode in which an image from the game apparatus 3 is displayed. Good. If the determination result in step S17 is affirmative, the CPU 10 may perform input switching control on the television 2 so as to be in the first mode. According to this, since it is not necessary for the user to perform input switching operation of the television 2 at the time of transition from the first television operation state to another state, the user operation can be simplified. The input switching control may be performed by the first method or the second method in step S24.

  Note that, depending on the model of the television 2, only an instruction for sequentially switching a plurality of types of inputs (for example, an instruction for sequentially switching five types of inputs from “input 1” to “input 5” one by one) is possible. In some cases, it is impossible to designate the second mode for displaying the television broadcast and to switch the input. In that case, the number of input switching instructions required for switching between the first mode and the second mode is registered in advance, and the CPU 10 instructs the input switching for the registered number of times. It may be. For example, “input 1” is to display a television broadcast, “input 3” is to display an image from the game apparatus 3, and an input switching instruction is used to switch from “input 1” to “input 3”. Is necessary twice, the game apparatus 3 stores “2” as the input switching instruction count. When the game system 1 is initially set, the game apparatus 3 can calculate the above number of times by causing the user to register an input number corresponding to the game apparatus 3.

(Browser processing)
FIG. 20 is a flowchart showing the flow of browser processing executed by the game apparatus 3. First, in step S21, the homepage image is displayed on the terminal device 7 as a browser image. That is, the CPU 10 generates a browser image and outputs (transmits) the generated browser image to the terminal device 7. The browser image may be any image as long as the image includes a web page acquired through the network 90. Hereinafter, the browser image in the present embodiment will be described with reference to FIG.

  FIG. 21 is a diagram illustrating an example of a browser image displayed on the terminal device 7. As shown in FIG. 21, the browser image includes a web page display area 141, a scroll bar 142, a back button 143, a forward button 144, a keyword area 145, a search button 146, and a remote control display button 147. Further, the browser image includes the menu icon 124 and the television icon 125 described above. The browser image shown in FIG. 21 is an operation image for displaying a web page and performing an operation on the web page.

  In FIG. 21, a web page display area 141 is an area in which an image of a web page generated based on data acquired from the network 90 is displayed. The web page displayed in the web page display area 141 can be scrolled up and down by moving the knob of the scroll bar 142 up and down. The return button 143 is an image representing an instruction to display the web page displayed immediately before the currently displayed web page. The forward button 144 is an image that represents an instruction to display the web page displayed after the web page currently displayed. The keyword area 145 is an area where the input search keyword is displayed. The search button 146 is an image representing an instruction to search a web page using a search keyword. The buttons 143 to 146 may have the same function as buttons in a conventional general browser.

  The remote control display button 147 is an image representing an instruction to display a remote control image (FIG. 24). Details of the remote control image will be described in the interruption processing described later. The menu icon 124 and the television icon 125 are the same as those shown in FIGS. Therefore, in the browser state, by performing an operation of selecting the menu icon 124 or the television icon 125, the user can easily transition the game system 1 to the menu state or the first television operation state.

  In step S31, the CPU 10 and the GPU 11b read data necessary for generating a browser image from the main memory and the VRAM 11d, and generate a browser image using the read data. In step S1 in the present embodiment, the image displayed in the web page display area 141 is generated using the home page data 110 stored in the main memory. Therefore, in this embodiment, the browser image can be quickly displayed on the terminal device 7 even immediately after the transition to the browser state. The generated browser image is stored in the VRAM 11d. Further, the CPU 10 outputs a browser image to the terminal device 7. The process of displaying the image on the terminal device 7 by outputting the image to the terminal device 7 is the same as in step S12. As a result, the browser image is displayed on the LCD 51. The CPU 10 may output at least a partial area of the browser image to the terminal device 7. For example, the CPU 10 may enlarge a part of the browser image to be displayed on the terminal device 7. According to this, it is possible to display the browser image in an easy-to-view manner even in the terminal device 7 having a screen smaller than that of the television 2 in general. In step S31, audio data may be transmitted to the terminal device 7 together with the image data, and the audio may be output from the speaker 77 of the terminal device 7. Following step S31, the process of step S32 is executed.

  In step S <b> 32, the CPU 10 acquires various data from the terminal device 7. The process in step S32 is the same as that in step S11 described above. Following step S32, the process of step S33 is executed.

  In step S33, the CPU 10 determines whether or not to transit to the menu state. The process in step S33 is the same as the process in step S25 described above. If the determination result of step S33 is affirmative, the CPU 10 ends the browser process and executes the menu process. On the other hand, when the determination result of step S33 is negative, the process of step S34 is executed.

  In step S34, the CPU 10 determines whether or not to transit to the first television operation state. The process in step S34 is the same as the process in step S16 described above. If the determination result of step S34 is affirmative, the CPU 10 ends the browser process and executes the television process. On the other hand, if the determination result of step S34 is negative, the process of step S35 is executed.

  In step S35, the CPU 10 executes other browser processing. The other browser process is a process executed in the browser process in addition to the processes in steps S31 to S34. For example, the CPU 10 executes a process corresponding to the button according to the selection operation for the buttons 143 to 146, or executes a process for scrolling and displaying the web page according to the scroll operation for the scroll bar 142. . In addition, the CPU 10 obtains a linked web page when a selection operation is performed on the link information included in the web page in step S35, a process of reproducing the moving image when the moving image is included in the web page, or the like. Execute. In step S35, various processes executed in a general browser may be executed.

  Further, the CPU 10 may execute a process of generating a television image and outputting it to the television 2. For example, the CPU 10 may output all or part of the image of the web page to the television 2. Specifically, when a moving image is included in the web page displayed on the terminal device 7, the CPU 10 may sequentially output images constituting the moving image to the television 2. According to this, the user can view the moving image of the web page on the larger screen of the television 2.

  In the present embodiment, the browser image is displayed at least on the terminal device 7, but the browser image may be displayed on at least one of the terminal device 7 and the television 2. When the browser image is displayed on the terminal device 7, there is an advantage that the television 2 can be used for other purposes. When the browser image is displayed on the television 2, the browser image can be displayed on a large screen in an easy-to-view manner. For example, a moving image included in a web page can be displayed on the television 2. In another embodiment, the user may be able to select a display device that is a target for outputting a browser image.

  Following step S35, the process of step S32 is executed again. Thereafter, browser processing is repeatedly executed until the determination result in any one of steps S33 and S34 is affirmative.

  According to the processing in the start state described above (menu processing, television processing, and browser processing), the game apparatus 3 can generate an application selection image, a device control image, and a browser image as the main operation image. is there. Then, the application selection image, the device control image, and the browser image are switched according to a predetermined change operation and output to the terminal device 7. Therefore, the user can easily switch between a screen for selecting an application, a screen for operating a television while watching a program guide, and a screen for browsing a web page by a change operation, and use the game system 1 for various purposes. Can do. In particular, in the present embodiment, in any of the three types of states in the start state, two icons for transitioning to the other two states are always displayed, so the user selects an icon. The game system 1 can be quickly transitioned to another desired state by one operation. In addition, according to the present embodiment, when a user wants to watch a TV program from the state of selecting an application or browsing a web page using the game system 1, the user performs a change operation to display the device control image on the terminal. It is only necessary to display the information on the device 7 and operate the control instruction, and the switching operation can be easily performed using the terminal device 7.

(Processing when interrupted)
Next, details of the interruption process will be described. FIG. 22 is a flowchart showing the flow of the application interruption process executed by the game apparatus 3. In the above-described application execution state, the application execution process is executed with the processing content corresponding to the application. Here, when a predetermined interruption operation is performed in the application execution state, the CPU 10 stops (temporarily stops) the application execution process and executes the application interruption process illustrated in FIG. The predetermined interruption operation may be any operation on the terminal device 7, but in the present embodiment, is an operation of pressing the home button 54C. In addition, it is preferable not to use the home button 54C for the operation in the application, and according to this, the user can transition to the suspended state at any timing during the execution of the application.

  In the present embodiment, the execution of the application itself is suspended (suspended) in the suspended state. However, in other embodiments, execution of the application may be continued. The CPU 10 may execute information processing in the application within a range that does not affect the use of the user, for example, by executing calculation of internal processing of the application.

  In the application interruption process, first, in step S41, the CPU 10 displays a setting image on the terminal device 7. That is, the CPU 10 generates a setting image and outputs (transmits) the generated setting image to the terminal device 7. Therefore, in the present embodiment, the game system 1 first transitions to the set state when transitioning to the suspended state. In other embodiments, the game system 1 may first transition to the second television operation state when transitioning to the interrupted state. Further, the setting image may be any operation image that represents a setting operation for the game system 1 (terminal device 7). Hereinafter, the setting image in the present embodiment will be described with reference to FIG.

  FIG. 23 is a diagram illustrating an example of a setting image displayed on the terminal device 7. As shown in FIG. 23, the setting image includes an application end button 151, an application stop button 152, an application restart button 153, a remote control display button 154, a terminal setting button 155, a controller setting button 156, a list button 157, and an instruction button 158. Including.

  The application end button 151 is an image representing an instruction to end an application whose execution has been interrupted. The application stop button 152 is an image representing an instruction to temporarily stop the application. The application restart button 153 is an instruction to restart the application. With these buttons 151 to 153, it is possible to end or temporarily stop the application from the suspended state and transition to the menu state, or to resume the application from the suspended state.

  The remote control display button 154 is an image representing an instruction to display a remote control image (FIG. 24) on the terminal device 7. Although details will be described later, the remote control image is an operation image for performing an operation on the television 2, and is an operation image including a button image representing an operation on the television 2.

  The terminal setting button 155 is an image representing an instruction to display a screen for performing various settings for the terminal device 7. The controller setting button 156 is an image representing an instruction to display a screen for performing various settings for the controller 5. The list button 157 is an image representing an instruction to display a list of other users (acquaintances and friends) registered in advance. For example, the user can play a game or communicate with another user selected from the list. The instruction button 158 is an image representing an instruction to display an instruction of the game system 1 (or the terminal device 7). With these buttons 155 to 158, the user can change various settings related to the game system 1.

  In step S41, the CPU 10 and the GPU 11b read from the VRAM 11d data necessary for generating a setting image, such as the images of the buttons 151 to 158, and generate a setting image using the read data. The generated setting image is stored in the VRAM 11d. Further, the CPU 10 outputs the setting image to the terminal device 7. The process of displaying the image on the terminal device 7 by outputting the image to the terminal device 7 is the same as in step S12. Thereby, the setting image is displayed on the LCD 51. Further, in step S41, audio data may be transmitted to the terminal device 7 together with the image data, and the audio may be output from the speaker 77 of the terminal device 7. Following step S41, the process of step S42 is executed.

  In step S <b> 42, the CPU 10 acquires various data from the terminal device 7. The process of step S42 is the same as that of step S11 described above. Following step S42, the process of step S43 is executed.

  In step S43, the CPU 10 determines whether or not to display a remote control image. That is, the CPU 10 determines whether or not an operation for changing from the setting state to the second television operation state has been performed. This operation may be any operation, and in this embodiment, is an operation of touching the remote control display button 154 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the remote control display button 154 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S43 is affirmative, the process of step S44 is executed. On the other hand, if the determination result of step S43 is negative, the process of step S44 is skipped and the process of step S45 is executed.

  In step S <b> 44, the CPU 10 displays a remote control image on the terminal device 7. That is, the CPU 10 generates a remote control image and outputs (transmits) the generated remote control image to the terminal device 7. The remote control image may be any operation image that represents an operation on the television 2. Hereinafter, the remote control image in the present embodiment will be described with reference to FIG.

  FIG. 24 is a diagram illustrating an example of a remote control image displayed on the terminal device 7. As shown in FIG. 24, the remote control image includes an application end button 151 and an application stop button 152 similar to the setting image. Therefore, in the present embodiment, in the second television operation state, as in the setting state, it is possible to end or pause the application and transition to the menu state. In another embodiment, for example, the CPU 10 causes the terminal device 7 to display an application restart button 153 similar to the setting image, thereby restarting the application from the second television operation state and transitioning to the application execution state. May be possible.

  The return button 167 is an image representing an instruction to change (return) from the second television operation state to the setting state. That is, in the present embodiment, the user can switch between the setting image and the remote control image by an operation of selecting the remote control display button 154 or the return button 167.

  The remote control image also includes a power button 161, an input switching button 162, a volume increase button 163, a volume decrease button 164, a channel increase button 165, and a channel decrease button 166. The buttons 161 to 166 are images representing control instructions for the television 2. The power button 161 is an image representing an instruction to switch on / off the power of the television 2. The input switching button 162 represents an instruction to switch input on the television 2 (that is, switching between a mode for inputting and displaying a video of a television program and a mode for inputting and displaying an image from the game apparatus 3). It is an image. The volume increase button 163 is an image representing an instruction to increase the volume of the television 2. The volume decrease button 164 is an image representing an instruction to increase the volume of the television 2. The channel increase button 165 is an image representing an instruction to change the channel selection of the television 2 one by one in ascending order. The channel decrease button 166 is an image representing an instruction to change the channel selection of the television 2 one by one in descending order. The buttons 161 to 166 allow the user to operate the power source, input switching, volume, and channel selection of the television 2. In other embodiments, in addition to the above buttons, for example, a button representing each channel, a button representing an instruction to display a program guide acquired from a television broadcast, or (when the television 2 has a recording function) ) A button of a general television remote controller, such as a button indicating a recording instruction, may be displayed on the terminal device 7.

  As described above, in the present embodiment, the remote control image is an image including images representing various control instructions. In another embodiment, the remote control image may be an image representing assignment of the content of the control instruction (for example, increase in volume) and a button for performing the control instruction (for example, the second R button 54L). Good. In addition, the remote control image may include a program guide as with the device control image.

  In step S44, the CPU 10 and the GPU 11b read from the VRAM 11d data necessary for generating a remote control image, such as the images of the buttons 151 to 153 and 161 to 166, and generate a remote control image using the read data. . The generated remote control image is stored in the VRAM 11d. Further, the CPU 10 outputs a remote control image to the terminal device 7. The process of displaying the image on the terminal device 7 by outputting the image to the terminal device 7 is the same as in step S12. As a result, the remote control image is displayed on the LCD 51. By the process of step S44, the game system 1 transits from the set state to the second television operation state. In step S44, audio data may be transmitted to the terminal device 7 together with the image data, and the audio may be output from the speaker 77 of the terminal device 7. Following step S44, the process of step S45 is executed.

  In other embodiments, the input of the television 2 may be switched (to display a television broadcast) at the time of transition to the second television operation state. That is, when the television 2 can switch between the first mode for displaying an image input from the game apparatus 3 and the second mode for displaying an image of the television broadcast, the CPU 10 performs a sub-operation on the terminal device 7. Control may be performed to switch the mode of the television 2 to the second mode in response to the display of an image (more specifically, a remote control image). Specifically, in step S44, the CPU 10 may perform input switching control on the television 2 so as to be in the second mode for displaying the television broadcast. According to this, since it is not necessary for the user to perform an input switching operation of the television 2 at the time of transition to the second television operation state, the user's operation can be further simplified. The input switching control may be performed by the first method or the second method in step S24.

  In step S45, the CPU 10 determines whether or not there is a control instruction. The control instruction is an instruction to perform control on the television 2. The operation for performing the control instruction may be any operation. In the present embodiment, the control instruction is performed by touching the buttons 161 to 166 representing the control instruction displayed on the terminal device 7 as in step S23. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether any of the buttons 161 to 166 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S45 is affirmative, the process of step S46 is executed. On the other hand, if the determination result of step S45 is negative, the process of step S46 is skipped and the process of step S47 is executed.

  In step S46, the CPU 10 performs control on the television 2 in accordance with the control instruction. The processing in step S46 is the same as that in step S24 described above. That is, the CPU 10 uses a first method in which the game apparatus 3 outputs a control signal to the television 2 via the terminal device 7 and / or a second method in which the game apparatus 3 directly outputs a control signal to the television 2. The television 2 is controlled. Following step S46, the process of step S47 is executed.

  In step S47, the CPU 10 determines whether or not to display a setting image. That is, the CPU 10 determines whether or not an operation for making a transition from the second television operation state to the setting state has been performed. This operation may be any operation, and in this embodiment, is an operation of touching the return button 167 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the return button 167 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S47 is affirmative, the process of step S48 is executed. On the other hand, if the determination result of step S47 is negative, the process of step S48 is skipped and the process of step S49 is executed.

  In step S <b> 48, the CPU 10 displays the setting image on the terminal device 7. The process in step S48 is the same as the process in step S41. By the process of step S48, the game system 1 changes from the second television operation state to the setting state. Following step S48, the process of step S49 is executed.

  In step S49, the CPU 10 executes various setting processes. That is, the CPU 10 determines whether or not the above-described terminal setting button 155, controller setting button 156, list button 157, and instruction button 158 are selected (touched). If any button is selected, processing corresponding to the selected button is executed. The setting process in step S49 may be the same as the setting process executed in a conventional general game system. Following step S49, the process of step S50 is executed.

  In step S50, the CPU 10 determines whether or not to transit to the menu state. That is, it is determined whether or not to terminate or suspend an application whose execution has been suspended. In this determination, it is determined whether or not an operation for ending or temporarily stopping the application (an operation for changing from the interrupted state to the menu state) has been performed. This operation may be any operation, and in this embodiment, is an operation of touching the application end button 151 or the application stop button 152 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the application end button 151 or the application stop button 152 displayed on the terminal device 7 is touched based on the touch data. If the determination result of step S50 is affirmative, the process of step S51 is executed. On the other hand, if the determination result of step S50 is negative, the process of step S52 is executed.

  In step S51, the CPU 10 performs an application termination process. Specifically, the data to be saved among the data used in the application is stored in the main memory as save data 111. Note that the save data 111 is stored in the flash memory 17 or a storage medium in which the application is stored at an appropriate timing. If the application end button 151 is selected in step S50, the application is started from a predetermined initial state when the application is started next time. On the other hand, when the application cancel button 152 is selected in step S50, when the application is started (restarted) next time, the application is started from the state at the time of interruption. Therefore, the CPU 10 may make the contents stored as the save data 111 different when the application end button 151 is selected and when the application stop button 152 is selected. After step S51, the CPU 10 ends the interruption process. Thereafter, the game system 1 transitions from the interrupted state to the menu state. That is, the CPU 10 executes menu processing (FIG. 16).

  As described above, in the present embodiment, when a predetermined operation (interruption operation) is performed on the terminal device 7 during execution of the application, the CPU 10 terminates an instruction that terminates the application being executed. A sub operation image including an image (application end button 151) is generated (step S41). Then, when an instruction to end the application is given by specifying the end instruction image, the execution of the application is ended (step S50). Further, when the execution of the application is terminated, the CPU 10 generates a main operation image. Thus, according to the present embodiment, when the running application is terminated, the main operation image (more specifically, the application selection image) is displayed on the terminal device 7. Therefore, the user can easily select an application to be executed next after ending the application.

  In the present embodiment, when the application is terminated in the application interruption process, the game system 1 transitions to the menu state. In other embodiments, in the above case, the game system 1 may transition to the first television operation state or the browser state. For example, the game system 1 may be able to transition to the first television operation state or the browser state in response to a predetermined operation being performed in the interrupted state. Specifically, the CPU 10 may display the television icon 125 and / or the browser icon 126 in the setting image and the remote control image. When an operation for selecting the TV icon 125 is performed, the CPU 10 ends the application and displays a device control image on the terminal device 7. Thereby, the game system 1 transits to the first television operation state. When an operation for selecting the browser icon 126 is performed, the CPU 10 ends the application and displays a browser image on the terminal device 7. Thereby, the game system 1 transits to the browser state. According to the above, the user can change the game system 1 from the suspended state to a desired state among the three states of the start state. Further, for example, when the application is terminated, the CPU 10 may determine a state to be transitioned from among the three states according to a predetermined condition. Specifically, the CPU 10 may set the state to be transitioned to the initial state.

  In step S52, the CPU 10 determines whether or not to resume the application whose execution has been interrupted. That is, the CPU 10 determines whether or not an operation for resuming the application (an operation for making a transition from the suspended state to the application execution state) has been performed. This operation may be any operation, and in the present embodiment, is an operation of touching the application restart button 153 displayed on the terminal device 7. That is, the CPU 10 reads the touch data of the terminal operation data 104 from the main memory, and determines whether or not the application resume button 153 displayed on the terminal device 7 is touched based on the touch data. When the determination result of step S52 is affirmative, the CPU 10 ends the interruption process and executes an application execution process. Thereby, the execution of the application is resumed, and the game system 1 transitions from the suspended state to the application executing state. On the other hand, when the determination result of step S52 is negative, the process of step S42 is executed again. Thereafter, the interruption processing is repeatedly executed until the determination result in any one of steps S50 and S52 becomes affirmative.

  In another embodiment, the input of the television 2 may be switched (so as to display an image from the game apparatus 3) at the time of transition from the second television operation state to the application execution state. That is, when the television 2 can switch between the first mode for displaying an image input from the game device 3 and the second mode for displaying a television broadcast image, the CPU 10 is displayed on the terminal device 7. Control may be performed to switch the mode of the television 2 to the first mode in response to switching from the remote control image to the image generated by executing the application. Specifically, if the determination result in step S52 is affirmative, the CPU 10 may control input switching for the television 2 so as to be in the first mode in which an image from the game apparatus 3 is displayed. Good. According to this, since it is not necessary for the user to perform an input switching operation of the television 2 at the time of transition from the second television operation state to the application execution state, the user operation can be simplified. In another embodiment, the CPU 10 performs the above-described control at the transition from the second television operation state to the setting image state, the transition to the menu state, and the transition to the browser state. Also good. The input switching control may be performed by the first method or the second method in step S24.

  As described above, in this embodiment, when a predetermined operation (interruption operation) is performed on the terminal device 7 during execution of the application, the CPU 10 temporarily stops the execution of the application. In addition, in the state where the sub operation image is displayed on the terminal device 7 (suspended state), the CPU 10 uses the terminal operation data to determine whether or not there has been a resumption instruction (operation for selecting the application resumption button 153) to resume execution of the application. Judgment based on. Then, when there is a restart instruction, the CPU 10 restarts the execution of the application. Thus, in the present embodiment, when an application being executed is interrupted, the user can easily restart the application by a restart instruction.

  As described above, in the present embodiment, the CPU 10 generates a predetermined sub-operation image when a predetermined operation (interruption operation) is performed on the terminal device 7 during execution of the application. And in the state (interruption state) in which a sub operation image is displayed on the display part of the terminal device 7, it is determined based on terminal operation data whether there was a control instruction (step S45). Therefore, in the present embodiment, when using the application of the game system 1 using the terminal device 7, the user can operate the television 2 by interrupting the use of the application. According to this, since the user can perform an operation on the television 2 without ending the execution of the application, the state in which the television 2 is used in the game system 1 and the state in which the television 2 for displaying a program is used. Switching operation is easier. For example, the user uses the terminal device 7 to suspend the use of the application and switch the input of the television 2 to view the television program, or to suspend the viewing of the television program and use the application. Can be done easily.

  In the present embodiment, the remote control image is displayed even in the browser state described above. That is, in the processing loop of steps S32 to S35 in the browser processing (FIG. 20), the CPU 10 selects (touches) the remote control display button 147 whether or not an operation for displaying a remote control image has been performed. ) Determine whether or not an operation has been performed. If the determination result is affirmative, the same processing as step S44 is executed. As a result, a remote control image is displayed on the terminal device 7. Note that the application end button 151 and the application stop button 152 are not included in the remote control image displayed in the browser state. When the return button 167 is selected, the CPU 10 displays a browser image on the terminal device 7. As described above, in the present embodiment, the user can display the remote control image even in the browser state, and can operate the television 2.

  In another embodiment, the transition from the three types of states in the start state to a state in which the sub operation image (setting image and remote control image) is displayed may be possible. That is, when a predetermined operation is performed in the above three types of states, the CPU 10 may execute a process similar to the above interruption process. At this time, the application end button 151 and the application stop button 152 are not included in the setting image and the remote control image. Further, instead of the application restart button 153, an image of a button representing an instruction to transition to the original state is displayed. According to this, since the sub operation image can be easily displayed even in the start state, the user can more easily perform operations on the television 2 in the start state.

  In another embodiment, a remote control image is displayed on the terminal device 7 in the interrupted state, and a program guide image may be displayed on the television 2. That is, in a state where the sub operation image is displayed on the terminal device 7, the CPU 10 may generate an image including a television program guide and output the image to the television 2. The program guide image may be acquired from the external device 91 via the network 90 in the same manner as the television processing. Moreover, the timing at which the program guide is displayed on the television 2 may be any timing. For example, the timing at which the program guide is displayed on the television 2 may be the time when the interruption process starts, or the time when the remote control image is displayed on the terminal device 7, or may be predetermined by the user. May be the point in time when the display instruction is issued.

(Operation at the end of the game system 1)
In the present embodiment, the user may end the game system 1 at an arbitrary timing in the above-described information processing (start processing, menu processing, television processing, browser processing, application execution processing, and interruption processing). Is possible. That is, when a predetermined end operation (such as pressing the power button 54M of the terminal device 7 or the power button 24 of the game apparatus 3) is performed, the CPU 10 executes a predetermined end process, and the game apparatus 3 and the terminal apparatus 7 The power is turned off. Note that the termination process includes a process of storing data representing the state of the game system 1 (the menu state and the like) at that time in the flash memory 17. With this data, the initial state can be determined at the next startup.

  According to the information processing described above, the user can use the game system 1 for various purposes such as viewing a program guide, browsing web pages, and using various applications. In addition, by using the screen of the terminal device 7 as an operation screen, the TV 2 can be used for other purposes such as watching TV programs while the game system 1 is used. Can be used in parallel. Thus, the user can smoothly switch between using the game system 1 and using the television 2 (for purposes other than the game system 1). For example, the user can use the game system 1 while watching a TV program, or can watch a TV program while using the game system 1 more smoothly. it can. Furthermore, since operations on both the game system 1 and the television 2 can be performed by the terminal device 7, between the state using the game system 1 and the state using the television (for purposes other than the game system 1). Switching operation can be easily performed.

[7. Modified example]
The above-described embodiment is an example for carrying out the present invention. In other embodiments, the present invention can be implemented with, for example, the configuration described below.

(Modification regarding start state)
In the above embodiment, the start state includes three types of states, and the images displayed on the terminal device 7 are different in each state. Here, in other embodiments, any number of states may be included in the start state. Further, in each state included in the start state, the images displayed on the terminal device 7 may be different or the same. For example, in each of the above states, the image displayed on the terminal device 7 may be the same, and possible operations in each state may be different.

(Modification regarding a predetermined display device used in the game system 1)
In the above embodiment, the television 2 that can receive a television broadcast and display a program has been described as an example of the predetermined display device used in the game system 1. Here, the predetermined display device is not limited to the television 2 and may be any device as long as it has a plurality of inputs and can display an image. For example, the predetermined display device may be a display device used as a monitor of a personal computer (different from the game device 3) or a display device used as a monitor of a recording / playback device.

(Modification regarding the configuration of the game system 1)
In the above embodiment, the game system 1 is configured to include the game device 3, the terminal device 7 including the display unit (LCD 51), the controller 5, and the television 2. Here, the game system includes a game device and an operation device, and may be configured to display an image on the display unit of the operation device and a predetermined display device. For example, the game system may be provided in a form that does not include the controller 5 or a predetermined display device (TV 2).

  Moreover, in the said embodiment, although demonstrated as an example the game system containing the game device which can run a game program, this invention is applied to the arbitrary information processing systems and information processing apparatuses which execute arbitrary applications. Can do. That is, in the above embodiment, the game application is included in one or more applications that can be executed by the game device, but the game application may not be included.

(Modification regarding information processing apparatus for executing game processing)
In the above embodiment, the game apparatus 3 executes a series of information processing executed in the game system 1, but part of the information processing may be executed by another apparatus. For example, in another embodiment, the terminal device 7 may execute part of information processing (for example, terminal image generation processing or application execution processing). In another embodiment, in an input system having a plurality of information processing apparatuses that can communicate with each other, the plurality of information processing apparatuses may share and execute information processing. That is, the game system includes a game device and an operation device, and (a) an acquisition unit that acquires operation data representing an operation on the operation device; A first operation image generation unit that generates a predetermined first operation image to be displayed on the unit; (c) an operation image output unit that outputs the first operation image to the operation device; and (d) a game in the start state. Whether there has been a selection instruction for selecting one of one or more applications that can be executed by the game device including the application, and whether there has been a control instruction for controlling a predetermined display device separate from the operation device An instruction determination unit that determines whether or not based on the operation data; and (e) when there is a selection instruction, the selected application is executed, and the execution of the application An application execution unit that outputs the generated image to at least one of the operation device and a predetermined display device; and (f) a device that performs control according to the control instruction to the predetermined display device when there is a control instruction. And a control unit. At this time, the structure provided with each part of said (a)-(f) may be sufficient as one apparatus, and the structure provided with each part of said (a)-(f) may be sufficient.

  In addition, when said information processing is performed in several information processing apparatus, it is necessary to synchronize the process performed by each information processing apparatus, and information processing will become complicated. On the other hand, as in the above embodiment, when information processing is executed by one game device 3 and the terminal device 7 receives and displays an image (that is, the terminal device 7 is a thin client terminal). In the case of functioning), it is not necessary to synchronize information processing among a plurality of information processing apparatuses, and information processing can be simplified.

  As noted above, the various systems, methods, and techniques described herein may be provided by digital electronic circuitry, computer hardware, firmware, software, or a combination of these elements. An apparatus for implementing the above technique includes a computer program product, an input / output device, and a computer processor that are embodied in a computer-readable non-transitory storage device for execution by a programmable processor. May be included. The process for realizing the above technique may be executed by a programmable processor that executes a program for executing a required function by processing input data and generating a desired output. The above technique is realized by one or more computer programs that can be executed on a computer system including a programmable processor that exchanges data and instructions with hardware resources such as an input device, an output device, and an information storage device. May be. Each computer program may be realized by a procedural or object-oriented high-level programming language, assembly language, or machine language, and may be compiled or interpreted as necessary. The above processor may be a general-purpose or dedicated microprocessor. The processor typically receives data and instructions from ROM or RAM. The storage device includes (a) a nonvolatile memory including a semiconductor memory device such as an EPROM, an EEPROM, or a flash memory, (b) a magnetic disk such as an internal hard disk or a removable external disk, and (c). It is not limited to a magneto-optical disk (d) CDROM, but includes all kinds of computer memory. The processor and the storage device described above may be supplemented by an ASIC (Application Specific Integrated Circuit) or may be implemented in a form incorporated in the ASIC.

  Further, the processing system (circuit) described in this specification is programmed for control processing such as game processing according to the contents described in this specification. A processing system including at least one CPU that executes an instruction according to the above contents may act as a “programmed logic circuit” for executing a processing operation defined by the above contents. .

  As described above, the present invention can be used as a game system, a game device, or an information processing program for the purpose of easily performing operations on a display device such as a television during execution of an application in the game system. It is.

DESCRIPTION OF SYMBOLS 1 Game system 2 Television 3 Game device 4 Optical disk 7 Terminal device 10 CPU
11e Internal main memory 12 External main memory 51 LCD
52 Touch panel 54 Operation button 122 Application icon 124 Menu icon 125 TV icon 126 Browser icon 133 Channel selection instruction button 151 Application end button 153 Application restart button 154 Remote control display button 161 Power button 162 Input switching button 163 Volume increase button 164 Volume decrease button 165 Channel increase button 166 Channel decrease button

Claims (26)

A game system including a game device and an operation device,
The operating device is:
An operation unit for outputting operation data representing an operation on the operation device;
A receiving unit for receiving an image output from the game device;
A display unit for displaying the received image,
The game device includes:
An acquisition unit for acquiring the operation data;
Run the application, the image generated by the execution, the operation device and the operating device and an application execution section to be output to at least one of the predetermined display device separate,
The game system includes:
Based on the operation data, it is determined whether or not a predetermined operation has been performed on the operating device during execution of the application . If the predetermined operation is performed , a predetermined operation image is generated. An operation image generation unit for outputting to the display unit of the operation device;
An instruction determination unit that determines, based on the operation data, whether or not there has been a control instruction to control the predetermined display device in a state in which the operation image is displayed on the display unit;
A game system comprising: a device control unit that performs control according to the control instruction to the predetermined display device when the control instruction is received. The instruction determination unit determines whether or not there is an end instruction to end an application being executed in a state where the operation image is displayed on the display unit, and based on the operation data,
The game system according to claim 1, wherein the application execution unit ends the execution of the application when the end instruction is given. When a predetermined operation is performed on the operating device during execution of the application, the application execution unit temporarily stops execution of the application,
The instruction determination unit determines whether or not there is a restart instruction to resume execution of the application in a state where the operation image is displayed on the display unit,
The game system according to claim 1, wherein the application execution unit resumes execution of the application when the restart instruction is issued. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The operation image generation unit generates an image including a TV program guide and outputs the generated image to the predetermined display device when the operation image is displayed on the display unit. The game system according to item 1. The operation image generation unit generates an image including a control instruction image representing the control instruction as the operation image,
The game system according to any one of claims 1 to 4, wherein the instruction determination unit determines that the control instruction has been given when the control instruction image is selected. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The game system according to any one of claims 1 to 4, wherein the operation image generation unit generates an image including a television program guide as the operation image and outputs the image to the operation device. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The predetermined display device can switch between a first mode for displaying an image input from the game device and a second mode for displaying an image of the television broadcast,
The device control unit according to any one of claims 1 to 6, wherein the device control unit performs control to switch a mode of the predetermined display device to the second mode in response to the operation image being displayed on the operation device. The game system according to item 1. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The predetermined display device can switch between a first mode for displaying an image input from the game device and a second mode for displaying an image of the television broadcast,
The device control unit changes the mode of the predetermined display device to the first mode in response to switching of an image displayed on the operation device from the operation image to an image generated by execution of the application. The game system according to claim 1, wherein switching control is performed. The operation device further includes a control signal output unit that outputs a control signal for controlling the predetermined display device,
The device control unit, a control signal indicating the control contents corresponding to the control instruction Ru is transmitted to the control signal output section, the game system according to any one of claims 1 to 8. The control signal output unit outputs an infrared signal as the control signal,
The game system according to claim 9, wherein the predetermined display device includes an infrared light receiving unit capable of receiving an infrared signal.   The game system according to any one of claims 1 to 8, wherein the device control unit outputs a control signal indicating a control content according to the control instruction to the predetermined display device. The operating device further includes a touch panel provided on the screen of the display unit,
The operation image generation unit generates an image including a control instruction image representing the control instruction as the operation image,
The game system according to any one of claims 1 to 11, wherein the instruction determination unit determines that the control instruction has been issued when the control instruction image is selected on the touch panel. An information processing method executed in a game system including a game device and an operation device,
An acquisition step of acquiring operation data representing an operation on the operation device;
An application execution step of executing an application and outputting an image generated by the execution to at least one of the operation device and a predetermined display device separate from the operation device;
Based on the operation data, it is determined whether or not a predetermined operation has been performed on the operating device during execution of the application . If the predetermined operation is performed , a predetermined operation image is generated. An operation image generation step to be displayed on a display unit included in the operation device;
An instruction determination step for determining, based on the operation data, whether or not there has been a control instruction for performing control on the predetermined display device in a state in which the operation image is displayed on the display unit;
An information processing method comprising: an apparatus control step of performing control according to the control instruction to the predetermined display device when the control instruction is received. In the instruction determination step, it is determined based on the operation data whether or not there is an end instruction to end the running application in a state where the operation image is displayed on the display unit.
The information processing method according to claim 13, wherein in the application execution step, the execution of the application is terminated when the termination instruction is issued. In the application execution step, when a predetermined operation is performed on the operation device during the execution of the application, the execution of the application is temporarily stopped.
In the instruction determination step, in a state where the operation image is displayed on the display unit, it is determined based on the operation data whether or not there is a restart instruction to resume execution of the application,
The information processing method according to claim 13 or 14, wherein, in the application execution step, execution of the application is resumed when the restart instruction is issued. The predetermined display device is capable of receiving a television broadcast and displaying a program,
16. In the operation image generation step, when the operation image is displayed on the display unit, an image including a TV program guide is generated and displayed on the predetermined display device. The information processing method according to any one of claims. In the operation image generation step, an image including a control instruction image representing the control instruction is generated as the operation image,
The information processing method according to any one of claims 13 to 16, wherein in the instruction determination step, it is determined that the control instruction has been given when the control instruction image is selected. The predetermined display device is capable of receiving a television broadcast and displaying a program,
17. The operation image generation step according to claim 13, wherein an image including a television program guide is generated as the operation image and displayed on a display unit of the operation device. Information processing method. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The predetermined display device can switch between a first mode for displaying an image input from the game device and a second mode for displaying an image of the television broadcast,
The control of switching the mode of the predetermined display device to the second mode is performed in the device control step in response to the operation image being displayed on the operation device. The information processing method according to any one of claims. The predetermined display device is capable of receiving a television broadcast and displaying a program,
The predetermined display device can switch between a first mode for displaying an image input from the game device and a second mode for displaying an image of the television broadcast,
In the device control step, the mode of the predetermined display device is changed to the first mode in response to switching of an image displayed on the operation device from the operation image to an image generated by execution of the application. The information processing method according to any one of claims 13 to 19, wherein control for switching to is performed. The operating device includes a control signal output unit that outputs a control signal for controlling the predetermined display device,
The device in the control step, a control signal indicating the control contents corresponding to the control instruction Ru is transmitted to the control signal output unit, information processing method according to any one of claims 20 to claim 13.   The information according to any one of claims 13 to 20, wherein in the device control step, a control signal indicating a control content according to the control instruction is output from the game device to the predetermined display device. Processing method. The operating device further includes a touch panel provided on the screen of the display unit,
In the operation image generation step, an image including a control instruction image representing the control instruction is generated as the operation image,
23. The information processing method according to claim 13, wherein in the instruction determination step, it is determined that the control instruction has been given when the control instruction image is selected on the touch panel. A game device capable of communicating with an operation device having a display unit,
An acquisition unit for acquiring operation data representing an operation on the operation device;
An application execution unit that executes an application and outputs an image generated by the execution to at least one of the operation device and a predetermined display device separate from the operation device;
Whether or not a predetermined operation has been performed on the operating device during execution of the application is determined based on the operation data, and when the predetermined operation is performed , for displaying on the display unit An operation image generating unit for generating a predetermined operation image;
An instruction determination unit that determines, based on the operation data, whether or not there has been a control instruction to control the predetermined display device in a state in which the operation image is displayed on the display unit;
A game device, comprising: a device control unit that performs control according to the control command to the predetermined display device when the control command is received. An information processing program executed in a computer of a game device capable of communicating with an operation device having a display unit,
Obtaining means for obtaining operation data representing an operation on the operation device;
Application execution means for executing an application and outputting an image generated by the execution to at least one of the operation device and a predetermined display device separate from the operation device;
Whether or not a predetermined operation has been performed on the operating device during execution of the application is determined based on the operation data, and when the predetermined operation is performed , for displaying on the display unit An operation image generating means for generating a predetermined operation image;
Instruction determination means for determining whether or not there has been a control instruction for performing control on the predetermined display device in a state where the operation image is displayed on the display unit;
An information processing program for causing the computer to function as device control means for performing control according to the control instruction to the predetermined display device when the control instruction is received.   An operating device capable of communicating with a game device,
  The game device executes an application, and outputs an image generated by the execution to at least one of the operation device and a predetermined display device separate from the operation device,
  An operation unit for outputting operation data representing an operation on the operation device;
  A receiving unit for receiving an image output from the game device;
  A display for displaying the received image;
  It is determined based on the operation data whether or not a predetermined operation has been performed on the operation device during execution of an application on the game device, and when the predetermined operation is performed, a predetermined operation image is generated And an operation image generation unit for outputting to the display unit,
  An instruction determination unit that determines, based on the operation data, whether or not there has been a control instruction to control the predetermined display device in a state in which the operation image is displayed on the display unit;
  An operation device comprising: a device control unit that performs control according to the control command to the predetermined display device when the control command is received.

Images