JP2018118129A - Information processing system, case, and cardboard member - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing system which can be implemented with a simple configuration, a case, and a cardboard member.SOLUTION: A data transmission apparatus including an imaging unit and transmitting data generated based on an image captured by the imaging unit can be contained in a case partially at least so that an imaging direction of the imaging unit faces an inside space of the case body. According to an operation from the outside of the case body, a member to be imaged which moves in the inside space of the case body is arranged in a position to be operated from the imaging unit. Information processing is executed on the basis of the transmitted data.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an information processing system including an information processing device and a case in which the data transmission device can be attached, a case in the information processing system, and a cardboard member to which the data transmission device can be attached.

  Conventionally, there is an accessory device that adds a function to a portable device by being connected to the portable device (see, for example, Patent Document 1). For example, Patent Document 1 discloses a technique for connecting another controller as an accessory device to a connector provided in a controller of a game device. According to this, various game operations can be performed by using two controllers.

JP 2010-017387 A

  However, since the accessory device has a circuit configuration for communicating with the portable device and a circuit configuration for information processing for generating information to be transmitted to the portable device, the improvement of simplifying the configuration of the accessory device is improved. There was room for.

  Therefore, an object of the present invention is to provide an information processing system, a case, and a cardboard member that can be realized with a simple configuration.

  In order to achieve the above object, the present invention may employ the following configuration, for example. When interpreting the description of the claims, it is understood that the scope should be interpreted only by the description of the claims, and the description of the claims and the description in this column are contradictory. In that case, priority is given to the claims.

  One configuration example of the information processing system of the present invention includes an information processing device, a data transmission device that transmits data to the information processing device, and a case in which the data transmission device can be attached. The data transmission device includes an imaging unit and a transmission unit. The transmission unit transmits data generated based on the image captured by the imaging unit to the information processing apparatus. The case includes a case main body, a data transmission device arrangement unit, an imaging target member, and an operation unit. The case main body has a space inside thereof. The data transmission device arrangement unit can be arranged so as to include at least a part of the data transmission device so that the imaging direction of the imaging unit faces the inner space of the case main body. The imaging target member is arranged in a space inside the case main body portion and at a position where the imaging device can take an image when the data transmission device is arranged in the data transmission device arrangement portion. The operation unit moves the imaging target member in the inner space of the case body in accordance with an operation from the outside of the case body. The information processing apparatus includes a receiving unit and a processing unit. The receiving unit receives data transmitted from the transmitting unit. The processing unit performs information processing based on the data received by the receiving unit.

  According to the above, the data transmission device that images the inside of the case is attached to the case, and the data generated based on the captured image obtained by capturing the imaging target member inside the case that moves according to the operation on the operation unit of the case is generated. Can be used for information processing. Therefore, the case can be realized with a simple configuration because it does not require an electrical configuration for detecting the operation content of the user or transmitting the detection result to the information processing apparatus.

  The data transmission device may further include a calculation unit. The calculation unit calculates at least the position of the imaging target member in the captured image captured by the imaging unit. In this case, the data transmitted by the transmission unit to the information processing apparatus may include at least the position of the imaging target member in the captured image.

  According to the above, since the data transmission device transmits the data indicating the position of the imaging target member captured in the captured image, the amount of data transmitted by the data transmission device can be reduced.

  The data transmission device may further include an infrared light emitting unit. The data transmission device arrangement unit may be capable of arranging the data transmission device so that the infrared light emission unit can emit infrared light in the inner space of the case main body. The imaging unit may be an infrared camera. The imaging target member may include a retroreflecting material.

  According to the above, the imaging target member is recognized by irradiating the imaging target member with infrared light and captured by the infrared camera, so that the imaging target member is recognized even if the captured image is an image of the inside of the case blocked from outside light. And the possibility of erroneous recognition due to external light can be reduced. Moreover, since the imaging target member includes a retroreflecting material that reflects infrared light, the imaging target member can be easily recognized from within the captured image.

  The case may further include a fixing unit that fixes the case to the user's body.

  According to the above, it is possible to cause the case to function as an extended operation device that can be fixed to the user's body.

  Further, the fixing portion may be a shoulder belt for the user to carry the case.

  According to the above, the case can be easily fixed to the user's body by carrying the case.

  The operation unit may include at least one string member connected to the imaging target member. At least one of the string members is configured to be capable of being pulled by the user's hand while the case is fixed to the user's body by the fixing portion. The operation unit may move the imaging target member connected to the string member in the inner space of the case main body by pulling the string member by the user.

  Based on the above, it is possible to perform information processing according to an operation of pulling the string member included in the case.

  The case may further include a slide portion. The slide portion supports the imaging target member to which the string member is connected so as to be slidable in a predetermined sliding direction in the inner space of the case main body portion. When the string member is pulled, the operation unit may move the imaging target member connected to the string member in one direction in the slide direction.

  Based on the above, it is possible to accurately recognize the operation amount for pulling the string member by allowing the imaging target member to slide in the case.

  The operation unit may be configured such that when the pulled string member is loosened, the imaging target member connected to the string member moves in the reverse direction in the sliding direction.

  According to the above, it is possible to accurately recognize the operation amount for loosening the string member.

  The information processing device may be a game device. The data transmission device may be a game controller used for operating the game device. The processing unit may perform a game process for controlling a predetermined object in the virtual space according to the position of the imaging target member.

  According to the above, the case can be used as an extended operation device used for game operations.

  The data transmission device may further include an inertial sensor. The transmission unit may further transmit data based on the output of the inertial sensor to the information processing apparatus. The processing unit may control the entire object according to data based on the output of the inertial sensor, and may control a part of the object according to the position of the imaging target member.

  According to the above, it is possible to control a part of the virtual object according to the operation using the operation unit of the case, and to control the movement of the entire virtual object according to the movement of the entire case.

  One configuration example of the case of the present invention can be equipped with a data transmission device that includes at least an imaging unit and transmits data generated based on an image captured by the imaging unit to at least another device. The case includes a case main body, a data transmission device arrangement unit, an imaging target member, and an operation unit. The case main body has a space inside thereof. The data transmission device arrangement unit can be arranged so as to include at least a part of the data transmission device so that the imaging direction of the imaging unit faces the inner space of the case main body. The imaging target member is arranged in a space inside the case main body portion and at a position where the imaging device can take an image when the data transmission device is arranged in the data transmission device arrangement portion. The operation unit moves the imaging target member in the inner space of the case body in accordance with an operation from the outside of the case body.

  According to the above, a data transmission device that images the inside of the case is mounted, and information processing is performed using data generated based on a captured image obtained by capturing an imaging target member inside the case that moves in response to an operation on the operation unit. be able to. Therefore, the case can be realized with a simple configuration because it does not require an electrical configuration for detecting the operation content of the user or transmitting the detection result to the information processing apparatus.

  The case may further include a shoulder belt portion and a slide portion. The shoulder belt portion can be fixed to the user's body by the user carrying the case. The slide portion supports the imaging target member so as to be slidable in a predetermined sliding direction in the inner space of the case main body portion. The operation unit may include at least one string member connected to the imaging target member. At least one of the string members is configured to be capable of being pulled by the user's hand while the case is fixed to the user's body by the shoulder belt portion. The operation unit moves the imaging target member in one direction in the sliding direction when the string member is pulled, and moves the imaging target member in the reverse direction in the sliding direction when the pulled string member is loosened. It may be configured as follows.

  According to the above, it is possible to perform information processing according to an operation of pulling the string member in a state of being fixed to the user's body by carrying the case on the back. In addition, by allowing the imaging target member to slide within the case, it is possible to accurately recognize the operation amount for pulling the string member and the operation amount for loosening the string member.

  In addition, the case main body, the data transmission device arrangement unit, the imaging target member, and the slide unit may be configured to be formed by folding at least one cardboard.

  According to the above, the case can be assembled by folding and combining the cardboard members.

  One configuration example of the cardboard member of the present invention can form an expansion attachment that can be attached to a data transmission device that transmits predetermined data to another device. The extension attachment includes a main body portion and a data transmission device arrangement portion. The data transmission device arrangement unit is formed in a part of the main body unit, and can be arranged by including at least a part of the data transmission device inside the main body unit. The cardboard member can be integrally formed by bending an expansion attachment having a main body portion and a data transmission device arrangement portion.

  Based on the above, it is possible to configure an extension attachment to which the data transmission device can be attached by bending the cardboard member.

  According to the present invention, a case that can be operated by a user can be realized with a simple configuration.

The figure which shows the state which mounted | wore the main body apparatus 2 with the left controller 3 and the right controller 4 The figure which shows an example of the state which removed the left controller 3 and the right controller 4 from the main body apparatus 2, respectively. Six views showing an example of the main unit 2 Six views showing an example of the left controller 3 Six views showing an example of the right controller 4 The block diagram which shows an example of an internal structure of the main body apparatus 2 The block diagram which shows an example of an internal structure of the main body device 2, the left controller 3, and the right controller 4 The figure which shows an example of a mode that a user wears case 200 and operates a game The figure which shows an example of the external appearance of case 200 The figure which shows an example of the state with which the right controller 4 was mounted | worn to the controller arrangement | positioning part 202 of case 200. The figure which shows an example of the state of the right controller 4 which images the imaging target member 205 provided in the case 200 The figure which shows an example which the imaging target member 205 slides according to the movement of the string member 204 The figure for demonstrating an example of the information of the imaging target member calculated in 1st mode (clustering mode) The figure for demonstrating an example of the information of the imaging target member calculated in 2nd mode (moment mode) The figure which shows the example of a combination of the main components at the time of assembling the case 200 It is a figure which shows an example of the cardboard member for assembling the front side main body 201a. The figure which shows an example of the cardboard member for assembling the back side main body 201b The figure which shows an example of a mode that a user operates a game using the control apparatus 250. The figure which shows an example of the internal structure of the control apparatus 250 The figure which shows an example of the data area set in the memory of the right controller 4 in this embodiment. The flowchart which shows an example of the information calculation process performed with CPU of the process part 121 The figure which shows an example of the data area set to DRAM85 of the main body apparatus 2 in this embodiment. The flowchart which shows an example of the information processing performed with the main body apparatus 2

  Hereinafter, an information processing system according to an example of the present embodiment will be described. An example of the information processing system in the present embodiment includes a main body device (information processing apparatus; functions as a game apparatus main body in the present embodiment) 2, a left controller 3, a right controller 4, and a case 200. The main device 2 can be attached to and detached from the left controller 3 and the right controller 4, respectively, and can be used as an integrated device by attaching the left controller 3 and the right controller 4 to the main device 2, respectively. The left controller 3 and the right controller 4 can be used as separate bodies (see FIG. 2). Further, the information processing system can be used in a mode in which an image is displayed on the main body device 2 and in a mode in which an image is displayed on another display device (for example, a stationary monitor) such as a television. In the former aspect, the information processing system can be used as a portable device (for example, a portable game machine). In the latter mode, the information processing system can be used as a stationary device (for example, a stationary game machine). Further, the case 200 in the information processing system can be used as an extended operation device by mounting a controller (for example, the right controller 4) therein.

  FIG. 1 is a diagram illustrating a state in which the left controller 3 and the right controller 4 are mounted on the main body device 2. As shown in FIG. 1, the left controller 3 and the right controller 4 are each attached to and integrated with the main body device 2. The main device 2 is a device that executes various processes (for example, game processes) in the information processing system. The main device 2 includes a display 12. The left controller 3 and the right controller 4 are devices including an operation unit for a user to input.

  FIG. 2 is a diagram illustrating an example of a state in which the left controller 3 and the right controller 4 are removed from the main body device 2. As shown in FIGS. 1 and 2, the left controller 3 and the right controller 4 are detachable from the main body device 2. The left controller 3 can be mounted on the left side surface (side surface on the x-axis positive direction side shown in FIG. 1) of the main body device 2 and is slid along the left side surface of the main body device 2 in the y-axis direction shown in FIG. Thus, it can be attached to and detached from the main unit 2. Further, the right controller 4 can be mounted on the right side surface (the side surface on the x-axis negative direction side shown in FIG. 1) of the main body device 2 and along the right side surface of the main body device 2 in the y-axis direction shown in FIG. The main body device 2 can be attached and detached by sliding. Hereinafter, the left controller 3 and the right controller 4 may be collectively referred to as “controller”. In the present embodiment, an “operation device” operated by one user may be a single controller (for example, one of the left controller 3 and the right controller 4) or a plurality of controllers (for example, the left controller 3 and the right controller 4). Or the other controller may be included), and the “operation device” can be configured by one or more controllers. Further, in the present embodiment, by mounting one controller (for example, the right controller 4) on the expansion operation device (for example, the case 200), the user operates the expansion operation device, and the control is executed in the main body device 2. Information processing (for example, game processing) can be controlled. Hereinafter, an example of a specific configuration of the main body device 2, the left controller 3, and the right controller 4 will be described.

  FIG. 3 is a six-sided view illustrating an example of the main device 2. As shown in FIG. 3, the main device 2 includes a substantially plate-shaped housing 11. In the present embodiment, the main surface of the housing 11 (in other words, the surface on the front side, that is, the surface on which the display 12 is provided) is generally rectangular. In the present embodiment, the housing 11 has a horizontally long shape. In other words, in the present embodiment, the longitudinal direction (that is, the x-axis direction shown in FIG. 1) of the main surface of the housing 11 is referred to as the lateral direction (also referred to as the left-right direction), and the short direction (that is, the figure) of the main surface. The y-axis direction shown in FIG. 1 is called the vertical direction (also called the vertical direction), and the direction perpendicular to the main surface (that is, the z-axis direction shown in FIG. 1) is called the depth direction (also called the front-back direction). . The main device 2 can be used in a direction in which the main device 2 is horizontally long. Further, the main body device 2 can be used in a vertically long direction. In that case, the housing 11 may be regarded as having a vertically long shape.

  The shape and size of the housing 11 are arbitrary. As an example, the housing 11 may be a portable size. Further, the main body device 2 alone or the integrated device in which the left controller 3 and the right controller 4 are attached to the main body device 2 may be a portable device. Further, the main body device 2 or the integrated device may be a handheld device. Further, the main body device 2 or the integrated device may be a portable device.

  As shown in FIG. 3, the main device 2 includes a display 12 provided on the main surface of the housing 11. The display 12 displays an image (a still image or a moving image) acquired or generated by the main device 2. In the present embodiment, the display 12 is a liquid crystal display (LCD). However, the display 12 may be any type of display device.

  The main device 2 also includes a touch panel 13 on the screen of the display 12. In the present embodiment, the touch panel 13 is a type capable of multi-touch input (for example, a capacitance type). However, the touch panel 13 may be of any type, and may be of a type capable of single touch input (for example, a resistance film type).

  The main device 2 includes a speaker (that is, the speaker 88 shown in FIG. 6) inside the housing 11. As shown in FIG. 3, speaker holes 11 a and 11 b are formed in the main surface of the housing 11. And the output sound of the speaker 88 is output from these speaker holes 11a and 11b, respectively.

  As shown in FIG. 3, the main body device 2 includes a left rail member 15 on the left side surface of the housing 11. The left rail member 15 is a member for detachably attaching the left controller 3 to the main body device 2. The left rail member 15 is provided on the left side surface of the housing 11 so as to extend along the vertical direction. The left rail member 15 has a shape that can be engaged with the slider of the left controller 3 (that is, the slider 40 shown in FIG. 4), and the left rail member 15 and the slider 40 form a slide mechanism. With this slide mechanism, the left controller 3 can be slidably and detachably attached to the main body device 2.

  The main device 2 also includes a left terminal 17. The left terminal 17 is a terminal for the main device 2 to perform wired communication with the left controller 3. The left terminal 17 is provided at a position in contact with the terminal of the left controller 3 (terminal 42 shown in FIG. 4) when the left controller 3 is attached to the main body device 2. The specific position of the left terminal 17 is arbitrary. In the present embodiment, as shown in FIG. 3, the left terminal 17 is provided on the bottom surface of the left rail member 15. In the present embodiment, the left terminal 17 is provided in the vicinity of the lower end of the bottom surface of the left rail member 15 and is disposed at a position where it is not exposed to the outside by a part of the left rail member 15.

  As shown in FIG. 3, the right side surface of the housing 11 is provided with the same configuration as that provided on the left side surface. That is, the main body device 2 includes a right rail member 19 on the right side surface of the housing 11. The right rail member 19 is provided on the right side surface of the housing 11 so as to extend along the vertical direction. The right rail member 19 has a shape that can be engaged with the slider of the right controller 4 (that is, the slider 62 shown in FIG. 5), and the right rail member 19 and the slider 62 form a slide mechanism. With this slide mechanism, the right controller 4 can be slidably and detachably attached to the main body device 2.

  The main device 2 also includes a right terminal 21. The right terminal 21 is a terminal for the main device 2 to perform wired communication with the right controller 4. The right terminal 21 is provided at a position in contact with the terminal of the right controller 4 (terminal 64 shown in FIG. 5) when the right controller 4 is attached to the main body device 2. The specific position of the right terminal 21 is arbitrary. In the present embodiment, as shown in FIG. 3, the right terminal 21 is provided on the bottom surface of the right rail member 19. In the present embodiment, the right terminal 21 is provided in the vicinity of the lower end of the bottom surface of the right rail member 19, and is disposed at a position that is not exposed to the outside by a part of the right rail member 19.

  As shown in FIG. 3, the main body device 2 includes a first slot 23. The first slot 23 is provided on the upper side surface of the housing 11. The first slot 23 has a shape capable of mounting the first type of storage medium. The first type of storage medium is, for example, a storage medium (for example, a dedicated memory card) dedicated to the information processing system and the same type of information processing apparatus. The first type of storage medium includes, for example, data used by the main device 2 (for example, application save data) and / or programs executed by the main device 2 (for example, application programs). Used for storing. The main device 2 also includes a power button 28. As shown in FIG. 3, the power button 28 is provided on the upper side surface of the housing 11. The power button 28 is a button for switching on / off the power supply of the main device 2.

  The main device 2 includes an audio input / output terminal (specifically, an earphone jack) 25. That is, the main device 2 can attach a microphone or earphone to the audio input / output terminal 25. As shown in FIG. 3, the audio input / output terminal 25 is provided on the upper side surface of the housing 11.

  The main device 2 includes volume buttons 26a and 26b. As shown in FIG. 3, the volume buttons 26 a and 26 b are provided on the upper side surface of the housing 11. The volume buttons 26 a and 26 b are buttons for giving instructions for adjusting the volume output by the main device 2. That is, the volume button 26a is a button for instructing to decrease the volume, and the volume button 26b is a button for instructing to increase the volume.

  The housing 11 has an exhaust hole 11c. As shown in FIG. 3, the exhaust hole 11 c is formed on the upper side surface of the housing 11. The exhaust hole 11 c is formed to exhaust (in other words, release) the heat generated inside the housing 11 to the outside of the housing 11. That is, it can be said that the exhaust holes 11c are exhaust heat holes.

  The main device 2 includes a lower terminal 27. The lower terminal 27 is a terminal for the main device 2 to communicate with the cradle. As shown in FIG. 3, the lower terminal 27 is provided on the lower surface of the housing 11. When the main device 2 is mounted on the cradle, the lower terminal 27 is connected to a terminal of the cradle. In the present embodiment, the lower terminal 27 is a USB connector (more specifically, a female connector). The cradle can mount only the main device 2 with the left controller 3 and the right controller 4 removed from the main device 2. As still another example, the cradle can be mounted with an integrated device in which the left controller 3 and the right controller 4 are mounted on the main body device 2. The cradle can communicate with a stationary monitor (for example, a stationary television) that is an example of an external display device that is separate from the main body device 2 (wired communication or wireless communication). Also good). When the integrated device or the main device 2 alone is placed on a cradle, the information processing system can display an image acquired or generated by the main device 2 on a stationary monitor. Further, in the present embodiment, the cradle has a function of charging the mounted integrated device or the main body device 2 alone. The cradle has a function of a hub device (specifically, a USB hub).

  The main device 2 also includes a second slot 24. In the present embodiment, the second slot 24 is provided on the lower surface of the housing 11. However, in other embodiments, the second slot 24 may be provided on the same surface as the first slot 23. The second slot 24 has a shape in which a second type of storage medium different from the first type can be mounted. The second type of storage medium may be a general-purpose storage medium, for example. For example, the second type of storage medium may be an SD card. The second type storage medium is, for example, the data used by the main device 2 (for example, saved data of an application) and / or executed by the main device 2 in the same manner as the first type storage medium. Used to store a program (for example, an application program).

  The housing 11 is formed with an intake hole 11d. As shown in FIG. 3, the intake hole 11 d is formed on the lower surface of the housing 11. The intake hole 11d is formed to suck (in other words, introduce) air outside the housing 11 into the housing 11. In the present embodiment, since the intake hole 11d is formed on the surface opposite to the surface where the exhaust hole 11c is formed, the heat radiation inside the housing 11 can be efficiently performed.

  The shape, number, and installation position of each component (specifically, buttons, slots, terminals, etc.) provided in the housing 11 described above are arbitrary. For example, in other embodiments, some of the power button 28 and each of the slots 23 and 24 may be provided on the other side or back of the housing 11. In other embodiments, main device 2 may be configured not to include some of the above-described components.

  FIG. 4 is a six-sided view illustrating an example of the left controller 3. As shown in FIG. 4, the left controller 3 includes a housing 31. In the present embodiment, the housing 31 has a substantially plate shape. Further, the main surface of the housing 31 (in other words, the surface on the front side, that is, the surface on the negative side in the z-axis direction shown in FIG. 1) is generally rectangular. In the present embodiment, the housing 31 has a vertically long shape, that is, a shape that is long in the vertical direction (that is, the y-axis direction shown in FIG. 1). The left controller 3 can also be held in a vertically long orientation when removed from the main unit 2. The housing 31 has a shape and size that can be gripped with one hand, particularly the left hand, when gripped in a vertically long orientation. Further, the left controller 3 can be held in a landscape orientation. When the left controller 3 is gripped in a landscape orientation, it may be gripped with both hands. In addition, the shape of the housing 31 is arbitrary, and the housing 31 may not be substantially plate-shaped in other embodiments. Further, the housing 31 does not have to be rectangular and may be, for example, a semicircular shape. Further, the housing 31 may not have a vertically long shape.

  The vertical length of the housing 31 is substantially the same as the vertical length of the housing 11 of the main device 2. The thickness of the housing 31 (that is, the length in the front-rear direction, in other words, the length in the z-axis direction shown in FIG. 1) is substantially the same as the thickness of the housing 11 of the main body device 2. Therefore, when the left controller 3 is mounted on the main device 2 (see FIG. 1), the user can hold the main device 2 and the left controller 3 as if they were an integrated device.

  As shown in FIG. 4, the main surface of the housing 31 is such that the left corner portion is rounder than the right corner portion. That is, the connection portion between the upper side surface and the left side surface of the housing 31 and the connection portion between the lower side surface and the left side surface of the housing 31 are the connection portion between the upper side surface and the right side surface, and the lower side surface and the right side surface. Is rounded (in other words, R in chamfering is large). Therefore, when the left controller 3 is attached to the main body device 2 (see FIG. 1), the left side of the integrated device has a rounded shape, which makes it easy for the user to hold.

  The left controller 3 includes an analog stick 32. As shown in FIG. 4, the analog stick 32 is provided on the main surface of the housing 31. The analog stick 32 is an example of a direction input unit capable of inputting a direction. The analog stick 32 has a stick member that can be tilted in all directions parallel to the main surface of the housing 31 (that is, a direction of 360 ° including vertical, horizontal, and diagonal directions). By tilting the stick member, the user can input a direction corresponding to the tilting direction (and an input corresponding to the tilted angle). The direction input unit may be a cross key or a slide stick. In the present embodiment, an input for pressing the stick member (in a direction perpendicular to the housing 31) is possible. In other words, the analog stick 32 is an input unit capable of inputting a direction and a size corresponding to the tilt direction and the tilt amount of the stick member, and pressing the stick member.

  The left controller 3 includes four operation buttons 33 to 36 (specifically, a right button 33, a down button 34, an up button 35, and a left button 36). As shown in FIG. 4, these four operation buttons 33 to 36 are provided below the analog stick 32 on the main surface of the housing 31. In the present embodiment, four operation buttons are provided on the main surface of the left controller 3, but the number of operation buttons is arbitrary. These operation buttons 33 to 36 are used for giving instructions according to various programs (for example, an OS program or an application program) executed by the main device 2. In the present embodiment, since the operation buttons 33 to 36 may be used for inputting a direction, the operation buttons 33 to 36 are replaced with a right button 33, a down button 34, and an up button. 35 and the left direction button 36. However, each operation button 33-36 may be used in order to give instructions other than direction input.

  The left controller 3 includes a recording button 37. As shown in FIG. 4, the recording button 37 is provided on the main surface of the housing 31, more specifically, in the lower right region on the main surface. The recording button 37 is a button for giving an instruction to save an image displayed on the display 12 of the main device 2. For example, when a game image is displayed on the display 12, the user presses the recording button 37 to save the game image displayed at the time of pressing in the storage unit of the main device 2, for example. be able to.

  The left controller 3 includes a − (minus) button 47. As shown in FIG. 4, the − button 47 is provided on the main surface of the housing 31, more specifically, in the upper right region on the main surface. The − button 47 is used for giving an instruction according to various programs (for example, an OS program or an application program) executed by the main device 2. The-button 47 is used as a select button (for example, a button used for switching selection items) in a game application, for example.

  Each operation unit (specifically, the analog stick 32 and each of the buttons 33 to 37, 47) provided on the main surface of the left controller 3 is an integrated device when the left controller 3 is attached to the main body device 2. For example, it is operated with the thumb of the left user's hand. When the left controller 3 is used while being held sideways with both hands in a state where the left controller 3 is detached from the main body device 2, the operation units are operated with the thumbs of the left and right hands of the user holding the left controller 3, for example. The Specifically, in this case, the analog stick 32 is operated with the thumb of the user's left hand, and the operation buttons 33 to 36 are operated with the thumb of the user's right hand.

  The left controller 3 includes a first L button 38. The left controller 3 includes a ZL button 39. These operation buttons 38 and 39 are used for giving instructions according to various programs executed in the main body device 2, as with the operation buttons 33 to 36. As shown in FIG. 4, the first L button 38 is provided on the upper left portion of the side surface of the housing 31. The ZL button 39 is provided in an upper left portion (strictly speaking, an upper left portion when the housing 31 is viewed from the front side) from the side surface to the back surface of the housing 31. That is, the ZL button 39 is provided on the rear side (the z-axis positive direction side shown in FIG. 1) of the first L button 38. In the present embodiment, since the upper left part of the housing 31 has a rounded shape, the first L button 38 and the ZL button 39 have a rounded shape corresponding to the roundness of the upper left part of the housing 31. When the left controller 3 is attached to the main body device 2, the first L button 38 and the ZL button 39 are arranged in the upper left portion of the integrated device.

  The left controller 3 includes the slider 40 described above. As shown in FIG. 4, the slider 40 is provided on the right side surface of the housing 31 so as to extend in the vertical direction. The slider 40 has a shape that can be engaged with the left rail member 15 (more specifically, the groove of the left rail member 15) of the main body device 2. Therefore, the slider 40 engaged with the left rail member 15 is fixed and cannot be removed in the direction perpendicular to the sliding direction (in other words, the direction in which the left rail member 15 extends).

  The left controller 3 also includes a terminal 42 for the left controller 3 to perform wired communication with the main body device 2. The terminal 42 is provided at a position in contact with the left terminal 17 (FIG. 3) of the main device 2 when the left controller 3 is attached to the main device 2. The specific position of the terminal 42 is arbitrary. In the present embodiment, as shown in FIG. 4, the terminal 42 is provided at a position where it is not exposed to the outside by the mounting surface of the slider 40. In the present embodiment, the terminal 42 is provided in the vicinity of the lower end of the mounting surface of the slider 40.

  The left controller 3 includes a second L button 43 and a second R button 44. These buttons 43 and 44 are used for giving instructions according to various programs executed in the main body device 2, as with the other operation buttons 33 to 36. As shown in FIG. 4, the second L button 43 and the second R button 44 are provided on the mounting surface of the slider 40. The second L button 43 is provided on the mounting surface of the slider 40 above the center in the vertical direction (y-axis direction shown in FIG. 1). The second R button 44 is provided below the center in the vertical direction on the mounting surface of the slider 40. The second L button 43 and the second R button 44 are arranged at positions where the left controller 3 cannot be pressed when the left controller 3 is attached to the main body device 2. That is, the second L button 43 and the second R button 44 are buttons used when the left controller 3 is removed from the main body device 2. The second L button 43 and the second R button 44 are operated with, for example, the index fingers or middle fingers of the left and right hands of the user holding the left controller 3 removed from the main body device 2.

  The left controller 3 includes a pairing button 46. In the present embodiment, the pairing button 46 is used to instruct setting (also referred to as pairing) processing related to wireless communication between the left controller 3 and the main device 2 and to instruct reset processing of the left controller 3. It is done. In other embodiments, the pairing button 46 may have only one function of the setting process and the reset process. For example, when a short press operation is performed on the pairing button 46 (specifically, when the pairing button 46 is pressed for a time shorter than a predetermined time), the left controller 3 performs the above setting process. Run. Further, when a long press operation is performed on the pairing button 46 (specifically, when the pairing button 46 is continuously pressed for the predetermined time or more), the left controller 3 executes a reset process. In the present embodiment, the pairing button 46 is provided on the mounting surface of the slider 40 as shown in FIG. Thus, the pairing button 46 is disposed at a position that cannot be seen when the left controller 3 is attached to the main body device 2. That is, the pairing button 46 is used when the left controller 3 is removed from the main body device 2.

  In the present embodiment, the buttons (specifically, the second L button 43, the second R button 44, and the pairing button 46) provided on the mounting surface of the slider 40 do not protrude from the mounting surface. Provided. That is, the upper surface of the button (in other words, the pressing surface) is disposed on the same surface as the mounting surface of the slider 40 or is disposed at a position recessed from the mounting surface. Accordingly, the slider 40 can be smoothly slid with respect to the left rail member 15 in a state where the slider 40 is mounted on the left rail member 15 of the main body device 2.

  FIG. 5 is a six-sided view illustrating an example of the right controller 4. As shown in FIG. 5, the right controller 4 includes a housing 51. In the present embodiment, the housing 51 has a substantially plate shape. The main surface of the housing 51 (in other words, the surface on the front side, that is, the surface on the negative side of the z axis shown in FIG. 1) is generally rectangular. Further, in the present embodiment, the housing 51 has a vertically long shape, that is, a shape that is long in the vertical direction. The right controller 4 can also be gripped in a portrait orientation when removed from the main device 2. The housing 51 has a shape and size that can be gripped by one hand, particularly the left hand, when gripped in a vertically long orientation. The right controller 4 can also be held in a landscape orientation. When the right controller 4 is gripped in a landscape orientation, it may be gripped by both hands.

  Like the housing 31 of the left controller 3, the housing 51 of the right controller 4 has a vertical length that is substantially the same as the vertical length of the housing 11 of the main device 2, and the thickness thereof is the main device 2. The thickness of the housing 11 is substantially the same. Therefore, when the right controller 4 is attached to the main device 2 (see FIG. 1), the user can hold the main device 2 and the right controller 4 as if they were an integrated device.

  Further, as shown in FIG. 5, the main surface of the housing 51 has a shape in which the right corner portion is rounder than the left corner portion. That is, the connection portion between the upper side surface and the right side surface of the housing 51 and the connection portion between the lower side surface and the right side surface of the housing 51 are the connection portion between the upper side surface and the left side surface, and the lower side surface and the left side surface. Is rounded (in other words, R in chamfering is large). Therefore, when the right controller 4 is attached to the main body device 2 (see FIG. 1), the right side of the integrated device has a rounded shape, which makes it easy for the user to hold.

  Similar to the left controller 3, the right controller 4 includes an analog stick 52 as a direction input unit. In the present embodiment, the analog stick 52 has the same configuration as the analog stick 32 of the left controller 3. Similarly to the left controller 3, the right controller 4 includes four operation buttons 53 to 56 (specifically, an A button 53, a B button 54, an X button 55, and a Y button 56). In the present embodiment, these four operation buttons 53 to 56 have the same mechanism as the four operation buttons 33 to 36 of the left controller 3. As shown in FIG. 5, the analog stick 52 and the operation buttons 53 to 56 are provided on the main surface of the housing 51. In the present embodiment, four operation buttons are provided on the main surface of the right controller 4, but the number of operation buttons is arbitrary.

  Here, in the present embodiment, the positional relationship between the two types of operation units (analog stick and operation button) in the right controller 4 is opposite to the positional relationship between these two types of operation units in the left controller 3. . That is, in the right controller 4, the analog stick 52 is disposed above the operation buttons 53 to 56, whereas in the left controller 3, the analog stick 32 is disposed below the operation buttons 33 to 36. The With such an arrangement, it is possible to use the left controller 3 and the right controller 4 with an operational feeling similar to that when the left controller 3 and the right controller 4 are removed from the main body device 2.

  The right controller 4 includes a + (plus) button 57. As shown in FIG. 5, the + button 57 is provided on the main surface of the housing 51, more specifically, in the upper left area of the main surface. Like the other operation buttons 53 to 56, the + button 57 is used to give instructions according to various programs (for example, an OS program or an application program) executed by the main body device 2. The + button 57 is used as, for example, a start button (for example, a button used for instructing a game start) in a game application.

  The right controller 4 includes a home button 58. As shown in FIG. 5, the home button 58 is provided on the main surface of the housing 51, more specifically, in the lower left region of the main surface. The home button 58 is a button for displaying a predetermined menu screen on the display 12 of the main device 2. The menu screen is, for example, a screen that can start an application designated by the user from one or more applications that can be executed in the main device 2. For example, the menu screen may be displayed when the main device 2 is activated. In the present embodiment, when the home button 58 is pressed in a state where an application is being executed in the main device 2 (that is, a state where an image of the application is displayed on the display 12), a predetermined operation screen is displayed. It may be displayed on the display 12 (at this time, a menu screen may be displayed instead of the operation screen). The operation screen is a screen on which, for example, an instruction to end the application and display the menu screen on the display 12 and an instruction to restart the application can be performed.

  The operation units (specifically, the analog stick 52 and the buttons 53 to 58) provided on the main surface of the right controller 4 hold the integrated device when the right controller 4 is mounted on the main body device 2. It is operated by the user's right thumb, for example. Further, when the right controller 4 is used while being held sideways with both hands in a state where the right controller 4 is detached from the main body device 2, the operation units are operated by the thumbs of the right and left hands of the user holding the right controller 4, for example. The Specifically, in this case, the analog stick 52 is operated with the thumb of the user's left hand, and the operation buttons 53 to 56 are operated with the thumb of the user's right hand.

  The right controller 4 includes a first R button 60. The right controller 4 includes a ZR button 61. As shown in FIG. 5, the first R button 60 is provided in the upper right portion of the side surface of the housing 51. The ZR button 61 is provided in an upper right portion (strictly speaking, an upper right portion when the housing 51 is viewed from the front side) from the side surface to the back surface of the housing 51. That is, the ZR button 61 is provided on the rear side of the first R button 60 (the z-axis positive direction side shown in FIG. 1). In the present embodiment, since the upper right portion of the housing 51 has a rounded shape, the first R button 60 and the ZR button 61 have a rounded shape corresponding to the roundness of the upper right portion of the housing 51. When the right controller 4 is attached to the main body device 2, the first R button 60 and the ZR button 61 are arranged in the upper right part of the integrated device.

  The right controller 4 includes a slider mechanism similar to that of the left controller 3. That is, the right controller 4 includes the slider 62 described above. As shown in FIG. 5, the slider 62 is provided on the left side surface of the housing 51 so as to extend in the vertical direction. The slider 62 has a shape that can be engaged with the right rail member 19 (more specifically, the groove of the right rail member 19) of the main body device 2. Therefore, the slider 62 engaged with the right rail member 19 is fixed in the direction perpendicular to the sliding direction (in other words, the direction in which the right rail member 19 extends) and is not removed.

  The right controller 4 includes a terminal 64 for the right controller 4 to perform wired communication with the main body device 2. The terminal 64 is provided at a position in contact with the right terminal 21 (FIG. 3) of the main device 2 when the right controller 4 is attached to the main device 2. The specific position of the terminal 64 is arbitrary. In the present embodiment, as shown in FIG. 5, the terminal 64 is provided at a position where it is not exposed to the outside by the mounting surface of the slider 62. In the present embodiment, the terminal 64 is provided near the lower end of the mounting surface of the slider 62.

  The right controller 4 includes a second L button 65 and a second R button 66 as in the left controller 3. These buttons 65 and 66 are used to give instructions according to various programs executed in the main body device 2, similarly to the other operation buttons 53 to 56. As shown in FIG. 5, the second L button 65 and the second R button 66 are provided on the mounting surface of the slider 62. The second L button 65 is provided on the mounting surface of the slider 62 below the center in the vertical direction (y-axis direction shown in FIG. 1). The second R button 66 is provided above the center in the vertical direction on the mounting surface of the slider 62. Similar to the second L button 43 and the second R button 44 of the left controller 3, the second L button 65 and the second R button 66 are arranged at positions where the right controller 4 cannot be pressed when the main device 2 is mounted. These buttons are used when the right controller 4 is removed from the main unit 2. The second L button 65 and the second R button 66 are operated by, for example, the index fingers or middle fingers of the left and right hands of the user holding the right controller 4 removed from the main body device 2.

  The right controller 4 includes a pairing button 69. Similarly to the pairing button 46 of the left controller 3, the pairing button 69 instructs setting (also referred to as pairing) processing related to wireless communication between the right controller 4 and the main device 2 and reset processing of the right controller 4. Used to indicate Since the setting process and the reset process are the same as those in the left controller 3, detailed description thereof is omitted. In the present embodiment, the pairing button 69 is provided on the mounting surface of the slider 62 as shown in FIG. That is, the pairing button 69 is disposed at a position that cannot be seen when the right controller 4 is attached to the main body device 2 for the same reason as the pairing button 46 of the left controller 3.

  Similarly to the left controller 3, the right controller 4 also has buttons (specifically, the second L button 65, the second R button 66, and the pairing button 69) provided on the mounting surface of the slider 62 on the mounting surface. It is provided so as not to protrude. Accordingly, the slider 62 can be smoothly slid with respect to the right rail member 19 in a state where the slider 62 is mounted on the right rail member 19 of the main body device 2.

  A window portion 68 is provided on the lower side surface of the housing 51. As will be described in detail later, the right controller 4 includes an infrared imaging unit 123 and an infrared light emitting unit 124 disposed inside the housing 51. The infrared imaging unit 123 images the periphery of the right controller 4 through the window unit 68 with the downward direction of the right controller 4 (the negative y-axis direction shown in FIG. 5) as the imaging direction. The infrared light emitting unit 124 sets the window 68 on the imaging target member to be imaged by the infrared imaging unit 123, with a predetermined range centered on the downward direction of the right controller 4 (the negative y-axis direction shown in FIG. 5). Irradiate with infrared light through. In the example of FIG. 5, in order to increase the infrared light irradiation range of the infrared light emitting unit 124, a plurality of infrared light emitting units 124 having different irradiation directions are provided. The window unit 68 is for protecting the lens of the camera of the infrared imaging unit 123, the light emitter of the infrared light emitting unit 124, and the like, and emits light of a wavelength detected by the camera or light emitted by the light emitter. A transparent material (for example, a transparent material) is used. Note that the window 68 may be a hole formed in the housing 51. In the present embodiment, the infrared imaging unit 123 itself has a filter member that suppresses transmission of light having a wavelength other than light detected by the camera (in this embodiment, infrared light). However, in other embodiments, the window 68 may have a filter function.

  Although details will be described later, the right controller 4 includes an NFC communication unit 122. The NFC communication unit 122 performs near field communication based on NFC (Near Field Communication) standards. The NFC communication unit 122 includes an antenna 122a used for short-range wireless communication and a circuit (for example, an NFC chip) that generates a signal (radio wave) to be transmitted from the antenna 122a. For example, the antenna 122a is used when another wireless communication device (for example, an NFC tag) that is a target in the short-range wireless communication is disposed in another device that is close to or in contact with the lower region of the main surface of the housing 51. It is provided at a position inside the housing 51 that enables the short-range wireless communication with the wireless communication device. Note that the near field communication is not limited to the one based on the NFC standard, and may be any near field communication (also referred to as non-contact communication). Proximity communication includes, for example, a communication method in which an electromotive force is generated in the other device by radio waves from one device (for example, by electromagnetic induction).

  In the left controller 3 and the right controller 4, the shape, number, and installation position of each component (specifically, slider, stick, button, etc.) provided in the housing 31 or 51 are arbitrary. For example, in another embodiment, the left controller 3 and the right controller 4 may include a direction input unit of a type different from the analog stick. Further, the slider 40 or 62 may be disposed at a position corresponding to the position of the rail member 15 or 19 provided in the main body device 2, and may be disposed on the main surface or the back surface of the housing 31 or 51, for example. In another embodiment, the left controller 3 and the right controller 4 may be configured not to include some of the above components.

  FIG. 6 is a block diagram illustrating an example of the internal configuration of the main device 2. The main unit 2 includes the components 81 to 98 shown in FIG. 6 in addition to the configuration shown in FIG. Some of these components 81 to 98 may be mounted on the electronic circuit board as electronic components and accommodated in the housing 11.

  The main device 2 includes a CPU (Central Processing Unit) 81. The CPU 81 is an information processing unit that executes various types of information processing executed in the main device 2. The CPU 81 stores an information processing program (for example, a game program) stored in a storage unit (specifically, an internal storage medium such as the flash memory 84 or an external storage medium attached to each of the slots 23 and 24). By executing, various information processing is executed.

  The main device 2 includes a flash memory 84 and a DRAM (Dynamic Random Access Memory) 85 as an example of an internal storage medium built therein. The flash memory 84 and the DRAM 85 are connected to the CPU 81. The flash memory 84 is a memory mainly used for storing various data (may be programs) stored in the main device 2. The DRAM 85 is a memory used for temporarily storing various data used in information processing.

  The main device 2 includes a first slot interface (hereinafter abbreviated as “I / F”) 91. In addition, the main device 2 includes a second slot I / F 92. The first slot I / F 91 and the second slot I / F 92 are connected to the CPU 81. The first slot I / F 91 is connected to the first slot 23, and reads and writes data to and from the first type storage medium (for example, an SD card) attached to the first slot 23 according to instructions from the CPU 81. Do it. The second slot I / F 92 is connected to the second slot 24, and reads and writes data to and from the second type storage medium (for example, a dedicated memory card) attached to the second slot 24 according to instructions from the CPU 81. Do it accordingly.

  The CPU 81 reads and writes data as appropriate between the flash memory 84, the DRAM 85, and each of the storage media, and executes the above information processing.

  The main device 2 includes a network communication unit 82. The network communication unit 82 is connected to the CPU 81. The network communication unit 82 performs communication (specifically, wireless communication) with an external device via a network. In the present embodiment, the network communication unit 82 communicates with an external device by connecting to a wireless LAN by a method based on the Wi-Fi standard as a first communication mode. Moreover, the network communication part 82 performs radio | wireless communication between the other main body apparatuses 2 of the same kind with a predetermined | prescribed communication system (For example, communication by an original protocol, infrared communication) as a 2nd communication mode. The wireless communication according to the second communication mode can be wirelessly communicated with other main body devices 2 arranged in the closed local network area, and directly communicates between the plurality of main body devices 2. This realizes a function that enables so-called “local communication” in which data is transmitted and received.

  The main device 2 includes a controller communication unit 83. The controller communication unit 83 is connected to the CPU 81. The controller communication unit 83 performs wireless communication with the left controller 3 and / or the right controller 4. The communication method between the main device 2 and the left controller 3 and the right controller 4 is arbitrary, but in this embodiment, the controller communication unit 83 communicates with the left controller 3 and between the right controller 4 and the Bluetooth ( Performs communication according to the registered trademark standard.

  The CPU 81 is connected to the left terminal 17, the right terminal 21, and the lower terminal 27 described above. When performing wired communication with the left controller 3, the CPU 81 transmits data to the left controller 3 through the left terminal 17 and receives operation data from the left controller 3 through the left terminal 17. Further, when performing wired communication with the right controller 4, the CPU 81 transmits data to the right controller 4 through the right terminal 21 and receives operation data from the right controller 4 through the right terminal 21. Further, the CPU 81 transmits data to the cradle via the lower terminal 27 when communicating with the cradle. Thus, in the present embodiment, the main device 2 can perform both wired communication and wireless communication with the left controller 3 and the right controller 4, respectively. When the left controller 3 and the right controller 4 are mounted on the main body device 2 or the single body device 2 is mounted on the cradle, the main body device 2 receives data (for example, image data or sound) via the cradle. Data) can be output to a stationary monitor or the like.

  Here, the main device 2 can perform communication simultaneously with the plurality of left controllers 3 (in other words, in parallel). Further, the main body device 2 can perform communication simultaneously with the plurality of right controllers 4 (in other words, in parallel). Therefore, the user can input to the main body device 2 using the plurality of left controllers 3 and the plurality of right controllers 4.

  The main device 2 includes a touch panel controller 86 that is a circuit that controls the touch panel 13. The touch panel controller 86 is connected between the touch panel 13 and the CPU 81. The touch panel controller 86 generates data indicating, for example, a position where touch input is performed based on a signal from the touch panel 13 and outputs the data to the CPU 81.

  The display 12 is connected to the CPU 81. The CPU 81 displays the generated image (for example, by executing the above information processing) and / or the image acquired from the outside on the display 12.

  The main device 2 includes a codec circuit 87 and a speaker (specifically, a left speaker and a right speaker) 88. The codec circuit 87 is connected to the speaker 88 and the audio input / output terminal 25 and to the CPU 81. The codec circuit 87 is a circuit that controls input / output of audio data to / from the speaker 88 and the audio input / output terminal 25. That is, when the codec circuit 87 receives audio data from the CPU 81, the codec circuit 87 outputs an audio signal obtained by performing D / A conversion on the audio data to the speaker 88 or the audio input / output terminal 25. As a result, sound is output from the audio output unit (for example, earphone) connected to the speaker 88 or the audio input / output terminal 25. When the codec circuit 87 receives an audio signal from the audio input / output terminal 25, the codec circuit 87 performs A / D conversion on the audio signal and outputs audio data in a predetermined format to the CPU 81. The volume button 26 is connected to the CPU 81. Based on the input to the volume button 26, the CPU 81 controls the volume output from the speaker 88 or the audio output unit.

  The main device 2 includes a power control unit 97 and a battery 98. The power control unit 97 is connected to the battery 98 and the CPU 81. Although not shown, the power control unit 97 is connected to each part of the main body device 2 (specifically, each part that receives power supply from the battery 98, the left terminal 17, and the right terminal 21). The power control unit 97 controls power supply from the battery 98 to each of the above parts based on a command from the CPU 81. In addition, the power control unit 97 is connected to the power button 28. The power control unit 97 controls power supply to each of the above parts based on an input to the power button 28. That is, when an operation to turn off the power is performed on the power button 28, the power control unit 97 stops the power supply to all or a part of the above parts and turns on the power to the power button 28. When the operation is performed, the power control unit 97 starts supplying power to all or some of the above-described units. Further, the power control unit 97 outputs information indicating an input to the power button 28 (specifically, information indicating whether or not the power button 28 is pressed) to the CPU 81.

  The battery 98 is connected to the lower terminal 27. When an external charging device (for example, a cradle) is connected to the lower terminal 27 and electric power is supplied to the main body device 2 via the lower terminal 27, the supplied electric power is charged to the battery 98.

  The main device 2 also includes a cooling fan 96 for radiating heat inside the main device 2. By operating the cooling fan 96, air outside the housing 11 is introduced from the intake hole 11d, and air inside the housing 11 is released from the exhaust hole 11c, so that heat inside the housing 11 is released. . The cooling fan 96 is connected to the CPU 81, and the operation of the cooling fan 96 is controlled by the CPU 81. In addition, the main device 2 includes a temperature sensor 95 that detects the temperature in the main device 2. The temperature sensor 95 is connected to the CPU 81, and the detection result of the temperature sensor 95 is output to the CPU 81. The CPU 81 controls the operation of the cooling fan 96 based on the detection result of the temperature sensor 95.

  FIG. 7 is a block diagram illustrating an example of an internal configuration of the main body device 2, the left controller 3, and the right controller 4. The details of the internal configuration related to the main unit 2 are shown in FIG. 6 and are not shown in FIG.

  The left controller 3 includes a communication control unit 101 that performs communication with the main device 2. As shown in FIG. 7, the communication control unit 101 is connected to each component including the terminal 42. In the present embodiment, the communication control unit 101 can communicate with the main device 2 by both wired communication via the terminal 42 and wireless communication not via the terminal 42. The communication control unit 101 controls a communication method performed by the left controller 3 for the main body device 2. That is, when the left controller 3 is attached to the main body device 2, the communication control unit 101 communicates with the main body device 2 via the terminal 42. Further, when the left controller 3 is disconnected from the main body device 2, the communication control unit 101 performs wireless communication with the main body device 2 (specifically, the controller communication unit 83). Wireless communication between the controller communication unit 83 and the communication control unit 101 is performed, for example, according to the Bluetooth (registered trademark) standard.

  The left controller 3 includes a memory 102 such as a flash memory. The communication control unit 101 is configured by, for example, a microcomputer (also referred to as a microprocessor), and executes various processes by executing firmware stored in the memory 102.

  The left controller 3 includes buttons 103 (specifically, buttons 33 to 39, 43, 44, and 46). The left controller 3 includes an analog stick 32 (described as “stick” in FIG. 7). Each button 103 and the analog stick 32 repeatedly outputs information related to operations performed on itself to the communication control unit 101 at an appropriate timing.

  The left controller 3 includes an acceleration sensor 104. In the present embodiment, the acceleration sensor 104 detects the magnitude of linear acceleration along predetermined three axis directions (for example, the xyz axis shown in FIG. 4). Note that the acceleration sensor 104 may detect an acceleration in a uniaxial direction or a biaxial direction. The left controller 3 includes an angular velocity sensor 105. In the present embodiment, the angular velocity sensor 105 detects angular velocities about predetermined three axes (for example, the xyz axis shown in FIG. 4). The angular velocity sensor 105 may detect an angular velocity around one axis or around two axes. The acceleration sensor 104 and the angular velocity sensor 105 are each connected to the communication control unit 101. The detection results of the acceleration sensor 104 and the angular velocity sensor 105 are repeatedly output to the communication control unit 101 at appropriate timing.

  The communication control unit 101 receives information related to input (specifically, information related to an operation or a detection result by the sensor) from each input unit (specifically, each button 103, analog stick 32, and each sensor 104 and 105). ) To get. The communication control unit 101 transmits operation data including the acquired information (or information obtained by performing predetermined processing on the acquired information) to the main body device 2. The operation data is repeatedly transmitted at a rate of once every predetermined time. Note that the interval at which the information related to the input is transmitted to the main device 2 may or may not be the same for each input unit.

  By transmitting the operation data to the main device 2, the main device 2 can obtain an input made to the left controller 3. That is, the main body device 2 can determine the operation on each button 103 and the analog stick 32 based on the operation data. Further, the main device 2 can calculate information related to the movement and / or posture of the left controller 3 based on operation data (specifically, detection results of the acceleration sensor 104 and the angular velocity sensor 105).

  The left controller 3 includes a vibrator 107 for notifying the user by vibration. In the present embodiment, the vibrator 107 is controlled by a command from the main device 2. That is, when the communication control unit 101 receives the command from the main device 2, the communication control unit 101 drives the vibrator 107 according to the command. Here, the left controller 3 includes an amplifier 106. When receiving the above command, the communication control unit 101 outputs a control signal corresponding to the command to the amplifier 106. The amplifier 106 amplifies the control signal from the communication control unit 101, generates a drive signal for driving the vibrator 107, and gives the drive signal to the vibrator 107. As a result, the vibrator 107 operates.

  The left controller 3 includes a power supply unit 108. In the present embodiment, the power supply unit 108 includes a battery and a power control circuit. Although not shown, the power control circuit is connected to the battery and is connected to each part of the left controller 3 (specifically, each part that receives power from the battery). The power control circuit controls power supply from the battery to each of the above parts. The battery is connected to the terminal 42. In the present embodiment, when the left controller 3 is attached to the main body device 2, the battery is charged by power supply from the main body device 2 via the terminal 42 under predetermined conditions.

  As illustrated in FIG. 7, the right controller 4 includes a communication control unit 111 that performs communication with the main body device 2. The right controller 4 includes a memory 112 connected to the communication control unit 111. The communication control unit 111 is connected to each component including the terminal 64. The communication control unit 111 and the memory 112 have the same functions as the communication control unit 101 and the memory 102 of the left controller 3. Therefore, the communication control unit 111 communicates with the main body device 2 both in wired communication via the terminal 64 and wireless communication not through the terminal 64 (specifically, communication in accordance with the Bluetooth (registered trademark) standard). Communication is possible, and the right controller 4 controls a communication method performed with respect to the main body device 2.

  The right controller 4 includes input units (specifically, buttons 113, an analog stick 52, an acceleration sensor 114, and an angular velocity sensor 115) similar to the input units of the left controller 3. Each of these input units has the same function as each input unit of the left controller 3 and operates in the same manner.

  The right controller 4 includes a vibrator 117 and an amplifier 116. The vibrator 117 and the amplifier 116 operate in the same manner as the vibrator 107 and the amplifier 106 of the left controller 3. That is, the communication control unit 111 operates the vibrator 117 using the amplifier 116 in accordance with a command from the main body device 2.

  The right controller 4 includes a power supply unit 118. The power supply unit 118 has the same function as the power supply unit 108 of the left controller 3 and operates in the same manner. That is, the power supply unit 118 controls power supply to each unit that receives power from the battery. When the right controller 4 is attached to the main body device 2, the battery is charged by power feeding from the main body device 2 via the terminal 64 under predetermined conditions.

  The right controller 4 includes an NFC communication unit 122 that performs short-range wireless communication based on the NFC standard. The NFC communication unit 122 has a so-called NFC reader / writer function. Here, short-range wireless communication in this specification refers to the other device (here, the device close to the antenna 122a) by radio waves from one device (here, the right controller 4) (for example, by electromagnetic induction). A communication method for generating electromotive force is included. The other device can operate by the generated electromotive force, and may or may not have a power source. The NFC communication unit 122 can communicate with the communication target when the right controller 4 (antenna 122a) and the communication target are close to each other (typically, when the distance between the two is less than ten centimeters). It becomes. The communication target is an arbitrary device capable of short-range wireless communication with the NFC communication unit 122, for example, an NFC tag or a storage medium having an NFC tag function. However, the communication target may be another device having an NFC card emulation function. As an example, in the present embodiment, the NFC communication unit 122 acquires type information, identification information, and the like of the case 200 by performing wireless communication with a communication target (for example, an NFC tag) mounted in the case 200. May be.

  The right controller 4 includes an infrared imaging unit 123. The infrared imaging unit 123 includes an infrared camera that captures an image around the right controller 4. In the present embodiment, the infrared imaging unit 123 is used for imaging an imaging target member arranged inside the extended operation device (for example, the case 200). The main body device 2 and / or the right controller 4 calculates information of the captured imaging target member (for example, average luminance, area, barycentric coordinates, etc. of the imaging target member), and based on the information, The operation content operated is determined. In addition, the infrared imaging unit 123 may perform imaging with ambient light, but in this embodiment, the infrared imaging unit 123 includes an infrared light emitting unit 124 that emits infrared light. For example, the infrared light emitting unit 124 emits infrared rays in synchronization with the timing at which the infrared camera captures an image. Then, the infrared light irradiated by the infrared light emitting unit 124 is reflected by the imaging target member, and the reflected infrared light is received by the infrared camera, whereby an infrared image is acquired. Thereby, a clearer infrared image can be obtained. The infrared imaging unit 123 and the infrared light emitting unit 124 may be provided in the right controller 4 as separate devices, or may be provided in the right controller 4 as a single device provided in the same package. It may be provided. In this embodiment, an infrared imaging unit 123 having an infrared camera is used. However, in other embodiments, a visible light camera (a visible light image sensor is used as an imaging unit) instead of the infrared camera. Camera) may be used.

  The right controller 4 includes a processing unit 121. The processing unit 121 is connected to the communication control unit 111 and is also connected to the NFC communication unit 122. The processing unit 121 executes management processing for the NFC communication unit 122 in response to a command from the main device 2. For example, the processing unit 121 controls the operation of the NFC communication unit 122 in accordance with a command from the main device 2. Further, the processing unit 121 controls the activation of the NFC communication unit 122, and controls the operation (specifically, reading and writing) of the NFC communication unit 122 with respect to a communication target (for example, an NFC tag). In addition, the processing unit 121 receives information to be transmitted to the communication target from the main device 2 via the communication control unit 111 and passes the information to the NFC communication unit 122, or receives information received from the communication target to the NFC communication unit 122. Or transmitted to the main device 2 via the communication control unit 111.

  The processing unit 121 includes a CPU, a memory, and the like, and a predetermined program (for example, image processing and various calculations) stored in a storage device (not shown) (for example, a nonvolatile memory or a hard disk) provided in the right controller 4. On the basis of the application program) for executing the management process on the infrared imaging unit 123 in accordance with a command from the main device 2. For example, the processing unit 121 causes the infrared imaging unit 123 to perform an imaging operation, and acquires and / or calculates information based on the imaging result (information on the captured image or information calculated from the information). Or transmitted to the main device 2 via the communication control unit 111. In addition, the processing unit 121 executes management processing for the infrared light emitting unit 124 in accordance with a command from the main device 2. For example, the processing unit 121 controls the light emission of the infrared light emitting unit 124 in accordance with a command from the main device 2. Note that the memory used when the processing unit 121 performs processing may be provided in the processing unit 121 or the memory 112.

  Next, with reference to FIG. 8, the operation using the case 200 which is an example of the extended operation device will be described. FIG. 8 is a diagram illustrating an example of a state in which the user wears the case 200 and performs a game operation. In this embodiment, it is possible to attach the right controller 4 to the case 200, and processing corresponding to the operation content for the case 200 is executed based on data transmitted from the right controller 4 attached to the case 200. Is done. Here, although details will be described later, the operation content for the case 200 is detected based on a captured image captured by the infrared imaging unit 123. Therefore, the case 200 does not require an electrical configuration such as an electronic circuit that detects the operation content by the user or transmits the detection result to the main body device 2. Therefore, according to the present embodiment, the configuration of the case 200, which is an example of the extended operation device, can be simplified.

  For example, in this embodiment, as shown in FIG. 8, the user uses the case 200 by wearing the case 200 on his back and pulling or loosening a plurality of string members 204 provided in the case 200. Perform the operation that was performed. For example, the user wears one end of the string member 204 on both feet, both hands, and the head (head right and head left). Accordingly, when the user moves the left foot, the string member 204 attached to the left foot is pulled or loosened, and when the user moves the right foot, the string member 204 attached to the right foot is pulled or loosened. Or Further, when the user moves the left hand, the string member 204 attached to the left hand is pulled or loosened, and when the user moves the right hand, the string member 204 attached to the right hand is pulled or loosened. To do. Further, when the user moves his / her head back and forth and right and left, the string members 204 respectively attached to the left and right of the head are pulled or loosened. Here, in the embodiment illustrated in FIG. 8, the user wears a member on the head, and one end of another string member 204 is connected to the lower left portion and the lower right portion of the member. Accordingly, when the user tilts his / her head from side to side, one of the string members 204 is pulled and the other string member 204 is loosened. Thus, the different string members 204 are pulled or loosened depending on the manner in which the user moves both hands, both feet or the head.

  Note that the user's head member illustrated in FIG. 8 may be assembled by bending a cardboard member, as in a case 200 described later. In this case, the user's head member may be commercialized as a set with the case 200 and configured to be assembled by the user. As an example, a member that can be put on a head such as a helmet or a hat may be formed by assembly. Further, the member worn by the user when one end of the string member 204 is connected is not limited to an aspect covering the head, but may be a member gripped by the user's hand or a member wound around the hand, It may be a member to be worn on or a member wound around a foot.

  In response to an operation using such a case 200, the player object PO arranged in the virtual space moves, and an image in the virtual space reflecting the movement of the player object PO is displayed on the display device (for example, the main body device 11). Displayed on the display 12). For example, when the user operates so that the left hand on which the string member 204 is attached is moved from the contracted state to the extended state, the string member 204 attached to the left hand is pulled. By the operation of pulling the string member 204, the string member 204 moves so as to be pulled out from the case main body 201 (see FIG. 9) of the case 200. By detecting such movement of the string member 204 inside the case 200, it is presumed that the user carrying the case 200 has moved the left hand, so that the player object PO also extends the left hand in the virtual space. Operate. That is, when the user operates the part to which the string member 204 is attached, a part of the player object PO corresponding to the part (for example, the same part as the part moved by the user) operates. In addition, in response to an operation using the case 200, not only the player object PO arranged in the virtual space moves, but the player object PO may change in another manner in the virtual space. For example, in accordance with a predetermined operation using the case 200, the display mode of the player object PO may be changed (for example, the color changes, changes to another object, the size changes, or the ability changes).

  Further, the right controller 4 attached to the case 200 is provided with an inertial sensor (acceleration sensor 114 and angular velocity sensor 115), and using the detection result of the inertial sensor, the attitude of the right controller 4 and / or It is possible to calculate the movement (ie, the posture and / or movement of the case 200). In the present embodiment, it is possible to control the movement of the player object PO in accordance with the posture and / or movement of the case 200. For example, when the user carrying the case 200 moves to tilt the entire body to the right, the case 200 itself also moves to tilt to the right. Therefore, by detecting such a movement with the inertial sensor of the right controller 4, It can be inferred that the user carrying the 200 performed an action such that the user fell to the right. When it is estimated that the user has moved to the right, the player object PO also moves to the right in the virtual space. In addition, when the user carrying the case 200 jumps, the case 200 itself moves up and down. Therefore, by detecting such a movement with the inertial sensor of the right controller 4, it is estimated that the user carrying the case 200 has jumped. it can. When it is estimated that the user has jumped, the player object PO also jumps in the virtual space or flies in the air. That is, when the user moves the entire case 200, the entire player object PO moves according to the movement and posture of the entire case 200.

  As described above, the player object PO that moves in response to an operation using the case 200 is displayed on the display device (for example, the display 12 of the main body device 11). Therefore, a user who operates the user carrying the case 200 enjoys the game by looking at the display device on which the player object PO is displayed. The viewpoint for displaying the player object PO on the display device is a virtual space. It may be arranged at any position. As a first example, a virtual space image of the player object PO viewed from behind the player object PO may be displayed on the display device. As a second example, a virtual space image may be displayed on the display device from the first person viewpoint of the player object PO. As a third example, a virtual space image of the player object PO viewed from the front of the player object PO may be displayed on the display device. Here, depending on the viewpoint and line-of-sight direction arranged in the virtual space, it may be difficult for the user to grasp the movement of the player object PO relative to his / her own movement. It is possible to provide a realistic game play by setting the part and the movement direction of the moving player object PO according to the viewpoint and the line-of-sight direction.

  Next, a case 200 that is an example of an extended operation device will be described with reference to FIGS. FIG. 9 is a diagram illustrating an example of the appearance of the case 200. FIG. 10 is a diagram illustrating an example of a state in which the right controller 4 is attached to the controller placement unit 202 of the case 200. FIG. FIG. 11 is a diagram illustrating an example of a state of the right controller 4 that images the imaging target member 205 provided in the case 200. FIG. 12 is a diagram illustrating an example in which the imaging target member 205 slides in accordance with the movement of the string member 204.

  In FIG. 9, the case 200 includes a case main body 201, a controller placement unit 202, a shoulder belt 203, and a string member 204. The case 200 has a shape like a backpack as a whole, and the two shoulder belts 203 are attached to the case main body 201, so that the user can carry the case 200 on the back. The case main body 201 is attached with two shoulder belts 203 and becomes a back side portion when the user carries the case 200 on the back side when the user carries the case 200 on the back, and a front side main body 201a that becomes a portion that contacts the user's back. The rear main body 201b is configured. By combining the front main body 201a and the rear main body 201b, an internal space in which light from the outside is blocked is formed in the case main body 201.

  The front main body 201a is provided with a plurality of (six in FIG. 9) string members 204 that can be pulled out from the case main body 201 or returned to the case main body 201. Specifically, the string member 204 is disposed in a state where one end is drawn out from a plurality of holes formed on the upper surface of the front main body 201a, and the other end is provided inside the front main body 201a. Each of the imaging target members 205 (see FIGS. 11 and 12) is connected to each other. As described with reference to FIG. 8, when the user wears the case 200 on the back, the case 200 is used by connecting one end of the string member 204 to the user's hands, both feet, the head, etc. The operation is performed.

  The rear main body 201b has a controller arrangement portion 202 near the center of the back surface. The controller placement section 202 is formed with an opening that opens the internal space of the case main body 201 formed by the front main body 201a and the rear main body 201b to the back side outside of the rear main body 201b (that is, outside the case main body 201). The As shown in FIG. 10, the opening of the controller arrangement portion 202 has a vertical length that is substantially the same as the thickness of the housing 51 of the right controller 4 (the length in the z-axis direction shown in FIG. 1). The length in the left-right direction is substantially the same as the length in the left-right direction of the housing 51 of the right controller 4 (the maximum length in the x-axis direction shown in FIG. 1). Therefore, when the right controller 4 is placed in the opening of the controller placement portion 202 so that the lower surface of the right controller 4 enters the inside of the case body 201, the opening is closed by the housing 51. It will be in a state that can be removed. A cutout portion Na is formed on the upper surface of the opening of the controller placement portion 202. Accordingly, when the right controller 4 is inserted into the opening so that the y-axis negative direction of the right controller 4 is the front direction of the case 200 with the upper surface of the right controller 4 facing upward, the right controller 4 The operation shaft portion of the analog stick 52 is fitted into the cutout portion Na, and the right controller 4 is placed in the controller placement portion 202 (see the lower diagram of FIG. 10). As described above, when the right controller 4 is arranged in the controller arrangement unit 202 and attached to the case 200, a part of the right controller 4 is included in the internal space of the case 200, and the infrared imaging unit 123 is included in the internal part. The space can be imaged. Note that a part of the right controller 4 illustrated in FIG. 10 is included and mounted in the internal space of the case 200, but all of the right controller 4 may be mounted and mounted in the internal space of the case 200. And only the lower surface (surface in which the window part 68 is provided) may be mounted so as to face the internal space of the case 200.

  Note that the rear main body 201b may be provided with a communication target (for example, an NFC tag) of the NFC communication unit 122 in the vicinity of the controller placement unit 202. For example, in a state where the right controller 4 is arranged in the controller arrangement unit 202, the communication target is arranged at a position close to the antenna 122a of the right controller 4 in the rear main body 201b. As a result, the right controller 4 is mounted on the controller placement unit 202 and at the same time, short-range wireless communication is possible with the communication target. In addition, by storing unique identification information and the like, the reliability of the extended operation device to which the right controller 4 is attached can be confirmed.

  As shown in FIGS. 11 and 12, a plurality of imaging target members 205 are arranged in the internal space of the case main body 201. Each of the plurality of imaging target members 205 is disposed in a slide unit 206 that is slidable in the vertical direction (A direction and B direction shown in FIG. 12). The slide unit 206 includes a partition unit 207 that divides the space in the slide unit 206 into lanes in which the plurality of imaging target members 205 slide in the slide direction. In addition, the slide portion 206 is disposed in the front main body 201a in the internal space of the case main body 201 so that the upper opening of each lane is disposed below the plurality of holes formed on the upper surface of the front main body 201a. It is fixed.

  Each of the imaging target members 205 has a predetermined weight, and is connected to the other end of the string member 204. The string member 204 connected to the imaging target member 205 passes through a hole formed in the upper surface of the front main body 201a formed in the lane of the slide portion 206 and above the lane, and is external to the case main body 201. Has been pulled out. Therefore, when the string member 204 connected to the imaging target member 205 is pulled outside the case body 201 with a force larger than the weight of the imaging member 205, the imaging member 205 is moved along the lane of the slide unit 206. It will rise upward (direction A in FIG. 12). Further, when the string member 204 that raises the imaging member 205 is loosened, the imaging member 205 is lowered downward (direction B in FIG. 12) along the lane of the slide portion 206 by its own weight. As described above, the plurality of imaging target members 205 are slidably moved up and down in the internal space of the case body 201 in accordance with an operation in which the connected string members 204 are pulled or loosened.

  As shown in FIG. 11, the slider unit 206 is located at a position facing the infrared imaging unit 123 of the right controller 4 attached to the controller arrangement unit 202. Therefore, in a state where the right controller 4 is mounted on the controller arrangement unit 202, the infrared imaging unit 123 can image the slider unit 206. Further, in the above state, the infrared light emitting unit 124 can irradiate the slider unit 206 with infrared light. Accordingly, the internal space of the case main body 201 is in a state in which external light is blocked, but since the infrared light is applied to the slider unit 206 by the infrared light emitting unit 124, the infrared imaging unit 123 uses the slider. The part 206 can be imaged.

  The slider portion 206 has a plurality of slits formed on the surface (the rear surface) facing the right controller 4 mounted on the controller arrangement portion 202. The slit is formed for each lane with the sliding direction as the major axis direction so that the position of each imaging target member 205 disposed on the slider unit 206 can be seen when viewed from the right controller 4 side. Therefore, the captured image captured by the infrared imaging unit 123 can recognize the position of each imaging target member 205 in the slider unit 206 through the slit.

  At least a part of the imaging target member 205 (typically, a part including at least a part visible from the infrared imaging unit 123 through the slit) is distinguished from other parts in the captured image captured by the infrared imaging unit 123. It is comprised by the recognition part 205a comprised with a possible material. For example, the recognition unit 205a may be made of a material having a retroreflection characteristic. Accordingly, more infrared light emitted from the infrared light emitting unit 124 is reflected, so that the recognition unit 205a, that is, the imaging target member 205 can be more easily recognized in the captured image captured by the infrared imaging unit 123.

  In the above-described embodiment, the example in which the operation on the case 200 is performed by pulling or loosening the string member 204 provided in the case 200 is used. However, the imaging in the case main body 201 is performed by another operation method. Operation means for moving the target member 205 may be included. For example, a part of the rod-shaped member inserted into the case 200 may be used as an imaging target member, and the position of the imaging target member and whether the imaging target member exists in the case 200 may be detected. In this case, the operation on the case 200 is performed by changing the depth at which the rod-shaped member inserted into the case 200 is inserted, pulling out the rod-shaped member, or returning the rod-shaped member.

  Next, with reference to FIG. 13 and FIG. 14, information of the imaging target member calculated from the captured image captured by the infrared imaging unit 123 will be described. FIG. 13 is a diagram for explaining an example of information on the imaging target member calculated in the first mode (clustering mode). FIG. 14 is a diagram for describing an example of information on the imaging target member calculated in the second mode (moment mode).

  In FIG. 13, in the first mode, a cluster (cluster) of high-luminance pixels is detected from the captured image captured by the infrared imaging unit 123, and the position and size of the cluster in the captured image are calculated. For example, as shown in FIG. 13, a predetermined coordinate axis (for example, for an entire captured image captured by the infrared imaging unit 123 or a partial region (an analysis window described later) of the captured image for which information is calculated from the captured image. , A coordinate axis with the horizontal direction as the X axis and the vertical direction as the Y axis). Then, a pixel group adjacent to pixels having a luminance equal to or higher than a predetermined threshold is extracted from the image in which the coordinate axis is set, and the pixel group has a size within a predetermined range (first threshold or higher and first If the size is equal to or smaller than 2 thresholds, it is determined to be a cluster. For example, in the case of a captured image obtained by capturing the slider unit 206 in the case 200, the recognition unit 205a that reflects more infrared light emitted from the infrared light emitting unit 124 is extracted as a high-luminance pixel. Each of the recognition units 205a imaged via is determined as a cluster. In the first mode, the total number of clusters extracted in this way is calculated, and the average luminance, area (number of pixels), barycentric coordinates, and circumscribed rectangle of each cluster are calculated. Calculated as information on the imaging target member.

  In the first mode, when calculating the position of each cluster (the center of gravity coordinates or the position of the circumscribed rectangle), the reference position set in the captured image may be used as a reference. As an example, the reference position may be set to a position of a predetermined object to be imaged captured in the captured image. Specifically, apart from the above-described imaging target member 205 (recognition unit 205a), a reference marker configured in a predetermined shape with a material having retroreflective properties is set near the slider unit 206 (infrared imaging unit). The position of the reference marker is set to a predetermined position, and the position of the reference marker functions as the reference position. Then, when the captured image of the infrared imaging unit 123 is analyzed, the position where the reference marker is captured from the captured image is calculated by performing image processing using pattern matching on the captured image. To do. By calculating the position of each cluster based on the imaging position of such a reference marker, the position, orientation, shape, etc. of each cluster can be detected with high accuracy. In addition, by using the positional relationship between the reference marker captured in the captured image and the other imaging target member 205 and the shape in which the reference marker is captured, it becomes easy to identify each imaging target member 205. At the same time, by making the positional relationship and the shape of the reference marker unique to the extended operation device, it is possible to determine the type of the extended operation device from the captured image of the infrared imaging unit 123.

  In FIG. 14, in the second mode, the captured image captured by the infrared imaging unit 123 is divided into a plurality of blocks, and the average luminance and the barycentric position for each block are calculated. For example, as illustrated in FIG. 14, a predetermined coordinate axis (for example, the horizontal direction) is set on the entire captured image captured by the infrared imaging unit 123 or on a partial region (analysis window described later) of the captured image for which information is calculated from the captured image. Is set as the X axis and the vertical direction is the Y axis), and the image in which the coordinate axis is set is divided into predetermined blocks (for example, 8 × 6 48 blocks). Then, for pixels having a luminance equal to or higher than a predetermined threshold in each block, the total luminance of each block, the horizontal moment of each block, and the vertical moment of each block are calculated. The calculation result is calculated as information on the imaging target member. For example, the sum of the brightness of each block is a value obtained by summing the brightness of the pixels in each block for all the pixels, and a parameter for obtaining the average brightness of the block by dividing the sum by the number of pixels of the block. It is. Further, the horizontal moment of each block is a value obtained by multiplying the luminance of the pixels in each block by the X-axis direction position of the pixel for all pixels, and the horizontal moment of the block. Is a parameter for obtaining the horizontal barycentric position of the luminance in the block by dividing by the sum of the luminance of the block. The vertical moment of each block is a value obtained by multiplying the luminance of the pixels in each block by the Y-axis direction position of the pixel for all pixels, and the vertical moment of the block. Is a parameter by which the vertical center-of-gravity position of the luminance in the block is obtained by dividing by the sum of the luminance of the block.

  The information on the imaging target member in the first mode and the information on the imaging target member in the second mode are calculated in the right controller 4 and transmitted to the main body device 2. Therefore, as compared with the case where data indicating the captured image itself captured by the infrared imaging unit 123 is transmitted, the amount of data transmitted to the main device 2 can be reduced, and the processing load on the main device 2 side is also reduced. be able to. In addition, since the data amount of information on the imaging target member is small, it is possible to transmit the information on the imaging target member to the main body device 2 by including it in other operation data transmitted from the right controller 4. The communication processing itself between the computer and the main device 2 can be simply configured.

  Next, an assembling method of the case 200 will be described with reference to FIGS. FIG. 15 is a diagram illustrating an example of a combination of main components when the case 200 is assembled. FIG. 16 is a diagram illustrating an example of a cardboard member for assembling the front main body 201a. FIG. 17 is a diagram illustrating an example of a cardboard member for assembling the rear main body 201b. 16 and 17, the solid line indicates a line cut from the cardboard member, the broken line indicates a line that folds the cardboard member, and the alternate long and short dash line indicates a line that folds the cardboard member.

  In FIG. 15, each component in the case 200 is configured by bending a cardboard member. For example, the slide unit 206 is a partition unit 207 in a state where a plurality of imaging target members 205 connected to the string member 204 are arranged in each lane (the string member 204 and the imaging target member 205 are not shown in FIG. 15). Is assembled into the rectangular tube member 208. Specifically, the partition unit 207 is assembled by inserting a plurality of partition plate members into a holding plate member in which a cut for each lane is generated, and each plate member is configured by a cardboard member. Further, the square tube member 208 is assembled by bending the plate material in which the slit is formed into a square tube shape having an open sliding direction, and the plate material is configured by a cardboard member.

  The cardboard member in the present embodiment may be a paperboard having a laminated structure or a member obtained by bonding a plurality of paperboards. As an example, a so-called corrugated cardboard member may be used in which paperboard is bonded to one or both sides of a corrugated paperboard.

  Further, the front main body 201a and the rear main body 201b install the slide portion 206 by fitting the four insertion pieces Pb of the rear main body 201b into the four insertion holes Hc formed in the front main body 201a. The previous case body 201 is assembled.

  As shown in FIG. 16, the front main body 201a includes a plurality of through holes Ha for penetrating the string member 204, four mounting holes Hb for mounting the shoulder belt 203, and four of the rear main body 201b. It is assembled by bending the plate material on which the four insertion holes Hc for inserting the insertion pieces Pb and the insertion pieces Pa for inserting into the insertion holes Hf formed in the rear main body 201b are bent. Specifically, the plate member is constituted by a cardboard member, and the front main body 201a is assembled by bending the cardboard member into a substantially hexahedron shape with a part of the rear surface opened.

  As shown in FIG. 17, the rear main body 201b includes four insertion pieces Pb for inserting into four insertion holes Hc formed in the front main body 201a, an insertion hole Hf for inserting the insertion piece Pa included in the front main body 201a, Assembling by bending the plate material in which the insertion hole Hd for inserting the controller arrangement portion 202 and the fixing hole He for penetrating and fixing a part of the right controller 4 attached to the case 200 are formed. It is done. Specifically, the plate material is constituted by a cardboard member. A part of the cardboard member is formed with a cutout region that forms the controller placement unit 202 by cutting out and bending the cardboard member. By penetrating the insertion hole Hd, the controller arrangement portion 202 is formed in the rear main body 201b. Then, the rear main body 201b is assembled by folding the cardboard member into a substantially hexahedron shape with the controller arrangement portion 202 formed.

  The shoulder belt 203 (the shoulder belt 203 is not shown in FIG. 15) is attached to the four attachment holes Hb of the case body 201 assembled in this way. Then, in a state where the upper surface of the case main body 201 is opened, the slide portion 206 is installed in the internal space on the front side of the case main body 201. The upper surface of the case main body 201 is fitted so that the insertion piece Pa included in the front main body 201a is fitted in the insertion hole Hf formed in the rear main body 201b in a state where the string members 204 are inserted in the through holes Ha. The case 200 is assembled by closing.

  As described above, the case 200 described above is configured by combining the components assembled by bending the cardboard member, and thus realizes an expansion operation device in which the user assembles a plate-shaped member group as a product form. be able to. In addition, as described above, the extended operation device does not require an electrical configuration such as an electronic circuit for detecting the operation content by the user and transmitting the detection result to the main body device 2. It can be realized as a product.

  In addition, about the joining method of the surfaces at the time of forming the polyhedron by bending the cardboard member described above, any method may be used. For example, the bonding sides of the surfaces may be bonded with an adhesive tape, the insertion side and the insertion hole may be formed in the bonding side, and the bonding sides may be bonded by the insertion. May be joined with an adhesive. Moreover, although the example which each part which comprises the case 200 comprises a cardboard member was used, at least one part of each of these parts may be a thin paper member and the board | plate material of another material. For example, at least a part of each of the above components may be assembled by bending a thin plate material or a thick plate material made of resin, wood, or metal, and the shoulder belt 203 may be made of a belt-like fiber material. Moreover, parts of other materials may be assembled to some of the above parts. For example, since it is considered that the hole through which the string member 204 is inserted is scraped by the movement of the string member 204, it is considered that a ring member made of resin or the like is fitted in a portion that contacts the string member 204 in the hole. It is done. Further, at least a part of each of the above components may be formed in advance as a polyhedral component.

  In addition, the case 200 is used as an example of the extended operation device according to the present embodiment. However, the extended operation device may be an operation device according to another aspect. Hereinafter, with reference to FIGS. 18 and 19, an operation using the control device 250 as another example of the extended operation device will be described. FIG. 18 is a diagram illustrating an example of a state in which a user performs a game operation using the control device 250. FIG. 19 is a diagram illustrating an example of the internal structure of the control device 250. The right controller 4 can also be attached to the control device 250, and processing corresponding to the operation content on the control device 250 is executed based on data transmitted from the right controller 4 attached to the control device 250. . Here, the operation content for the control device 250 is also detected based on the captured image captured by the infrared imaging unit 123. Therefore, the control device 250 also does not require an electrical configuration such as an electronic circuit for detecting the operation content by the user or transmitting the detection result to the main body device 2. Therefore, the configuration of the control device 250, which is another example of the extended operation device, can be simplified.

  As shown in FIG. 18, the user performs an operation using the control device 250 by riding on the control device 250 and operating an operation pedal 252 and an operation rod 253 provided in the control device 250. In the example shown in FIG. 18, the control device 250 includes a plurality of operation pedals 252 and a plurality of operation rods 253 on the device main body 251.

  For example, the user sits on the apparatus main body 251 and operates by stepping on a plurality of operation pedals 252 installed on the upper surface of the apparatus main body 251 with both feet. The operation pedal 252 is urged in the direction in which the pedal surface that the user steps on rises, and when the user steps on the pedal surface with a force larger than the urging force, the pedal surface is stepped toward the upper surface of the apparatus main body 251. Be lowered. In addition, the user operates a plurality of operation rods 253 installed on the upper surface of the apparatus main body 251 with both hands. Specifically, the user operates by tilting the operating rod 253 forward, backward, left, and right around the support portion where the operating rod 253 is supported by the apparatus main body 251. Further, the user operates by holding the operation levers 254 provided near the upper end of the operation rod 253 with both hands. The operation lever 254 is urged in a direction in which the opening angle of the operation lever 254 increases. When the user grips the operation lever 254 with a force larger than the urging force, the opening angle of the operation lever 254 decreases. The right controller 4 is attached to a controller arrangement portion provided in the apparatus main body 251 in order to detect the operation contents of the operation pedal 252 and the operation rod 253.

  As shown in FIG. 19, when the right controller 4 is attached to a controller arrangement portion provided in the apparatus main body 251, a part of the right controller 4 is included in the internal space of the apparatus main body 251 and an infrared imaging unit. 123 becomes a state in which the inside space can be imaged. The imaging target members 255 to 257 are arranged in the imaging range of the infrared imaging unit 123 in the internal space of the apparatus main body 251 and in the infrared light irradiation range of the infrared light emitting unit 124.

  A plurality of imaging target members 255 are provided in the internal space of the apparatus main body 251 corresponding to each of the plurality of operation pedals 252. The imaging target member 255 is connected to the other end of the string member 252a whose one end is connected to the pedal surface of the operation pedal 252. As shown in FIG. 19, the string member 252a moves up and down in accordance with the upward / downward movement of the pedal surface of the operation pedal 252, so that the operation target 255 is also moved in accordance with the vertical movement of the string member 252a. Move in the interior space of the. Therefore, the operation content for the operation pedal 252 can be grasped based on the position of the imaging target member 255 calculated using the captured image captured by the infrared imaging unit 123.

  The imaging target member 256 is provided in the vicinity of the lower ends of the plurality of operation rods 253 protruding into the internal space of the apparatus main body 251. As shown in FIG. 19, when the operation rod 253 tilts and moves around the support portion on the upper surface of the apparatus main body 251 on which the operation rod 253 is supported, the lower end portion of the operation rod 253 is also inside the device main body 251. Tilt and move in space. Accordingly, the imaging target member 256 provided at the lower end of the operation rod 253 also moves in the internal space of the apparatus main body 251. Therefore, it is possible to grasp the tilting operation content with respect to the operating rod 253 based on the position of the imaging target member 256 calculated using the captured image captured by the infrared imaging unit 123. Note that when the operating rod 253 tilts back and forth with respect to the imaging direction of the infrared imaging unit 123, the imaging target member 256 also moves back and forth with respect to the infrared imaging unit 123. In this case, in the captured image captured by the infrared imaging unit 123, by calculating a change in luminance of the captured imaging target member 256 and a change in size of the captured imaging target member 256, the infrared imaging unit 123 It is possible to calculate the front-rear position of the imaging target member 256.

  A plurality of imaging target members 257 are provided in the internal space of the apparatus main body 251 corresponding to each of the plurality of operation levers 254. The imaging target member 257 is connected to the other end of the string member 254a whose one end is connected to the operation lever 257, respectively. Since the string member 254a moves up and down according to the opening angle of the operation lever 254, the operation target 257 also moves within the internal space of the apparatus main body 251 according to the vertical movement of the string member 254a. Therefore, the operation content for the operation lever 257 can be grasped based on the position of the imaging target member 257 calculated using the captured image captured by the infrared imaging unit 123.

  The player object PO arranged in the virtual space moves in response to an operation using such a control device 250, and an image of the virtual space reflecting the movement of the player object PO is displayed on the display device (for example, the main body device 11). Display 12). For example, when the user performs an operation of stepping on the operation pedal 252, the speed at which the player object PO moves is increased or decreased. In addition, when the user performs an operation of tilting the operation rod 253 back and forth, left and right, the moving direction of the player object PO changes according to the tilt direction of each operation rod 253. In addition, when the user performs an operation of gripping the operation lever 254, the player object PO performs a predetermined action (for example, attack or defense).

  In the above-described embodiment, the operation rod 253 is provided as one of the operation means provided in the control device 250, but the operation rod 253 is an operation unit capable of indicating a direction such as an analog stick. Also good. For example, an imaging target member that moves in conjunction with the tilting operation of the analog stick is provided in the space below the analog stick, and the movement of the imaging target member is imaged by the infrared imaging unit 123, whereby the tilting direction of the analog stick or The tilt amount can also be calculated in the same manner as the tilt direction and tilt angle of the operating rod 253.

  Next, an example of specific processing executed by the right controller 4 in the present embodiment will be described with reference to FIGS. FIG. 20 is a diagram illustrating an example of a data area set in a memory (for example, a memory included in the processing unit 121) of the right controller 4 in the present embodiment. The memory of the right controller 4 stores data used in other processes in addition to the data shown in FIG. 20, but detailed description thereof is omitted. Further, in the information calculation process executed by the right controller 4 to be described later, an example in which information on the imaging target member is calculated in the first mode (clustering mode) is used.

  Various programs Ps executed by the processing unit 121 are stored in the program storage area of the memory of the right controller 4. In the present embodiment, the various programs Ps store an information calculation program for acquiring and calculating information based on the imaging result of the captured image described above, a program for transmitting the calculated information to the main device 2, and the like. The The various programs Ps may be stored in the right controller 4 in advance, or may be a storage medium that can be attached to and detached from the main unit 2 (for example, a first type storage medium that is installed in the first slot 23, a first storage medium, 2nd type storage medium mounted in the two slots 24) and may be stored in the memory of the right controller 4 or acquired from another device via a network such as the Internet. It may be stored in a memory. The processing unit 121 executes various programs Ps stored in the memory of the right controller 4.

  Also, various data used in the processing executed by the processing unit 121 is stored in the data storage area of the memory of the right controller 4. In the present embodiment, the memory of the right controller 4 stores captured image data Dsa, analysis window data Dsb, extracted pixel data Dsc, extracted pixel group data Dsd, cluster data Dse, cluster number data Dsf, and the like.

  The captured image data Dsa is data indicating a captured image captured by the infrared imaging unit 123. In the present embodiment, the infrared imaging unit 123 performs an imaging process at a period based on an instruction from the processing unit 121, and the captured image data Dsa is appropriately updated using the captured image. Note that the update cycle of the captured image data Dsa may be updated for each cycle of processing executed in the processing unit 123 described below and / or processing executed in the CPU 81, or for each cycle in which the captured image is captured. May be updated.

  The analysis window data Dsb is data indicating an image in a range (analysis window) to be analyzed in order to calculate information based on the imaging result among the captured images stored in the captured image data Dsa.

  The extracted pixel data Dsc is data indicating pixels having a luminance equal to or higher than a predetermined threshold extracted from the image in the analysis window. The extracted pixel group data Dsd is data indicating a pixel group adjacent to the extracted pixel in the image in the analysis window.

  The cluster data Dse is data indicating information for each cluster determined in the image in the analysis window, and includes average luminance data Dse1, area data Dse2, barycentric coordinate data Dse3, circumscribed rectangle data Dse4, and the like. The average luminance data Dse1 is data indicating the average luminance for each cluster. The area data Dse2 is data indicating the area (number of pixels) for each cluster. The barycentric coordinate data Dse3 is data indicating barycentric coordinates for each cluster. The circumscribed rectangle data Dse4 is data indicating the position and size of a rectangle circumscribed in a predetermined direction for each cluster.

  The cluster number data Dsf is data indicating the number of clusters extracted from the image in the analysis window.

  Next, a detailed example of processing for calculating information based on the imaging result of a captured image in the present embodiment (hereinafter simply referred to as information calculation processing) will be described. FIG. 21 is a flowchart illustrating an example of information calculation processing executed by the CPU of the processing unit 121. In the present embodiment, the series of processing illustrated in FIG. 21 is performed by the CPU of the processing unit 121 executing an information calculation program included in various programs Ps. Moreover, the timing at which the information calculation process shown in FIG. 21 is started is arbitrary.

  Note that the processing of each step in the flowchart shown in FIG. 21 is merely an example, and if the same result is obtained, the processing order of each step may be changed, and in addition to the processing of each step ( Alternatively, another process may be performed. In the present embodiment, the processing of each step in the flowchart is described as being executed by the CPU of the processing unit 121. However, the processing of some of the steps in the flowchart may be performed by a processor other than the CPU of the processing unit 121 or a dedicated process. The circuit may execute. A part of the processing executed in the right controller 4 is executed by another information processing device (for example, the main device 2 or a server that can communicate with the main device 2 via a network) that can communicate with the right controller 4. May be. That is, each process shown in FIG. 21 may be executed by cooperation of a plurality of information processing apparatuses including the right controller 4.

  In FIG. 21, the CPU of the processing unit 121 performs initial setting in the information calculation process (step S141), and proceeds to the next step. For example, in the above initial setting, the CPU of the processing unit 121 initializes parameters for performing the processing described below. In the initial setting, the CPU of the processing unit 121 sets the type and identification information of the extended operation device to which the right controller 4 is attached as necessary. For example, the type of the extended operation device may be initialized by the user performing an operation of selecting the type of the extended operation device using the main body device 2, the left controller 3, or the right controller 4. As another example, when the right controller 4 can perform short-range wireless communication with the communication target provided in the extended operation device via the NFC communication unit 122, the type or identification of the extended operation device is determined from the communication target. By acquiring the information, the type and identification information of the extended operation device may be initialized. Furthermore, when the above-described reference marker is captured in the captured image of the infrared imaging unit 123, the extended operation device is based on the imaging shape of the reference marker and the imaging positional relationship between the reference marker and another imaging target member. May be initialized. In the initial setting, the CPU of the processing unit 121 initially sets a range (analysis window) to be analyzed in order to calculate information based on the imaging result in the captured image. The range of the analysis window may be set based on an instruction from the main body device 2 or may be determined in advance according to the type of the extended operation device. A plurality of analysis window ranges may be set. In this case, information based on the imaging result is calculated for each analysis window that is sequentially different in time series.

  Next, the CPU of the processing unit 121 acquires the captured image captured by the infrared imaging unit 123 (step S142), and proceeds to the next step. For example, the CPU of the processing unit 121 acquires a captured image from the infrared imaging unit 123 and updates the captured image data Dsa.

  Next, the CPU of the processing unit 121 sets an analysis window (step S143), and proceeds to the next step. For example, the CPU of the processing unit 121 cuts out the image in the analysis window from the captured image stored in the captured image data Dsa based on the initially set analysis window range, and uses the extracted image. The analysis window data Dsb is updated.

  Next, the CPU of the processing unit 121 extracts pixels having a luminance equal to or higher than a predetermined threshold from the image in the analysis window (step S144), and proceeds to the next step. For example, the CPU of the processing unit 121 extracts a pixel having a luminance equal to or higher than a predetermined threshold as an extraction pixel in the image stored in the analysis window data Dsb, and updates the extraction pixel data Dsc using the extraction pixel. .

  Next, the CPU of the processing unit 121 extracts an extracted pixel group to which the extracted pixels extracted in step S144 are adjacent from the image in the analysis window (step S145), and proceeds to the next step. For example, the CPU of the processing unit 121 processes one of the extracted pixel groups that have not been subjected to the processing of steps S146 to S150 described later from among the extracted pixel groups extracted from the image in the analysis window. The extracted pixel group data Dsd is updated using the data indicating the extracted pixel group.

  Next, the CPU of the processing unit 121 calculates the size S of the extracted pixel group to be processed (step S146), and proceeds to the next step. For example, the CPU of the processing unit 121 calculates the size S (for example, the number of pixels S) of the extracted pixel group indicated by the extracted pixel group data Dsd.

  Next, the CPU of the processing unit 121 determines whether or not the size S calculated in step S146 is equal to or larger than the first threshold T1 and equal to or smaller than the second threshold T2 (T2> T1) (step S147). ). Here, the first threshold value T1 is a value indicating a minimum size that is recognized as a cluster (cluster) of high-luminance pixels in which the imaging target member is imaged in the image in the analysis window. The second threshold value T2 is a value indicating the maximum size that is recognized as a cluster in which the imaging target member is imaged in the image in the analysis window. Then, when the size S is equal to or larger than the first threshold value T1 and equal to or smaller than the second threshold value T2, the CPU of the processing unit 121 advances the process to step S148. On the other hand, if the size S is not less than the first threshold value T1 and not more than the second threshold value T2, the CPU of the processing unit 121 advances the process to step S150.

  In step S <b> 148, the CPU of the processing unit 121 sets the extracted pixel group to be processed as a cluster. Then, the CPU of the processing unit 121 calculates cluster information related to the cluster set in step S148 (step S149), and proceeds to step S151. For example, in step S149, the CPU of the processing unit 121 calculates the average luminance of the extracted pixel group indicated by the extracted pixel group data Dsd, and uses the calculated average luminance to calculate the cluster set in step S148. The average luminance data Dse1 in the corresponding cluster data Dse is updated. Further, the CPU of the processing unit 121 updates the area data Dse2 in the cluster data Dse corresponding to the cluster set in step S148, using the size S calculated in step S146. Further, the CPU of the processing unit 121 calculates the centroid position of the extracted pixel group indicated by the extracted pixel group data Dsd, and the coordinate value indicating the centroid position in the image in the analysis window (for example, the XY coordinates exemplified in FIG. 13). Value) is used to update the barycentric coordinate data Dse3 in the cluster data Dse corresponding to the cluster set in step S148. Further, the CPU of the processing unit 121 circumscribes the extracted pixel group indicated by the extracted pixel group data Dsd and is in a predetermined direction with respect to the image in the analysis window (for example, parallel to the X axis or the Y axis illustrated in FIG. 13). The cluster set in step S148 is set using each coordinate value (for example, the XY coordinate values illustrated in FIG. 13) indicating the positions of two vertices facing each other in the rectangle. The circumscribed rectangle data Dse4 in the cluster data Dse corresponding to is updated. When the above-described reference marker is captured in the captured image of the infrared imaging unit 123, the position of the center of gravity of the extracted pixel group and the position of the circumscribed rectangle are calculated based on the position where the reference marker is captured. It doesn't matter.

  On the other hand, in step S150, the CPU of the processing unit 121 proceeds to step S151 as it is without setting the extraction pixel group to be processed as a cluster.

  In step S151, the CPU of the processing unit 121 determines whether or not an extracted pixel group that has not been subjected to the processes in steps S145 to S150 remains in the image in the analysis window. When the unprocessed pixel group remains, the CPU of the processing unit 121 returns to step S145 and repeats the process. On the other hand, if no unprocessed pixel group remains, the CPU of the processing unit 121 deletes the cluster data Dse whose cluster information has not been updated in step S149, and advances the process to step S152.

  In step S152, the CPU of the processing unit 121 calculates the number of clusters set from the image in the analysis window in step S148, and proceeds to the next step. For example, the CPU of the processing unit 121 updates the cluster number data Dsf using the number of clusters set from the image in the analysis window.

  Next, the CPU of the processing unit 121 performs processing for transmitting data indicating information based on the imaging result of the captured image to the main device 2 (step S153), and proceeds to the next step. For example, the CPU of the processing unit 121 uses the cluster data Dse and the cluster number data Dsf to generate data to be transmitted to the main device 2 and outputs the data to the communication control unit 111. Thereby, the communication control unit 111 generates information based on the imaging result of the captured image together with other operation data (button operation data, acceleration data, angular velocity data, etc.) as operation data, and the operation data is generated at predetermined transmission cycles. To the main unit 2.

  Next, the CPU of the processing unit 121 determines whether or not to end the information calculation process (step S154). The conditions for ending the information calculation process in step S154 include, for example, receiving an instruction to end the information calculation process from the main device 2, or performing an operation for ending the information calculation process by the user. The CPU of the processing unit 121 returns to step S142 when the information calculation process is not ended, repeats the process, and ends the process according to the flowchart when the information calculation process is ended. Thereafter, the series of processing from step S142 to step S154 is repeatedly executed until it is determined in step S154 that the information calculation processing is to be ended.

  Next, with reference to FIG. 22 and FIG. 23, an example of specific processing executed by the main device 2 in the present embodiment will be described. FIG. 22 is a diagram illustrating an example of a data area set in the DRAM 85 of the main device 2 in the present embodiment. The DRAM 85 stores data used in other processes in addition to the data shown in FIG. 22, but detailed description thereof is omitted. Further, in the information processing executed by the main body device 2 to be described later, an example in which information on the imaging target member calculated in the first mode (clustering mode) is used is used.

  Various programs Pa executed by the main device 2 are stored in the program storage area of the DRAM 85. In the present embodiment, the various programs Pa are based on the above-described communication program for wireless communication with the left controller 3 and / or the right controller 4 and data acquired from the left controller 3 and / or the right controller 4. An application program for performing information processing (for example, game processing) is stored. Note that the various programs Pa may be stored in advance in the flash memory 84, or a storage medium that can be attached to and detached from the main body device 2 (for example, a first type storage medium installed in the first slot 23, a second type The second type storage medium mounted in the slot 24) may be acquired and stored in the DRAM 85, or may be acquired from another device via a network such as the Internet and stored in the DRAM 85. The CPU 81 executes various programs Pa stored in the DRAM 85.

  In the data storage area of the DRAM 85, various data used in processing such as communication processing and information processing executed in the main device 2 are stored. In the present embodiment, the DRAM 85 stores operation data Da, operation content data Db, posture data Dc, object motion data Dd, image data De, and the like.

  The operation data Da is operation data appropriately acquired from the right controller 4. As described above, the operation data transmitted from the right controller 4 includes information related to input from each input unit (specifically, each button, analog stick, and each sensor) (specifically, information related to the operation, Or the detection result by a sensor) and the information based on the imaging result of the captured image of the infrared imaging unit 123 are included. In the present embodiment, operation data is transmitted from the right controller 4 at a predetermined cycle by wireless communication, and the operation data Da is appropriately updated using the received operation data. The update cycle of the operation data Da may be updated every frame, which is a cycle of processing executed by the main body device 2 described later, or updated every cycle when the operation data is transmitted by the wireless communication. May be. The operation data Da includes button operation data Da1, cluster data Da2, cluster number data Da3, angular velocity data Da4, and acceleration data Da5. The button operation data Da1 is data indicating information related to input from each button and analog stick of the right controller 4. The cluster data Da2 is data indicating information for each cluster calculated by the processing unit 121 of the right controller 4. The cluster number data Da3 is data indicating the number of clusters extracted by the processing unit 121 of the right controller 4. The angular velocity data Da4 is data indicating information regarding the angular velocity generated in the right controller 4 detected by the angular velocity sensor 115 of the right controller 4. For example, the angular velocity data Da4 includes data indicating the angular velocity around the xyz axis generated in the right controller 4. The acceleration data Da5 is data indicating information related to acceleration generated in the right controller 4 detected by the acceleration sensor 114 of the right controller 4. For example, the acceleration data Da5 includes data indicating the acceleration in the xyz axis direction generated in the right controller 4.

  The operation content data Db is data indicating the operation content using the case 200 calculated based on the cluster data Da2 and the cluster number data Da3.

  The posture data Dc is data indicating the posture of the right controller 4 with reference to the direction of gravity acceleration in real space. For example, the posture data Dc includes data indicating the direction of gravitational acceleration acting on the right controller 4, data indicating the xyz-axis direction with respect to the gravitational acceleration direction, and the like.

  The object motion data Dd is data related to the motion of the player object placed in the virtual world. The image data De is data for displaying an image (for example, an image of a player object, an image of another object, a background image, etc.) on a display device (for example, the display 12 of the main device 2) during a game.

  Next, a detailed example of information processing (for example, game processing) in the present embodiment will be described. FIG. 23 is a flowchart illustrating an example of information processing executed by the main device 2. In the present embodiment, the series of processing shown in FIG. 23 is performed by the CPU 81 executing a communication program or a predetermined application program (for example, a game program) included in various programs Pa. Also, the timing at which the information processing shown in FIG. 23 is started is arbitrary.

  Note that the processing of each step in the flowchart shown in FIG. 23 is merely an example, and if the same result is obtained, the processing order of each step may be changed, and in addition to the processing of each step ( Alternatively, another process may be performed. In the present embodiment, the processing of each step in the flowchart is described as being executed by the CPU 81. However, the processing of some steps in the flowchart may be executed by a processor or a dedicated circuit other than the CPU 81. Good. A part of the processing executed in the main device 2 may be executed by another information processing device that can communicate with the main device 2 (for example, a server that can communicate with the main device 2 via a network). That is, each process illustrated in FIG. 23 may be executed by cooperation of a plurality of information processing apparatuses including the main body apparatus 2.

  In FIG. 23, the CPU 81 performs initial setting in the game process (step S161), and proceeds to the next step. For example, in the above initial setting, the CPU 81 initializes parameters for performing processing described below. In the initial setting, the CPU 81 sets the type and identification information of the extended operation device to which the right controller 4 is attached as necessary. For example, the type of the extended operation device may be initialized by the user performing an operation of selecting the type of the extended operation device using the main body device 2, the left controller 3, or the right controller 4. As another example, when the right controller 4 can perform short-range wireless communication with the communication target provided in the extended operation device via the NFC communication unit 122, the type or identification of the extended operation device is determined from the communication target. By acquiring the information, the type and identification information of the extended operation device may be initialized. Further, when the above-described reference marker is captured in the captured image of the infrared imaging unit 123, the expansion specified based on the imaging shape of the reference marker and the imaging positional relationship between the reference marker and another imaging target member The type of the extended operation device may be initially set by acquiring the type of the operation device. In the initial setting, the CPU 81 initially sets a range (analysis window) to be analyzed in order to calculate information based on the imaging result among the captured images of the infrared imaging unit 123, and sets the setting to the right controller 4. The contents may be indicated.

  Next, the CPU 81 acquires operation data from the right controller 4 and updates the operation data Da (step S162), and proceeds to the next step. For example, the CPU 81 updates the button operation data Da1, the cluster data Da2, the cluster number data Da3, the angular velocity data Da4, and the acceleration data Da5 according to the operation data acquired from the right controller 4.

  Next, the CPU 81 calculates the operation content using the case 200 based on the barycentric coordinates for each cluster extracted from the captured image of the infrared imaging unit 123 (step S163), and proceeds to the next step. For example, the CPU 81 calculates the operation content using the case 200 based on the centroid coordinate data and the number of clusters for each cluster in the cluster data Da2 and the cluster number data Da3 updated in step S162. Specifically, when the position of the imaging target member 205 to which the string member 204 attached to the user's right hand is connected, that is, the position of the cluster corresponding to the imaging target member 205 rises, the user extends the right hand. It is determined that such an operation has been performed. In addition, when the position of the imaging target member 205 to which the string member 204 attached to both feet of the user is connected, that is, the position of the cluster corresponding to the imaging target member 205 is lowered, the user squats down. It is determined that

  Next, the CPU 81 calculates the attitude of the right controller 4 (step S164), and proceeds to the next step. For example, the CPU 81 acquires data indicating the acceleration generated in the right controller 4 from the acceleration data Da5, calculates the direction of gravitational acceleration acting on the right controller 4, and uses the data indicating the direction to perform posture. Data Dc is updated. An arbitrary method may be used as a method for extracting the gravitational acceleration. For example, an acceleration component generated on the right controller 4 on average may be calculated and the acceleration component may be extracted as the gravitational acceleration. The CPU 81 acquires data indicating the angular velocity generated in the right controller 4 from the angular velocity data Da4, calculates the angular velocity around the xyz axis of the right controller 4, and uses the data indicating the angular velocity to determine the direction of gravity acceleration. The orientation data Dc is updated by calculating the xyz-axis direction of the right controller 4 with reference to.

  Note that the posture of the right controller 4 may be updated only according to the angular velocity around each axis of the xyz after the xyz axis direction based on the gravitational acceleration is calculated. However, in order to prevent the relationship between the posture of the right controller 4 and the direction of gravitational acceleration from shifting due to the accumulation of errors, the xyz-axis direction with respect to the direction of gravitational acceleration is calculated for each predetermined period, and the right controller 4 The posture may be corrected.

  Next, the CPU 81 sets the operation of each part (part) of the player object PO based on the operation content using the case 200 (step S165), and proceeds to the next step. For example, based on the operation content calculated in step S163, the CPU 81 sets the motion of each part of the player object PO with a motion similar to the user's operation content, and uses the motion to determine the player object motion data Dd. Update. Specifically, when it is determined in step S163 that the user has performed an operation to extend the left hand on the case 200, the player object PO is also set to the operation of extending the left hand in the virtual space, and the operation is performed. The player object motion data Dd is updated by using it. Thus, when the user performs an operation to move a user part using the case 200, a part of the player object PO corresponding to the part (for example, the same part as the part moved by the user) moves. Will be set. Note that the action of the player object PO set in step S165 is not limited to the action of moving the part of the player object PO corresponding to the part moved by the user, and other actions may be set. For example, when the user performs a predetermined operation (for example, a squatting operation) using the case 200, an action of the player object PO that is not related to the moved user part may be set, or the player object PO May be set to an operation that changes to another aspect in the virtual space.

  Next, the CPU 81 sets the operation of the entire player object PO based on the posture of the right controller 4 (step S166), and proceeds to the next step. For example, when the posture of the right controller 4 indicated by the posture data Dc is a posture that tilts the case 200 so as to tilt to the right, the CPU 81 sets the player object PO to tilt to the right in the virtual space. The player object motion data Dd is updated using the motion.

  Next, the CPU 81 performs processing for displaying an image of the virtual space in which the player object PO is arranged on the display device (step S167), and proceeds to the next step. For example, the CPU 81 changes the posture and position of the player object PO based on the player object motion data Dd, and arranges the player object PO in the virtual space based on the changed posture and position. And CPU81 produces | generates the virtual space image which looked at the virtual space where the player object PO is arrange | positioned from the virtual camera arrange | positioned in the predetermined position (For example, the viewpoint behind the player object PO, or a 1st person viewpoint). The virtual space image is displayed on a display device (for example, the display 12 of the main device 11).

  Next, the CPU 81 determines whether or not to end the game (step S168). Examples of the condition for ending the game in step S168 include that the condition for ending the game is satisfied, and that the user performs an operation for ending the game. When the game is not finished, the CPU 81 returns to step S162 and repeats the process. When the game is finished, the CPU 81 finishes the process according to the flowchart. Thereafter, the series of processing from step S162 to step S168 is repeatedly executed until it is determined in step S164 that the game is to be ended.

  As described above, in this embodiment, the operation content for the case 200 which is an example of the extended operation device is detected based on the captured image captured by the infrared imaging unit 123. Therefore, the case 200 does not require an electrical configuration such as an electronic circuit that detects the operation content by the user or transmits the detection result to the main body device 2. Therefore, according to the present embodiment, the configuration of the case 200, which is an example of the extended operation device, can be simplified. In this embodiment, the data transmitted from the right controller 4 to the main device 2 is data indicating the captured image itself captured by the infrared imaging unit 123, such as data relating to the position of the cluster captured in the captured image. is not. Therefore, the amount of data transmitted from the right controller 4 to the main device 2 can be reduced. In the present embodiment, the operation content for the extended operation device that can be detected by the main body device 2 can be calculated based on the position where the imaging target member is imaged. In addition, an operation amount (analog input amount) for operating the operation means can be calculated.

  In the above-described embodiment, the data based on the captured image transmitted from the right controller 4 to the main body device 2 may be data indicating the captured image itself captured by the infrared imaging unit 123. In this case, it is conceivable that the amount of data transmitted from the right controller 4 to the main device 2 increases. However, on the main device 2 side, various analysis processes relating to the captured image can be performed.

  Further, in the above-described extended operation device, the imaging target member provided in the extended operation device main body moves at least in any of the vertical direction, the horizontal direction, and the front-rear direction, and is based on the position of the imaging target member. Although the operation content for the extended operation device is calculated, the operation target may move in another direction according to the operation for the extended operation device. As an example, an operation target provided in the main body of the extended operation device may be rotated according to an operation on the extended operation device. As a first example, when an imaging target member rotates around a rotation axis provided in parallel to the imaging direction of the infrared imaging unit 123, a marker capable of pointing in a predetermined direction is set as the imaging target member. Is attached to the surface of the infrared imaging unit 123 side. Then, by detecting the direction indicated by the marker in the captured image of the infrared imaging unit 123, it is possible to calculate the rotation angle at which the imaging target member is rotating, and based on the rotation angle, the extended operation device The operation content for can be calculated. As a second example, when the imaging target member rotates around a rotation axis provided perpendicular to the imaging direction of the infrared imaging unit 123, the length and pattern at the position facing the infrared imaging unit 123 rotate. A marker that varies depending on the angle is attached to the side surface of the imaging target member. Then, by detecting the length and pattern of the marker in the captured image of the infrared imaging unit 123, it is possible to calculate the rotation angle at which the imaging target member is rotated, and the expansion operation is performed based on the rotation angle. The operation content for the apparatus can be calculated.

  In the above-described embodiment, the method for detecting the movement and posture of the right controller 4 is merely an example, and the movement and posture of the right controller 4 may be detected using other methods and other data. Good. For example, the posture of the right controller 4 is calculated using only the angular velocity generated in the right controller 4 or a combination of the angular velocity and acceleration generated in the right controller 4. May be calculated. Even when only the acceleration generated in the right controller 4 is detected, the direction in which the gravitational acceleration generated in the right controller 4 can be calculated, and the xyz axis direction based on the gravitational acceleration is sequentially calculated as described above. It goes without saying that the same processing as in the embodiment is possible. In the above-described embodiment, the game image corresponding to the operation using the extended operation device to which the right controller 4 is attached is displayed on the display 12 of the main body device 2, but is displayed on the stationary monitor via the cradle. May be.

  The main device 2, the left controller 3, and the right controller 4 may be any device, such as a portable game device, an arbitrary portable electronic device (PDA (Personal Digital Assistant), a cellular phone, a personal phone, and the like. Computer, camera, tablet, etc.).

  In the above description, an example in which information processing (game processing) is performed by the main body device 2 is used. However, at least a part of the processing steps may be performed by another device. For example, when the main device 2 is configured to be able to communicate with another device (for example, another server, another image display device, another game device, or another portable terminal), the processing step further includes Other devices may work together. In this way, by performing at least a part of the processing steps with another device, processing similar to the processing described above can be performed. Further, the above-described information processing (game processing) can be executed by cooperation between one processor or a plurality of processors included in an information processing system including at least one information processing apparatus. In the above embodiment, the CPU 81 of the main device 2 can perform information processing by executing a predetermined program. However, a part or all of the above processing is performed by a dedicated circuit included in the main device 2. It may be broken.

  Here, according to the above-described modification, the present invention can be realized even in a so-called cloud computing system form, a distributed wide area network system form, and a local network system form. For example, in a distributed local network system configuration, the above-described processing is performed in cooperation between a stationary information processing device (stationary game device) and a portable information processing device (portable game device). It is also possible to do. In these system forms, there is no particular limitation as to which apparatus performs the above-described processing, and it goes without saying that the present invention can be realized regardless of the processing sharing.

  In addition, it is needless to say that the processing order, setting values, conditions used for determination, and the like used in the information processing described above are merely examples, and that the present embodiment can be realized with other orders, values, and conditions.

  The program may be supplied not only to the main apparatus 2 and the right controller 4 through an external storage medium such as an external memory, but also to the apparatus through a wired or wireless communication line. The program may be recorded in advance in a nonvolatile storage device inside the apparatus. As the information storage medium for storing the program, in addition to the nonvolatile memory, CD-ROM, DVD, or similar optical disk storage medium, flexible disk, hard disk, magneto-optical disk, magnetic tape, etc. But you can. Further, the information storage medium for storing the program may be a volatile memory for storing the program. Such a storage medium can be referred to as a computer-readable recording medium. For example, the various functions described above can be provided by causing a computer or the like to read and execute the programs of these recording media.

  Although the present invention has been described in detail above, the above description is merely illustrative of the present invention in all respects and is not intended to limit the scope thereof. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. It is understood that the scope of the present invention should be construed only by the claims. Moreover, it is understood that those skilled in the art can implement the equivalent range from description of the specific Example of this invention based on description and technical common sense of this invention. In addition, it is to be understood that the terms used in the present specification are used in the meaning normally used in the art unless otherwise specified. Thus, unless defined otherwise, all technical and technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. In case of conflict, the present specification, including definitions, will control.

  As described above, the present invention can be used as an information processing system, a case, a cardboard member, and the like that can be realized with a simple configuration.

2 ... Main unit 3 ... Left controller 4 ... Right controller 11 ... Housing 12 ... Display 81 ... CPU
83 ... Controller communication unit 85 ... DRAM
DESCRIPTION OF SYMBOLS 111 ... Communication control part 121 ... Processing part 123 ... Infrared imaging part 124 ... Infrared light emission part 200 ... Case 201 ... Case main body 202 ... Controller arrangement part 203 ... Shoulder belt 204 ... String members 205, 255, 256, 257 ... Imaging Target member 206 ... Slide part 250 ... Steering device 251 ... Device main body 252 ... Operation pedal 253 ... Operation rod 254 ... Operation lever

Claims (14)

An information processing system including an information processing device, a data transmission device that transmits data to the information processing device, and a case in which the data transmission device can be attached,
The data transmission device includes:
An imaging unit;
A transmission unit that transmits data generated based on an image captured by the imaging unit to the information processing apparatus;
The case is
A case body having a space inside,
A data transmission device arrangement unit capable of including and arranging at least a part of the data transmission device so that the imaging direction of the imaging unit faces the inner space of the case body unit;
An imaging target member that is arranged in an inner space of the case main body and is arranged at a position where the data transmission device can be imaged when the data transmission device is arranged in the data transmission device arrangement unit;
An operation unit that moves the imaging target member in the inner space of the case body in response to an operation from the outside of the case body part,
The information processing apparatus includes:
A receiver for receiving the data transmitted from the transmitter;
An information processing system comprising: a processing unit that performs information processing based on the data received by the receiving unit. The data transmission device further includes an arithmetic unit that calculates at least a position of the imaging target member in a captured image captured by the imaging unit,
The information processing system according to claim 1, wherein the data transmitted by the transmission unit to the information processing apparatus includes at least a position of the imaging target member in the captured image. The data transmission device further includes an infrared light emitting unit,
The data transmitter arrangement unit can arrange the data transmitter so that the infrared light emitter can emit infrared light in the inner space of the case body.
The imaging unit is an infrared camera,
The information processing system according to claim 1, wherein the imaging target member includes a retroreflecting material.   The information processing system according to any one of claims 1 to 3, wherein the case further includes a fixing unit that fixes the case to a user's body.   The information processing system according to claim 4, wherein the fixing unit is a shoulder belt for a user to carry the case on the back. The operation unit includes at least one string member connected to the imaging target member,
At least one of the string members is configured to be capable of being pulled by the user's hand in a state where the case is fixed to the user's body by the fixing portion.
6. The operation unit according to claim 4, wherein a user pulls the string member to move the imaging target member connected to the string member in an inner space of the case main body. Information processing system. The case further includes a slide part that slidably supports the imaging target member to which the string member is connected in a predetermined sliding direction in the inner space of the case body part,
The information processing system according to claim 6, wherein when the string member is pulled, the operation unit moves the imaging target member connected to the string member in one direction in the slide direction.   The said operation part is comprised so that the said imaging target member connected to the said string member may move to the reverse direction in the said sliding direction, when the said pulled string member is loosened. Information processing system. The information processing device is a game device,
The data transmission device is a game controller used for operation of the game device,
The game system according to claim 1, wherein the processing unit performs a game process for controlling a predetermined object in a virtual space in accordance with a position of the imaging target member. The data transmission device further includes an inertial sensor,
The transmission unit further transmits data based on the output of the inertial sensor to the information processing device,
The information according to claim 9, wherein the processing unit controls the entire object according to data based on an output of the inertial sensor, and controls a part of the object according to a position of the imaging target member. Processing system. A case that includes at least an imaging unit and that can be equipped with a data transmission device that transmits data generated based on an image captured by the imaging unit to at least another device,
A case body having a space inside,
A data transmission device arrangement unit capable of including and arranging at least a part of the data transmission device so that the imaging direction of the imaging unit faces the inner space of the case body unit;
An imaging target member that is arranged in an inner space of the case main body and is arranged at a position where the data transmission device can be imaged when the data transmission device is arranged in the data transmission device arrangement unit;
A case that includes an operation unit that moves the imaging target member in an inner space of the case body in response to an operation from the outside of the case body. A shoulder belt portion on which the user can carry the case and fix it to the user's body;
A slide part that slidably supports the imaging target member in a predetermined slide direction in the inner space of the case body part;
The operation unit includes at least one string member connected to the imaging target member,
At least one of the string members is configured to be capable of being pulled by the user's hand in a state where the case is fixed to the user's body by the shoulder belt portion.
The operation unit moves the imaging target member in one direction in the sliding direction when the string member is pulled, and moves the imaging target member in the sliding direction when the pulled string member is loosened. The case of claim 11, configured to move in the reverse direction.   The case according to claim 12, wherein the case main body, the data transmission device arrangement unit, the imaging target member, and the slide unit are configured to be formed by folding at least one cardboard. A cardboard member capable of forming an expansion attachment to which a data transmission device that transmits predetermined data to another device can be attached,
The extension attachment is
The main body,
A data transmission device arrangement unit that is formed on a part of the main body unit and can be arranged by including at least a part of the data transmission device inside the main body unit;
A cardboard member that can be integrally formed by bending the expansion attachment having the main body and the data transmission device arrangement portion.

Images