JP2016091423A - Information processor, information process system, real object system, and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processing technique capable of actualizing various forms utilizing bodies in the real world.SOLUTION: An imaging device 12 of an information processing system 1 photographs real objects 120a, 120b placed on a play field 20. An information processor 10 acquires a state on the play field 20 from photographic images etc., and determines and controls a movement of the real object 120a according to position relation between the real objects 120a, 120b and user's requests received through a microphone 16 and an input device 14. According to user operations done through the input device 14, the real object 120b is controlled. The play field 20 is formed of a projection image by a projector included in a display device 18. According to movements of the real objects 120a, 120b, the projection image and a sound form a speaker 19 are changed.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing technique using an object in real space.

  In recent years, a technique for measuring a parameter related to an object such as a person or an object in a real space by some means, and importing the parameter as an input value for analysis or displaying it as an image has been used in various fields. In the field of computer games, intuitive and easy operations are realized by acquiring the movement of the user and the marker held by the user and moving the character in the virtual world in the display screen accordingly. (For example, refer to Patent Document 1). As described above, the technique for reflecting the movement and shape change of an object in the real space on a display image is expected to be applied not only to a game but also toys and learning materials (for example, see Non-Patent Document 1 and Patent Document 2). ).

WO 2007/050885 A2 publication JP 2008-73256 A

Posey: Instrumenting a Poseable Hub and Strut Construction Toy, Michael Philetus Weller, Ellen Yi-Luen Do, Mark D Gross, Proceedings of the Second International Conference on Tangible and Embedded Interaction, 2008, pp 39-46

  The above-mentioned conventional technology replaces simple things in the real world with various and attractive things in the virtual world or communicates with the equipment by using the main output means as image display on the display device. Is easily made. On the other hand, as the degree of dependence on image representation is reduced, the use of real-world objects and the room for computers to exhibit their performance are limited, making it difficult to diversify. Information processing technology using objects in the real world is very effective in that it is easy for the user to understand intuitively and easily obtain a sense of reality. Therefore, there is a need for a technique that can realize the same aspect with an object in the real world, even if it is less dependent on image representation.

  The present invention has been made in view of such problems, and an object of the present invention is to provide an information processing technique that can realize various forms using real-world objects.

  One embodiment of the present invention relates to an information processing apparatus. The information processing apparatus sequentially acquires image frames of moving images taken by the imaging apparatus, and detects the state information of the real object existing in the object space at predetermined time intervals by detecting the images in the image frames. An information processing unit that determines operation details of the real object to be controlled among the real objects at predetermined time intervals according to a predetermined rule based on the state information detected by the state specifying unit, and a control target And a real object control unit that controls the real object to operate according to the content determined by the information processing unit.

  Another aspect of the present invention relates to an information processing system. This information processing system is an information processing system including an information processing device and a real object that moves under the control of the information processing device, and the information processing device sequentially processes image frames of moving images taken by the imaging device. By acquiring and detecting an image in the image frame, a state specifying unit that detects state information of a real object existing in the object space at a predetermined time interval, and a predetermined based on the state information detected by the state specifying unit By rule, an information processing unit that determines the operation content of the real object to be controlled among the real objects at predetermined time intervals, and a control target real object to be operated with the content determined by the information processing unit And a real object control unit.

  Yet another aspect of the present invention relates to a real object system. This real object system includes a communication unit that receives a control signal from an information processing device and a drive unit that operates an actuator according to the received control signal, thereby including a plurality of real objects that operate based on the control signal. In the body system, one of the real objects is operated based on a control signal reflecting a user operation performed via an input device connected to the information processing device, and another real object is determined by the information processing device. The operation is based on a control signal reflecting the movement.

  Yet another embodiment of the present invention relates to an information processing method. This information processing method sequentially acquires image frames of moving images taken by the imaging device, and detects state information of the real object existing in the object space at predetermined time intervals by detecting images in the image frames. A step of determining the content of operation of the real object to be controlled among the real objects at predetermined time intervals according to a predetermined rule based on the detected state information, and a step of determining the real object to be controlled Controlling to operate according to the determined content.

  Note that any combination of the above-described components, and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a recording medium on which the computer program is recorded, and the like are also effective as an aspect of the present invention. .

  According to the present invention, play and learning using real-world objects can be easily realized in various forms.

It is a figure which shows the structural example of the information processing system which can apply this Embodiment. It is a figure which shows the internal circuit structure of the information processing apparatus in this Embodiment. It is a figure which shows the structure of the real object and information processing apparatus in this Embodiment in detail. It is a figure which shows the example of an external appearance structure of the real object in this Embodiment. It is a figure which shows the example of a coupling | bonding of the some real object in this Embodiment with a bottom view. It is a figure which shows the modification of the number and position of the marker provided in the real object of this Embodiment with a top view. As another example of the marker provided on the real object of the present embodiment, it is a diagram showing a case where the light emission of the marker is used for distinguishing the front and back of the real object by a top view of the real object It is a flowchart which shows the process sequence which an information processing apparatus controls a real object according to the state on a play field in this Embodiment. It is a figure which shows the example of implementation | achievement of a "match / competition" among the aspects which can be implement | achieved in this Embodiment. It is a figure which shows the example of data prepared for a scenario memory | storage part in order to implement | achieve the form shown in FIG. It is a figure which shows the implementation example of "cooperation and assistance" among the aspects which can be implement | achieved in this Embodiment. It is a figure which shows the implementation example of "act" among the aspects which can be implement | achieved in this Embodiment. It is a figure which shows the example of the aspect implement | achieved when the function to hold an object to a real object is added in this Embodiment. It is a figure which shows another example of the aspect implement | achieved when the function to hold an object to a real object is added in this Embodiment. It is a figure for demonstrating the example of the connection method of the block in the aspect in which the real object of this Embodiment assembles a block. It is a figure which shows another example of the aspect implement | achieved when the function which grasps | emits an object in this Embodiment is added. It is a figure which shows the example of the aspect implement | achieved when the function to hold | maintain a pen is added to the real object in this Embodiment. It is a figure which shows the example of the aspect which generate | occur | produces interaction with a real object and a user's body in this Embodiment. It is a figure which shows the example of the aspect which included the rolling of the ball in the motion of the real object in this Embodiment. It is a figure for demonstrating the aspect in which a some user participates in one game using a network in this Embodiment.

  FIG. 1 shows a configuration example of an information processing system to which this embodiment can be applied. The information processing system 1 moves the real objects 120a and 120b placed on the play field 20, the imaging device 12 that captures the space on the play field 20, and moves at least one of the real objects 120a and 120b by performing predetermined information processing. The information processing device 10 includes an input device 14 that accepts user operations, a microphone 16 that acquires ambient sound, a display device 18 that displays an image, and a speaker 19 that outputs sound. Note that the input device 14, the microphone 16, the display device 18, and the speaker 19 may not be included in the information processing system 1 depending on the mode of implementation.

  The play field 20 is a plane that defines an area serving as a reference for the information processing apparatus 10 to recognize the real objects 120a and 120b and specify the position coordinates thereof. The play field 20 is not limited in material and shape as long as such a planar area can be defined, and may be any of paper, board, cloth, desk top board, game board, and the like. Or the image etc. which the projector contained in the display apparatus 18 projected on the desk or the floor may be sufficient.

  The shapes of the real objects 120a and 120b are not limited as long as they are objects that exist in real space. That is, it may be a simple shape as shown in the figure, or may be a more complex shape such as a doll or a miniature of a real world object such as a doll or a minicar, its parts, or a game piece. Further, the size, material, color, number of used objects, etc. of the real objects 120a, 120b are not limited. Furthermore, it may be a structure that can be assembled or disassembled by the user, or may be a finished product. At least one of the real objects 120a and 120b includes an actuator that establishes communication with the information processing apparatus 10 and is driven by the transmitted control signal.

  Communication may be established by connection using a wireless interface such as Bluetooth (registered trademark) protocol or IEEE802.11, or via a cable. In the illustrated example, each of the real objects 120a and 120b includes wheels 122a and 122b, and is configured to self-run by rotating an axle motor according to a control signal from the information processing apparatus 10. Here, for example, the information processing apparatus 10 moves the real object 120b based on a user operation via the input device 14, while the real object 120a is moved according to the position or movement of the real object 120b. Move.

  As described above, in this embodiment, the information processing apparatus 10 basically moves any real object (for example, the real object 120a) according to a predetermined rule corresponding to the state on the play field 20. In the following description, the real object for which the information processing apparatus 10 determines the movement is particularly referred to as “real object controlled by the information processing apparatus 10”, and is distinguished from the real object operated by the user via the input device 14. The real object (for example, the real object 120b) other than the real object controlled by the information processing apparatus 10 may be any object that can be freely placed and moved by the user, and may not include an actuator inside. Even if an actuator is provided, the operation means is not limited to the input device 14 and may be directly operated using a dedicated controller or the like.

  Furthermore, the object driven by the actuator is not limited to the wheel. For example, a gripper or an arm may be attached to a real object and the movable part thereof may be moved, or any of mechanisms that are controlled by a general robot or toy may be employed. In addition to such mechanical movement, a light emitting element, a display, a speaker, a vibrator, and the like may be incorporated to operate them. A plurality of mechanisms may be operated simultaneously. In any case, these mechanisms are controlled by the information processing apparatus 10.

  The imaging device 12 is a general digital video camera provided with an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS), and is a space on the play field 20 in which real objects 120a and 120b are placed. Take a video. Alternatively, a camera that detects invisible light such as near-infrared light or a general camera that detects visible light may be combined. The frame data of the moving image is sequentially transmitted to the information processing apparatus 10 together with the photographing, and is used to acquire the position coordinates of the real objects 120a and 120b on the plane formed by the play field 20. Therefore, the imaging device 12 is preferably arranged so as to overlook the play field 20.

  However, if the position coordinates of the real objects 120a and 120b are found, the position and angle of the imaging device 12 are not particularly limited. For example, when a marker of a predetermined size is provided on the real objects 120a and 120b and the distance from the imaging surface of the imaging device 12 is determined with the apparent size, the position coordinates in the camera coordinate system are acquired using the distance, May be converted into position coordinates in the world coordinate system with the play field 20 as a horizontal plane. Or you may acquire the distance of the real objects 120a and 120b from the imaging surface by comprising the imaging device 12 with the stereo camera which image | photographs the same space from the right-and-left viewpoint which has a known space | interval.

  A technique for acquiring the position of a subject in the depth direction using a stereo camera is widely known. A viewpoint moving camera may be used instead of the stereo camera. Alternatively, a device that irradiates reference light such as near infrared rays and detects the reflected light may be provided, and the positions of the real objects 120a and 120b may be specified by an existing method such as TOF (Time of Flight). Further, the positions of the real objects 120a and 120b may be specified by detecting the contact position using the upper surface of the play field 20 as a touch pad. When a TOF or touchpad is used, the real objects 120a and 120b are distinguished by integrating with color information of each image in the captured image. Further, the real objects 120a and 120b once detected can be tracked using the existing visual tracking technology, so that the subsequent position coordinate acquisition can be made efficient.

  The information processing apparatus 10 acquires the position and movement of a real object other than the real object 120a on the play field 20, mainly the real object 120a that it controls, and determines the movement of the real object 120a that it controls based on it. And the real object 120a is operated by transmitting the control signal according to the determination result. For example, the information processing apparatus 10 may be a game device or a personal computer, and may implement an information processing function by loading a necessary application program. Further, as will be described later, the information processing apparatus 10 may establish communication with another information processing apparatus or server via a network and send and receive necessary information.

  Here, the movements of the real objects 120a and 120b are acquired by tracking the time change of the position coordinates using the moving image data captured by the imaging device 12 as described above. When the user moves the real object 120b via the input device 14, if the initial position of the real object 120b is acquired, the time change of the position is specified from the user's operation content on the input device 14 thereafter. it can. As described above, the movements of the real objects 120a and 120b may be acquired by means other than the captured image, or accuracy may be improved by integrating information acquired by a plurality of means.

  The input device 14 receives a user operation such as start / end of processing or driving of the real object 120b, and inputs a signal representing the content of the operation to the information processing device 10. The input device 14 may be any common input device such as a game controller, a keyboard, a mouse, a joystick, or a touch pad, or any combination thereof. The microphone 16 acquires ambient sound, converts it into an electrical signal, and inputs it to the information processing apparatus 10. The information processing apparatus 10 mainly recognizes the user's voice out of the voice signal acquired from the microphone 16 and reflects it in the movement of the real object 120a controlled by the information processing apparatus 10 itself.

  The display device 18 displays an image generated by the information processing device 10. In the illustrated example, it is assumed that the display device 18 is a projector and an image is displayed on the play field 20. On the other hand, the display device 18 may be a display such as a general television monitor, or both the projector and the display may be included in the configuration. Alternatively, the play field 20 may be a display.

  The speaker 19 may be a general speaker, a sounding device such as a buzzer, or a combination thereof, and outputs a predetermined sound or voice as sound according to a request from the information processing device 10. Connection between the information processing apparatus 10 and other apparatuses may be wired or wireless, and may be via various networks. Alternatively, any two or more of the information processing device 10, the imaging device 12, the input device 14, the microphone 16, the display device 18, and the speaker 19 may be combined or integrally provided. Further, an external device such as a speaker may be connected to the information processing apparatus 10.

  FIG. 2 shows an internal circuit configuration of the information processing apparatus 10. The information processing apparatus 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. These are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a peripheral device interface such as USB or IEEE 1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a nonvolatile memory, a display device 18, a speaker 19, and the like. An output unit 36 that outputs data to the output device, an imaging device 12, an input device 14, and an input unit 38 that inputs data from the microphone 16, a recording medium drive unit that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory 40 is connected.

  The CPU 22 controls the processing and signal transmission in the components inside the information processing apparatus 10 by executing the operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium and loaded into the main memory 26 or downloaded via the communication unit 32. The GPU 24 has a function of a geometry engine and a function of a rendering processor. The GPU 24 performs a drawing process according to a drawing command from the CPU 22 and outputs it to the display device 18 as appropriate. The main memory 26 is constituted by a RAM (Random Access Memory), and stores programs and data necessary for processing.

  The communication unit 32 establishes communication with the real objects 120a and 120b including the actuator and transmits a control signal. In a mode in which sound is output or an image is displayed on the real object 120a or the like, those data are also transmitted. As will be described later, in a mode in which a sensor is provided on the real object 120a or the like, the communication unit 32 may receive a measurement value from the sensor from the real object 120a or the like. Further, the communication unit 32 may establish communication with the network as necessary, and send and receive necessary files and data to and from an external server or information processing apparatus.

  FIG. 3 shows the configuration of the real objects 120a and 120b and the information processing apparatus 10 in detail. In FIG. 3, each element described as a functional block for performing various processes may be configured by a CPU (Central Processing Unit), a memory, a microprocessor, other LSIs, actuators, sensors, and the like in hardware. In terms of software, it is realized by a program loaded in a memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  In the figure, a real object controlled by the information processing apparatus 10 is a real object 120a, and a real object operated by the user via the input device 14 is a real object 120b. However, as described above, there may be an actual object that is directly operated by the user or an actual object that is simply placed or moved by hand. In the case where the object is composed only of such a real object, the real object 120b may not be provided. A real object that the user simply moves or places by hand may be a flat card, a doll without an internal structure, a game piece, a block, or an ornament. Even in this case, it is possible to identify each image with high accuracy from the photographed image by adding a change on the exterior such as different colors, patterns, and shapes, or printing a two-dimensional barcode. it can.

  Each of the real objects 120 a and 120 b includes drive units 106 a and 106 b that operate according to a control signal from the information processing apparatus 10, and communication units 108 a and 108 b that receive necessary control signals and data from the information processing apparatus 10. However, the control signal received by the real object 120a is determined by the information processing apparatus 10, and the control signal received by the real object 120b reflects a user operation via the input device 14. The drive units 106 a and 106 b include an actuator that is driven by a control signal from the information processing apparatus 10. As shown in FIG. 1, when the real objects 120a and 120b are self-propelled by the wheels 122a and 122b, the axle is rotated or the rudder angle is changed by the actuator.

  Thereby, the real objects 120a and 120b can be made to travel at the direction and speed according to the control signal from the information processing apparatus 10. In addition, as described above, the driving units 106a and 106b may include an actuator that generates a motion other than the wheel, a light emitting element, a display, a speaker, a vibrator, and the like and a mechanism that operates the actuator. These mechanisms are also operated by a control signal from the information processing apparatus 10 by utilizing an existing technology.

  The communication units 108a and 108b receive the control signal transmitted from the information processing apparatus 10 and notify the respective driving units 106a and 106b. The communication units 108a and 108b hold the individual identification information of their own real objects 120a and 120b in an internal memory. Then, it is determined based on the individual identification information transmitted together with the control signal whether or not the control signal transmitted from the information processing apparatus 10 is transmitted to the real objects 120a and 120b.

  The real objects 120a and 120b may further include a sensor (not shown) that measures its own state. In addition to acquiring the position coordinates of the real objects 120a and 120b using the captured images as described above, each real object measures its own state, so that the positional relationship, the change in the shape of the real object, and the like can be more accurately determined. You may ask for. In this case, the communication units 108a and 108b transmit the measurement value obtained by the sensor to the information processing apparatus 10 together with its individual identification information. As shown in FIG. 1, when wheels 122a and 122b are provided on the real objects 120a and 120b, a rotary encoder and a rudder angle sensor may be provided on the wheels 122a and 122b to specify the actual movement amount and movement direction.

  Similarly, a position sensor that acquires the absolute positions of the real objects 120a and 120b, and a motion sensor such as an acceleration sensor, a gyro sensor, and a geomagnetic sensor may be provided. Although the real objects 120a and 120b are basically moved by a control signal from the information processing apparatus 10, if actual measurement values are acquired in this way, feedback control can be performed so as to correct the error. A joint that can be bent and stretched by the user may be provided on the real objects 120a and 120b, and a potentiometer for specifying the angle may be introduced. As a result, it is possible to recognize any change in the shape of the real objects 120a and 120b, and therefore it is possible to accurately track the image in the captured image. However, in view of the required position detection accuracy, manufacturing cost, etc., the necessity of the sensor is determined.

  The information processing apparatus 10 includes a communication unit 50 that transmits a control signal to the real objects 120a and 120b, a state specifying unit 52 that specifies a state on the play field 20 including the positional relationship between the real objects 120a and 120b, and information on the real objects. The stored real object information storage unit 54, the information processing unit 62 that performs information processing according to the positional relationship between the real objects 120a and 120b, and the scenario storage unit 56 that sets the correspondence between the positional relationship and processing to be performed accordingly. In addition, a real object control unit 60 that generates a control signal as a result of a user operation or processing in the information processing unit 62 is included.

  The state specifying unit 52 sequentially acquires image frames of moving images from the imaging device 12 and analyzes them to specify the positions of the real objects 120a and 120b at predetermined time intervals. Various techniques are widely known as techniques for detecting and tracking an image of an object by image analysis, and any of them may be adopted in the present embodiment. In addition, as described above, a technique for identifying a position in a three-dimensional space by integrating image information in a captured image and position information in the depth direction using a stereo image, TOF, etc. also uses a general method. can do.

  Furthermore, when providing the sensors on the real objects 120a and 120b as described above, the state specifying unit 52 uses the measurement values transmitted from the real objects 120a and 120b to specify the movement, position, shape, posture, and the like in detail. Also good. In this case, the communication unit 50 receives the measurement values transmitted from the real objects 120 a and 120 b and supplies them to the state specifying unit 52. In addition, in order to distinguish the real object 120a controlled by the information processing apparatus 10 and the real object 120b operated by the user on the captured image, the real objects 120a and 120b have markers or two-dimensional barcodes that emit light with different specific colors. May be provided.

  The real objects 120a and 120b may be distinguished by other appearance features such as color, shape, pattern, and size. The real object information storage unit 54 stores information associating individual identification information of the real objects 120a and 120b with features on their appearance. The state specifying unit 52 refers to the information and specifies individual identification information corresponding to the appearance feature of the image detected from the captured image, thereby distinguishing the real objects 120a and 120b at each position. The individual identification information and the position coordinates of the real objects 120a and 120b are sequentially supplied to the information processing unit 62.

  The information processing unit 62 performs information processing to be performed based on the positional relationship between the real objects 120a and 120b, the user operation on the input device 14, the audio signal acquired by the microphone 16, and the like. For example, when realizing a game in which a real object 120b operated by a user competes with a real object 120a controlled by the information processing apparatus 10, the information processing unit 62 determines the movement of the real object 120a controlled by the information processing apparatus 10. The game progresses while calculating the score, determining the win / loss, generating the display image, and determining the output sound. A game program, rules for determining movement, data necessary for image generation, and the like are stored in the scenario storage unit 56.

  Note that, when using the audio signal acquired by the microphone 16, the information processing unit 62 is provided with a voice recognition function using a general technique in order to detect a predetermined keyword from the audio signal. The information processing unit 62 outputs the generated display image data to the display device 18 or outputs to the speaker 19 by decoding audio data to be generated. When the display device 18 is a projector that projects an image onto the play field 20, the image on the play field 20 can be changed according to the progress of the game, the positions of the real objects 120a and 120b, and the like. Information such as a score may be included in the image.

  The real object control unit 60 generates a control signal so that the real object 120a moves with the movement determined by the information processing unit 62. Further, it interprets the contents of the user operation on the input device 14 and generates a control signal for the real object 120b that is the operation target of the user. These control signals are associated with the individual identification information of each of the real objects 120a and 120b supplied from the information processing unit 62. The communication unit 50 establishes communication with the real objects 120a and 120b, and transmits the control signal generated by the real object control unit 60. Specifically, the signal to be transmitted varies depending on the control method, and a technique generally used in the field of robot engineering or the like may be appropriately employed.

  FIG. 4 shows an external configuration example of the real object 120a. The same applies to the real object 120b. The real object 120a of this example includes wheels 122a at the lower part of a rectangular parallelepiped main body as shown in FIG. In addition, a marker 126 that emits light of a predetermined color is provided at the top. The state specifying unit 52 of the information processing apparatus 10 acquires the position coordinates of the real object 120a by detecting the image of the marker 126 in the captured image. That is, the center 129 of the marker 126 is set as the tracking point of the real object 120a. In the illustrated example, the marker 126 is disposed at a position close to one side of the rectangle that forms the upper surface 124a of the real object 120a. By arranging the marker 126 so as to be shifted from the center of the real object 120a in this way, for example, the side surface close to the marker 126 can be defined as the front of the real object, and recognition of forward and backward movement can be shared with the user.

  The marker 126 is mounted with, for example, an infrared light emitting diode or a white light emitting diode. The color may be switched by attaching a color filter cap to the surface of the white light emitting diode. By making the emission color different between the real object 120a controlled by the information processing apparatus 10 and the real object 120b operated by the user, they can be distinguished by the color of the image in the captured image. In this case, the user himself can also distinguish between the two. Further, the orientation of the real object 120a may be defined by the color by providing a plurality of markers on one real object 120a and changing the emission color. In this way, it may be possible to switch whether the light emission is invisible light such as infrared rays or visible light depending on whether it is necessary to make some distinction by color or not. Therefore, the marker 126 may incorporate a wavelength conversion element such as an up-conversion phosphor.

  Further, as shown on the lower surface 124b of the real object 120a, each wheel 122a is provided with a motor 128 for rotating the axle. The motor 128 corresponds to the drive unit 106a shown in FIG. 3 and is driven by a control signal transmitted from the information processing apparatus 10 to rotate the wheel 122a at the requested rotation speed and direction. The wheel 122a may be provided with a mechanism for changing the rudder angle by an actuator. As a result, a movement such as changing the moving direction of the real object 120a or turning on the spot is also generated. In addition, the real object 120a may be provided with a switch for releasing the rotation of the wheel 122a from the motor 128 so that the user can run the real object 120a by hand.

  One real object may be assembled using the real object 120a shown in FIG. 4 as a minimum unit. FIG. 5 shows a connection example of a plurality of real objects in a bottom view. First, the real object 130a shown on the left side of the figure has a configuration in which two real objects 120a of the smallest unit are connected to the front and rear. In this case, the real object 130a travels in the longitudinal direction as indicated by the arrow. The coupling is realized by, for example, a plate capable of fitting two real objects 120a, a cover covering the whole, a binding band, and the like. Each real object 120a may be provided with a concavo-convex connection portion, a hook-and-loop fastener, a magnet, and the like so as to be connected. The same applies to other examples.

  The real object 130b shown in the upper right of FIG. 5 has a configuration in which two minimum real objects 120a are connected to the left and right. In this case, as indicated by the arrow, the real object 130b travels in a direction perpendicular to the longitudinal direction. The real object 130c shown in the lower right of the drawing has a configuration in which the smallest real object 120a is connected to each other so as to form a triangle inside. In this case, the real object 130c turns at the same place as shown by the arrow. By allowing multiple real objects 120a to be connected, even if it is a mode of carrying or towing another real object on top, it prepares the real object according to its weight and size flexibly it can. In addition, the direction of movement and movement can be changed according to the user's preference and the purpose of play.

  FIG. 6 is a top view showing a modification example of the number and positions of markers provided on a real object. First, the real object 120c shown on the left side of the figure includes two markers 126a and 126b. The state specifying unit 52 of the information processing apparatus 10 acquires the position of the midpoint 132 of the line segment connecting the markers 126a and 126b in the captured image as a tracking point of the real object 120c. Similar to the example shown in FIG. 4, the middle point 132 can be distinguished from the front and back by shifting the center point 132 from the center of the upper surface of the real object 120c. Also, as shown in the figure, a boom 134 is provided on the real object 120c, and markers 126a and 126b are arranged at both ends thereof, so that the position thereof can be adjusted so that it enters the field of view of the imaging device 12 regardless of the surrounding state. Also good.

  The real object 120d shown on the right side of the figure includes three markers 126c, 126d, and 126e. The state specifying unit 52 detects the images of the markers 126c, 126d, and 126e in the photographed image, and acquires the position of the center of gravity 136 of the triangle having the vertexes as the tracking points of the real object 120d. In this case, the front and rear can be distinguished by changing the number of markers before and after the real object 120d.

  FIG. 7 shows, as another example of the marker provided on the real object, a top view of the real object in the case where the light emission of the marker is used to distinguish the front and back of the real object. The real object 120e includes markers 126f and 126g at both ends of the boom 134 as in the real object 120c of FIG. However, the boom 134 may be disposed at an arbitrary position such as the center of the upper surface of the real object 120e as illustrated. The positions of the markers 126f and 126g may be freely arranged by the user. In some cases, the booms 134 may not be provided, and the markers 126f and 126g may be directly installed on the real object 120e main body. In any case, a plurality of markers 126f and 126g are arranged on a straight line that can define the front-rear direction, such as a straight line connecting the left and right of the upper surface of the real object 120e.

  In such a configuration, only one of the two markers 126f and 126g is caused to emit light to define the left and right of the real object 120e, and thus clearly indicate the front and back of the real object 120e. For example, a rule is provided in advance that the left marker 126f is turned on and the right marker 126g is turned off in the forward direction. When the direction of the arrow in the figure is determined to be the forward direction from the arrangement and rotation direction of the motor, the information processing apparatus 10 lights the left marker 126f according to the information because the left and right of the marker are determined naturally.

  If the forward direction is not fixed, the wheel is rotated in two forward and reverse directions with one of the markers 126f and 126g lit, and the rotation direction in which the lit marker is on the left is determined from the movement of the marker in the captured image. Identify and determine the direction of forward rotation. The state specifying unit 52 of the information processing apparatus 10 records the information thus determined in the real object information storage unit 54 in association with the individual identification information of the real object 120e, and uses it for subsequent real object control. If the user recognizes only the above rules, there is no discrepancy with the control of the information processing apparatus 10.

  As shown in FIG. 7, even if one of the markers is turned off, if the image of the marker is detected from the photographed image based on the shape, color, etc., the tracking point is the same as the real object 120c in FIG. This is the midpoint 132 of the line segment connecting the markers 126f and 126g. In addition to distinguishing the left and right of the markers 126f and 126g by lighting / extinguishing, the markers 126f and 126g may be lit / flashing, may emit visible / invisible light, or may have different emission colors. Note that the arrangement and number of markers shown in FIGS. 6 and 7 are examples, and may be appropriately determined according to the size and shape of the real object. In addition, as described above, when an actual object can be recognized by appearance features other than the marker, the marker need not be provided.

  In the present embodiment, even if a real object has a simple shape as described above, an intended world is constructed by considering it as various objects, so that play and games can be realized. That is, it is not necessary to change the appearance of the real object according to the world to be constructed. Even in such a case, for example, the user can distinguish the former as an ambulance, the latter as a normal car, etc. by emitting one real object's marker in red and the other real object's marker in white. Intuitive grasp. At this time, if an effect such as blinking red light emission is given, the user can further experience a sense of reality.

  In addition, by changing the color of the marker as described above, the user himself / herself can distinguish between the real object controlled by the information processing apparatus 10 and the real object operated by the user. In this case, the information processing apparatus 10 may clearly indicate in advance which color of the real object is the target operated by the user. For example, the information processing apparatus 10 turns on the markers in the respective colors and moves one of the real objects slightly. If the moved real object is ruled as a real object controlled by the information processing apparatus 10 (or a real object operated by the user), the user can know which color of the real object he / she operates. You can play and play smoothly.

  Next, the basic operation of the information processing apparatus 10 configured as described above will be described. FIG. 8 is a flowchart illustrating a processing procedure in which the information processing apparatus 10 controls a real object according to the state on the play field 20. This flowchart is started when a user inputs a processing start request via the input device 14 or the like in a state where the real objects 120a and 120b are placed on the play field 20. Note that when the user operates the real object 120b via the input device 14, the processing for that is performed at an independent timing and is not included in the illustrated flowchart.

  First, the state specifying unit 52 of the information processing device 10 requests the imaging device 12 to start shooting (S10). When the imaging device 12 starts capturing a moving image accordingly, an image frame is acquired at a predetermined frame rate and transmitted to the information processing device 10. Here, the progress of the time step t for acquiring the image frame to be processed is expressed as t = 0, 1, 2,. The period of the time step t may be the same as the imaging period of the image frame in the imaging device 12, or may be longer than that. That is, all image frames of a moving image may be processed, or processing may be performed while thinning out image frames every predetermined number.

  First, when an image frame of the first time step t = 0 is acquired (S12, S14), the state specifying unit 52 analyzes the image frame, and states such as the position coordinates of the real objects 120a and 120b on the plane of the play field 20 Information is acquired (S16). Most simply, the position coordinates on the play field 20 are obtained by detecting an image of each real object marker from the image frame and appropriately performing coordinate conversion based on the detected image. However, as described later, the information acquired here may vary depending on the embodiment. As described above, the value measured by the sensor provided on the real object, the content of the user operation via the input device 14, the audio information acquired by the microphone 16 and the like may be further used. These pieces of information are reflected in subsequent information processing as needed.

  Next, based on the rules stored in the scenario storage unit 56, the information processing unit 62 determines the movement of the real object 120a according to the current state while a game or the like is progressing (S18). For example, when the real object 120a is moved so as to follow the real object 120b operated by the user, a direction vector from the real object 120a to the real object 120b is calculated, and the moving direction of the real object 120a is determined based on the direction. Furthermore, in time steps after time step t = 1, the velocity vector and acceleration vector of each real object may be obtained by taking the difference in position between the time steps. By using these pieces of information together, it is possible to perform more advanced motion control such as, for example, predicting the position of the real object 120b operated by the user and moving the real object 120a to be controlled.

  Even if the movement is not limited to the moving direction, if the movement in the direction in which it is improved is determined for each time step on the play field 20, the change in the surroundings can be dealt with flexibly. Real objects can be realized. Further, even in a complicated system where there are many real objects, the motion can be determined relatively easily by selecting the best directivity by statistical processing one by one.

  As described above, the information processing unit 62 not only changes the moving direction of the real object, but also the amount and direction of change of the movable part of the real object such as a joint, the lighting start of the light emitting element, the image to be displayed on the display built in the real object, The sound generated from a speaker with a built-in real object may be determined. Further, an image to be newly displayed on the display device 18 such as a projector or a sound effect generated from the speaker 19 may be determined. Then, the real object control unit 60 generates a control signal based on the determination result, adds image data and audio data as necessary, and transmits the control signal from the communication unit 50, thereby controlling the real object 120a (S20). At this time, the information processing unit 62 may appropriately output display image and audio data to the display device 18 and the speaker 19.

  While the process is not requested by the user via the input device 14 (N in S22), the processes from S14 to S20 are repeated for the next time step t = 1 (S24, S14 to S20). As described above, in the present embodiment, since the situation is recognized and the movement is determined in a minute time period such as the shooting period of the image frame of the moving image, the process is simplified rather than calculating the change in the long period. As well as being able to respond to changes in the situation in detail. When the user requests the end of the process, the entire process is terminated (Y in S22).

  By providing audiovisual equipment such as displays and speakers on real objects, it can be assumed that the immersive feeling will increase because the expression ability of each real object will increase, but there are also disadvantages such as increase in manufacturing cost, power consumption, and enlargement Arise. Therefore, instead of such a configuration, an image or light may be projected onto a real object by a projector or a projector connected to the information processing apparatus 10, or a sound that matches the movement may be generated by the speaker 19. Both may be performed simultaneously, or only one of them may be performed. In either case, it is possible to make it appear as if a real object is emitting light or uttering, and it is possible to realize as much expression as possible while suppressing the above disadvantages.

Next, specific modes that can be realized by using the configurations and methods described so far are exemplified. The forms described below are roughly divided into the following eight.
(1) Competition / competition: User and information processing device compete with each other / friends (2) Cooperation / Assistance: User and information processing device team up (3) Disturbance: Either information processing device or user is the other (4) Education: Information processing device teaches the user (5) Acting: Information processing device entertains the user (6) Recording / playback: Information processing device records / plays back the movement generated by the user (7) Imitation: imitation of movement generated by either information processing device or user (8) Activity: Information processing device works independently for some reason

  FIG. 9 shows an implementation example of “competition / competition” among the modes that can be realized in the present embodiment. In the play field 20, a real object 120a controlled by the information processing apparatus 10 and a real object 120b operated by the user are placed. In addition, the play field 20 includes miniatures such as trees 140 and houses 142a and 142b. These are real objects that do not have a drive mechanism and can be placed freely by the user. In the play field 20, a river 144, a bridge 146, and sand 148 are displayed. These may be projected by a projector or drawn on paper or the like. In the latter case, the information processing apparatus 10 acquires information on the position and range of those areas on the play field 20. When the play field 20 printed in such a manner is used, the information may be stored in the scenario storage unit 56. Even when the user draws on the spot, the information can be acquired by detecting from the captured image.

  In such an environment, the user moves the real object 120b to play against or compete with the real object 120a controlled by the information processing apparatus 10. Various rules can be considered in this case. For example, the speed to a predetermined goal such as the tree 140 may be competed or directly hit. In the process, the score may be lowered when hitting a placed real object such as the tree 140 or the houses 142a and 142b. Further, a laser irradiation means (not shown) may be provided on the real objects 120a and 120b so as to be regarded as a gun, and a score may be given if the object can be fired while aiming at the opponent's real object. When realizing such a battle game, the real objects 120a and 120b may have a shape such as a tank, for example. Further, when a traveling sound of a tank or the like is generated from the speaker 19 or the like in accordance with the movement of the real objects 120a and 120b, a more realistic feeling is given.

  In such a battle game, the information processing apparatus 10 controls the river 144 and the sand 148 displayed on the play field 20 so that the movements of the real objects 120a and 120b are restricted. For example, when the user moves his / her real object 120b in the direction of the river 144 where the bridge 146 is not applied, the user is forced to stop at the edge of the river 144. That is, even if the input device 14 is moved in the direction of the river 144, the information processing unit 62 controls the real object 120b so that the speed becomes zero at the edge of the river 144. As a result, the user resumes the movement by retreating or changing the direction of the real object 120b. In the sand 148, for example, the moving speed is slowed or the direction change is slowed. Also in this case, the information processing unit 62 controls the real object 120b by reducing the moving speed requested by the user to the input device 14 by a predetermined rate or reducing the response of the change in the steering angle.

  The speed and direction are calculated so that the same restriction is imposed on the real object 120a controlled by the information processing apparatus 10. By providing an entry prohibition area such as the river 144 outside the side surrounding the play field 20, the real objects 120 a and 120 b may be prevented from going out of the play field 20. In addition to imposing restrictions on movement, a region where speed and direction change can be accelerated may be provided. A team battle may be made by placing a plurality of real objects 120a and 12b. For example, a plurality of users may each bring their own real objects 120b to form a team and play against the same number of real objects 120a controlled by the information processing apparatus 10.

  FIG. 10 shows an example of data prepared in the scenario storage unit 56 in order to realize the form shown in FIG. When the play field 20 is an image projected by the projector, the scenario storage unit 56 prepares data of the image 110 to be projected. In the example shown in FIG. 9, a river 144, a bridge 146, and a sand 148 are drawn on the image 110. The information processing unit 62 reads the data of the image 110 from the scenario storage unit 56 at the start of the battle, and outputs the data to the projector that is the display device 18 for projection. The deformation rule of the image 110 may also be stored in the scenario storage unit 56, and the projected image may be changed over time. For example, by displaying an image in which the water of the river 144 overflows over time or the bridge 146 collapses at the timing when the real object 120b crosses, the real object and the image are fused, and a more thrilling battle realizable.

  Regardless of whether the image 110 is projected or the drawn play field 20 is used, as described above, the area of the river 144 and the sand 148 is acquired and the data 112 mapped to the plane corresponding to the play field 20 is prepared. In the mapping data, a motion limit amount in each region is set. In the above example, “V = 0” is set in order to set the speed to 0 in the region of the river 144 other than the bridge 146. Further, if the speed is limited to 80% in the area of the sandy ground 148, “V = 80%” is set.

  In the figure, the data 112 displays a dotted line region indicating the position of the real object acquired at each time step. This data is obtained by converting and mapping the position of the image detected from the captured image by the state specifying unit 52 into the position coordinates on the plane of the play field 20 (xy plane in the drawing). As described above, since individual identification information of each real object is obtained based on the appearance characteristics, the identification information “# T0”, “# T1” of the tree 140, and the house 142a are located in the vicinity of the area of each real object in FIG. , 142b identification information “# H0” and “# H1”, and real object 120a and 120b identification information “# C0” and “# C1” are also shown.

  The state specifying unit 52 creates individual identification information of each real object and information related to the position for each time step, and supplies the information to the information processing unit 62. Here, the positions of the real objects 120a and 120b (areas corresponding to the identification information “# C0” and “# C1”) are changed by a user operation or control by the information processing apparatus 10. The information processing unit 62 monitors whether or not they enter the area where the speed limit is imposed, and limits the speed by a set amount. In addition, the position of the real object 120a controlled by the information processing apparatus 10 (identification information “# C0”) is compared with the position of the real object 120b operated by the user (identification information “C1”), and the position is easily moved. For example, the optimum moving direction of the real object 120a at that time is determined and controlled. The handling of the data related to the position information is the same in the forms described later.

  FIG. 11 shows an implementation example of “cooperation / assistance” among the modes that can be implemented in the present embodiment. An annular road 150 and a pedestrian crossing 156 are displayed in the play field 20. Then, a real object 120a controlled by the information processing apparatus 10 and a real object 120b operated by the user are placed. In addition, miniatures such as trees 152a and 152b and a signal 158 and dolls 154a and 154b are placed in the play field 20. The trees 152a and 152b and the dolls 154a and 154b are real objects that do not have a driving mechanism, and can be freely placed by the user or moved by hand. The signal 158 includes blue, yellow, and red light emitting diodes, and is configured so that the user can switch the light emission color via the input device 14 or the information processing device 10. In any case, since the actual control is performed by the information processing apparatus 10, the information processing apparatus 10 can naturally recognize the switching of the emission color.

  In such an environment, the real objects 120a and 120b travel on the road 150 counterclockwise. The user causes the real object 120b that he / she operates to travel ahead of the real object 120a controlled by the information processing apparatus 10, and excludes from the road 150 objects that obstruct the traveling of the real object 120a controlled by the information processing apparatus 10. Thus, “cooperation” is performed so that the real object 120a can travel quickly. In the illustrated example, the tree 152b falls down and blocks the road 150, so the user pushes the tree 152b with the real object 120b to get out of the road 150. Even if the real object 120a controlled by the information processing apparatus 10 reaches the real object 120b, the real object 120a is stopped there until the tree 152b is eliminated.

  That is, the shorter the time required to eliminate the tree 152b, the faster the real object 120a can go around the road 150. In the example shown in the figure, there is only one set of the real object 120a controlled by the information processing apparatus 10 and the real object 120b operated by the user. You may fight. In such an aspect, the real object 120a controlled by the information processing apparatus 10 can be represented as an ambulance, and the real object 120b operated by the user can be regarded as a private car, thereby producing a situation such as a citizen cooperating to run the ambulance quickly. . Each real object may be in the shape of their car. Further, a driving sound or a brake sound of a car may be generated from the speaker 19 or the like in accordance with the movement of each real object.

  In such a battle, the above-described “disturbance” mode may be realized by interfering with the real object 120b operated by the user so that the real object 120a of the opponent team cannot pass. In this case, the information processing apparatus 10 can realize a movement that moves forward while avoiding interference by moving the real object 120a by moving away from the disturbing real object 120b and determining a direction with a small retraction amount. .

  Alternatively, the third real object to be controlled by the information processing apparatus 10 or operated by another user is prepared, and the real object 120a controlled by the information processing apparatus 10 rushes to the third real object. It is good also as a game which the real object 120b which a user operates obstructs going. In this case, if the third real object is an ambulance, the real object 120a controlled by the information processing apparatus 10 is a tank, and the real object 120b operated by the user is a private car, the ambulance that is likely to be attacked by the tank is protected by the private car. It is possible to produce a situation such as Or conversely, it is good also as a game which the real object 120a which the information processing apparatus 10 obstructs that the real object 120b which a user operates tries to connect with a 3rd real object.

  In addition, using the same play field 20, an “education” mode in which a child can learn traffic rules can be realized. In this case, for example, the child moves the dolls 154a and 154b by hand. While the signal 158 is red, the real object 120 a controlled by the information processing apparatus 10 continues to travel on the road 150. Then, when the signal 158 turns blue, it stops before the pedestrian crossing 156 so that the child can move the doll 154b or the like to cross the pedestrian crossing 156. When the signal 158 tries to cross in red or cross the road without a pedestrian crossing, the real object 120a or the speaker 19 may suddenly stop or generate a brake sound just before that. To recognize that it is a dangerous act.

  In this way, the target to which the real object 120a controlled by the information processing apparatus 10 reacts may be a thing that can be held and moved by hand instead of the real object 120b having the same configuration. Such a mode is particularly effective for a user who is difficult to operate with the input device 14 such as an infant. In the above-mentioned “cooperation” mode, the tree 152b may be excluded using a hand instead of the real object 120b. Conversely, in the “education” mode, the user operates the real object 120b instead of the dolls 154a and 154b, and generates an interaction with the real object 120a controlled by the information processing apparatus 10 while traveling on a more complicated road. By doing so, you may be allowed to learn the traffic regulations of automobiles.

  Moreover, in the same play field 20, it is good also as a rule which stops the real object 120a which the information processing apparatus 10 controls in front of the dolls 154a and 154b on the roadside. The user leaves the real object 120a in a state where the doll is put on the stopped real object 120a, and stops again in front of the miniature of the house placed separately. This makes it possible to play a taxi. It is a rule to stop the real object 120a in front of the miniature at the bus stop, and you can play with the bus by getting on and off the doll.

  FIG. 12 shows an implementation example of “act” among the modes that can be realized in the present embodiment. A plurality of real objects 120a controlled by the information processing apparatus 10 are placed in the play field 20, and the user appreciates playing a mass game with a unified movement. In this aspect, there is no need to interact with a real object operated by the user, but even if the user randomly places the real object 120a on the play field 20, the position is acquired from the captured image. Thus, it is possible to return to the set formation and start the mass game. Even if the user obstructs the real object 120a by hand during the mass game, the user can return to the specified position and continue the mass game.

  In this case, data in which the time change of the position coordinates of each real object 120a is set is stored in the scenario storage unit 56. As a result, the initial formation and the position in the middle are defined, so that the actual object 120a deviating from it can be returned to the defined position. The information processing unit 62 determines an optimal moving direction at each time step while referring to the setting of the position coordinates so as not to hinder the movement of the other real object 120a in the returning process.

  During the mass game, the information processing apparatus 10 may add effects by the movement of light such as applying a spotlight using a projector or a lighting device (not shown), or may play music from the real object 120 a itself or the speaker 19. . Moreover, the contents of the mass game and the music may be switched depending on the combination of the shape of the placed real object 120a and the color of the marker. Furthermore, some change may occur depending on the user's behavior. For example, when the applause or cheering sound made by the user when the mass game is finished, the mass game may be started again as an encore. In this case, when the size of the audio signal acquired by the microphone 16 exceeds the threshold value, the information processing unit 62 determines the start of encore.

  Furthermore, a mass game may proceed in accordance with the user's command, or the rhythm, tempo, and tune may be switched. For example, when the user presses a predetermined button of the input device 14 with a desired time signature, the real object 120a is moved by the movement according to the time signature. The voice for which the user counts may be acquired by the microphone 16. Alternatively, the imaging device 12 may take a picture of a user actually conducting a command by hand or shaking a marker that imitates a command stick, and the time may be adjusted.

  One of the plurality of real objects 120a in this example may be a real object operated by the user or a real object moved by the user's hand. At this time, the information processing apparatus 10 controls the other real object 120a so as to move in accordance with the real object. The other real object 120a may move with the same movement as the real object moved by the user, or the other real object 120a may move in a line with the real object moved by the user. Good. These correspond to the “imitation” forms described above.

  Alternatively, another real object may be surrounded or aligned so that the real object moved by the user is always at the center, so that the real object moved by the user may move. In this case, the distance and positional relationship with the real object moved by the user, the distance and positional relationship between the other real objects 120a, and their temporal changes are set in advance and stored in the scenario storage unit 56. If the next moving direction is calculated for every time step for all the real objects 120a to be controlled, a movement that matches the real object that the user moves can be realized.

  These aspects including the actual objects operated by the user may be enjoyed only by the user on the spot, or the time change of the position of each actual object is recorded so that it can be reproduced later at the timing requested by the user. Thus, the above-described “recording / reproduction” may be realized. At the time of reproduction, the movement of the real object that was originally operated by the user is also reproduced by the real object 120a controlled by the information processing apparatus 10. As a result, it is possible to provide the pleasure of creating a mass game or a march with a desired movement.

  In addition, a real object operated by the user may be mixed in a plurality of real objects 120a controlled by the information processing apparatus 10, and the game may be challenged to execute a mass game as illustrated. In this case, the information processing device 10 plays music from the speaker 19 and causes the display device 18 to display a moving image representing a model movement. The user places an actual object that he / she operates in one of the initial formations, and operates it so that he / she follows the example. The information processing unit 62 controls other real objects and detects an event in which the real object operated by the user deviates from the model based on the captured image. Then, a final score is calculated by deducting points according to the number of deviations and displayed on the display device 18 at the end of the mass game.

  As described above, when the user operates the real object 120b, the information processing apparatus 10 may assist the operation. Specifically, if the actual movement of the real object 120b is different from the movement requested via the input device 14 due to some circumstances, fine adjustment is made to the control signal reflecting the operation content to the input device 14 To make the actual movement closer to the request. For example, when an excessive load is applied to the real object 120b, it may be meandering even if the user's operation is straight ahead. In such a case, the information processing unit 62 adjusts the rudder angle in a direction to suppress meandering so that the real object 120b can actually go straight.

  Specifically, the actual difference based on the change in the position of the real object in the photographed image or the measurement value by the sensor is compared with the user's operation content input from the input device 14, whereby the request and the actual difference are To detect. And a control signal is adjusted so that a difference may become small. This form corresponds to the “auxiliary” described above. Note that the assist target is not limited to that related to the traveling of the real object, but the same applies to the movement of the gripper and the arm. Further, such adjustment can be performed at any time when any other mode is implemented.

  The examples shown so far are modes that can be realized mainly by using the running of the real object, but the modes can be diversified by adding functions to the real object. FIG. 13 shows an example of a mode realized when a function for grasping an object is added. In the play field 20, a real object 120a controlled by the information processing apparatus 10, a real object 120b operated by a user, and a plurality of blocks 162 are placed. The block 162 may be a lump such as a synthetic resin having no mechanism inside. Each real object 120 a and 120 b includes grippers 160 a and 160 b for gripping the block 162.

  For example, the user operates the real object 120b via the input device 14 to bring it close to one block 162, and opens and closes the gripper 160 to grip it. Then, the grasped block 162 is carried into his position 164b displayed on the play field 20. Similarly, the information processing apparatus 10 controls the real object 120a to move the block 162 to another position 164a. The winner is the one who brings more blocks 162 to his position. Thereby, the above-described “match” and “disturbance” modes can be realized.

  The grippers 160a and 160b sandwich the block 162 from the left and right sides by opening and closing. However, the block 162 may be lifted so that the root portion can be moved in the vertical direction. By providing a gripper at the tip of the arm capable of controlling the joint angle, a more complicated operation such as lifting the block 162 to a high position or turning the block 162 upside down may be performed. Instead of the gripper, a forklift mechanism for inserting and lifting a claw between the block 162 and the floor may be provided. Or you may make it carry the block 162 on a loading platform by making a real object into the form of the crane vehicle with a loading platform. In accordance with the form of these real objects, the shape of the block 162 is also appropriately optimized to facilitate carrying.

  FIG. 14 shows another example of a mode realized when a function for grasping an object is added. This example realizes the above-mentioned form of “cooperation / assistance”. In the play field 20, a real object 120a having a gripper and a plurality of blocks 170 having various colors, which are controlled by the information processing apparatus 10, are placed. The user 8 assembles the block 170 in the work area 172 provided in the play field 20. As shown in the figure, they may be assembled three-dimensionally or arranged in a plane. In the former case, it may be simply placed like a building block, or the blocks 170 may be assembled as a mutually connectable structure.

  In any case, the user 8 specifies the color of the next necessary block by voice during assembly. In the figure, “RED! (Red!)” Is designated. Then, the real object 120a searches for the block of the designated color among the blocks 170 in the area other than the work area 172 on the play field 20, and carries it close to the user. If the work area 172 is fixed, it can be estimated that the user 8 is in the vicinity thereof. The information processing unit 62 recognizes the designated color based on the audio signal acquired by the microphone 16 and controls the block 170 of that color to be grasped by the real object 120a and carried to the estimated position.

  The position of the user 8 may be detected by providing a rule that the user 8 is in the field of view of the imaging device 12. In this case, even if the work area 172 is not clearly defined, if it is carried from the block 170 placed farther from the detected position of the user 8, it is not necessary to carry the block being assembled. The designation of the block by voice is not limited to color, and any attribute such as size and shape may be used, or a combination thereof. In this form, the user can assemble the blocks as he / she likes, and the real object 120a can carry out the work efficiently by carrying blocks randomly selected in the process. Since hands are used for assembling, by realizing the designation of the block by voice, the work efficiency is not reduced by the designation.

  On the other hand, a model selected from a completed model prepared in advance may be assembled. In this case, the user selects an object to be assembled from the models displayed on the paper or the display device, and designates it using the input device 14. The scenario storage unit 56 of the information processing apparatus 10 stores an assembly order and block identification information used for each model. Accordingly, it is possible to realize a mode in which the real object 120a selects and carries the next necessary block by determination by the information processing apparatus 10 without designating the block by the user. At this time, information relating to how the carried blocks are connected may be displayed on the display device 18 or projected onto the play field 20.

  The real object 120a may sort the plurality of blocks 170 according to a predetermined standard such as color, shape, and size. In this case, the information processing unit 62 first specifies the number of types of the blocks 170 on the play field 20, for example, the number of colors based on the captured image. Then, the corresponding number of areas are set on the play field 20. Then, by repeating the control of bringing the block 170 held by the real object 120a to the corresponding area according to the attribute such as the color, a group of blocks 170 is formed for each type. This makes it much easier to find the desired block 170, especially when there are a large number of blocks 170. As shown in FIG. 14, the user 8 may perform the sorting operation in parallel during the assembly of the block 170, or may be performed not only during the assembly but also during a play game described later, for example. Further, the blocks may be cleared on the real object 120a by collecting scattered blocks 170 in one place regardless of the attribute.

  In addition to the work area 172 of the user 8, a work area for the real object 120a may be provided, and the block 170 may be assembled in parallel with the real object 120a. For example, the user and the real object 120a assemble the same model selected by the user separately. If the user 8 assembles the actual object 120a as an example, the assembly procedure is equivalent to that shown. Alternatively, the completed form may be displayed on the display device 18 and simultaneously assembled to compete for the speed to completion. You may assemble one thing in cooperation in one work area. When the real object 120a is assembled as described above, the blocks 170 need only be placed at the position when arranged in a plane, but when the three-dimensional assembly is performed, the above-described arm or crane is provided on the real object 120a. Thus, the block 170 can be lifted to a high position.

  However, when assembling while being connected by some kind of connection mechanism rather than simply being placed like a building block, it is necessary to consider the force required for connection and its reaction. For example, when a method of fitting a convex portion into the concave portion of the block is adopted as a general connection method, a large force is required for the arm of the real object 120a. In addition, the stability of the real object 120a may not be maintained due to the reaction. Therefore, the connection state may be realized by the control of the information processing apparatus 10 with the block itself as a control target of the information processing apparatus 10. FIG. 15 is a diagram for explaining an example of a block connection method in a mode in which a real object assembles a block. This figure is a side view of the process of connecting the blocks by fitting the cylindrical concave portions 176 provided in the block 170b into the columnar convex portions 174 provided in the block 170a.

  First, in the state (a) in which the block is not connected, the diameter r2 of the concave portion 176 of the block 170b is larger than the diameter r1 of the convex portion 174 of the block 170a. When the state where the concave portion 176 of the block 170b is fitted into the convex portion 174 of the block 170a as in the state (b) is detected, the information processing unit 62 performs the concave portion 176 of the block 170b as in the state (c). The convex portion 174 is tightened by the internal mechanism. As a result, the force required for the real object 120a in the fitting operation can be equivalent to lifting the block 170b and placing it on the block 170a, and the internal mechanism strengthens the connection, so that the assembled object collapses even when tilted. You can avoid it.

  Here, the concave portion 176 has a clamp structure in which at least a part of the inner wall is narrowed by, for example, rotation of a screw. When the control target real object 120a places the block 170b on the block 170a, the information processing unit 62 transmits a control signal for operating the actuator that rotates the screw to the block 170b to tighten the screw. The convex portion 174 of the block 170a is sandwiched. On the contrary, when the connection is removed from the state (c), the state may be shifted to the state (b) by loosening a screw. The block connection method is not limited to this, and an air suction mechanism may be provided in the connection portion, and the connection surface may be connected by forming a vacuum on the connection surface under the control of the information processing apparatus 10. Alternatively, the connection may be made by using an adhesive surface such as a hook-and-loop fastener, and when removing, the eject pin may be protruded and peeled off under the control of the information processing apparatus 10.

  Further, two gripper arms may be provided on the real object 120a so that the main body of the real object 120a does not lose stability due to a reaction at the time of connection. Accordingly, if both the blocks 170a and 170b to be connected are held and connected, the reaction is unlikely to occur. Alternatively, if the play field 20 can be connected to the block 170a that is in contact with the block 170b and the block 170a that is the connection destination is fixed when the block 170b is connected, the real object 120a is less likely to lose stability.

  On the other hand, the mode of using the voice command in the environment as shown in FIG. 14 can be used not only for assembling blocks but also for playing. For example, the block 170 is likened to fruits and vegetables such as bananas, apples and melons depending on the color. When the user 8 says “Please give me a banana”, the real object 120a carries the yellow block 170 to the vicinity of the user 8. At this time, a sound such as “Yes, please” is generated from the real object 120a or the speaker 19 at the same time, thereby making it possible to produce a situation such as shopping at a fruit and vegetable store.

  In this case, the scenario storage unit 56 stores the color of the block 170 in association with keywords such as “banana”, “apple”, and “melon”. When the information processing unit 62 detects a keyword from the audio signal acquired by the microphone 16, the information processing unit 62 controls the real object 120a so as to carry the block 170 of the corresponding color. The block 170 may be shaped like an actual fruit or vegetable. The blocks may be associated with rough classifications such as fruits and vegetables, meat, fish, etc. and produced as if shopping at different stores such as fruit and vegetable stores, butchers, and fish stores. At the start of this game of play, the area of each store may be formed on the play field 20 by separating the blocks according to the color of the real object 120a as described above. Further, by placing a block that the user 8 regards as money on the play field 20, the real object 120 a may take it away and make it more like shopping.

  The block provided by the user 8 may be regarded as a laundry, and the real object 120a may take away the block in response to a voice such as “Cleaner, take this with you”. In this case, a keyword such as “Take this with you” and a rule of movement that grabs a block placed in the vicinity of the user 8 and puts them together in a predetermined area on the play field 20 are associated with each other. It is stored in the storage unit 56. A block may be regarded as an animal or a character, or a figure having such a shape may be used instead of the block 170. These modes are those in which real objects "cooperate" with pretend play, but they can also be used for the purpose of "education" in the sense that an infant learns how to shop.

  FIG. 16 shows still another example of a mode realized when a function of grasping an object is added. This example realizes the above-described "match" form, and the play field 20 is a board game board. In the play field 20, an actual object 120a having a gripper and pieces 180 and 182 used for a board game, which are controlled by the information processing apparatus 10, are placed. The pieces 180 and 182 may be a lump of synthetic resin or the like having no mechanism inside. Further, the shapes of the pieces 180 and 182 and the grids of the board are not limited to those shown in the drawings, and may be determined as appropriate depending on the game. The game played here may be a general game such as chess, shogi, and reversi.

  The real object 120a moves his piece 180 using a gripper, and the user 8 moves his piece 182 by hand. The information processing unit 62 determines the piece 180 to which the real object 120a should move and the destination to move according to the battle situation including the position of the piece 182 moved by the user. Such a strategy itself may be the same processing as a program in a conventional computer game. Instead of the real object 120a becoming an opponent, the real object 120a may play an auxiliary role in a situation where two users are competing. For example, when the opponent's piece is picked up by one of the users with chess or shogi, the piece is carried toward the user. In the case of reversi, the front and back of the piece are reversed. In either case, the scenario storage unit 56 stores rules for each game, particularly rules for changing other pieces by each hand. This aspect is not limited to a board game, and can be similarly realized in a card game or a six game.

  Further, a battle game may be realized in which the pieces 180 and 182 are character figures and attacked by making the opponent figure touch the body. In this case, the real object 120a avoids being hit or bumped against the figure of the user 8 by grasping or pushing his / her own figure. Alternatively, an actual object having a gripper is also placed on the user side, and the user operates to oppose it. Further, a third real object controlled by the information processing apparatus 10 may be introduced so that the weapon held by the figure is brought from the place where the weapon is placed. At this time, the user may be able to designate a weapon by voice, as shown in FIG.

  FIG. 17 shows an example of an aspect realized when a function for holding a pen is added to a real object. In the play field 20, a real object 120a controlled by the information processing apparatus 10 and a real object 120b operated by the user are placed. Each real object 120a, 120b includes a holding mechanism 190a, 190b that can hold the pen downward. The holding mechanisms 190 a and 190 b are configured to be able to change the pen height so that the pen tip can be brought into contact with or separated from the play field 20 by a control signal from the information processing apparatus 10. Then, when the real objects 120a and 120b move while the pen tip is in contact, a line drawing corresponding to the movement is created on the play field 20.

  For example, according to the line drawing drawn by the real object 120b operated by the user, the real object 120a controlled by the information processing apparatus 10 draws a line drawing so as to compensate for the line drawing. Complete the picture. At this time, the information processing apparatus 10 may decorate the user's original line drawing according to a predetermined rule, or a complete picture of a sample picture may be displayed on the display device 18 and both may draw for that purpose. You may make it go. In the former case, a rule regarding how to decorate the user's line drawing is stored in the scenario storage unit 56. In the latter case, the completed form of the picture is stored in the scenario storage unit 56, and the deficient portion is drawn on the real object 120a by the control of the information processing apparatus 10 by comparing with the progress of drawing at each time step.

  The play field 20 may be a replaceable paper or a surface-treated plate that can be redrawn by erasing line drawings. The above is a form of “cooperation / assistance” using a pen. However, the form of “activity” may be realized by drawing only a real object 120 a controlled by the information processing apparatus 10. In this case, for example, the user designates an object to be drawn by voice or an operation via the input device 14. The information processing unit 62 reads out a picture associated with the designated object from the scenario storage unit 56 and draws the picture as it is, thereby realizing an interaction with the user. Alternatively, the user may give points to the picture drawn by the real object 120a or correct the picture. At this time, the user may directly write the score and correction to the play field 20 with a pen held in his hand.

  The information processing apparatus 10 acquires and learns the score and the corrected portion from the captured image. For example, a picture with a bad score below the threshold is not drawn again because it is not the user's preference. The picture corrected by the user is reproduced when the picture is drawn again. Therefore, the information processing unit 62 associates the score of the user and the corrected picture data with the completed picture data stored in the scenario storage unit 56. In this aspect, instead of actually holding the pen on the real objects 120a and 120b, a projector is used to project an image on which a line is drawn on the path through which the real objects 120a and 120b have passed. Can also be realized. This eliminates the need to change paper or erase lines, and can easily store the resulting picture electronically.

  FIG. 18 shows an example of a mode in which an interaction is generated between a real object and a user's body. In the play field 20, a real object 120a controlled by the information processing apparatus 10 is placed. Then, the real object 120 a self-runs so as to bounce off three sides of the four sides of the edge of the play field 20. The user protects the remaining side with his / her hand 196. That is, the real object 120a moves like a ball of a pinball, but by hitting it back with the hand 196, a "matching / competition" form is realized.

  The information processing unit 62 physically calculates the movement and rebound of the ball when it is assumed that the play field 20 is inclined at a predetermined angle, and moves the real object 120a as such. On the other hand, the state specifying unit 52 detects the image of the hand 196 from the captured image and tracks it with an existing method. As a result, the information processing unit 62 can also reproduce the manner in which the ball rebounds by the movement of the hand with the real object 120a. Even if the user does not actually hit back the real object 120a, if the hand 196 catches up with the real object 120a before the real object 120a crosses the side, the success of the hit is determined, and the real object 120a is controlled to return to the opposite side. To do. If it does not catch up, the real object 120a goes out of the play field 20 and loses the user. The inside of the play field 20 may be surrounded by a block, and the real object 120a may be moved so as to bounce off the block.

  Furthermore, a block may be placed as a rebound point in the interior. Even if the user places the block in a free position or shape, the position coordinates of each block can be determined from the captured image, so the speed and moving direction of the real object 120a can be determined for each time step by simple physical calculation. . If the material is assumed according to the color of the block, the change in the coefficient of restitution can be reflected in the speed calculation. When rebounding, a rebounding sound may be generated from the speaker 19 or the like.

  As another example of the interaction between the real object 120a and the hand 196, a “competition” mode in which the hand 196 follows the real object 120a may be realized. In this case, a plurality of real objects 120a may be placed so as to escape all at once according to the movement of the hand 196. The information processing unit 62 determines the moving direction of each real object 120a so as to move away from the hand based on the position information of the hand at each time step. At this time, if the moving direction is determined by generating random numbers or the like so that the directions are dispersed as much as possible, it becomes difficult to catch and the game performance is further improved. The user may be in a state where the user does not actually grab the real object 120a but has caught it in a predetermined area of the play field 20. By making the real object 120a into the shape of a living creature such as sheep, a mouse, or a fish, or generating a cry from the speaker 19, a more realistic feeling can be given.

  In the examples described so far, the movement of the real object is driven by the actuator. Furthermore, play can be diversified in combination with real objects that move due to naturally occurring forces such as gravity. FIG. 19 shows an example of a “cooperation” or “act” mode in which ball rolling is included in the movement of a real object. In this example, a device such as a Rube Goldberg machine is assumed in which other objects and devices move so as to respond to the rolling of the ball 200. In the play field 20, several real objects 120a controlled by the information processing apparatus 10 and some real objects 120b operated by the user are placed. In addition, an approximate course in which the ball 200 rolls is formed by the user in the block 202.

  Real objects 120m, 120n, and 120p are further placed as real objects controlled by the information processing apparatus 10. The real object 120m is composed of a quadrangular prism main body and a slide. The quadrangular column is provided with holes 204 and 206 at the lower part and the upper part, and a mechanism for raising and lowering the internal cavity like an elevator. The actuator that moves the table up and down is driven by a control signal from the information processing apparatus 10. In addition, the real objects 120n and 120p are provided with a mechanism that causes the whole to emit light or generate a sound effect according to a control signal from the information processing apparatus 10. As described above, the real objects 120n and 120p may appear to emit light by projecting light from the projector, or sound effects may be generated from the speaker 19. In such a configuration, the real object 120a controlled by the information processing apparatus 10 puts the ball 200 carried by the gripper into the hole 204 below the real object 120m.

  Then, under the control of the information processing apparatus 10, the platform on which the ball 200 is placed rises inside the quadrangular column of the real object 120m, and the ball 200 comes out of the upper hole 206 by tilting the platform. As a result, the ball 200 rolls on the slide and collides with the real object 120n. When the collision of the ball 200 is detected based on the captured image, the information processing apparatus 10 reacts by causing the real object 120n to emit light or generate sound. The ball 200 that bounces and rolls off from the real object 120n is guided to a desired direction by the user operating the real object 120b to bounce or push it. Next, the real object 120a controlled by the information processing apparatus 10 is rebounded or pushed into the opening of the real object 120p, and the ball 200 thus rolled is set as a goal.

  At this time, the information processing apparatus 10 tracks the position of the ball 200 based on the captured image, moves the real object 120a to a position where it rebounds in the direction of the real object 120p, or pushes the ball 200 with the real object 120a. Further, it is detected from the photographed image that the ball 200 has entered the opening, and the real object 120p is caused to emit light or a sound is generated to produce the goal moment. As described above, in the present embodiment, since the change in the field is detected based on the captured image, various functions of the real object can be activated in accordance with the movement of the ball that is not controlled. The scenario storage unit 56 stores the rules related to the movement of the ball 200 as described above and the movement of the real object according to the movement.

  Although the course shown in the figure has a relatively simple configuration, a domino-inclined portion in which flat plates arranged one after another due to the collision of the balls 200 are collapsed or the ball 200 is jumped by releasing the contracted spring by an actuator. More complicated configurations such as, etc. may be added. Moreover, you may make it the moving main body switch in the middle by telling the force which the ball 200 rolls to another thing. By combining various basic real objects such as real objects 120m, 120n, and 120p, a unique device that combines the movements of the natural force and the movements of the drive system is created according to the user's ideas. be able to.

  You may implement | achieve the form of the battle and competition illustrated in FIG.9,11,16 etc. via a network between several users. FIG. 20 is a diagram for explaining a mode in which a plurality of users participate in one game using a network. In the example of the figure, the information processing systems 1a, 1b, and 1c are similarly constructed in three places. The information processing apparatuses 10a, 10b, and 10c of each system are connected to the server 210 via the network 212. In the play field of the information processing system 1a, real objects that are moved by the users of the information processing systems 1b and 1c via the network 212 are placed in addition to the real objects operated by the users there.

  Similarly, in the other information processing systems 1b and 1c, the real object operated by the user and the real object operated by the user of the other information processing system via the network 212 are placed on the play field. Therefore, in the illustrated example, three real objects are placed in the play fields of all the information processing systems 1a, 1b, and 1c. Before the game is started, the server 210 associates the information processing system of the participating user with the real object, and notifies the information processing apparatuses 10a, 10b, and 10c. When each user of the information processing systems 1a, 1b, and 1c operates his / her real object via the input device, the server 210 notifies the other information processing system of the information.

  Each of the information processing apparatuses 10a, 10b, and 10c moves a corresponding real object in accordance with an operation of the user via the input device 14, and responds to each of them according to operation information of other users notified from the server 210. Move a real object. As a result, in all the information processing systems 1a, 1b, and 1c, the real object corresponding to each user moves in the same manner. Further, the number of real objects may be increased, and at least one of the server 210 or the information processing apparatuses 10a, 10b, and 10c may determine the movement. In this way, for example, the battle game is executed in the play field as shown in FIG.

  As described above, if a real object is used as a tank or a score is given by shelling, it is possible to realize a situation in which a user at a distant place actually fights in the world of a miniature garden. Here, since the actual movement of the real object is controlled by the information processing apparatuses 10a, 10b, and 10c, the game can be made more interesting by performing various adjustments. For example, the number of times the user has played in the past and the battle record are recorded in the server 210, and the information processing apparatuses 10a, 10b, and 10c are notified at the start of the game. The information processing apparatuses 10a, 10b, and 10c adjust the movement of the real object according to the information.

  For example, if it is determined by comparison with a threshold value that the number of battles of a certain user is low or the performance is not good, a handicap is given to the user, and the real object of another user is moved at a higher speed, or a single bombardment Score higher than other users. In the board game, the number of the first piece is made different. You may divide the proficiency level into multiple levels. By doing so, since it is possible to fight equally regardless of age and proficiency level, it is possible to enjoy the game without selecting an opponent. Moreover, depending on the case, it is also possible to select that only users with the same skill level fight without handicap.

  According to the present embodiment described above, the position and movement of a real object on the play field are detected using an image obtained by photographing the play field. Then, the information processing apparatus controls and moves any of the real objects according to a preset rule. This makes it possible to create various spaces in the real world where objects react in various ways. At this time, by determining the movement of the real object at each time step of a minute time interval, it is possible to make it appear as if it is responding to the movement of the other real object flexibly with a relatively simple calculation.

  In addition, a real object that is operated by the user via the input device or a real object that is placed or moved by a hand is introduced to influence the movement of the real object that is controlled by the information processing apparatus. Thus, play such as a computer game and collaborative work with a device can be realized using a real object, and even an infant who is not familiar with an input device can easily play and use it. The same effect can be obtained by acquiring a voice and reacting a real object to it.

  By moving the real object operated by the user via the information processing apparatus, it is possible to adjust the moving speed to be limited depending on the place on the play field or to give a handicap by the user. As a result, various adjustments that were easy in a computer game are possible in the real world, and the rules can be made more complex and multifunctional. Moreover, even if it is the same game using the same play field, the difficulty level can be changed easily. In addition, since the movement of the real object is determined based on the captured image, it is possible to interact with the real object controlled by the information processing device and the image or output sound displayed in the play field, the movement of an object or human body that does not have a drive system It becomes. If the image is projected by the projector, the real object and the image can be linked, and thus the world view to be constructed can be easily changed.

  Furthermore, if a system of a plurality of users is connected via a network, it is possible to realize a situation where a user who is in a remote place plays with a real object. Also in this case, since the local information processing apparatus controls the real object, it is possible to make adjustments such as restriction of operation and handicap by the user. In addition to the user, the information processing apparatus can be included as a participating member. For example, it is possible to realize a relatively large aspect in which a multi-user Allied Force and an information processing device army battle each other using a large number of real objects.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  DESCRIPTION OF SYMBOLS 1 Information processing system, 10 Information processing apparatus, 12 Imaging apparatus, 14 Input apparatus, 16 Microphone, 18 Display apparatus, 19 Speaker, 20 Play field, 22 CPU, 24 GPU, 26 Main memory, 50 Communication part, 52 State specification part 54 real object information storage unit, 56 scenario storage unit, 60 real object control unit, 62 information processing unit, 106a drive unit, 108a communication unit, 120a real object, 122 wheel, 126 marker, 128 motor.

Claims (23)

A state specifying unit that sequentially acquires image frames of moving images taken by the imaging device and detects state information of a real object existing in the object space at a predetermined time interval by detecting images in the image frames;
An information processing unit that determines the operation content of the real object to be controlled among the real objects at the time interval according to a predetermined rule based on the state information detected by the state specifying unit;
A real object control unit that controls the real object to be controlled to operate according to the content determined by the information processing unit;
An information processing apparatus comprising:   The said real object control part further controls so that the real object of a user's operation target among the said real objects may operate | move by the content which reflects user operation via an input device. Information processing device.   The real object control unit performs control so that light emitting elements provided in the real object to be controlled and the real object to be operated by the user are in different light emission states, and either one of the real objects is predetermined. The information processing apparatus according to claim 2, wherein a distinction between the real object to be controlled and the real object to be operated by the user is clearly indicated to the user by the difference in the light emission state. .   The information processing apparatus according to claim 1, wherein the information processing unit generates a sound determined based on the state information from a speaker.   5. The information processing apparatus according to claim 1, wherein the information processing unit gives a predetermined restriction to an operation according to a position where the real object exists.   The said information processing part produces | generates the image projected on the said to-be-photographed space, displays it on a projector, and changes the said image according to the predetermined rule based on the said status information. An information processing apparatus according to claim 1. The real object control unit causes the real object to travel by rotation of a wheel of the real object to be controlled,
The information processing unit is a position of at least one of another real object traveling by a user operation, another real object placed by the user, a real object that moves without using a driving system, and a part of the user's body The information processing apparatus according to claim 1, wherein a traveling direction of the real object to be controlled is determined according to a relationship.   The information processing unit determines the content of the subsequent operation of the real object using the state information of the real object to be controlled that is operating under the control of the real object control unit. The information processing apparatus according to claim 1.   The information processing unit determines an adjustment amount according to a predetermined rule with respect to a movement that occurs when a user operation via the input device is reflected as it is, and the real object control unit operates with the adjusted movement. The information processing apparatus according to claim 2, wherein the real object to be operated is controlled.   The information processing unit according to claim 9, wherein the information processing unit determines the adjustment amount so as to reduce a difference between a motion requested by a user operation and an actual motion based on the state information. apparatus.   The information processing apparatus according to claim 9, wherein the information processing unit determines the adjustment amount based on a user's individual operation history.   The information processing unit determines the content of the operation of the real object to be controlled so as to continue to move in a predetermined positional relationship with respect to the real object to be operated by the user, and the real object to be controlled by the user and the control target The movement recorded by the real object to be controlled is reproduced at another timing by storing data that records the time change of the position of the real object in a storage device. 2. The information processing apparatus according to 2. The real object control unit controls a real object to be controlled so that another real object is transported by a transport mechanism included in the real object,
The information processing apparatus according to claim 1, wherein the information processing unit determines the content of the operation of the real object to be controlled so that a predetermined creation is completed by the conveyed real object. . The creation includes a real object transported by a transport mechanism included in a real object operated by a user,
The information processing apparatus according to claim 13, wherein the information processing unit determines an operation content of the real object to be controlled so as to reduce a difference between a creation process and a completed body. The information processing unit determines the content of the action of the real object to be controlled so that a plurality of real objects constituting the creation are placed in contact with each other,
The information processing apparatus according to claim 13 or 14, wherein the real object control unit controls a coupling mechanism of the real objects so that real objects placed in contact with each other are coupled.   The real object control unit further controls a real object corresponding to the other information processing apparatus so as to operate with a content reflecting an operation on an input device connected to the other information processing apparatus connected via the network. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus.   The information processing unit further recognizes a word uttered by a user from an audio signal acquired by a microphone, and determines a content of operation of a real object to be controlled among the real objects according to a predetermined rule based on the word in the time interval. The information processing apparatus according to claim 1, wherein the information processing apparatus is determined by:   The information processing unit determines the content of the operation of the real object to be controlled so that a line drawing is drawn on a horizontal plane of the subject space according to a predetermined rule by a drawing tool attached to the real object. The information processing apparatus according to claim 1 or 2. An information processing system including an information processing device and a real object that moves under the control of the information processing device,
The information processing apparatus includes:
A state specifying unit that sequentially acquires image frames of moving images taken by the imaging device and detects state information of a real object existing in the object space at a predetermined time interval by detecting images in the image frames;
An information processing unit that determines the operation content of the real object to be controlled among the real objects at the time interval according to a predetermined rule based on the state information detected by the state specifying unit;
A real object control unit that controls the real object to be controlled to operate according to the content determined by the information processing unit;
An information processing system comprising: A communication unit that receives a control signal from the information processing device;
A drive unit for operating the actuator according to the received control signal;
A real object system including a plurality of real objects that operate based on the control signal,
Any one of the real objects operates based on a control signal reflecting a user operation performed via an input device connected to the information processing apparatus, and another real object has a movement determined by the information processing apparatus. An actual object system that operates based on a reflected control signal. Detecting sequentially the state information of the real object existing in the object space at predetermined time intervals by sequentially acquiring the image frames of the moving images taken by the imaging device and detecting the images in the image frames;
Determining a content of operation of a real object to be controlled among the real objects at the time interval according to a predetermined rule based on the detected state information;
Controlling the real object to be controlled to operate with the content determined in the determining step;
An information processing method by an information processing apparatus, comprising: A function of sequentially acquiring image frames of a moving image captured by the imaging device and detecting state information of a real object existing in the object space at a predetermined time interval by detecting an image in the image frame;
A function of determining the content of the operation of the real object to be controlled among the real objects at the time interval according to a predetermined rule based on the detected state information;
A function of controlling the real object to be controlled to operate according to the content determined in the determining step;
A computer program for causing a computer to realize the above. A function of sequentially acquiring image frames of a moving image captured by the imaging device and detecting state information of a real object existing in the object space at a predetermined time interval by detecting an image in the image frame;
A function of determining the content of the operation of the real object to be controlled among the real objects at the time interval according to a predetermined rule based on the detected state information;
A function of controlling the real object to be controlled to operate according to the content determined in the determining step;
A computer-readable recording medium having recorded thereon a computer program that causes a computer to realize the above.

Images