JP7299478B2 - Object attitude control program and information processing device - Google Patents

Description

The present invention relates to an object attitude control program and an information processing apparatus.

In recent years, with the development of electronic devices, it has become possible to display high-definition still images and moving images on terminals such as smartphones and personal computers. Furthermore, research and development of information processing technology is progressing, and a display technology that combines virtual space and real space using augmented reality (AR) is becoming practical. For example, Patent Literature 1 discloses a technique for displaying an object arranged in a virtual space in a display area of an information terminal.

JP 2015-41126 A

On the other hand, in order to control the posture of an object displayed in the display area, the input processing may become complicated when specifying the target part of the object.

In view of such problems, one of the objects of the present invention is to easily control the orientation of an object placed in a virtual space.

According to one embodiment of the present invention, the information processing apparatus superimposes a first image representing an object having a plurality of parts on a second image captured by an imaging section and displays the image on the display section. 2. An object for controlling the posture of the object by changing the position information of at least a part of a plurality of parts of the object to be operated based on the positional relationship with the operating body in the image. An attitude control program is provided.

In the object posture control program, a process for notifying a user may be executed when the position of the operating tool and the position of the operated part at least partially overlap.

In the object posture control program, the position of the operated part is changed based on a change in the position information of the operating body while the position of the operating body and the position of the operated part overlap at least partially. may

In the object posture control program, the position of the operated part and the position of the operated part may be defined as positions in a coordinate system of a space in which the object is arranged.

According to one embodiment of the present invention, there is provided an information processing apparatus for controlling the posture of an object, in which a first image representing an object having a plurality of parts is superimposed on a second image captured by an imaging unit and displayed on a display unit. and changing the position information of at least a part of the plurality of parts of the object to be operated based on the positional relationship between the object and the operating body in the second image to change the posture of the object. An information processing device is provided.

By using an embodiment of the present invention, the pose of an object can be easily controlled.

3 is a diagram showing the hardware configuration of the object attitude control system according to the first embodiment of the present invention; FIG. 4 is a functional block diagram of an object attitude control unit according to the first embodiment of the present invention; FIG. FIG. 4 is a flowchart of object attitude control processing according to the first embodiment of the present invention; FIG. 4 is a flowchart of superimposed display processing according to the first embodiment of the present invention; It is an example of alignment between the coordinates of the virtual space and the coordinates displayed on the information processing device in the superimposed display process according to the first embodiment of the present invention. It is an example of the coordinates of each part of the object according to the first embodiment of the present invention. It is an example of the user interface displayed on the display unit of the information processing apparatus in the superimposed display process according to the first embodiment of the present invention. FIG. 4 is a flowchart of first object orientation detection processing according to the first embodiment of the present invention; 6 is an example of a user interface displayed on the display unit of the information processing apparatus in the first object orientation detection process according to the first embodiment of the present invention; FIG. 9 is a flow diagram of second object orientation detection processing according to the first embodiment of the present invention; 8 shows the position coordinates of the operating body before and after movement and the position coordinates of the object in the second object orientation detection process according to the first embodiment of the present invention; It is an example of a user interface displayed on the display unit of the information processing device in the second object orientation detection process according to the first embodiment of the present invention. FIG. 10 is a functional block diagram of an object attitude control section according to the second embodiment of the present invention; FIG. 11 is a flowchart of first object orientation detection processing according to the second embodiment of the present invention; It is an example of the user interface displayed on the display unit of the information processing apparatus in the first object orientation detection process according to the second embodiment of the present invention. FIG. 11 is a flowchart of second object orientation detection processing according to the second embodiment of the present invention; It is an example of a user interface displayed on the display unit of the information processing apparatus according to the second embodiment of the present invention. It is a modification of the user interface displayed on the display unit of the information processing apparatus according to the first embodiment of the present invention.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention can be implemented in many different aspects and should not be construed as being limited to the description of the embodiments exemplified below. Although the drawings may be represented schematically in order to make the description clearer, they are only examples and do not limit the interpretation of the present invention. In addition, the letters "first" and "second" for each element are labels for convenience used to distinguish each element, and unless otherwise specified, have more meaning. do not have. In the drawings referred to in this embodiment, the same parts or parts having similar functions are denoted by the same reference numerals or similar reference numerals (numerals xxx with A and B only), and the repetition thereof Description may be omitted. Also, part of the configuration may be omitted from the drawing. In addition, no particular description will be given if it is something that can be recognized by a person who has ordinary knowledge in the field to which the present invention belongs.

A "part" described in one embodiment of the present invention refers to a movable part of an object, and a plurality of parts are combined to form an interlocking part.

<First Embodiment>
An object attitude control system according to a first embodiment of the present invention will be described in detail with reference to the drawings.

(1-1. Hardware Configuration of Object Attitude Control System)
FIG. 1 shows a hardware configuration and functional block diagram of an object attitude control system 1. As shown in FIG. As shown in FIG. 1, object attitude control system 1 includes terminal 10 and server 20 . The terminal 10 and the server 20 may be collectively referred to as an information processing device.

The terminal 10 is one of computers and has a display section 11 , a control section 12 , a storage section 13 , an operation section 14 , a communication section 15 , a sensor section 16 , an imaging section 17 and a speaker section 18 . In this example, a smart phone is used as the terminal 10 . In addition, it is not limited to smartphones, and may be mobile phones (feature phones), tablet terminals, notebook PCs (Personal Computers), IoT devices (devices equipped with power supply mechanisms, communication functions, and information storage mechanisms), etc., and servers through networks. Any device that can communicate with 20 is applicable.

The display unit 11 is a display device such as a liquid crystal display or an organic EL display, and display contents are controlled by signals input from the control unit 12 .

The control unit 12 includes a CPU (Central Processing Unit), ASIC (Application Specific Integrated Circuit), FPGA (Flexible Programmable Gate Array), or other arithmetic processing circuit. The control unit 12 causes the application including the object posture control program stored in the storage unit 13 to be executed based on the operations of the display unit 11 and the operation unit 14 .

The storage unit 13 has a function as a database that stores an object attitude control program and spatial information used in the object attitude control program. A memory, SSD, or other memory-capable device is used for the storage unit 13 .

The operation unit 14 includes controllers, buttons, or switches. When the operation unit 14 is moved up, down, left and right, pressed, or rotated, information based on the operation is transmitted to the control unit 12 . In this embodiment, since the terminal 10 is a display device (touch panel) having a touch sensor, the display unit 11 and the operation unit 14 may be arranged at the same place.

The communication unit 15 has a function of transmitting/receiving data to/from the server 20 . A LAN transmitter/receiver (for example, a Wi-Fi transmitter/receiver) is used for the communication unit 15 . Note that the transceiver is not limited to a LAN transceiver. When the terminal is a portable terminal, a transmitter/receiver for mobile terminal communication (for example, LTE communication) may be provided, or a transmitter/receiver for short-range wireless communication may be provided. Terminal 10 is connected to server 20 via network 50 .

The sensor unit 16 has a function of detecting position information of markers, operating tools, objects, and the like. The sensor section 16 includes a position sensor, a distance sensor, and a displacement sensor.

The imaging unit 17 has a function of capturing an environment image, and captures an object provided with a marker portion for displaying the object. For example, a CMOS image sensor is used for the imaging unit 17 .

The speaker unit 18 has a function of outputting sound information to the outside based on the detected information.

The server 20 has a communication section 21 , a storage section 22 , a control section 23 and a display section 24 . Server 20 functions as a database and application server. Note that the server 20 does not necessarily have to be used when the terminal 10 is preloaded with all the information related to the object attitude control program.

The communication unit 21 has a transmitter/receiver and performs information communication of object control information with the terminal 10 via the network 50 . A transmitter/receiver for the Internet is used for the communication unit 21 . Note that the transmitter/receiver is not limited to a LAN transmitter/receiver for the Internet, and a communicable device similar to the terminal 10 is used.

The storage unit 22 has a function as a database of information used in the object attitude control program by using a hard disk and an SSD.

The control unit 23 controls processing of the object attitude control program using a CPU, ASIC, FPGA, or other arithmetic processing circuit. Also, in some cases, a user interface for executing the object attitude control program is provided to the display section 24 by a command from the control section 23 .

(1-2. Configuration of Object Posture Control Unit 100)
FIG. 2 shows an object attitude control section 100 which is configured by each component of the terminal 10 and each component of the server 20 in the object attitude control system 1 and which controls a program (object attitude control program) for realizing the object attitude control function. 1 shows a functional block diagram. In this example, an example in which an object attitude control program is provided in the terminal 10 will be described.

The object posture control unit 100 includes a captured image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operating body detection unit 150, an overlapping part detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate changer 190 .

The captured image acquisition unit 110 has a function of acquiring an environmental image of the real space captured by the imaging unit 17 .

The marker detection unit 120 has a function of detecting markers from the image acquired by the captured image acquisition unit 110 . In this example, marker information is stored in the storage unit 13 in advance. It should be noted that the information of the marker does not necessarily have to be stored in the storage unit 13, and a feature point that can be the marker may be extracted from the captured image.

The space definition unit 130 has a function of aligning the coordinates of the virtual space and the display unit 11 . In this embodiment, the space defining unit 130 aligns the coordinates of the virtual space and the display unit 11 with the detected markers as a reference using a marker-based alignment method.

The object display unit 140 has a function of displaying on the display unit 11 an object whose orientation is to be controlled and which is placed in the virtual space.

The operating body detection unit 150 has a function of detecting an operating body from an object included in a captured image and setting it as the operating body. The overlapping part detection unit 160 has a function of detecting that the operating body overlaps with a part of a plurality of parts of the object, and setting the part as an operated part.

The notification unit 170 has a function of notifying the user that the operating body and the operated part overlap. It should be noted that the fact that the operating body and the operated part overlap means, for example, that the operating body and the operated part overlap in a space (including a virtual space) where the operating body and the operated part are arranged or in the display area of the display unit 11. It means that position coordinates overlap at least partly with the operated part. In this example, notification is given via sound information emitted from the speaker unit 18 of the terminal 10 .

The movement information acquisition unit 180 has a function of acquiring movement information based on the set coordinates of the operating body before and after movement.

The object coordinate changing unit 190 has a function of changing and recording the coordinates of the non-operating part superimposed on the operating tool and transmitting the position information to the object display unit 140 in order to display the object after movement on the display unit 11 . have The movement information acquisition unit 180, the object coordinate change unit 190, and the object display unit 140 acquire movement information of the operated part of the object 60 times per second, and change the spatial coordinates of the operating body and the coordinates on the display screen. Thereby, the movement (change in posture) of the object can be displayed as a still image or moving image.

(1-3. Object posture control processing)
Next, object attitude control processing based on commands from the object attitude control program in the object attitude control section 100 will be described. The object orientation control processing includes superimposed display processing S100 and object orientation detection processing. FIG. 3A is a flowchart of superimposed display processing. FIG. 3B is a flow chart of object orientation detection processing. As shown in FIG. 3B, the object orientation detection processing includes first object orientation detection processing S200 and second object orientation detection processing S300. The superimposed display process and the object orientation detection process can be performed in parallel.

The superimposed display processing S100 includes captured image acquisition/display processing, marker information detection processing, space definition processing, object coordinate acquisition processing, and captured image/object superimposed display processing. The first object orientation detection processing S200 includes an operating tool detection process, an operating tool setting process, an overlapping portion detection process, an operated part setting process, and an overlapping notification process. The second object orientation detection processing S300 includes operation body movement amount acquisition processing, object coordinate change processing, and determination processing as to whether or not movement of the terminal has ended. Each object attitude control process will be described separately.

(1-3-1. Superimposed Display Processing S100)
FIG. 4 shows the superimposed display processing S100. The superimposed display processing S100 is started when an application including an object attitude control program is activated.

In the superimposed display processing S100, the captured image acquisition unit 110 captures an environment image of the real space, acquires the captured image (S110), and obtains the image captured on the display unit 11 of the terminal 10 (also referred to as the second image). to display).

FIG. 5 is an example when the terminal 10 captures an image of the real space. In FIG. 5, in the actual space, the terminal 10, the table 60, the seat 70, and the user's hand 90 serving as an operating body are arranged. Sheet 70 includes four markers 75 . In this example, a portion of table 60 , sheet 70 and user's hand 90 are imaged by terminal 10 to include four markers 75 .

Returning to FIG. 4, description will be made. Next, the marker detection unit 120 performs marker detection processing (S120). In this example, information on four markers is registered in advance in the object attitude control program. Marker information relates to marker morphology and placement. When the four markers are not detected (S120; No), the process returns to the captured image acquisition/display process (S110).

Next, when four markers are detected (S120; Yes), the space definition unit 130 aligns the coordinates of the real space and the virtual space (space definition) using the marker-based alignment method. (S130). At this time, the space definition unit 130 collates the real space marker 75 displayed on the display unit 11 with the virtual space marker stored in the object attitude control program, and converts the virtual space coordinates to the real space. The coordinates are adjusted so that they can be displayed on the display unit 11 . For example, the position coordinates of the marker 75-1 are such that the position coordinates in the real space are (X _r75-1 , Y _r75-1 , Z _r75-1 ), and the position coordinates in the display unit 11 are (X _d75-1 , Y _{d75- 1} ), the position coordinates in the virtual space are defined as (X _v75-1 , Y _v75-1 , Z _v75-1 ).

Next, the space definition unit 130 performs a process of acquiring object coordinates in the virtual space (S140). FIG. 6 shows the coordinates of each part of the object in the virtual space. As shown in FIG. 6, the coordinate information of each part of the object is set with respect to the origin set based on the coordinates of the four markers. By acquiring the information on the display section 11 of the marker 75 through the above processing, the coordinates of each part of the object are acquired.

Next, the object display unit 140 superimposes and displays the captured image (second image) and the image of the object (also referred to as the first image) (S150). FIG. 7 shows a user interface of objects displayed on the display unit 11 of the terminal 10. As shown in FIG. In FIG. 7, an object 80 in virtual space placed on a sheet 70 is displayed together with a user's hand 90 . In this example, an object 80 has a humanoid body shape and has multiple parts 81 and multiple bones 82 . The part 81 is a movable part and corresponds to a human joint. The site 81 (site 81a) is connected to the adjacent site 81 (site 81b) via a bone 82 (bone 82a). That is, it can be said that the part 81a and the part 81b are interlocking parts. Note that the sheet 70 does not have to be displayed when the object 80 is displayed.

Next, it is determined whether or not to end the superimposed display process (S160). The determination processing may be performed based on information input by the user, or may be performed based on whether or not a predetermined condition is satisfied. For example, if there is a moving object in the captured image, it is determined that the superimposed display process is not to end (S160; No), the process returns to the captured image acquisition process (S110) again, and the superimposed display process S100 ends. Repeated. On the other hand, if it is determined to end the superimposed display process (S160; Yes), the superimposed display process S100 ends.

(1-3-2. First Object Posture Detection Processing S200)
FIG. 8 shows a flowchart of the first object orientation detection processing S200. The first object orientation detection processing S200 is triggered by input of a detection start signal. The detection start signal may be input by the user, or may be a predetermined signal input from a program.

First, the operating object detection unit 150 performs an operating object detection process (S210). The detection process is determined based on the parallax that occurs in the captured image.

First, the distance from the terminal 10 (imaging unit 17 ) to the object is calculated based on changes in the position of the object (parallax) in a plurality of images captured by the imaging unit 17 . Next, an object whose distance from the terminal 10 to the object is smaller than the distance from the imaging unit 17 to the table is detected as the operating object. In this example, since the distance from the terminal 10 to the user's hand 90 is smaller than the distance from the terminal 10 to the table 60, the user's hand 90 is detected as the operating body (S210; Yes) and set. (S220). At this time, the portion of the user's hand 90 that is closest to the sheet 70 (more specifically, the tip portion 90a of the index finger) may be defined as the operating body. When the operating object is not detected (S210; No), the above processing is performed in a loop.

Next, the overlapping part detection unit 160 detects that the object 80 and the operating body overlap (S230). Whether or not they are superimposed is determined based on whether or not the coordinates of the target portion of the object 80 in the virtual space match the coordinates of the operating tool in the real space.

FIG. 9 is an example of a user interface showing when the operating body and the operated region are superimposed. As shown in FIG. 9, in this example, a tip portion 90a of the index finger and a tip portion 90b of the thumb of the user's hand 90 partially overlap a portion 81a (corresponding to the joint of the left wrist) of the object 80. . A method for detecting that the tip portion 90a of the index finger and the portion 81a of the object 80 are superimposed will be described below.

First, the real space position (coordinates) of the tip 90a of the index finger is obtained. For the coordinates of the tip 90a of the index finger, the center of the area surrounded by the four markers 75 arranged on the sheet 70 is set as the origin, and relative coordinates are used with the origin as a reference. At this time, the coordinates of the tip _90a of the index finger in the real space are ( _{Xr_90a , Yr_90a} , _{Zr_90a )} . Based on _{the coordinates in the real space, the coordinates (Xv_90a, Yv_90a, Zv_90a ) when the} tip _90a of the index finger is placed in the _virtual space are calculated.

Next, the coordinates in the virtual space of the tip 90a of the index finger and the coordinates in the virtual space of one of the plurality of parts 81 of the object 80 (in this example, the coordinates of the part 81a of the object (X _v — _81a , Y _v — _81a , Z _v — _81a )) match, it is determined that the overlapping portion has been detected (S230; Yes). At this time, the part 81a of the object is set as the operated part (S240). When the overlapping portion is not detected (S230; No), the overlapping portion detection process is looped.

When the overlapping portion is detected, the notification unit 170 notifies that the operating body and the operated portion overlap (S250). In this example, the notification is made using sound information emitted from the speaker unit 18 of the terminal 10 . Thus, the first object orientation detection processing S200 is completed.

(1-3-3. Second Object Posture Detection Processing S300)
FIG. 10 is a flowchart of the second object orientation detection processing S300. The second object orientation detection processing S300 is started upon completion of the first object orientation detection processing S200.

In the second object orientation detection process S300, first, the movement information acquisition unit 180 acquires the amount of movement of the operating body (S310). In this example, the V-SLAM (Visual Simultaneous Localization and Mapping) method is used to track the posture of the object as the positional information of the operating body (tip 90a of the index finger of the user's hand 90) changes. The position (coordinates) of the tip portion 90 a of the index finger is measured and processed using the sensor section 16 and the imaging section 17 and acquired by the movement information acquisition section 180 . An image sensor, a position sensor, a distance sensor, a displacement sensor, or other elements capable of position detection are used for the sensor unit 16 and the imaging unit 17 .

FIG. 11A shows the initial value (before movement), the position (coordinates) of the operating body (index finger tip 90a) after time t1 (after movement), and the index finger tip 90a obtained from the coordinates before and after the movement. is a data structure indicating the amount of movement of "Before movement" indicates the time when it is detected that the operating body and the operated portion overlap each other.

Next, based on the acquired amount of movement of the operating body, the object coordinate changing unit 190 changes and records the coordinates of the operated part (part 81a), which is the movement target part of the object 80 (S320). . FIG. 11B shows the data structure of the initial values (before movement), the coordinates in the virtual space of the operated part (part 81a) of the object after time t1, and the coordinates in the display unit 11. FIG. As shown in FIG. 11B, the coordinate information of the operated part (part 81a) of the object 80 in the virtual space and the position of the object 80 on the display unit 11 are displayed based on the amount of movement of the operating body (tip part 90a of the index finger). The coordinate information of the operated part (part 81a) is changed, and the changed coordinate information is stored (the coordinate information is rewritten).

Returning to FIG. 10, description will be made. At this time, the object display unit 140 displays the object 80 on the display unit 11 using the information of the operated part after movement (after time t1) in the superimposed display processing S100. FIG. 12 shows a user interface displaying an object 80 in which the position of the operated part is changed on the display unit 11 . As shown in FIG. 12, the operated part (part 81a) moves from the position displayed in FIG. 7 with reference to a different part (part 81b, corresponding to the elbow joint). In other words, it is possible to move some of the multiple parts of the object based on the information about the movement of the operating body. The above process is repeated as long as the operating body moves (S330; Yes). In this example, the amount of movement of the operating body is acquired 60 times per second, so the change in the posture of the object is displayed smoothly. When the movement of the terminal 10 ends (S330; No), the second object orientation detection processing S300 ends.

As described above, by using the object attitude control program of the present embodiment, it is possible to easily control the attitude of an object having a plurality of parts without performing complicated control.

<Second embodiment>
In the present embodiment, posture control processing for an object that is different from that in the first embodiment will be described. Specifically, an example in which a plurality of parts move based on the movement of one terminal will be described. Note that the same configuration and method as in the first case may be omitted as appropriate.

(2-1. Configuration of Server and Object Attitude Control Unit 100A)
FIG. 13 shows a functional block diagram of the object attitude control section 100A in the object attitude control system 1A. As shown in FIG. 22, the object posture control section 100A includes a captured image acquisition section 110, a marker detection section 120, a space definition section 130, an object display section 140, an operating body detection section 150, an overlapping portion detection section 160, and a notification section 170. , a movement information acquisition unit 180 and an object coordinate change unit 190, as well as a second operating body detection unit 195 and a second overlapping part detection unit 200.

The second operating tool detection unit 195 has a function of detecting an object different from the target object detected as the operating tool from among multiple objects included in the captured image, and setting it as the second operating tool. The second overlapping portion detection unit 200 has a function of detecting that the second operating body overlaps with a part of a plurality of parts of the object, and setting the relevant part as a second operated part. .

(2-2. Object posture control processing)
FIG. 14 shows a flowchart of the first object attitude detection processing S200A based on the command of the object attitude control program in the object attitude control section 100A. As shown in FIG. 14, after the superimposed portion is notified (S250), it is determined whether or not to set an additional operating body (S255). The above determination may be made based on whether or not there is an object in motion other than the object set as the operating object during a predetermined period, or may be made based on input information from the user. Further, the display unit 11 may display "Do you want to add a new operating object?" and "Yes" and "No" buttons. If there is no additional setting (S255; No), the first object orientation detection processing S200A ends.

If it is determined to additionally set (S255; Yes), the second operating body detection unit 195 detects an object that will be the second operating body (S260). In this example, the user's left hand 91 is detected as the second operating body. As a detection method, a method similar to the method described in the first embodiment of the present invention can be used. At this time, the portion of the left hand 91 of the user that is closest to the sheet 70 (more specifically, the tip portion 91a of the index finger of the left hand) may be set as the operating object (S265).

Next, the second overlapping portion detection section 200 detects an overlapping portion (second overlapping portion) between the second operating body and a part of the object 80 (S270). A method similar to the overlapping portion detection method of the first embodiment of the present invention is used for the detection method of the second overlapping portion. When it is determined that the second operating body overlaps the part 81c of the object 80 (S270; Yes). At this time, the part 81c of the object is set as the second operated part (S280). When the second overlapping portion is not detected (S270; No), the second overlapping portion detection process is looped.

After the second overlapping portion is detected, the notification unit 170 may notify that the second overlapping is detected (S290). As for the notification method, the same method as in the first embodiment can be used. Thus, the first object orientation detection processing S200A is completed.

FIG. 16 is a flowchart of the third object control processing S300A. In this embodiment, the movement of the operated portion (the portion 81a) of the object accompanying the movement of the operating body (the tip portion 90a of the index finger of the user's hand 90) is realized by the same method as in the first embodiment. In the case of this embodiment, as shown in FIG. 17, the operated part (part 81a) is moved, and at the same time, the second operating body (tip 91a of the index finger of the user's left hand 91), which is different from the operating body, is moved. Based on the amount, the coordinate changes of the operated portion and the second operated portion are changed so that the second operated portion (the portion 81c, corresponding to the ankle joint) moves relative to the portion 81e (S320). . In other words, it is possible to control the attitude of the object by moving a plurality of operated parts according to the movements of a plurality of operating bodies.

In addition, in this embodiment, an example of controlling the posture of an object using two operating tools has been described, but the present invention is not limited to this. The same can be applied to the case of having three or more operating bodies.

It should be noted that within the scope of the idea of the present invention, those skilled in the art can conceive of various modifications and modifications, and it is understood that these modifications and modifications also belong to the scope of the present invention. . For example, additions, deletions, or design changes of components, or additions, omissions, or changes in conditions of the above-described embodiments by those skilled in the art are also subject to the gist of the present invention. is included in the scope of the present invention as long as it has

(Modification)
In the first embodiment of the present invention, an example of using the X, Y, and Z coordinates in the movement information of the operating body has been described, but the present invention is not limited to this. For example, the angle θ may be used in addition to the X, Y, and Z coordinates. In this case, any of the X, Y, and Z coordinates may not be used. The information of the operated portion (part 81a) of the object 80 in the virtual space and the position coordinates of the operated portion (part 81a) of the object 80 in the display section 11 are changed in accordance with the movement of the operating body, and the position coordinates after the change are changed. is stored (the coordinate information is rewritten). This makes it possible to control the orientation of the object, particularly the orientation of the twist.

Further, in the first embodiment of the present invention, an example of detecting and setting an operating object and then detecting a superimposed portion and setting an operated part has been described, but the present invention is not limited to this. For example, when the objects are superimposed on each other, the superimposed object may be detected and set as the operating object, and the superimposed portion of the object may be set as the operated portion.

Also, in the first embodiment of the present invention, an example in which the tip portion 90a of the index finger mainly overlaps a part of the object (the portion 81a) has been described, but the present invention is not limited to this. The overlapping portion detection section 160 may detect that there is overlap when there are a plurality of overlapping portions. For example, when two or more points are overlapped, the user's fingers can have a shape of pinching the object, and the posture of the object can be controlled more naturally.

Also, when it is determined that multiple parts overlap, additional control different from the normal attitude control of the object may be provided. FIG. 18 is an example of a user interface when it is determined that multiple parts overlap and the operating body moves. As shown in FIG. 18, in this example, the tip 90a of the index finger and the tip 90b of the thumb overlap the object portion 81(a). At this time, according to the movement of the tip 90a of the index finger and the tip 90b of the thumb, the part 81a of the object 80 also moves based on preset conditions, and the bone 82 (82a) extends. In this way, the posture of the object can be controlled in various forms by performing additional control different from normal posture control of the object.

It should be noted that detection may be performed by moving the operating body so as to surround the portion corresponding to the portion 81a, or by surrounding the portion corresponding to the portion 81a with a plurality of fingers.

In the first embodiment of the present invention, an example in which the notification unit 170 notifies using sound information has been described, but the present invention is not limited to this. For example, the notification unit 170 may notify by vibrating the terminal 10 when the superimposed portion is detected. Alternatively, when the superimposed portion is detected, the terminal 10 may emit light to notify. Alternatively, when a superimposed portion is detected, it may be notified using character information. Alternatively, when the overlapping portion is detected, it may be highlighted by changing the color of the overlapping portion and notified. Also, the notification unit 170 may not necessarily be used.

Note that detection of an operating body is not limited to the method using parallax. For example, an object that satisfies a predetermined condition may be detected as an operating object based on the color or shape of the object.

Also, an object that moves faster than a predetermined speed may be detected as the operating object. Further, when the distance derived from the coordinates of the virtual space and the coordinates of the virtual space of the object when the object is placed in the virtual space becomes smaller than a predetermined distance, the object is detected as the operating object. may

In the first embodiment of the present invention, an example in which a captured image and an object in a virtual space are aligned by a marker-based alignment method and displayed in a superimposed manner has been described, but the present invention is not limited to this. In addition to marker-based registration methods, vision-based registration methods such as natural feature-based registration methods that do not use specific markers and model-based registration methods are available as methods for aligning virtual space including objects with captured images. Techniques, sensor-based registration techniques using sensors, and the like may be used as appropriate.

Also, in the first embodiment of the present invention, an example of displaying an object using augmented reality was shown, but the present invention is not limited to this. An embodiment of the present invention can be applied not only to augmented reality but also to mixed reality (MR: Mixed Reality) as long as the coordinates of an object can be specified.

Further, in the first embodiment of the present invention, an example of displaying on the display unit of a planar terminal having a form like a smart phone was shown, but the present invention is not limited to this. For example, a goggle type terminal may be used as a terminal for displaying. Further, at this time, a two-lens camera (stereo camera) may be used for the imaging unit 17 . As a result, the displayed image can be displayed more stereoscopically.

DESCRIPTION OF SYMBOLS 1... Object attitude control system, 10... Terminal, 11... Display part, 12... Control part, 13... Storage part, 14... Operation part, 15... Communication part, 16... sensor section, 17... imaging section, 18... speaker section, 20... server, 21... communication section, 22... storage section, 23... control section, 24... Display section 50 Network 60 Table 70 Sheet 75 Marker 80 Object 81 Site 82 Bone 90 Hand 90a... Tip part 90b... Tip part 91... Hand 91a... Tip part 100... Object posture control unit 110... Captured image acquisition unit 120. Marker detection unit 130 Space definition unit 140 Object display unit 150 Operation body detection unit 160 Overlapping part detection unit 170 Notification unit 180 Movement information acquisition unit 190 Object coordinate change unit 195 Second operating body detection unit 200 Second superimposed part detection unit

Claims (6)

displaying on a display unit a first image representing an object having a plurality of parts superimposed on a second image captured by the imaging unit, in a control unit of the information processing device;
setting the second object, which has a short distance from the imaging unit, as an operating body, out of the first object and the second object existing in the real space imaged in the second image;
Controlling the posture of the object by changing position information of at least a part of a plurality of parts of the object to be operated, based on the positional relationship between the object and the operating body in the second image. Object attitude control program for executing The distance between the imaging unit and the first object or the second object is calculated based on parallax that occurs during imaging.
Object attitude control program according to claim 1. executing a process for notifying a user when the position of the operating body and the position of the operated part at least partially overlap;
3. The object attitude control program according to claim 1 or 2 . changing the position of the operated part based on a change in position information of the operating body while the position of the operating body and the position of the operated part overlap at least partially;
4. The object attitude control program according to any one of claims 1 to 3. The position of the operating body and the position of the operated part are each defined as a position in a coordinate system of the space in which the object is arranged,
5. The object attitude control program according to claim 3 or 4. An information processing device for controlling the posture of an object,
displaying on a display unit a first image representing an object having a plurality of parts superimposed on a second image captured by the imaging unit;
setting the second object, which has a short distance from the imaging unit, as an operating body, out of the first object and the second object existing in the real space imaged in the second image;
Control for controlling the posture of the object by changing position information of at least a part of a plurality of parts of the object to be operated based on the positional relationship between the object and the operating body in the second image. An information processing device comprising :

Images