JP2024103514A - Object attitude control program and information processing device - Google Patents

Abstract

[Problem] To easily control the attitude of an object. [Solution] An object attitude control program causes an information processing device to execute the following: displaying a first image representing an object having multiple parts on a display unit, superimposed on a second image captured by an imaging unit, and controlling the attitude of the object by changing position information of at least some of the operated parts of the object based on the positional relationship between the object and an operating body in the second image. [Selected Figure] Figure 2

Description

The present invention relates to an object posture control program and an information processing device.

In recent years, advances in electronic devices have made it possible to display high-definition still images and videos on devices such as smartphones and personal computers. Furthermore, research and development of information processing technology has progressed, and display technology that combines virtual space and real space using augmented reality (AR) is becoming practical. For example, Patent Document 1 discloses a technology for displaying objects placed in a virtual space on the display area of an information terminal.

JP 2015-41126 A

On the other hand, controlling the posture of an object displayed in the display area can require complex input processing to specify the target part of the object.

In view of these problems, one of the objectives of the present invention is to easily control the orientation of objects placed in a virtual space.

According to one embodiment of the present invention, an object posture control program is provided for causing an information processing device to execute the following: superimposing a first image representing an object having multiple parts on a second image captured by an imaging unit and displaying the first image on a display unit; and controlling the posture of the object by changing position information of at least some of the operated parts of the object based on the positional relationship between the object and an operating body in the second image.

In the above object posture control program, a process may be executed to notify the user when the position of the operating body and the position of the operated part overlap at least partially.

In the above object posture control program, the position of the operated part may be changed based on a change in position information of the operating body while the position of the operating body and the position of the operated part overlap at least partially.

In the above object posture control program, the position of the operated part and the position of the operated part may each be defined as a position in a coordinate system of a space in which the object is placed.

According to one embodiment of the present invention, there is provided an information processing device that controls the attitude of an object, in which a first image representing an object having multiple parts is displayed on a display unit by superimposing it on a second image captured by an imaging unit, and the attitude of the object is controlled by changing position information of at least some of the operated parts of the multiple parts of the object based on the positional relationship between the object and an operating body in the second image.

By using one embodiment of the present invention, the pose of an object can be easily controlled.

1 is a diagram showing a hardware configuration of an object posture control system according to a first embodiment of the present invention. FIG. 2 is a functional block diagram of an object attitude control unit according to the first embodiment of the present invention. FIG. 4 is a flowchart of an object attitude control process according to the first embodiment of the present invention. FIG. 4 is a flowchart of a superimposed display process according to the first embodiment of the present invention. 11 is an example of alignment between coordinates in a virtual space and coordinates displayed on the information processing device in the superimposed display processing according to the first embodiment of the present invention. 4 is an example of coordinates of each part of an object according to the first embodiment of the present invention. 5 is a diagram illustrating an example of a user interface displayed on a display unit of an information processing device in the superimposed display process according to the first embodiment of the present invention. FIG. 4 is a flowchart of a first object pose detection process according to the first embodiment of the present invention. 11 is a diagram illustrating an example of a user interface displayed on a display unit of the information processing device in the first object pose detection process according to the first embodiment of the present invention. FIG. 11 is a flowchart of a second object pose detection process according to the first embodiment of the present invention. 13A and 13B show position coordinates before and after movement of an operating body and position coordinates of an object in a second object orientation detection process according to the first embodiment of the present invention. 11 is a diagram illustrating an example of a user interface displayed on a display unit of the information processing device in the second object posture detection process according to the first embodiment of the present invention. FIG. 11 is a functional block diagram of an object attitude control unit according to a second embodiment of the present invention. FIG. 11 is a flowchart of a first object pose detection process according to the second embodiment of the present invention. 13 is a diagram illustrating an example of a user interface displayed on a display unit of an information processing device in a first object pose detection process according to a second embodiment of the present invention. FIG. 11 is a flowchart of a second object pose detection process according to the second embodiment of the present invention. 13 is a diagram illustrating an example of a user interface displayed on a display unit of an information processing device according to a second embodiment of the present invention. 10 is a modified example of a user interface displayed on a display unit of the information processing device according to the first embodiment of the present invention.

The following describes the embodiments of the present invention with reference to the drawings. However, the present invention can be implemented in many different ways, and should not be interpreted as being limited to the description of the embodiments exemplified below. The drawings may be shown in a schematic manner to make the explanation clearer, but they are merely examples and do not limit the interpretation of the present invention. Furthermore, the letters "first" and "second" attached to each element are convenient labels used to distinguish each element, and have no further meaning unless otherwise explained. In the drawings referred to in this embodiment, the same parts or parts having similar functions are given the same or similar symbols (symbols consisting of the number xxx with A or B added), and repeated explanations may be omitted. In addition, some of the configuration may be omitted from the drawings. In addition, if a person with ordinary knowledge in the field to which the present invention belongs can recognize something, no special explanation will be given.

In one embodiment of the present invention, a "part" refers to a movable part that an object has, and a linked part is formed by combining multiple parts.

First Embodiment
An object attitude control system according to a first embodiment of the present invention will be described in detail with reference to the drawings.

(1-1. Hardware configuration of object attitude control system)
Fig. 1 shows a hardware configuration and a functional block diagram of an object posture control system 1. As shown in Fig. 1, the object posture control system 1 includes a terminal 10 and a server 20. The terminal 10 and the server 20 may be collectively referred to as an information processing device.

The terminal 10 is a type of computer, and has a display unit 11, a control unit 12, a memory unit 13, an operation unit 14, a communication unit 15, a sensor unit 16, an imaging unit 17, and a speaker unit 18. In this example, a smartphone is used as the terminal 10. Note that the terminal 10 is not limited to a smartphone, and may be a mobile phone (feature phone), a tablet terminal, a notebook PC (Personal Computer), an IoT device (a device equipped with a power supply mechanism, a communication function, and an information storage mechanism), or the like, and is applicable as long as it is capable of communicating with the server 20 via a network.

The display unit 11 is a display device such as a liquid crystal display or an organic EL display, and the display content is controlled by a signal input from the control unit 12.

The control unit 12 includes a central processing unit (CPU), an application specific integrated circuit (ASIC), a flexible programmable gate array (FPGA), or other arithmetic processing circuit. The control unit 12 executes an application including an object attitude control program stored in the memory unit 13 based on the operation of the display unit 11 and the operation unit 14.

The storage unit 13 functions as a database that stores the object attitude control program and the spatial information used in the object attitude control program. The storage unit 13 may be a memory, an SSD, or any other storage device.

The operation unit 14 includes a controller, a button, or a switch. When the operation unit 14 is moved up, down, left, right, pressed, rotated, or the like, information based on the operation is transmitted to the control unit 12. In this embodiment, the terminal 10 is a display device (touch panel) having a touch sensor, so that the display unit 11 and the operation unit 14 may be located in the same place.

The communication unit 15 has a function of transmitting and receiving data to and from the server 20. A LAN transceiver (e.g., a Wi-Fi transceiver) is used for the communication unit 15. Note that the transceiver is not limited to a LAN transceiver. If the terminal is a portable terminal, a transceiver for portable terminal communication (e.g., LTE communication) may be provided, or a transceiver for short-range wireless communication may be provided. The terminal 10 is connected to the server 20 via a network 50.

The sensor unit 16 has a function of detecting position information of a marker, an operating body, an object, etc. The sensor unit 16 includes a position sensor, a distance sensor, and a displacement sensor.

The imaging unit 17 has a function of capturing an environmental image, and captures an image of an object on which a marker is provided for displaying the object. For example, a CMOS image sensor is used for the imaging unit 17.

The speaker unit 18 has the function of outputting sound information to the outside based on the detected information.

The server 20 has a communication unit 21, a memory unit 22, a control unit 23, and a display unit 24. The server 20 functions as a database and an application server. Note that if all information related to the object attitude control program is pre-installed in the terminal 10, it is not necessary to use the server 20.

The communication unit 21 has a transceiver, and communicates control information of objects with the terminal 10 via the network 50. A transceiver for the Internet is used for the communication unit 21. Note that the transceiver is not limited to a LAN transceiver for the Internet, and any device capable of communicating in the same way as the terminal 10 is used.

The storage unit 22 functions as a database of information used in the object attitude control program by using a hard disk and an SSD.

The control unit 23 controls the processing of the object attitude control program using a CPU, ASIC, FPGA, or other arithmetic processing circuit. In addition, a user interface for executing the object attitude control program may be provided to the display unit 24 by instructions from the control unit 23.

(1-2. Configuration of Object Pose Control Unit 100)
2 shows a functional block diagram of an object posture control unit 100 that is configured by the components of the terminal 10 and the server 20 in the object posture control system 1 and controls a program (object posture control program) that realizes an object posture control function. In this example, an example in which the object posture control program is provided in the terminal 10 will be described.

The object posture control unit 100 includes a captured image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operating body detection unit 150, an overlapping part detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190.

The captured image acquisition unit 110 has the function of acquiring an environmental image of the real space captured by the imaging unit 17.

The marker detection unit 120 has a function of detecting markers from the image acquired by the captured image acquisition unit 110. In this example, the marker information is stored in advance in the storage unit 13. Note that the marker information does not necessarily have to be stored in the storage unit 13, and feature points that can be markers may be extracted from the captured image.

The space definition unit 130 has a function of aligning the virtual space with the coordinates of the display unit 11. In this embodiment, the space definition unit 130 uses a marker-based alignment method to align the virtual space with the coordinates of the display unit 11 based on the detected marker.

The object display unit 140 has the function of displaying on the display unit 11 an object whose posture is to be controlled and which is placed in a virtual space.

The operating body detection unit 150 has a function of detecting an operating body from objects included in a captured image and setting it as the operating body. The overlapping part detection unit 160 has a function of detecting that the operating body overlaps with a part of multiple parts of an object and setting that part as the part to be operated.

The notification unit 170 has a function of notifying the user that the operating object and the operated part overlap. Note that the operating object and the operated part overlap, for example, when the operating object and the operated part overlap in position coordinates at least partially in the space (including virtual space) in which the operating object and the operated part are placed or in the display area of the display unit 11. In this example, the notification is made via sound information emitted from the speaker unit 18 of the terminal 10.

The movement information acquisition unit 180 has a function of acquiring movement information based on the coordinates before and after the movement of the set operating object.

The object coordinate modification unit 190 has a function of modifying and recording the coordinates of the non-operated part that overlaps with the operating body in order to display the moved object on the display unit 11, and transmitting the position information to the object display unit 140. The movement information acquisition unit 180, object coordinate modification unit 190, and object display unit 140 acquire movement information of the operated part of the object 60 times per second, and can display the movement (posture change) of the object as a still image or video by modifying the spatial coordinates of the operating body and the coordinates on the display screen.

(1-3. Object Attitude Control Processing)
Next, an object posture control process based on commands from an object posture control program in the object posture control unit 100 will be described. The object posture control process includes a superimposed display process S100 and an object posture detection process. Fig. 3(A) is a flow diagram of the superimposed display process. Fig. 3(B) is a flow diagram of the object posture detection process. As shown in Fig. 3(B), the object posture detection process includes a first object posture detection process S200 and a second object posture detection process S300. The superimposed display process and the object posture detection process can be performed in parallel.

The superimposed display process S100 includes captured image acquisition and display process, marker information detection process, space definition process, object coordinate acquisition process, and captured image/object superimposed display process. The first object posture detection process S200 includes operating body detection process, operating body setting process, superimposed portion detection process, operated part setting process, and superimposition notification process. The second object posture detection process S300 includes operating body movement amount acquisition process, object coordinate change process, and process of determining whether the movement of the terminal has ended. Each object posture control process will be explained separately.

(1-3-1. Overlay display processing S100)
The superimposed display process S100 is shown in Fig. 4. The superimposed display process S100 is started when an application including an object posture control program is started.

In the superimposed display process S100, the captured image acquisition unit 110 captures an environmental image of the real space, acquires the captured image (S110), and displays the captured image (also called the second image) on the display unit 11 of the terminal 10.

Figure 5 is an example of capturing an image of a real space with the terminal 10. In Figure 5, the real space includes the terminal 10, a table 60, a sheet 70, and a user's hand 90, which serves as an operating object. The sheet 70 includes four markers 75. In this example, the terminal 10 captures an image of a portion of the table 60, the sheet 70, and the user's hand 90, so as to include the four markers 75.

Returning to FIG. 4, the marker detection unit 120 then performs marker detection processing (S120). In this example, information on four markers is preregistered in the object posture control program. The marker information relates to the shape and arrangement of the markers. When the four markers are not detected (S120; No), the process returns to the capture and display processing of the captured image (S110).

Next, if four markers are detected (S120; Yes), the space definition unit 130 uses the marker-based alignment method to align the coordinates of the real space and the virtual space (space definition) (S130). At this time, the space definition unit 130 compares the markers 75 in the real space displayed on the display unit 11 with the markers in the virtual space stored in the object attitude control program, and adjusts them so that the virtual space coordinates can be displayed on the display unit 11 as real space coordinates. For example, the position coordinates of marker 75-1 are defined as (Xr75-1, Yr75-1, Zr75-1) in the real space, (Xd75-1, Yd75-1) on the display unit 11, and (Xv75-1, Yv75-1, Zv75-1) in the virtual space.

Next, the space definition unit 130 performs a process of acquiring object coordinates in the virtual space (S140). FIG. 6 shows the coordinates of each part of the object in the virtual space. As shown in FIG. 6, the coordinate information of each part of the object is set with respect to an origin that is set based on the coordinates of the four markers. Through the above process, the information on the display unit 11 of the marker 75 is acquired, and the coordinates of each part of the object are acquired.

Next, the object display unit 140 displays the captured image (second image) and the image of the object (also called the first image) in a superimposed manner (S150). FIG. 7 shows a user interface of an object displayed on the display unit 11 of the terminal 10. In FIG. 7, an object 80 in a virtual space arranged on a sheet 70 is displayed together with a user's hand 90. In this example, the object 80 has a humanoid body shape and has multiple parts 81 and multiple bones 82. The part 81 is a movable part and corresponds to a human joint. The part 81 (part 81a) is connected to the adjacent part 81 (part 81b) via a bone 82 (bone 82a). In other words, it can be said that the parts 81a and 81b are linked parts. It should be noted that the sheet 70 does not need to be displayed when the object 80 is displayed.

Next, a determination process is performed as to whether or not to end the superimposed display process (S160). This determination process may be performed based on information input by the user, or may be performed based on whether or not a predetermined condition is satisfied. For example, if there is a moving object in the captured image, it is determined that the superimposed display process should not be ended (S160; No), and the process returns to the captured image acquisition process (S110) again, and the superimposed display process S100 is repeated. On the other hand, if it is determined that the superimposed display process should be ended (S160; Yes), the superimposed display process S100 is ended.

(1-3-2. First object posture detection process S200)
8 shows a flow diagram of the first object posture detection process S200. The first object posture detection process S200 is triggered by inputting a detection start signal. The detection start signal may be input by a user, or a predetermined signal may be input from a program.

First, the operating object detection unit 150 performs an operating object detection process (S210). This detection process is determined based on the parallax that occurs in the captured image.

First, the distance from the terminal 10 (imaging unit 17) to the object is calculated based on the change in the position of the object in multiple images captured by the imaging unit 17 (parallax). Next, an object whose distance from the terminal 10 to the object is shorter than the distance from the imaging unit 17 to the table is detected as the operating object. In this example, since the distance from the terminal 10 to the user's hand 90 is shorter than the distance from the terminal 10 to the table 60, the user's hand 90 is detected as the operating object (S210; Yes) and set (S220). At this time, the part of the user's hand 90 closest to the sheet 70 (more specifically, the tip 90a of the index finger) may be defined as the operating object. When the operating object is not detected (S210; No), the above process is repeated in a loop.

Next, the overlap detection unit 160 detects that the object 80 and the operating body overlap (S230). The detection of whether or not there is overlap is determined based on whether or not the coordinates in the virtual space of the target part of the object 80 match the coordinates in the real space of the operating body.

Figure 9 is an example of a user interface showing when the operating object and the operated part overlap. As shown in Figure 9, in this example, the tip 90a of the index finger and the tip 90b of the thumb of the user's hand 90 overlap part of the part 81a (corresponding to the left wrist joint) of the object 80. The following describes a method for detecting the overlap of the tip 90a of the index finger and the part 81a of the object 80.

First, the real-space position (coordinates) of the tip 90a of the index finger is obtained. The coordinates of the tip 90a of the index finger are calculated using relative coordinates based on the origin, which is the center of the area surrounded by the four markers 75 arranged on the sheet 70. At this time, the real-space coordinates of the tip 90a of the index finger are (Xr_90a, Yr_90a, Zr_90a). Then, based on the real-space coordinates, the coordinates (Xv_90a, Yv_90a, Zv_90a) of the tip 90a of the index finger when it is arranged in the virtual space are calculated.

Next, when the virtual space coordinates of the tip 90a of the index finger match any one of the multiple parts 81 of the object 80 (in this example, the virtual space coordinates (Xv_81a, Yv_81a, Zv_81a) of the object part 81a), it is determined that an overlapping portion has been detected (S230; Yes). At this time, the object part 81a is set as the part to be operated (S240). Note that when an overlapping portion is not detected (S230; No), the overlapping portion detection process is repeated in a loop.

When an overlapping portion is detected, the notification unit 170 notifies that the operating object and the operated part overlap (S250). In this example, the notification is made using sound information emitted from the speaker unit 18 of the terminal 10. With this, the first object posture detection process S200 ends.

(1-3-3. Second object posture detection process S300)
10 is a flow diagram of the second object posture detection process S300. The second object posture detection process S300 is started upon completion of the first object posture detection process S200.

In the second object posture detection process S300, first, the movement information acquisition unit 180 acquires the amount of movement of the operating body (S310). In this example, tracking of the object posture accompanying changes in position information of the operating body (tip 90a of the index finger of the user's hand 90) is performed by the V-SLAM (Visual Simultaneous Localization and Mapping) method. The position (coordinates) of the tip 90a of the index finger is measured and calculated using the sensor unit 16 and the imaging unit 17, and acquired by the movement information acquisition unit 180. The sensor unit 16 and the imaging unit 17 may be an image sensor, a position sensor, a distance sensor, a displacement sensor, or other elements capable of detecting position.

Figure 11 (A) shows a data structure indicating the initial value (before movement), the position (coordinates) of the operating body (tip 90a of the index finger) after time t1 (after movement), and the amount of movement of the tip 90a of the index finger calculated from the coordinates before and after the movement. Before movement indicates the time when it is detected that the operating body and the operated part overlap.

Next, based on the acquired amount of movement of the operating body, the object coordinate change unit 190 changes and records the coordinates of the operated part (part 81a) of the object 80 that is the part to be moved (S320). FIG. 11(B) shows the data structure of the initial value (before movement) and the coordinates in virtual space of the operated part (part 81a) of the object after time t1, and the coordinates on the display unit 11. As shown in FIG. 11(B), the coordinate information of the operated part (part 81a) of the object 80 in the virtual space and the coordinate information of the operated part (part 81a) of the object 80 on the display unit 11 are changed based on the amount of movement of the operating body (tip 90a of the index finger), and the changed coordinate information is stored (the coordinate information is rewritten).

Returning to FIG. 10, the explanation will be given. At this time, the object display unit 140 displays the object 80 on the display unit 11 using the information of the operated part after the movement (after time t1) in the superimposed display process S100. FIG. 12 is a user interface in which the object 80 in which the position of the operated part has changed is displayed on the display unit 11. As shown in FIG. 12, the operated part (part 81a) moves from the position displayed in FIG. 7 based on a different part (part 81b, corresponding to the elbow joint). In other words, based on information on the movement of the operating body, some of the parts of the object can be moved. The above process is repeated as long as the operating body moves (S330; Yes). In this example, the amount of movement of the operating body is obtained 60 times per second, so that the change in the posture of the object is displayed smoothly. When the movement of the terminal 10 is completed (S330; No), the second object posture detection process S300 is completed.

As described above, by using the object posture control program of this embodiment, it is possible to easily control the posture of an object having multiple parts without performing complex control.

Second Embodiment
In this embodiment, an object attitude control process different from that in the first embodiment will be described. Specifically, an example in which multiple parts move based on the movement of one terminal will be described. Note that configurations and methods similar to those in the first embodiment may be omitted as appropriate.

(2-1. Configuration of Server and Object Attitude Control Unit 100A)
Fig. 13 shows a functional block diagram of an object posture control unit 100A in the object posture control system 1A. As shown in Fig. 22, the object posture control unit 100A includes a captured image acquisition unit 110, a marker detection unit 120, a space definition unit 130, an object display unit 140, an operation body detection unit 150, an overlapping portion detection unit 160, a notification unit 170, a movement information acquisition unit 180, and an object coordinate change unit 190, as well as a second operation body detection unit 195 and a second overlapping portion detection unit 200.

The second operating body detection unit 195 has a function of detecting an object different from the object detected as the operating body from among multiple objects contained in the captured image, and setting it as the second operating body. The second overlapping portion detection unit 200 has a function of detecting that the second operating body overlaps with a part of multiple parts of the object, and setting that part as the second operated part.

(2-2. Object Attitude Control Processing)
FIG. 14 shows a flow diagram of the first object posture detection process S200A based on a command by the object posture control program in the object posture control unit 100A. As shown in FIG. 14, after the notification of the overlapping part is given (S250), it is determined whether to set an additional operating body (S255). The above determination may be made based on whether there is a moving object other than the object set as the operating body during a predetermined period, or may be made based on input information from the user. In addition, the display unit 11 may display "Do you want to add a new operating body?" along with buttons "Yes" and "No". If no additional setting is given (S255; No), the first object posture detection process S200A is terminated.

When it is determined that an additional operation body is to be set (S255; Yes), the second operation body detection unit 195 detects an object to be the second operation body (S260). In this example, the user's left hand 91 is detected as the second operation body. The detection method may be the same as that described in the first embodiment of the present invention. At this time, the part of the user's left hand 91 closest to the sheet 70 (more specifically, the tip 91a of the index finger of the left hand) may be set as the operation body (S265).

Next, the second overlapping portion detection unit 200 detects an overlapping portion (second overlapping portion) between the second operating body and a portion of the object 80 (S270). The second overlapping portion is detected using a method similar to the overlapping portion detection method of the first embodiment of the present invention. If it is determined that the second operating body overlaps with a portion 81c of the object 80 (S270; Yes), the object portion 81c is set as the second operated portion (S280). Note that if the second overlapping portion is not detected (S270; No), the second overlapping portion detection process is repeated in a loop.

After the second overlapping portion is detected, the notification unit 170 may notify that the second overlapping portion has been detected (S290). The notification method may be the same as that in the first embodiment. This completes the first object pose detection process S200A.

Figure 16 is a flow diagram of the third object control process S300A. In this embodiment, the movement of the operated part (part 81a) of the object accompanying the movement of the operating body (tip 90a of the index finger of the user's hand 90) is realized in the same manner as in the first embodiment. In this embodiment, as shown in Figure 17, at the same time that the operated part (part 81a) moves, the coordinate change of the operated part and the second operated part is changed so that the second operated part (part 81c, corresponding to the ankle joint) moves with respect to part 81e based on the amount of movement of a second operating body (tip 91a of the index finger of the user's left hand 91) different from the operating body (S320). In other words, the posture of the object can be controlled by moving multiple operated parts in accordance with the movement of multiple operating bodies.

In this embodiment, an example in which two operating bodies are used to control the attitude of an object is shown, but the present invention is not limited to this. It can also be applied to a case in which there are three or more operating bodies.

It should be noted that within the scope of the concept of the present invention, a person skilled in the art may come up with various modifications and alterations, and it is understood that these modifications and alterations also fall within the scope of the present invention. For example, those embodiments described above in which a person skilled in the art has appropriately added or deleted components or modified the design, or added or omitted steps or changed conditions, are also included in the scope of the present invention as long as they contain the gist of the present invention.

(Modification)
In the first embodiment of the present invention, an example in which the X coordinate, Y coordinate, and Z coordinate are used in the movement information of the operating body is shown, but is not limited thereto. For example, not only the X coordinate, Y coordinate, and Z coordinate but also the angle θ may be used. In this case, any of the X coordinate, Y coordinate, and Z coordinate may not be used. In accordance with the movement of the operating body, the information of the operated part (part 81a) of the object 80 in the virtual space and the position coordinates of the operated part (part 81a) of the object 80 in the display unit 11 are changed, and the information of the changed position coordinates is stored (the coordinate information is rewritten). This allows the posture of the object, especially the twist posture, to be controlled.

In the first embodiment of the present invention, an example is shown in which an operating body is detected and set, and then an overlapping portion is detected and an operated part is set, but this is not limiting. For example, when a target object and an object overlap, the overlapping target object may be detected and set as the operating body, and the overlapping portion of the object may be set as the operated part.

In the first embodiment of the present invention, an example in which the tip 90a of the index finger overlaps a portion of the object (portion 81a) has been mainly described, but this is not limiting. The overlap detection unit 160 may detect overlap when there are multiple overlapping portions. For example, when there is overlap at two or more points, the user's fingers may have a shape that pinches the object, and the posture of the object can be controlled more naturally.

In addition, when it is determined that multiple parts overlap, additional control different from the normal object posture control may be provided. FIG. 18 is an example of a user interface when it is determined that multiple parts overlap and the operating body is moved. As shown in FIG. 18, in this example, the tip 90a of the index finger and the tip 90b of the thumb overlap with the part 81(a) of the object. At this time, in accordance with the movement of the tip 90a of the index finger and the tip 90b of the thumb, the part 81a of the object 80 also moves based on a preset condition, and further the bone 82 (82a) is extended. In this way, the posture of the object can be controlled in various forms by performing additional control different from the normal object posture control.

The detection may be performed by moving the operating object so as to surround the part corresponding to part 81a, or by surrounding the part corresponding to part 81a with multiple fingers.

In the first embodiment of the present invention, the notification unit 170 provides notification using sound information, but this is not limited to this example. For example, the notification unit 170 may provide notification by vibrating the terminal 10 when an overlapping portion is detected. Alternatively, the notification unit 170 may provide notification by causing the terminal 10 to emit light when an overlapping portion is detected. Alternatively, the notification unit 170 may provide notification using text information when an overlapping portion is detected. Alternatively, the notification unit 170 may provide notification by highlighting the overlapping portion by changing its color when an overlapping portion is detected. Also, the notification unit 170 does not necessarily have to be used.

The detection of the operating object is not limited to the method using parallax. For example, an object that satisfies a predetermined condition based on the color or shape of the object may be detected as the operating object.

An object moving at a speed faster than a predetermined speed may be detected as the operating body. An object may also be detected as the operating body when the distance derived from the virtual space coordinates of the object when placed in the virtual space and the virtual space coordinates of the object becomes smaller than a predetermined distance.

In the first embodiment of the present invention, an example has been shown in which a captured image and an object in a virtual space are aligned using a marker-based alignment method and displayed in a superimposed manner, but this is not limiting. As a method for aligning a virtual space including an object with a captured image, in addition to the marker-based alignment method, a natural feature-based alignment method that does not use a specific marker, a vision-based alignment method such as a model-based alignment method, or a sensor-based alignment method that uses a sensor may also be used as appropriate.

In the first embodiment of the present invention, an example of displaying an object using augmented reality is shown, but the present invention is not limited to this. One embodiment of the present invention can also be applied to mixed reality (MR) and other technologies in addition to augmented reality, as long as the coordinates of the object can be specified.

In the first embodiment of the present invention, an example is shown in which the image is displayed on the display unit of a flat terminal having a form similar to that of a smartphone, but the present invention is not limited to this. For example, a goggle-type terminal may be used as the terminal for display. In this case, a two-lens camera (stereo camera) may be used for the imaging unit 17. This allows the displayed image to be displayed in a more three-dimensional manner.

1...Object posture control system, 10...Terminal, 11...Display unit, 12...Control unit, 13...Memory unit, 14...Operation unit, 15...Communication unit, 16...Sensor unit, 17...Imaging unit, 18...Speaker unit, 20...Server, 21...Communication unit, 22...Memory unit, 23...Control unit, 24...Display unit, 50...Network, 60...Table, 70...Sheet, 75...Marker, 80...Object, 81...Part, 82...Bone , 90...hand, 90a...tip, 90b...tip, 91...hand, 91a...tip, 100...object posture control unit, 110...captured image acquisition unit, 120...marker detection unit, 130...space definition unit, 140...object display unit, 150...operation body detection unit, 160...overlapped portion detection unit, 170...notification unit, 180...movement information acquisition unit, 190...object coordinate change unit, 195...second operation body detection unit, 200...second overlapped portion detection unit

Claims (5)

a display unit that displays a first image representing an object having a plurality of parts on a second image captured by an imaging unit;
An object posture control program for controlling the posture of the object by changing position information of at least a part of a plurality of parts of the object to be operated based on the positional relationship between the object and an operating body in the second image. executes a process for notifying a user when a position of the operating object and a position of the operated portion at least partially overlap with each other;
2. The object attitude control program according to claim 1. changing the position of the operated portion based on a change in position information of the operating body while the position of the operating body and the position of the operated portion at least partially overlap with each other;
3. The object attitude control program according to claim 1. The position of the operated part and the position of the operated part are each defined as a position in a coordinate system of a space in which the object is placed.
4. The object attitude control program according to claim 2 or 3. An information processing device for controlling an attitude of an object,
A first image representing an object having a plurality of parts is superimposed on a second image captured by the imaging unit and displayed on the display unit;
An information processing device that controls a posture of the object by changing position information of at least a part of an operated portion among a plurality of portions of the object based on a positional relationship between the object and an operating body in the second image.

Images