JP2021009552A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To provide an information processing apparatus, an information processing method, and a program capable of correctly selecting an object intended by a user even when an object-in-object is included in a plurality of objects existing in a field of view of the user.SOLUTION: An information processing apparatus 100 comprises a detection unit 12, a calculation unit 17, and a selection unit 18. The detection unit detects a designated area designated by an operation body that moves within a field of view of a user according to a motion of the user. The calculation unit calculates the degree of overlapping with the designated area for an object that overlaps with the designated area in the field of view of the user, among a plurality of objects that exist in the field of view of the user and include one or more objects contained within another object. The selection unit selects a target object from the plurality of objects based on the degree of overlapping and the size of the object.SELECTED DRAWING: Figure 4

Description

Embodiments of the present invention relate to information processing devices, information processing methods and programs.

In recent years, information processing devices such as HMDs (Head Mounted Display) that allow users to experience virtual reality (VR) and augmented reality (AR) have become widespread. In this type of information processing device, when a desired object is selected from a plurality of objects existing in the user's field of view, a predetermined process is executed based on the selected object. As a user interface for selecting an object, a virtual pointer is generally placed in the user's field of view so that the user can select the object by moving the pointer.

However, with such a conventional method, it may not be possible to uniquely identify which object the user is trying to select from among the plurality of objects existing in the user's field of view. For example, if multiple objects include a small object contained in a large object (hereinafter referred to as "in-object object"), the object with a large virtual pointer position and the object within it. It overlaps with both in-object objects, and it may not be possible to uniquely identify whether the user is trying to select a large object or an in-object object, so improvement is required.

JP-A-2002-42172 JP-A-2015-203889 Japanese Unexamined Patent Publication No. 2011-203823

The problem to be solved by the present invention is an information processing device and information that can correctly select an object intended by the user even when a plurality of objects existing in the user's field of view include an object in the object. It is to provide a processing method and a program.

The information processing device of the embodiment includes a detection unit, a calculation unit, and a selection unit. The detection unit detects a designated area designated by the operating body that moves in the user's field of view according to the user's operation. The calculation unit is a plurality of objects existing in the user's field of view, and among the plurality of objects including one or more objects included in the other objects, the calculation unit and the designated area in the user's field of view. For the overlapping objects, the degree of superimposition with the designated area is calculated. The selection unit selects a target object from the plurality of objects based on the degree of superimposition and the size of the object.

FIG. 1 is a diagram illustrating a problem of a conventional method of selecting an object. FIG. 2 is a diagram showing a state in which the user wears the information processing device according to the first embodiment. FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus according to the first embodiment. FIG. 4 is a block diagram showing a functional configuration example of the information processing apparatus according to the first embodiment. FIG. 5-1 is a diagram illustrating an example of a designated area. FIG. 5-2 is a diagram illustrating an example of a designated area. FIG. 5-3 is a diagram illustrating an example of a designated area. FIG. 5-4 is a diagram illustrating an example of a designated area. FIG. 5-5 is a diagram illustrating an example of a designated area. FIG. 5-6 is a diagram illustrating an example of a designated area. FIG. 6 is a diagram illustrating an outline of processing by the image interpolation unit. FIG. 7-1 is a diagram illustrating a specific example of object selection. FIG. 7-2 is a diagram illustrating a specific example of object selection. FIG. 7-3 is a diagram illustrating a specific example of object selection. FIG. 7-4 is a diagram illustrating a specific example of object selection. FIG. 7-5 is a diagram illustrating a specific example of object selection. FIG. 8 is a diagram showing an example of a method for causing the user to recognize the selected object. FIG. 9 is a diagram showing an example of a method for causing the user to recognize the selected object. FIG. 10 is a diagram illustrating a process of enlarging / reducing an image. FIG. 11 is a flowchart showing an outline of the operation of the information processing apparatus. FIG. 12 is a block diagram showing a functional configuration example of the information processing apparatus according to the second embodiment.

Hereinafter, the information processing apparatus, the information processing method, and the program of the embodiment will be described in detail with reference to the accompanying drawings.

<Outline of Embodiment>
In an information processing device such as an HMD using VR technology or AR technology, a user interface for selecting a desired object from a plurality of objects existing in the user's field of view is arranged in the user's field of view. Generally, a virtual pointer is moved by a user through a gesture or the like so that a desired object can be selected. However, in such an object selection method using a virtual pointer, when multiple objects that are candidates for selection include objects within the object, it is not possible to uniquely identify which object the user is trying to select. There is.

For example, as shown in FIG. 1, there is a PC (personal computer) image or CG (computer graphics) part in the user's field of view, and the entire PC, the keyboard in the PC, and each key in the keyboard are candidates for selection. It is assumed that it is recognized as an object. Here, when the user moves the pointer P to the position shown in FIG. 1, the position pointed to by the pointer P overlaps the entire PC, the keyboard in the PC, and the keys in the keyboard, so that the object the user is trying to select. It is not possible to uniquely identify whether is the entire PC, the keyboard in the PC, or the keys in the keyboard.

Therefore, in the present embodiment, a plurality of objects included in the user's field of view are used by using the degree of superimposition with the designated area specified by the operating body that moves in the user's field of view according to the user's movement and the size of the object. We propose a new technical proposal that enables the user to correctly select the object intended by the user. In the following, an example of application to the HMD using the AR technology will be described as the first embodiment, and an example of application to the HMD using the VR technology will be described in order as the second embodiment. In the following description, components having the same function are designated by the same reference numerals, and duplicate description will be omitted as appropriate.

<First Embodiment>
The information processing device according to the first embodiment is configured as an HMD using AR technology. HMDs using AR technology are commonly referred to as "AR glasses". FIG. 2 shows a state in which the user U wears the information processing device 100 according to the first embodiment. The information processing device 100 configured as AR glasses can be worn by the user U on the head and used in the same manner as general glasses.

The information processing device 100 is equipped with a camera 104 that captures a real space in front of the user U, and the image (video) of the real space captured by the camera 104 corresponds to the eye of the user U corresponding to the lens portion of the glasses. It is displayed on the display 106 arranged in front of the. Further, the information processing device 100 can generate various information as, for example, virtual CG parts, superimpose the CG parts on the real space image captured by the camera 104, and display the CG parts on the display 106. As a result, the user U equipped with the information processing device 100 can experience the AR space in which the CG parts are contained in the real space.

In the information processing apparatus 100, a process such as semantic segmentation by SegNet described in Reference 1 below is performed on the image in the real space captured by the camera 104. Semantic segmentation is a technique for identifying an object in an image by assigning a label such as an object represented by the pixel to each pixel of the image. As a result, an object (object) existing in the real space in front of the user U is recognized. The recognition of this object is carried out with a relatively fine particle size, and objects in the object such as the keyboard in the PC and the keys in the keyboard illustrated in FIG. 1 are also recognized. The object recognized here is a candidate for an object selected by the user U wearing the information processing device 100 when the information processing device 100 is made to execute a predetermined process.
(Reference 1) Vijay Badrinarayanan et al. , "SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentation", arXiv: 1511.00561v3, [cs.CV], 10. Oct. 2016.

Further, the CG part displayed by superimposing the image in the real space captured by the camera 104 may also be generated so as to include the object in the object. These CG parts are also candidates for object selection. That is, in the field of view of the user U equipped with the information processing device 100, as candidates for object selection, a plurality of objects reflected in the image in the real space captured by the camera 104, CG parts generated by the information processing device 10, and the like. Objects exist, and among these multiple objects, there are objects within the object.

When a desired object is selected from a plurality of objects included in the field of view of the user U equipped with the information processing device 100, a predetermined process based on this object is executed. For example, a similar image search for searching an image database for an image showing an object similar to the selected object according to a search query including the selected object is given as an example of a predetermined process. The result of this processing can be superimposed on the real space image captured by the camera 104 and displayed on the display 106.

In the information processing device 100 of the present embodiment, as described above, in order to enable the user to correctly select the object intended by the user from a plurality of objects included in the field of view of the user U who wears the information processing device 100. As a criterion for selecting an object, the degree of superimposition with a designated area specified by an operating body that moves in the field of view of the user U according to the operation of the user U and the size of the object are used. In particular, in the present embodiment, the hand of the user U reflected in the image in the real space captured by the camera 104 is assumed as the operating body that moves in the field of view of the user U according to the movement of the user U. That is, a target object is selected from a plurality of objects included in the field of view of the user U based on the degree of superimposition with the designated area specified by the user U and the size of the object.

FIG. 3 is a block diagram showing a hardware configuration example of the information processing apparatus 100 according to the first embodiment. As shown in FIG. 3, the information processing apparatus 100 includes a processor 101 such as a CPU (Central Processing Unit) constituting a computer system and a memory 102 such as a RAM (Random Access Memory) and a ROM (Read Only Memory). To be equipped. Further, the information processing device 100 includes a storage device 103 using, for example, a micro SD card as a recording medium, a camera 104 that captures a real space in front of the user U, and a microphone 105 into which a voice command spoken by the user U is input. It includes a display 106 that displays real-space images and CG parts captured by the camera 104, a communication I / F 107 for communicating with an external device such as a server, and the like, and these are connected to the above-mentioned computer system. In addition to the configuration shown in FIG. 3, the information processing device 100 may be provided with various sensing devices for detecting the movement of the user U, or may be connected to various sensing devices.

FIG. 4 is a block diagram showing a functional configuration example of the information processing apparatus 100 according to the first embodiment. As functional components of the information processing device 100, for example, as shown in FIG. 4, the image pickup unit 11, the detection unit 12, the image holding unit 13, the image interpolation unit 14, and the image scale changing unit 15 are included. A display unit 16, a calculation unit 17, a selection unit 18, a feedback unit 19, an object holding unit 20, a cancel processing unit 21, and a processing execution unit 22 are provided.

Among these functional components, the imaging unit 11 is realized by using the camera 104 shown in FIG. Further, the display unit 16 is realized by using the display 106 shown in FIG. Further, the image holding unit 13 and the object holding unit 20 are realized by using the memory 102 and the storage device 103 shown in FIG. The other detection unit 12, image interpolation unit 14, image scale change unit 15, calculation unit 17, selection unit 18, feedback unit 19, cancellation processing unit 21, and processing execution unit 22 are, for example, the processor 101 shown in FIG. Is realized by executing a predetermined program stored in the storage device 103 or the like using the memory 102. In addition, some or all of these parts are equipped with dedicated hardware such as ASIC (Application Specific Integrated Circuit) and FPGA (Field-Programmable Gate Array) (dedicated processor instead of general-purpose processor 101 such as CPU). It may be realized by using.

The imaging unit 11 images the real space in front of the user U equipped with the information processing device 100. The real space image (video) in front of the user U captured by the imaging unit 11 is sent to the detection unit 12, the image holding unit 13, and the image interpolation unit 14.

The detection unit 12 detects the user U's hand (an example of an “operation body”) reflected in this image from the image in the real space in front of the user U captured by the image pickup unit 11, and thereby the user U. Detects the specified area specified by the hand of. The user U's hand reflected in the image can be detected, for example, by the above-mentioned semantic segmentation.

The designated area is an area to be superimposed on the object in order for the user U to select the object, and is designated by the user U reflected in the image captured by the imaging unit 11. The range of this designated area is determined according to the shape of the user U's hand reflected in the image captured by the imaging unit 11 or the movement thereof. Specific examples of the designated area assumed in the present embodiment are shown in FIGS. 5-1 to 5-6.

First, an example of a designated area determined according to the shape of the hand of the user U will be described. For example, when the palm is open, the area from the fingertip to the wrist is designated as the designated area R, as shown in FIG. 5-1. Further, for example, in the case where only the index finger is raised, the area from the tip of the index finger to the base is designated as the designated area R, as shown in FIG. 5-2. Further, for example, in the case where a circle is formed by the index finger and the thumb, the circle area formed by the index finger and the thumb is designated as the designated area R as shown in FIG. 5-3. Further, for example, in the case where a square is formed by the index fingers and thumbs of both hands, as shown in FIG. 5-4, the square area formed by the index fingers and thumbs of both hands is designated as the designated area R. In addition, the shape of the hand of the user U who designates the designated area R can include various predetermined patterns.

The shape of the user U's hand reflected in the image captured by the imaging unit 11 can be identified by using, for example, a CNN (Convolutional Neural Network) such as VGGNet described in Reference 2 below. That is, by applying the CNN in which the above-mentioned hand shape pattern is learned in advance to the hand region of the user U detected from the image captured by the image pickup unit 11, the user U reflected in the image. The shape of the hand can be identified.
(Reference 2) Karen Simonyan and Andrew Zisserman, “VERY DEEP CONVOLUTIONAL NETWORKS FOR LARGE-SCALE IMAGE RECOGNITION”, arXiv: 1409.1556v6, [cs.CV], 10. Apr. 2015.

Next, an example of the designated area determined according to the movement of the hand of the user U will be described. For example, if the palm is open and all four fingers other than the thumb are moving, the area from the fingertips to the base of the four fingers other than the thumb is specified as shown in FIG. 5-5. It is defined as the area R. Further, when the entire palm is moved so as to be stroked, as shown in FIG. 5-6, the region of the entire locus in which the palm has moved is designated as the designated region R. These hand movements can be identified using, for example, a classifier that handles time-series data such as an RNN (Recurrent Neural Network).

As described above, since the designated area R is the area designated by the hand of the user U reflected in the image captured by the imaging unit 11, the size of the designated area R is determined by the user U wearing his / her hand. The closer it is to the information processing device 100, the larger it becomes, and the farther it is, the smaller it becomes. The position information of the designated area R detected by the detection unit 12 is sent to the calculation unit 17. Further, the position information indicating the area of the user U's hand detected by the detection unit 12 is sent to the image interpolation unit 14.

The image holding unit 13 holds the image captured by the imaging unit 11 as a past image used for processing by the image interpolation unit 14 for a certain period of time.

The image interpolation unit 14 performs a process of interpolating the background of the region in which the hand of the user U is reflected in the image newly captured by the image pickup unit 11 by using the past image held by the image holding unit 13. The image captured by the image pickup unit 11 is occlusioned by the hand of the user U reflected in the image, and the background of the region of the hand of the user U becomes invisible. As a result, there is a concern that it becomes difficult for the user U to accurately superimpose the designated area R on the desired object, or it becomes difficult to accurately calculate the degree of superimposition between the designated area R and the object. Therefore, the image interpolation unit 14 performs a process of interpolating the background of the hand region of the user U.

FIG. 6 is a diagram for explaining the outline of the processing by the image interpolation unit 14, and the image holding unit 13 holds the region X in which the hand of the user U is reflected in the image Im1 newly captured by the image capturing unit 11. An example of interpolation using the past image Im2 is shown.

The image interpolation unit 14 first selects, among the past images held by the image holding unit 13, the past image Im2 in which the amount of displacement between the image holding unit 13 and the image Im1 is small and the area in which the user U's hand is reflected does not overlap. To do. Specifically, the image interpolation unit 14 divides, for example, past images held by the image holding unit 13 into tiles in which the areas in which the user U's hand is reflected do not overlap, and the user U's hand The optical flow between the tile image excluding the reflected region and the corresponding region of the image Im1 newly captured by the imaging unit 11 is calculated. Then, the past image containing the most tile images with the least optical flow to and from the image Im1 is selected as the past image Im2 used for interpolation of the region X.

Next, the image interpolation unit 14 aligns the image Im1 with the past image Im2 by using, for example, a SIFT (Scale-Invariant Feature Transform) feature having position invariance. Specifically, the image interpolation unit 14 detects the feature points of SIFT for both the image Im1 and the past image Im2, and calculates the feature amount at each feature point. Then, feature points having a sufficiently small distance (difference in feature amount) are associated with each other between the image Im1 and the past image Im2, and scale conversion, rotation, shift movement, etc. are performed so that these feature points have the same coordinates. Perform image processing. As a result, the misalignment between the image Im1 and the past image Im2 is corrected.

Next, the image interpolation unit 14 cuts out a region Y in the past image Im2 corresponding to the region X in which the hand of the user U is reflected in the image Im1, and cuts out the region X in the image Im1 from the past image Im2. The outline of the area X is marked while being replaced with the image of. As a result, the background of the region X in the image Im1 is interpolated, and the image Im3 on which the marker M representing the outline of the user U's hand is superimposed is generated. At this time, the brightness of the image of the region Y cut out from the past image Im2 is lowered, or the image is replaced after performing predetermined image processing such as blurring, so that the user U's hand is transparent and vague. It is possible to give a visual effect that makes the background visible.

The image Im3 (an image in which the background of the region in which the hand is reflected) generated by the image interpolation unit 14 is sent to the display unit 16 and the calculation unit 17 via the image scale changing unit 15. The image scale changing unit 15 is a functional unit that enlarges / reduces the image Im3 as needed, and a specific description thereof will be described later.

The display unit 16 displays the image Im3 generated by the image interpolation unit 14 (when the image scale changing unit 15 performs the enlargement / reduction processing, the enlarged / reduced image Im3). As shown in FIG. 6, this image Im3 is superposed with a marker M indicating the contour of the user U's hand. Therefore, the user U moves his / her hand so as to superimpose the designated area R on the desired object from among the plurality of objects existing in the field of view of the user U while looking at the image Im3 displayed by the display unit 16. Can be done.

The calculation unit 17 calculates the degree of superimposition with the designated area R for the object overlapping the designated area R designated by the user U among the plurality of objects existing in the field of view of the user U. The calculation of the superposition degree by the calculation unit 17 is started by a predetermined action by the user U as a trigger.

Here, various actions of the user U that trigger the start of the calculation of the superposition degree by the calculation unit 17 can be considered. For example, the calculation unit 17 may start the calculation of the degree of superimposition triggered by the user U performing an act of stopping the movement of the hand for a certain period of time. This can be realized, for example, by determining whether or not the state in which the displacement amount of the user U's hand reflected in the image captured by the imaging unit 11 is equal to or less than a predetermined value continues for a preset number of frames.

In addition, a predetermined gesture such as a preset gesture, for example, putting a hand forward and stopping like a button pressing operation, or pointing to a designated area R with the other hand when a designated area R is specified with one hand, is performed. The calculation of the superposition degree by the calculation unit 17 may be started by using the user U as a trigger. Gesture detection can be realized, for example, by determining whether or not the shape and movement of the user U's hand reflected in the image captured by the imaging unit 11 matches the preset gesture.

Further, the calculation unit 17 starts the calculation of the superposition degree by using a preset voice command such as "select here" or "select this" as a trigger when the user U utters a predetermined voice command. It may be. The detection of a voice command can be realized, for example, by voice-recognizing the utterance of the user U input to the microphone 105 mounted on the information processing device 100 and determining whether or not it matches the preset voice command. ..

Further, when the information processing device 100 is equipped with a line-of-sight detection sensor and the line-of-sight detection sensor detects a predetermined line-of-sight movement of the user U, for example, it triggers that the line-of-sight direction of the user U overlaps with the designated area R. As a result, the calculation unit 17 may start the calculation of the superposition degree.

In addition, a specific facial expression change of the user U, for example, a facial expression change such as closing (winking) one eye may be detected, and this may be used as a trigger to start the calculation of the superposition degree by the calculation unit 17. This is because, as in the example shown in FIGS. 5-3 and 5-4, when the circled or square area surrounded by the fingers is set as the designated area R, the user tries to look into the designated area R. Then, one eye is often closed naturally, so it is considered to be a natural process.

As described above, the calculation unit 17 calculates the degree of superimposition on the object that overlaps with the designated area R designated by the user U among the plurality of objects existing in the field of view of the user U. Here, when an object in the object is included in the object overlapping the designated area R, the degree of superimposition with the designated area R is calculated even for a large object including the object in the object. For example, in the above-mentioned example of a PC, when the designated area R overlaps with any key in the keyboard, not only the key but also the entire keyboard including the key, and further the entire PC including the keyboard are also designated areas. The degree of superimposition with R is calculated.

Here, the degree of superimposition with the designated area R is
Superposition = Calculated by the size of the area within the object that overlaps the specified area R / the size of the object. The superposition degree calculated by the calculation unit 17 is sent to the selection unit 18.

The selection unit 18 is a target object (depending on the processing execution unit 22 described later) from among a plurality of objects existing in the field of view of the user U based on the superposition degree calculated by the calculation unit 17 and the size of the object. Select the object to be used for processing). Specifically, the selection unit 18 is the object having the largest size among the objects whose superposition degree calculated by the calculation unit 17 exceeds a predetermined threshold value (for example, 0.8) with respect to the object overlapping the designated area R. Select.

Here, a specific example of object selection by the selection unit 18 will be described with reference to the example of a PC as in FIG. 7-1 to 7-5 are diagrams for explaining a specific example of object selection by the selection unit 18 using a PC as a subject. Here, it is assumed that the objects that are candidates for selection are the entire PC, the monitor in the PC, the touch pad in the PC, the keyboard in the PC, and each key in the keyboard.

In the example shown in FIG. 7-1, the designated area R designated by the user U overlaps the entire PC, and is included in the entire PC, the monitor in the PC, the touch pad in the PC, the keyboard in the PC, and the keyboard. In all of the keys, the degree of superimposition with the designated area R is equal to or higher than the threshold value. In this case, the selection unit 18 selects the entire PC having the largest size as the target object.

In the example shown in FIG. 7-2, the designated area R designated by the user U mainly overlaps with the monitor in the PC, and the degree of superimposition with the designated area R is equal to or higher than the threshold value of the PC. Only the monitor inside. In this case, the selection unit 18 selects the monitor in the PC, which is the only object whose degree of superimposition with the designated area R is equal to or greater than the threshold value, as the target object.

In the example shown in FIG. 7-3, the designated area R designated by the user U overlaps the touch pad in the PC and some keys in the keyboard, and the degree of superimposition with the designated area R is equal to or higher than the threshold value. It is the touchpad in these PCs and some keys in the keyboard. In this case, the selection unit 18 selects the touchpad in the PC having the largest size among the touchpad in the PC and some keys in the keyboard as the target object.

In the example shown in FIG. 7-4, the designated area R designated by the user U overlaps only some keys in the keyboard, and the degree of superimposition with the designated area R is equal to or higher than the threshold value. , Only the keys in the keyboard. In this case, the selection unit 18 in this case selects the key in the keyboard, which is the only object whose degree of superimposition with the designated area R is equal to or greater than the threshold value, as the target object.

In the example shown in FIG. 7-5, the designated area R designated by the user U overlaps the keyboard in the PC and each key in the keyboard, and the degree of superimposition with the designated area R is equal to or higher than the threshold value. Is the keyboard in these PCs and each key in the keyboard. In this case, the selection unit 18 selects the keyboard in the PC having the largest size among the keyboard in the PC and each key in the keyboard as the target object.

When an object made of CG parts is arranged in the field of view of the user U as a candidate object for selection, a plurality of objects having the same size may be overlapped depending on the viewpoint of the user U. Here, when the designated area R designated by the user U overlaps with these plurality of objects and the degree of superimposition of these plurality of objects with the designated area R becomes equal to or more than the threshold value, the user U sets the object on the front side. Since it is presumed that the object is to be selected, the selection unit 18 selects the object closest to the user U (that is, the information processing device 100 worn by the user U) as the target object among these plurality of objects. You may try to do it.

The information of the object selected by the selection unit 18 is sent to the feedback unit 19 and the object holding unit 20. The object holding unit 20 is a buffer that holds information on the object selected by the selection unit 18.

The feedback unit 19 performs feedback processing for the user U to recognize that the object has been selected by the selection unit 18. Examples of this feedback process include a process of causing the user U to visually recognize the object selected by the selection unit 18. Specifically, as shown in FIG. 8, the feedback unit 19 is an area of an object (here, a touch pad in a PC) selected by the selection unit 18 in the image 50 displayed on the display unit 16. 51 may be detected and highlighting processing may be performed to make the area 51 of this object stand out. As the highlighting process, for example, processing such as inversion of the outline of the area 51 or increasing the brightness of each pixel in the area 51 to make it shine can be considered.

Further, as shown in FIG. 9, for example, the feedback unit 19 provides a presentation area 52 for presenting an object selected by the selection unit 18 in the image 50 displayed on the display unit 16, and this presentation area A cut-out image 53 cut out in the vicinity of the object (here, the touch pad in the PC) selected by the selection unit 18 may be displayed in the 52. By performing the process of visually recognizing the object selected by the selection unit 18 to the user U in this way, whether or not the user U has correctly selected the object while looking at the image 50 of the display unit 16. Can be confirmed.

Further, the feedback unit 19 may perform a process of causing the user U to recognize that the object has been selected by the selection unit 18 by vibration. For example, the feedback unit 19 may vibrate the information processing device 100 at the timing when the object is selected by the selection unit 18. Further, the feedback unit 19 vibrates a smart watch or the like connected to the information processing device 100 by short-range wireless communication such as Bluetooth (registered trademark) at the timing when the object is selected by the selection unit 18. May be good.

The cancel processing unit 21 performs a process of canceling the object selected by the selection unit 18 triggered by a predetermined action by the user U. Specifically, the cancel processing unit 21 deletes the latest object held by the object holding unit 20 when the user U performs a predetermined action predetermined as a trigger for the cancel processing. Further, the cancel processing unit 21 may delete all the objects held by the object holding unit 20 when the user U performs a predetermined action as a trigger for the cancel processing. By the processing of the cancel processing unit 21, when the object selected by the selection unit 18 is not intended by the user U, the object can be selected again.

Here, various actions of the user U that trigger the cancellation process can be considered. For example, the cancellation processing unit 21 holds the object as a trigger when the user U superimposes the designated area R on the same object as the object selected by the selection unit 18 and repeatedly selects the object. It may be deleted from the part 20.

In addition, the cancel processing unit 21 uses the latest object or the latest object from the object holding unit 20 as a trigger when the user U performs a predetermined gesture such as raising a thumb or flicking with the index finger, which is a preset gesture. You may want to delete all objects.

Further, when the user U utters a predetermined voice command such as a preset voice command such as "cancel" or "redo", the cancel processing unit 21 sends the latest object or all from the object holding unit 20. You may want to delete the object.

When the object to be selected by the user U is a smaller size object included in the object selected by the selection unit 18, but this small object is not recognized as a candidate object for selection. There is also. In such a case, after canceling the object selected by the selection unit 18, for example, the above-mentioned semantic segmentation or the like is performed with the user U uttering a voice command such as "more finely". The objects used may be recognized with a finer grain size so that smaller objects can be selected.

By the way, the size of the designated area R used for selecting an object changes as the user U moves his / her hand closer to or further from the information processing device 100 as described above, but the range in which the user U can move his / her hand is limited. There is a limit. For this reason, among the objects that are candidates for selection, there is a concern that the designated area R cannot be correctly superimposed on, for example, an object having an extremely small size, and such an object cannot be properly selected. ..

As a countermeasure against such a problem, the information processing apparatus 100 according to the present embodiment includes an image scale changing unit 15. The image scale changing unit 15 is triggered by a predetermined action by the user U, and among the images displayed on the display unit 16 (for example, the image Im3 shown in FIG. 6), the hand of the user U (for example, the image shown in FIG. 6). A process of enlarging or reducing an image excluding the marker M) in Im3 is performed. That is, the image scale changing unit 15 is displayed on the display unit 16 without changing the size of the user U's hand (for example, the marker M in the image Im3 shown in FIG. 6) in response to a predetermined action by the user U. The image (for example, the image Im3 shown in FIG. 6) is enlarged or reduced. As a result, the relative size of the object with respect to the designated area R can be changed, and the object can be selected more appropriately.

Here, various actions of the user U that trigger the enlargement / reduction processing of the image can be considered. For example, the posture change of the user U may be detected, and the image scale changing unit 15 may perform enlargement / reduction processing on the image displayed on the display unit 16 according to the posture change. Specifically, for example, as shown in FIG. 10, when the user U bends his head backward, the image is reduced, and when the user U puts his head forward, the image is enlarged. Be done. For the detection of the posture change, for example, the movement amount of the user U is calculated by using the optical flow of the image captured by the imaging unit 11, or the information processing device 100 is equipped with a sensor that detects the movement such as a gyro sensor. If so, it can be realized by calculating the movement amount of the user U using the sensor output.

Further, when a physical operation unit such as an operation dial is provided in the housing of the information processing device 100 and the user U operates this operation unit, the image scale change unit 15 is displayed on the display unit 16 according to the operation amount. The displayed image may be enlarged or reduced.

In addition, the image scale changing unit 15 is triggered by the user U performing a preset gesture, for example, a predetermined gesture such as pulling the hand with the fingers widely spread toward the front side or pushing it toward the back side. May perform enlargement / reduction processing on the image displayed on the display unit 16.

Further, when the user U utters a preset voice command, for example, a predetermined voice command such as "enlargement" or "reduction", the image scale changing unit 15 enlarges the image displayed on the display unit 16. -The reduction process may be performed.

In addition, a specific facial expression change of the user U, for example, a facial expression change such as closing the eyes small or opening the eyes wide is detected, and when the user U closes the eyes small, the image scale changing unit 15 is displayed on the display unit 16. When the user U opens his eyes wide, the image scale changing unit 15 may enlarge the image displayed on the display unit 16. Further, the intention of the user U may be understood by analyzing the brain waves of the user U, and the image scale changing unit 15 may perform enlargement / reduction processing on the image displayed on the display unit 16 accordingly.

The process execution unit 22 executes a predetermined process based on the object selected by the selection unit 18 and held in the object holding unit 20 without being deleted by the cancel processing unit 21. In the present embodiment, a similar image search is illustrated as an example of a predetermined process executed by the process execution unit 22.

The similar image search is a process of searching for an image similar to the image to be a query from the images stored in the image database when an image to be a query is specified. In the present embodiment, the processing execution unit 22 generates a search query including an image of an object selected by the selection unit 18 and held in the object holding unit 20 without being deleted by the cancel processing unit 21. Then, the processing execution unit 22 searches for an image showing an object similar to the object selected by the selection unit 18 from, for example, an external image database of the information processing device 100 according to the generated search query.

Object similarity is determined based on the features of the objects extracted from the image. That is, the feature amount is calculated for each of the object included in the search query and the object included in each image stored in the image database, and the object included in the search query is calculated from the images stored in the image database. An image showing an object whose distance from the feature amount of is less than or equal to a predetermined value is acquired as a search result.

Note that the search query can include multiple objects. That is, when there are a plurality of objects selected by the selection unit 18 and held in the object holding unit 20 without being deleted by the cancel processing unit 21, the processing execution unit 22 AND-joins or combines the images of these plurality of objects. A search query combined with OR is generated, and the image database is searched according to this search query. When a search query in which images of a plurality of objects are AND-combined is executed, an image showing all objects similar to each object is acquired as a search result. On the other hand, when a search query in which images of a plurality of objects are OR-combined is executed, all the images showing an object similar to any of the plurality of objects are acquired as search results.

Such a similar image search is extremely effective, for example, when the information processing apparatus 100 according to the present embodiment is used for work support purposes. That is, when the user U wears the information processing device 100 on his head and performs a predetermined work, it may be necessary to confirm whether or not the performed work has been performed accurately. In such a case, select an object that reflects the work result, search for a similar image from the image database that stores the past work result as an image according to the search query that includes the image of this object, and present it to user U. By doing so, the user U can easily confirm whether or not the performed work has been performed accurately.

Here, a similar image search is illustrated as an example of a predetermined process executed by the process execution unit 22, but the process executed by the process execution unit 22 is not limited to the similar image search. In addition to the similar image search, the process execution unit 22 may record, for example, information on the object selected by the selection unit 18, or acquire detailed information on the object selected by the selection unit 18 from an external database. Various processes such as a process of recognizing characters included in the object selected by the selection unit 18 may be executed.

Next, the main operation of the information processing apparatus 100 according to the present embodiment configured as described above will be briefly described with reference to the flowchart of FIG. FIG. 11 is a flowchart showing an outline of the operation of the information processing apparatus 100.

When the operation of the information processing device 100 is started, the detection unit 12 first detects the designated area R in the field of view of the user U (step S101). Then, the calculation unit 17 calculates the degree of superimposition with the designated area R for the object overlapping the designated area R among the plurality of objects existing in the field of view of the user U (step S102).

Here, if there is no object whose degree of superimposition with the designated area R is equal to or greater than the threshold value among the plurality of objects existing in the field of view of the user U (step S103: No), the process ends as it is. On the other hand, if there is an object whose degree of superimposition with the designated area R is equal to or higher than the threshold value (step S103: Yes), and if there is one object whose degree of superimposition with the designated area R is equal to or higher than the threshold value (step S104: Yes). The selection unit 18 selects the object as the target object (step S105). On the other hand, when there are a plurality of objects whose degree of superimposition with the designated area R is equal to or greater than the threshold value (step S104: No), the selection unit 18 selects the object having the largest size among these plurality of objects as the target object. (Step S106).

After that, the process execution unit 22 executes a predetermined process based on the object selected in step S105 or step S106 (step S107), and a series of processes is completed.

As described in detail above with reference to specific examples, in the information processing apparatus 100 according to the present embodiment, the detection unit 12 is reflected in the image captured by the imaging unit 11 (““ Detects the designated area R specified by (an example of the operating body). Then, the calculation unit 17 calculates the degree of superimposition with the designated area R for the object overlapping the designated area R detected by the detection unit 12 among the plurality of objects existing in the field of view of the user U. Then, the selection unit 18 selects a target object from a plurality of objects existing in the field of view of the user U based on the superposition degree calculated by the calculation unit 17 and the size of the object. There is. Therefore, according to the information processing apparatus 100 according to the present embodiment, even when the objects in the object are included in the plurality of objects existing in the field of view of the user U, the object intended by the user U is correctly selected. Can be done.

In the present embodiment, the hand of the user U reflected in the image in the real space captured by the camera 104 is assumed as the operating body that moves in the field of view of the user U according to the operation of the user U. Instead of the hand, the foot of the user U may be used as the operating body. In this case, the designated area R may be detected by detecting the area of the foot of the user U from the image in the real space captured by the camera 104.

Further, the operating body that moves in the field of view of the user U according to the movement of the user U may be, for example, an object (tool) that the user U moves by holding it by hand or attaching it to a hand or foot. In this case, the designated area R is detected by detecting the area of the object (tool) as a marker provided on the object (tool) used as the operating body from the image in the real space captured by the camera 104. do it.

Further, in the present embodiment, the information processing device 100 is configured as a so-called non-transmissive HMD in which the image of the real space captured by the camera 104 is displayed on the display 106, but the actual image in front of the user U is realized. If the camera 104 that captures the space is provided, the information processing device 100 is configured as a so-called transmissive HMD in which the optical image of the real space in front of the user U is transmitted through the display 106 and visually recognized by the user U. May be good.

Further, the present embodiment is not limited to the HDM worn by the user U on the head and used, and can also be applied to an information terminal such as a smartphone that the user U can use while placing it in front of the eyes with one hand.

<Second Embodiment>
Next, the second embodiment will be described. The information processing device according to the present embodiment is configured as an HMD using VR technology. In the following, the information processing device according to the present embodiment will be referred to as an information processing device 100'to distinguish it from the information processing device 100 according to the first embodiment described above.

The information processing device 100'according to the present embodiment can be used by the user U by being worn on the head in the same manner as the information processing device 100 according to the first embodiment. However, unlike the information processing device 100 according to the first embodiment, the information processing device 100'according to the present embodiment presents a virtual space that is completely CG to the user U. Therefore, the plurality of objects existing in the user U's field of view, that is, the plurality of objects that are candidates for object selection are all CG parts, and the plurality of objects composed of these CG parts include the objects in the object. There is.

Further, in the present embodiment, as an operating body that moves in the field of view of the user U according to the movement of the user U, a CG part of the hand that moves on the image of the virtual space according to the movement of the hand of the user U is assumed. That is, a target object is selected from a plurality of objects included in the field of view of the user U based on the degree of superimposition with the designated area specified by the CG part of the hand and the size of the object.

The hardware configuration of the information processing device 100'according to the present embodiment is substantially the same as the hardware configuration of the information processing apparatus 100 according to the first embodiment illustrated in FIG. However, since the information processing device 100'according to the present embodiment presents the virtual space to be a complete CG to the user U, the camera 104 may not be provided. Instead, various sensing devices for detecting the movement of the user U's hand are provided. Examples of the sensing device for detecting the movement of the user U's hand include a TOF (Time-of-Flight) camera that acquires a distance image in front of the user U, and a data glove worn by the user U in the hand. , An electromyographic sensor that estimates movement by measuring the myoelectric potential of the user U, and the like. The camera 104 that captures the real space in front of the user U may be used as a sensing device for detecting the movement of the user U's hand.

FIG. 12 is a block diagram showing a functional configuration example of the information processing apparatus 100'according to the present embodiment. Compared with the functional configuration example of the information processing apparatus 100 according to the first embodiment shown in FIG. 4, in this embodiment, motion detection is performed instead of the image capturing unit 11, the image holding unit 13, and the image interpolation unit 14. A unit 23 and a virtual space generation unit 24 are provided. The motion detection unit 23 is realized by using the various sensing devices described above. Further, the virtual space generation unit 24 is realized, for example, by the processor 101 shown in FIG. 3 using the memory 102 to execute a predetermined program stored in the storage device 103 or the like. Further, the virtual space generation unit 24 may be realized by using dedicated hardware such as ASIC or FPGA.

The motion detection unit 23 detects the motion of the hand of the user U who wears the information processing device 100'according to the present embodiment. That is, the motion detection unit 23 detects the position and shape of the user U's hand, which changes in time series, by using the sensor data obtained from the above-mentioned sensing device. For example, when a TOF camera is used as a sensing device, a distance image (an image that holds distance information to an object in front for each pixel) is acquired by the TOF camera, so that the user U's hand is included in this distance image. The position and shape of the user U's hand can be detected by regarding the area corresponding to the distance of the possible range as the hand area.

In addition, the data glove and the myoelectric sensor can detect the shape of the user U's hand. By using these data gloves and myoelectric sensors in combination with, for example, a TOF camera, it is possible to more accurately detect the position and shape of the user U's hand, which changes over time. The above-mentioned camera 104 may be used instead of the TOF camera.

The virtual space generation unit 24 generates a virtual space to be presented to the user U. As described above, a plurality of objects composed of CG parts exist in the virtual space generated by the virtual space generation unit 24, and the objects in the object are included in the plurality of objects composed of these CG parts. .. Further, this virtual space is generated so that the CG part imitating the hand of the user U moves according to the movement of the hand of the user U detected by the motion detection unit 23. In the present embodiment, the image of the virtual space generated by the virtual space generation unit 24 is displayed on the display unit 16.

Further, in the present embodiment, the CG part of the user U's hand that moves in the virtual space generated by the virtual space generation unit 24 is used as an operating body for designating the designated area R. Therefore, the detection unit 12 is a virtual space generation unit 24 generated by the virtual space generation unit 24 based on the movement of the user U's hand detected by the motion detection unit 23 (the position and shape of the user U's hand that changes in time series). The position and shape of the CG part of the user U's hand moving in the space are determined, and the designated area R designated by the CG part of the user U's hand is detected in the virtual space.

Further, in the present embodiment, the calculation unit 17 creates an object (CG part) in the virtual space that overlaps with the designated area R detected by the detection unit 12 on the image of the virtual space generated by the virtual space generation unit 24. The degree of superimposition with the designated area R is calculated for the target. Other processing is the same as that of the first embodiment described above.

As described above, in the information processing device 100'according to the present embodiment, the detection unit 12 is designated by the CG parts (an example of the "operation body") of the hand that moves in the virtual space according to the movement of the hand of the user U. The designated area R to be processed is detected. Then, the calculation unit 17 targets the object (CG part) in the virtual space that overlaps with the designated area R detected by the detection unit 12 on the image of the virtual space generated by the virtual space generation unit 24, and sets the designated area. The degree of superimposition with R is calculated. Then, the selection unit 18 selects a target object from a plurality of objects existing in the field of view of the user U based on the superposition degree calculated by the calculation unit 17 and the size of the object. There is. Therefore, in the information processing device 100'according to the present embodiment, as in the first embodiment described above, even if the plurality of objects existing in the field of view of the user U include the in-object object, the user The effect of being able to correctly select the object intended by U can be obtained.

In the present embodiment, a CG part that imitates the hand of the user U is assumed as an operating body that moves in the field of view of the user U according to the movement of the user U, but instead of the CG part that imitates the hand of the user U. In addition, a CG part imitating the foot of the user U may be used as an operating body. In this case, instead of detecting the movement of the user U's hand, the movement of the user U's foot may be detected, and the designated area R designated by the CG part of the user U's foot may be detected in the virtual space.

Further, the operating body that moves in the field of view of the user U according to the movement of the user U imitates the hand or foot of the user U, for example, a robot arm that moves in the virtual space according to the movement of the hand or foot of the user U. It may be a CG part other than the CG part.

Further, the operation of the user U who moves the CG part serving as the operating body in the virtual space is not limited to moving the hands and feet, and may be, for example, the operation of the controller connected to the information processing device 100'. ..

Further, the present embodiment is not limited to the HDM worn by the user U on the head and used, and can also be applied to an information terminal such as a smartphone that the user U can use while placing it in front of the eyes with one hand.

<Supplementary explanation>
Each function of the information processing apparatus 100 (100') according to the above-described embodiment is, for example, a collaboration between hardware including the computer system illustrated in FIG. 3 and a program (software) executed on the computer system. Can be realized by. A program executed on a computer system is provided as a computer program product, for example, in an installable or executable format file, recorded on a computer-readable recording medium such as an SD card. Further, the program executed on the computer system may be stored on a computer connected to a network such as the Internet and provided by downloading the program via the network. In addition, a program executed on a computer system may be configured to be provided or distributed via a network such as the Internet. Further, the program executed on the computer may be configured to be provided by incorporating it into a program ROM or the like in advance.

The program executed on the computer system described above has, for example, a module configuration including functional components such as a detection unit 12, a calculation unit 17, and a selection unit 18. For example, the processor 101 is used from the recording medium. By reading and executing the program, each part described above is loaded on the memory 102 and generated on the memory 102.

Although the embodiments of the present invention have been described above, the embodiments described here are presented as examples and are not intended to limit the scope of the invention. The novel embodiment described here can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. The embodiments and modifications thereof described here are included in the scope and gist of the invention, and are also included in the scope of the invention described in the claims and the equivalent scope thereof.

11 Imaging unit 12 Detection unit 13 Image holding unit 14 Image interpolation unit 15 Image scale changing unit 16 Display unit 17 Calculation unit 18 Selection unit 19 Feedback unit 20 Object holding unit 21 Cancel processing unit 22 Processing execution unit 23 Motion detection unit 24 Virtual space Generator

Claims (20)

A detection unit that detects a designated area designated by an operating body that moves within the user's field of view according to the user's movement,
Of the plurality of objects existing in the user's field of view, including one or more objects included in the other objects, the object overlapping the designated area in the user's field of view. A calculation unit that calculates the degree of superimposition with the designated area,
A selection unit that selects a target object from the plurality of objects based on the degree of superimposition and the size of the object.
Information processing device equipped with. The information processing device according to claim 1, wherein the selection unit selects an object having the largest size among objects whose superposition degree exceeds a predetermined threshold value. The information processing apparatus according to claim 1, wherein the selection unit selects an object closest to the user when there are a plurality of objects having the same size and the degree of superposition exceeds a predetermined threshold value. The information processing apparatus according to any one of claims 1 to 3, wherein the range of the designated area is determined according to the shape or movement of the operating body. The information processing device according to any one of claims 1 to 4, wherein the calculation unit starts calculating the degree of superimposition in response to a predetermined action by the user. The information processing device according to claim 5, wherein the predetermined action includes at least one of a gesture, a speech of a voice command, a movement of the line of sight, and a change in facial expression. The information processing apparatus according to any one of claims 1 to 6, further comprising a cancel processing unit that cancels an object selected by the selection unit in response to a predetermined action by the user. The information processing device according to claim 7, wherein the predetermined action includes at least one of an operation of repeatedly selecting the same object, a gesture, and an utterance of a voice command. The information processing apparatus according to any one of claims 1 to 8, further comprising a feedback processing unit that performs feedback processing for causing the user to recognize that an object has been selected by the selection unit. The feedback process includes at least one of a process of visually recognizing the object selected by the selection unit and a process of causing the user to recognize that the object has been selected by the selection unit by vibration. The information processing apparatus according to claim 9. An imaging unit that captures the real space in front of the user,
A display unit that displays a real space image captured by the image pickup unit is further provided.
The operating body is the user's hand or foot or tool reflected in the image in the real space.
Each of the plurality of objects is an object reflected in the real space image or a CG (computer graphics) part superimposed on the real space image.
The detection unit detects the designated area by detecting the user's hand or foot or tool reflected in the image in the real space.
The information processing device according to any one of claims 1 to 10, wherein the calculation unit calculates the degree of superimposition of an object that overlaps with the designated area on the image in the real space. An image holding unit that holds an image of the real space previously captured by the imaging unit,
An image interpolation unit that interpolates the background of the region in which the operating body is reflected in the real space image newly captured by the image capturing unit using the past real space image held by the image holding unit. And, with more
The information processing apparatus according to claim 11, wherein the selection unit calculates the degree of superimposition using an image in the real space in which the background of the region in which the operating body is reflected is interpolated by the image interpolation unit. The 12th aspect of the present invention, wherein the display unit visualizes and displays the contour of the user's hand or foot in an image of the real space in which the background of the region in which the operation body is reflected is interpolated by the image interpolation unit. Information processing equipment. Further provided with a display unit for displaying an image of the virtual space including the plurality of objects, each of which is composed of CG parts.
The operating body is a CG part of a hand or foot that moves on an image of the virtual space according to the movement of the user's hand or foot.
The detection unit detects the designated area by detecting the movement of the user's hand or foot.
The information processing device according to any one of claims 1 to 10, wherein the calculation unit calculates the degree of superimposition of an object that overlaps with the designated area on an image of the virtual space. The invention according to any one of claims 11 to 14, further comprising an image scale changing unit for enlarging or reducing an image excluding the operating body among the images displayed on the display unit in response to a predetermined action by the user. Information processing equipment. The information processing device according to claim 15, wherein the predetermined action includes at least one of a posture change, an operation of an operation unit, a gesture, a voice command utterance, and a facial expression change. The information processing apparatus according to any one of claims 1 to 16, further comprising a process execution unit that executes a predetermined process based on an object selected by the selection unit. The predetermined process is the process of searching an image in which an object similar to the object selected by the selection unit is displayed from an image database according to a search query including an object selected by the selection unit. Information processing device. An information processing method executed in an information processing device.
A step of detecting a designated area designated by an operating body that moves within the user's field of view according to the user's movement, and
Of the plurality of objects existing in the user's field of view, including one or more objects included in the other objects, the object overlapping the designated area in the user's field of view. The step of calculating the degree of superimposition with the designated area and
A step of selecting a target object from the plurality of objects based on the degree of superposition and the size of the object, and
Information processing methods including. On the computer
A function to detect a designated area specified by an operating body that moves within the user's field of view according to the user's movement, and
Of the plurality of objects existing in the user's field of view, including one or more objects included in the other objects, the object overlapping the designated area in the user's field of view. A function to calculate the degree of superimposition with the designated area and
A function to select a target object from the plurality of objects based on the degree of superimposition and the size of the object, and
A program to realize.

Images