JP6632298B2 - Information processing apparatus, information processing method and program - Google Patents

Description

  The present invention relates to an information processing device, an information processing method, and a program.

  2. Description of the Related Art Conventionally, a pointing device (mouse or touch pad) connected to a PC that controls a projector, a laser pointer, or the like is generally used for pointing on a projection surface on which an image is projected by a projector. In recent years, a method has been proposed in which pointing is performed by detecting a finger pointing to a display image of an operator from an image captured by a camera, and estimating a position on the projection surface pointed by the operator.

  In Patent Literature 1, when a plurality of people point at the display screen at the same time, the face and hand of the operator are detected from a captured image captured by a camera installed on a display that displays the screen. . Then, a technology is disclosed in which a face icon and a hand icon are displayed on a display screen, and the relevance between the operator and the pointer is presented by connecting each of the icons with a string. This makes it possible to associate each of the plurality of users with the pointing.

Japanese Patent No. 5343773

  However, in the related art, when there are users all around the projector, icons of a plurality of users may overlap and become indistinguishable. In this case, there is a problem that it is difficult to understand the correspondence between the user and the pointing projected by the projector.

  The present invention has been made in view of such a problem, and when a plurality of icons corresponding to respective instruction operations of a plurality of users are displayed on a projection plane, the user performing the instruction operation is distinguished. The purpose of the present invention is to present the correspondence between a user and an icon in an easily understandable manner.

Therefore, the present invention provides an information processing apparatus, comprising: a projection unit configured to project display data onto a projection surface; a photographing unit configured to photograph an image of the entire circumference of the projection unit in a horizontal direction; A user position specifying unit that specifies a user position based on the position of the projecting unit; and a pointing position on a projection surface, which is a target of a pointing operation performed by a user using an operating element, based on the image. Pointed position specifying means to pass through the position of the projection means and the user position, and corresponding to the projection plane, the positional relationship between the position of the projection means and the user position in a predetermined plane, It said position indicated by the user, based on the inclination determination unit operable shows the indicated position, and to determine the slope of the first icon indicating the relationship between the indication position and the user position, the projection Wherein the position of the unit based on the positional relationship between the user position, and further comprising a display control means for controlling said projecting means to project the first icon slope determined by the slope determining means I do.

  According to the present invention, when a plurality of icons corresponding to respective instruction operations of a plurality of users are displayed on the projection surface, the user performing the instruction operation is distinguished, and the correspondence between the user and the icon is determined. It can be presented clearly.

FIG. 1 is a diagram illustrating a projector system according to a first embodiment and a usage form thereof. FIG. 2 is a diagram illustrating an example of a hardware configuration of a projector. FIG. 2 is a diagram illustrating an example of a functional configuration of a projector. It is a flowchart which shows information processing. It is explanatory drawing of the process which detects a pointing shape. FIG. 9 is an explanatory diagram of a process for determining an operation element position. FIG. 9 is an explanatory diagram of a process of displaying a pointer icon. It is a figure showing the example of a change of an icon. FIG. 14 is an explanatory diagram of a mapping process according to the second embodiment. FIG. 11 is an explanatory diagram of a projector according to a third embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating a projector system according to a first embodiment and a usage form thereof. In the present embodiment, the projector 100 will be described as an example in which a plurality of users hold a conference or a presentation while viewing a display image projected by the projector 100. The projector system according to the present embodiment includes a projector 100 as an information processing device and a server device 160 arranged on a network. The projector 100 and the server device 160 are connected by a high-speed wireless line 161. The projector 100 projects information by acquiring various data (contents) stored in the server device 160 via a network. Of course, the data originally held by the projector 100 can be projected.

  In the present embodiment, the projector 100 is installed on a table 120 and projects display data 150 on a projection surface 140. Further, users 130a to 130c are present so as to surround the projector 100. In the following, when it is not necessary to particularly distinguish the users 130a to 130c, the users 130a to 130c are simply referred to as users 130. The users 130a to 130c hold a conference or a presentation while viewing the display data 150 projected by the projector 100. At this time, various operations on the display data 150 projected on the projection surface 140 are performed by gesture operations of the users 130a to 130c. Projector 100 detects a gesture operation based on an image captured by omnidirectional stereo camera 110 mounted on projector 100.

  Then, when the gesture operation is an instruction operation indicating a certain position on display data 150, that is, a position on projection plane 140, projector 100 displays a pointer icon. Here, the pointer icon is an icon indicating a point pointed by the pointing operation, and an icon indicating which of the users 130a to 130c the pointing icon corresponds to the pointing operation. In the example of FIG. 1, two pointer icons 141a and 141b are displayed on the projection plane 140 so as to be superimposed on the display data 150. In the present embodiment, the pointer icons 141a and 141b are images showing arrows. The tip of the arrow indicates the position specified by the pointing operation, and the inclination of the arrow indicates which of the users 130a to 130c the pointer icon corresponds to the pointing operation of the user. Therefore, the inclination of the pointer icon changes according to the positional relationship between the projector 100 and the users 130a to 130c. Details will be described later. Further, in the present embodiment, a case will be described in which the instruction operation is performed with the fingertip of the user, but the operator for performing the instruction operation is not limited to the fingertip. The operator may be, for example, a hand, an arm, a stick, or the like.

  The omnidirectional stereo camera 110 is mounted on the projector 100 of the present embodiment. The omnidirectional stereo camera 110 has two cameras 111 and 112. Each of the cameras 111 and 112 is an omnidirectional camera that can capture the entire circumference (360 °) in the horizontal direction, and can capture an image of a semi-celestial sphere (up and down and left and right omnidirectional directions). The two cameras 111 and 112 are mounted vertically as shown in the figure, and can function as a stereo camera by using images in all left and right directions (all around the y axis).

  In the present embodiment, the projector 100 is equipped with an omnidirectional stereo camera 110 as an imaging device so that the pointing operation of all the users 130a to 130c around the projector 100 can be detected. Furthermore, in order to be able to calculate the distance from the projector 100 to each object, the omnidirectional stereo camera 110 has a stereo configuration, and calculates the distance by the stereo method. That is, as shown in FIG. 1, the omnidirectional stereo camera 110 has two cameras 111 and 112. The cameras 111 and 112 are mounted vertically, and each of the cameras 111 and 112 captures an image in all left and right directions (all around the y axis). In the present embodiment, it is assumed that the cycle at which the imaging control unit 302 captures an image matches the frame rate of the image captured by the omnidirectional stereo camera 110. However, a predetermined time that does not depend on the frame rate can be set as the imaging cycle.

  FIG. 2 is a diagram illustrating an example of a hardware configuration of the projector 100. The CPU 200 controls each device connected via the bus 203 as a whole. An operating system (OS), various processing programs relating to functions and processes of the projector 100 described later, device drivers, and the like are stored in the ROM 202, temporarily stored in the RAM 201, and executed by the CPU 200 as appropriate. The RAM 201 is used as a temporary storage area such as a main memory and a work area of the CPU 200 which can be accessed at high speed. Here, the OS and each processing program may be stored in the storage 209. In this case, necessary information is read into the RAM 201 as needed when the power is turned on. The display I / F 204 converts display data generated by the projector 100 into a signal that can be output by the projection unit 205. The input I / F 206 converts an image captured by the omnidirectional stereo camera 110 into a format that the projector 100 can process.

  The storage I / F 208 inputs and outputs data to and from the storage 209. The storage 209 stores an image serving as a component of a UI (User Interface) of display data projected by the projector 100, a copy of data acquired from the server device 160, a program that the projector 100 starts internally, and the like. As the storage 209, a large-capacity storage device such as a disk device, a flash memory, a network drive, or a USB drive can be used. Further, the storage device may be configured by connecting a plurality of storage devices in parallel, or may be a storage included in the server device 160. The network I / F 210 remotely communicates with the server device 160 on the network. In the present embodiment, it is assumed that a high-speed wireless network is used. However, a configuration using a wired network may be used.

  Although not shown, the server device 160 is also configured with hardware such as a CPU 200, a RAM 201, a ROM 202, and a bus 203, like the projector 100. Further, it is configured by a storage I / F 208, a storage 209, and a network I / F 210. The function of each hardware is the same as that of projector 100. It is assumed that the storage device 209 of the server device 160 stores a large amount of data such as presentation materials, documents, images, and moving images, and the projector 100 can acquire the data via the high-speed wireless line 161.

  FIG. 3 is a diagram illustrating an example of a functional configuration of the projector 100. The projector 100 includes a communication control unit 300, a storage unit 301, a shooting control unit 302, an area detection unit 303, and a shape detection unit 304. Further, the projector 100 includes a three-dimensional position calculation unit 305, a reference point identification unit 306, a pointing detection unit 307, and a display control unit 308. Each of these functional units is realized by the CPU 200 developing a program stored in the ROM 202 in the RAM 201 and executing processing according to each flowchart described later. When hardware is configured as a substitute for software processing using the CPU 200, an arithmetic unit or a circuit corresponding to the processing of each functional unit described here may be configured.

  The communication control unit 300 controls input and output of data to and from the storage 209 via the storage I / F 208. Further, the communication control unit 300 performs data communication with the server device 160 on the network via the network I / F 210. The storage unit 301 is realized by the storage 209. As described above, the storage unit 301 stores various UI components, copies of data acquired from the server device 160, and programs executed by the projector 100.

  The photographing control unit 302 controls photographing by the omnidirectional stereo camera 110. The photographing control unit 302 acquires an RGB stereo image of the entire periphery captured by the omnidirectional stereo camera 110 at predetermined time intervals, and stores the acquired RGB stereo images in the RAM 201 as needed. The RGB stereo image is composed of two RGB images captured from two viewpoints. There is a gap (parallax) between these two images, and a distance (depth) can be calculated based on the parallax. The stored RGB stereo images are sequentially processed in each functional block of the projector 100. In the present embodiment, it is assumed that the cycle at which the imaging control unit 302 captures an image matches the frame rate of the image captured by the projector 100. However, a predetermined time that does not depend on the frame rate can be set as the imaging cycle. The image capturing control unit 302 processes an image captured by the camera 111 as a master image and an image captured by the camera 112 as a slave image, but is not limited thereto.

  The area detection unit 303 detects a skin color area (including the face and hands of the user 130) and other areas in the image by using the color information of the RGB stereo image stored in the RAM 201. Further, the area detection unit 303 labels the detected skin color area, and stores the result in the RAM 201. Further, the area detection unit 303 detects a face area from the skin color area stored in the RAM 201, labels the area, and stores the label in the RAM 201. Details will be described later. The shape detection unit 304 uses the information on the skin color area and the face area held in the RAM 201 to detect the pointing shape as an instruction operation from the skin color area other than the face area. Details will be described later. Further, the shape detecting unit 304 labels the detected pointing shape area, and stores the result in the RAM 201. In the present embodiment, the shape detected as the pointing operation is the pointing shape of the hand of the person, but is not limited thereto.

  The three-dimensional position calculation unit 305 calculates a three-dimensional position from the projector 100 to each pixel of each region using the information on the face region and the pointing shape region held in the RAM 201. Although a specific calculation method will be described later, a corresponding point is searched from two images (a master image and a slave image) constituting a stereo image, and a spatial position is calculated from the parallax using triangulation. Here, the corresponding point is a point in an image taken from each viewpoint when one point in space is photographed simultaneously from two viewpoints, and is generally referred to as a “stereo corresponding point”. It is called point. For example, when the point D shown in FIG. 1 is taken as one point in the space and the point D is imaged simultaneously from two viewpoints of the cameras 111 and 112, the point D in the image captured by the camera 111 and the point D Is the corresponding point. The three-dimensional position calculation unit 305 stores the calculated three-dimensional position information of each pixel in the RAM 201.

  In the present embodiment, the omnidirectional camera (omnidirectional stereo camera 110) is used as the means for measuring the three-dimensional position. However, the omnidirectional camera is not necessarily a color image, and the omnidirectional stereo image is acquired using the infrared stereo camera. May be. Alternatively, the three-dimensional position information may be obtained by using a range image sensor capable of capturing an omnidirectional image instead of the omnidirectional stereo camera. Further, three-dimensional position information may be calculated by imaging the entire periphery with a plurality of cameras instead of one camera.

  The reference point specifying unit 306 specifies the position of the face area as a user position by using the three-dimensional position information of the face area stored in the RAM 201, and stores the specified user position in the RAM 201. The reference point specifying unit 306 also specifies the position of the pointing shape area as a control element position using the three-dimensional position information of the pointing shape area stored in the RAM 201, and stores the specified control element position in the RAM 201. I do. Note that both the user position and the operator position are positions in the three-dimensional coordinate system. The pointing detection unit 307 detects a pointing operation on the projection surface 140 based on the user position and the operation element position held in the RAM 201. Details will be described later. The display control unit 308 generates display data from the data stored in the storage unit 301 and the data acquired from the server device 160 on the network, and controls the projection unit 205 to project the display data. The display control unit 308 further controls to display a pointer icon at a position on the projection plane 140 corresponding to the position pointed by the user when the pointing operation of the user is detected by the pointing detection unit 307. I do.

  FIG. 4 is a flowchart illustrating information processing by the projector 100. The projector 100 projects the display data 150 on the projection surface 140 by this processing, and when detecting a user's instruction operation, projects and displays a pointer icon corresponding to the instruction operation on the display data 150 in a superimposed manner. When the power of the projector 100 is turned on and started, the projector 100 first executes the process of step S400. In step S400, the display control unit 308 controls the projection unit 205 to generate display data and project the display data on the projection plane 140. Note that the display data projected here is a presentation material or the like stored in the storage 209 or the server device 160 and used in a conference. The projection unit 205 projects and displays the display data on the projection plane under the control of the display control unit 308 (projection processing).

  Next, in step S401, the imaging control unit 302 controls the omnidirectional stereo camera 110 to capture an RGB omnidirectional stereo image at a predetermined frame rate (imaging control processing). The imaging control unit 302 stores the obtained omnidirectional stereo image in the RAM 201. The omnidirectional stereo camera 110 captures an omnidirectional stereo image under the control of the capturing control unit 302 (photographing processing). Next, in step S402, the area detection unit 303 analyzes the omnidirectional stereo image held in the RAM 201, and detects whether or not there is a skin color area in the master image and the slave image. If the area detecting unit 303 has not detected a skin color area (No in step S402), the processing proceeds to step S411. When detecting the skin color region (Yes in step S402), the region detection unit 303 advances the process to step S403.

  In step S403, the area detecting unit 303 labels the detected skin color area, and stores the result in the RAM 201. Further, in step S403, the area detection unit 303 detects a face area from the skin color area held in the RAM 201. Various algorithms are known for face detection. In the present embodiment, a face is regarded as a set of local light-dark differences, and matching between light-dark in a skin color region and a plurality of simple light-dark patterns is performed. Thus, the face area is detected. Then, the area detection unit 303 labels the detected face area, and stores the result in the RAM 201.

  Next, in step S404, the shape detection unit 304 uses the label image of the skin color area and the label image of the face area held in the RAM 201 to detect the pointing shape as a pointing operation from the skin color area other than the face area. To detect. FIG. 5 is an explanatory diagram of the processing for detecting the pointing shape. While detecting the convex shape 500 shown in FIG. 5 as the pointing shape, the shape detecting unit 304 determines that the convex shape 501 is not the pointing shape, and does not detect it as the pointing shape. More specifically, the shape detection unit 304 pays attention to the difference between the ratio of the flesh color region in the region of the convex shape 500 and the ratio of the flesh color region in the region of the convex shape 501. 500 are detected.

  Note that the processing for detecting the pointing shape is not limited to the embodiment. As another example, the projector 100 stores in advance the image of the pointing shape to be detected in the ROM 202 or the like as a template, and the shape detecting unit 304 compares (matches) the captured image with the captured omnidirectional stereo image. The pointing shape may be detected. As another example, the shape detection unit 304 can detect a pointing shape by extracting a predetermined feature amount from the shape of the skin color region and performing machine learning. Then, the shape detection unit 304 labels the detected pointing area, and stores the result in the RAM 201. Next, in step S405, the shape detection unit 304 determines whether or not the pointing shape exists. When the pointing shape does not exist (No in Step S405), the shape detecting unit 304 proceeds with the process to Step S411. If the pointing shape exists (Yes in Step S405), the shape detecting unit 304 proceeds with the process to Step S406.

  In step S <b> 406, the three-dimensional position calculation unit 305 calculates a three-dimensional position for each pixel of the face area and the pointing area in the master image from the omnidirectional stereo image stored in the RAM 201. For example, when calculating the three-dimensional position of the pixel (m, n) in the pointing shape area in the master image, the three-dimensional position calculation unit 305 first determines the stereo correspondence point (m, n) corresponding to the pixel position (m, n). m1, n1) is searched from the slave image. Then, the three-dimensional position calculation unit 305 determines the pixel position (m, n) by triangulation from the parallax between the pixel position (m, n) in the master image and the stereo corresponding point (m1, n1) in the slave image. ) Is obtained (depth, depth), and a position in a three-dimensional space is calculated. The three-dimensional position calculation unit 305 calculates the above-described three-dimensional position for all the pixels in the face area and the pointing area in the master image, and stores the result in the RAM 201.

  Next, in step S <b> 407, the reference point specifying unit 306 specifies the user position and the operation device position using the three-dimensional position information of the face area and the pointing shape area held in the RAM 201. Hereinafter, the processing for specifying the position of the operator will be described first. The projector 100 according to the present embodiment determines the position of the fingertip in the pointing area as the operator position. FIG. 6 is an explanatory diagram of the processing for determining the operation device position. The reference point specifying unit 306 determines the position of the fingertip using the outline of the pointing shape area. Specifically, as shown in FIG. 6, the reference point specifying unit 306 first generates a finger-like point whose angle Θ formed by three equally-spaced points (a, b, c) on the contour is within a predetermined threshold value. Ask for multiple items. Then, the reference point specifying unit 306 detects a portion where points that are likely to be fingertips are continuous. The reference point specifying unit 306 further holds the average position of the plurality of consecutive points that are likely to be the fingertips in the RAM 201 as the fingertip position 600, that is, the operator position.

  Next, a process for specifying a user position will be described. The projector 100 according to the present embodiment specifies the position of the face jaw as the user position. The reference point specifying unit 306 obtains the position of the eyes and the position of the mouth from the local brightness difference of the face, and lowers the perpendicular from the line connecting both eyes to the direction of the mouth, as in the case of the face detection described above. The position where the line intersects with the outline of the region is specified as the position of the jaw. Then, the reference point specifying unit 306 holds the position of the jaw in the RAM 201 as the user position. The reason why the position of the jaw is set to the user position is a result in consideration of the actual pointing and pointing situation and usability. In the present embodiment, the reference point of the face area is the position of the jaw. However, the position of the center of gravity, the position of the nose, and the middle point between both eyes of the face area may be the user position. Note that the processing in step S407 is an example of the operator position specifying processing and the user position specifying processing.

  After the process of step S407, in step S408, the pointing detection unit 307 determines whether or not the distance between the manipulator position and the user position (hereinafter, referred to as a user distance) specified in step S407 is within the distance range. to decide. When the user distance is within the distance range (Yes in step S408), the pointing detection unit 307 determines that a pointing operation as an instruction operation has been detected, and advances the process to step S409. If the user distance is out of the distance range (No in step S408), the pointing detection unit 307 determines that the instruction operation has not been detected, and advances the process to step S411. In step S409, the pointing detection unit 307 determines that the pointing of the user 130 has been detected, and a detection event indicating that the pointing of the user 130 has been detected (including the three-dimensional position information of the two reference points). To the display control unit 308.

  In the projector 100 according to the present embodiment, the distance range is set to 20 to 70 cm. The reason why the upper limit of the distance range is set to 70 cm is a result in consideration of a range that can be reached by humans. Also, the reason why the lower limit of the distance range is set to 20 cm is that when the position of the jaw and the position of the fingertip are too close, the user does not try to perform a pointing operation (pointing), but accidentally points near the face. This is because there is a high possibility that it is merely a shape. In other words, this is the result of considering erroneous recognition of the pointing operation. Of course, the distance range may be set to a value other than “20 cm to 70 cm”.

  Next, in step S410, the display control unit 308 is notified of the pointing operation detection event from the pointing detection unit 307. Then, the display control unit 308 performs control such that the pointer icon indicating the positional relationship between the projector 100 and the user 130 is combined with the display data 150 being displayed on the projection plane 140 and projected and displayed (display control processing). Hereinafter, the process of displaying the pointer icon will be described with reference to FIGS. 1 and 7. As a premise, as shown in FIG. 1, two users 130a and 130b perform a pointing operation on the display data 150 displayed on the projection surface 140, and the pointing detection unit 307 detects the finger in step S409 in FIG. It is assumed that the insertion operation has been detected.

  In this case, based on the three-dimensional position information of the face area and the pointing area held in the RAM 201, the display control unit 308 determines the XZ plane (the plane perpendicular to the vertical direction) as shown in FIG. Create a map on a plane parallel to (). That is, the map corresponds to a diagram (top view) in which the positional relationship between the projector 100 and the users 130a and 130b is viewed from above like a map. That is, on the map, when the minus direction of the Z-axis is the upward direction, the user 130a is mapped to the lower side and the user 130b is mapped to the right side with respect to the projector 100. Note that a triangular area 701 indicates the range of the projection area, and a circular area 702 indicates an area where the omnidirectional stereo camera 110 can calculate three-dimensional position information. Reference numeral 703 denotes a center point when the projector 100 is viewed from above, and reference numerals 704a and 704b denote user positions (jaw positions) of the users 130a and 130b specified by the reference point specifying unit 306, respectively.

  Further, the display control unit 308 obtains straight lines 705a and 705b connecting the center point 703 of the projector 100 on the map and the user positions 704a and 704b. Then, the display control unit 308 associates the center point 703 of the projector 100 on the map with the corresponding center point 713 of the projection plane 140, as shown in FIG. Here, the corresponding center point 713 is a predetermined position, and in the present embodiment, is the center position of the projection plane 140. However, the position of the corresponding center point 713 may be a position near the center of the predetermined projection plane 140, and is not limited to the center position. Further, the display control unit 308 draws the two straight lines 715a and 715b by associating each of the two straight lines 705a and 705b with the corresponding center point 713 of the projection plane 140, and further draws each straight line 715a and 715b of the display data 150. Stretch to the border. Then, the display control unit 308 obtains intersections 720a and 720b between the straight lines 715a and 715b and the boundary of the display data 150. The display control unit 308 determines the determined intersections 720a and 720b as the entry positions of the pointer icons 141a and 141b to be assigned to the users 130a and 130b, respectively (entry position determination processing).

  Further, the display control unit 308 determines the position on the display data 150 (the position on the projection plane 140) pointed to by the user 130a based on the straight line connecting the user position and the operation device position for the user 130a. It is specified as the designated position of 130a. This process is an example of a designated position specifying process for specifying a designated position on the projection plane. Specifically, the display control unit 308 specifies a position 721a at which a straight line connecting the user position and the operation device position and the projection plane 140 intersect as the designated position. Similarly, the display control unit 308 specifies the designated position 721b of the user 130b based on a straight line connecting the user position and the operation device position with respect to the user 130b.

  The display control unit 308 controls to project and display an arrow-shaped pointer icon 141a having the designated position 721a as a tip position along a straight line connecting the entry position 720a and the designated position 721a. In addition, the display control unit 308 controls to project and display an arrow-shaped pointer icon 141b having the designated position 721b as a tip position along a straight line connecting the entry position 720b and the designated position 721b.

  Here, each of the pointer icons 141a and 141b is an example of an icon indicating the relationship between the user position and the designated position. The straight line connecting the entry position 720a and the designated position 721a is a straight line that determines the inclination of the pointer icon 141a, and the process of obtaining this straight line is an example of the inclination determination process that determines the inclination of the pointer icon. When the user position or the operation element position moves after the display of the pointer, the display control unit 308 appropriately updates the display of the pointer according to the moved position.

  Referring back to FIG. 4, in step S411, CPU 200 determines whether or not to terminate the operation of projector 100. CPU 200 determines that the operation of projector 100 is to be terminated when an instruction to terminate projector 100 is given by an input to a physical button or GUI provided on projector 100. When ending the operation (Yes in step S411), the CPU 200 ends the process. When not ending the operation (No in step S411), the CPU 200 advances the processing to S400 and continues the processing.

  As described above, the projector 100 according to the present embodiment obtains the pointer entry position from the positional relationship between the projection unit 205 and the user 130, and displays the pointer whose inclination changes according to the pointer entry position. In this manner, by synthesizing and displaying the user's positional relationship with the projector 100 on the projection surface 140 in accordance with the display form of the pointer, the user can recognize the approximate relative position of the pointing user with respect to the projector 100. it can.

  As represented by a map, people are accustomed to mapping from the top, that is, mapping to the horizontal plane, so once you understand the mapping rules, you can intuitively understand the positional relationship just by looking at the displayed pointer icon It is possible to grasp. The display area required for that is only the area for displaying the pointer icon, so that it is reasonable compared to the case of displaying the conventional pointer icon.

  Further, in a use case where a conference is performed using display data projected on the projection surface as described in the present embodiment, a user is often present around the projector 100. Therefore, as shown in the present embodiment, by integrating the omnidirectional stereo camera 110 into the projector 100, the user position can be easily mapped over the entire ambience.

  As a first modification of the present embodiment, the icons are not limited to arrow-type pointer icons. FIG. 8 is a diagram illustrating an example of changing icons. The display control unit 308 may display transparent shadow icons 800a and 800b imitating a user's arm, as shown in FIG.

  As a second modification, the display control unit 308 may display the pointer icon as an animation. For example, in the example shown in FIG. 7B, an animation may be displayed such that the pointer icons 141a and 141b move from the entry positions 720a and 720b to the designated positions 721a and 721b, respectively. This allows the user to clearly recognize the entry positions 720a and 720b of the pointer icons 141a and 141b. Furthermore, after the pointer icons 141a and 141b reach the designated positions 721a and 721b, respectively, the display control unit 308 hides the pointer icons 141a and 141b after a lapse of a predetermined time after the pointing operation is not detected. Such control may be performed.

  As a third modification, in the present embodiment, the user's face and pointing are detected from the omnidirectional stereo image, but the present invention is not limited to this. As another example, the projector 100 may detect a user's face or pointing using position information that can be obtained from a device that can obtain position information.

  As a third modification, the projector 100 may determine the length of the arrow as the pointer icon based on the distance between the designated position and the user position (length determination processing).

(Second embodiment)
Next, a projector 100 according to a second embodiment will be described. In step S410 of FIG. 4, the projector 100 according to the second embodiment maps the positional relationship between the projector 100 and the user 130 on a plane parallel to the XY plane, and displays a pointer icon based on the mapping result. . Hereinafter, differences between the projector 100 according to the second embodiment and the projector 100 according to the first embodiment will be described. FIG. 9 is an explanatory diagram of the mapping processing according to the second embodiment. In FIG. 9A, reference numeral 902 denotes an operator position, and 903 denotes a user position. Reference numeral 900 denotes a position where a straight line connecting the operator position 902 and the user position 903 intersects the projection plane 140, that is, a designated position.

  In the present embodiment, the display control unit 308 obtains the operation position 900 based on the operation element position 902 and the user position 903. Further, the display control unit 308 obtains x1 and y1, which are x and y components of a line connecting the operation position 900 and the operation element position 902. Here, x1 is a length obtained by projecting the line segment in the x direction as shown in FIG. 9B. Further, y1 is the length of the projection of the line segment in the y direction as shown in FIG. 9C. Further, the display control unit 308 adjusts the depth of the pointer (the perspective of the arrow) in accordance with the length of the perpendicular from the operator position 902 to the projection plane 140, that is, the size of the component z1 in the z direction. Other configurations and processes of the projector 100 according to the second embodiment are the same as those of the projector 100 according to the first embodiment.

  As described above, the projector 100 according to the second embodiment projects the pointer (icon) by mapping the positional relationship of the user with respect to the projector 100 from the first-person viewpoint, like the first-person viewpoint display of the car navigation system. Since the mapping of the first person viewpoint is left as it is when compared with the mapping to the position viewed from the top (top view), the user can intuitively recognize the approximate position of each user.

(Third embodiment)
FIG. 10 is an explanatory diagram of a projector 1010 according to the third embodiment. The projector 1010 according to the third embodiment is a flashlight type mobile device. That is, projector 1010 is movable. For this reason, the position of the projector 1010 is not constant, and the projection direction changes according to the position of the projector 1010. The projector 1010 displays a pointer according to the changing position of the projector 1010. Hereinafter, differences between the projector 1010 according to the third embodiment and the projector 1010 according to the other embodiments will be described. In the present embodiment, the projector 100 will be described as an example in a case where a curator of an art museum has the projector 1010 and gives a description of the artwork 1080 to the visitor while projecting and displaying display data by the projector 1010. I do. Reference numeral 1040 shown in FIG. 10 indicates a projection plane. The projection surface 1040 is provided behind the artwork 1080. Further, as another example, projection surface 1040 may be provided in front of art object 1080. In this case, the projection surface 1040 is a transparent surface.

  As shown in FIG. 10A, the omnidirectional stereo camera 1010 according to the third embodiment has a range image sensor 1013 in addition to the two cameras 1011 and 1012. The projector 1010 is a mobile device carried by a museum curator 1030a as a user, and changes a projection target and a projection surface. Further, the positional relationship between the projector 1010 and the visitors 1030a, 1030b, and 1030c as the user frequently changes. For this reason, the projector 1010 captures not only the omnidirectional stereo image but also the distance image at a predetermined frame rate using the detection result of the distance image sensor 1013. The projector 1010 processes the distance image captured by the area detection unit 303 and the shape detection unit 304 to identify what the projection target is. That is, it identifies that the projection target object is a specific artwork. Using this identification information, the projector 1010 acquires information on the identified artwork 1080 from the server device 160 on the network, and superimposes and displays the information on the artwork 1080 itself. The projector 1010 can, for example, superimpose and display additional information such as the name of the artwork and the creator.

  FIG. 10A shows that when the curator 1030a is explaining the artwork 1080 to the visitors 1030b and 1030c using the projector 1010, two of the visitors 1030b and 1030c are a part of the artwork 1080. Shows a scene where pointing is performed. As in the case of the first embodiment, the pointing operation of the visitors 1030b and 1030c is detected by the projector 1010, and as a result, pointer icons 1041b and 1041c corresponding to the visitors 1030b and 1030c are displayed on the projection surface 1040. Is displayed. Also, 1042b and 1042c in FIG. 10A indicate the entry positions of the pointer icons 1041b and 1041c, respectively.

  Here, it is assumed that the curator 1030a attempts to explain the position pointed by the visitor 1030b, that is, the position of the tip of the pointer icon 1041b, and rotates the projector 1010 in the direction of arrow A shown in FIG. In this case, as shown in FIG. 10B, the projection plane changes from 1040 to another plane 1050. At this time, it is assumed that the position to which the visitor 1030b or 1030c points is not changed, but the position between the projector 1010 and the visitor 1030b or 1030c due to the rotational movement of the projector 1010. Relationships change. Accordingly, the pointer icon 1041b corresponding to the visitor 1030b changes to the pointer icon 1051b shown in FIG. Here, 1052b is the entry position of the pointer icon 1051b. That is, the position of the artwork 1080 to which the pointer corresponding to the user 130b is pointed does not change, but the entry position and the inclination of the pointer icon change.

  On the other hand, due to the rotational movement of the projector 1010, the positional relationship between the projector 1010 and the visitor 1030c changes, and the pointer icon 1041c corresponding to the visitor 1030c becomes the pointer icon 1051c shown in FIG. Changes to However, the position of the pointer icon 1051c is outside the range of the projection plane 1050, that is, outside the projection range. As described above, when the position of the pointer icon is out of the range of the projection plane 1050, the projector 1010 ends the display of the pointer icon, and replaces the pointer icon with the arrowhead related icon 1060 of the projection plane 1050. Display at the boundary position. The related icon 1060 indicates that there is a position to which the visitor 1030c is pointing in that direction. Furthermore, when the pointer icon 1051c is again within the range of the projection plane 1050, the projector 1010 ends the display of the related icon 1060 and displays the pointer icon 1051c instead of the related icon 1060.

  By performing such display control by the projector 1010, the curator 1030a can explain the points pointed by the visitors 1030b and 1030c (points where he / she wants to hear the explanation) one by one in order. Can be. Further, the projector 1010 of the present embodiment records the designated position in a storage unit such as the RAM 201 (recording process). Then, when the pointing operation is continued for a certain period of time or longer, the projector 1010 may control to continue displaying the corresponding pointer icon or the related icon even after the pointing operation is completed. This eliminates the need for the visitor to continue the pointing operation until the turn of the explanation comes, and the convenience of the projector 1010 can be improved.

  As described above, even in an embodiment such as a flashlight-type mobile device, the approximate position of the user operating the pointer icon can be reported. That is, the user's positional relationship with the projector 1010 is fed back to the user using the entry position of the pointer icon and the inclination of the pointer, so that the user can recognize the approximate position of the pointing user.

  As described above, according to each of the above-described embodiments, even when a plurality of icons corresponding to a plurality of user's instruction operations are displayed on the projection surface, the correspondence between the user performing the instruction operation and the icons Can be presented in an easy-to-understand manner.

  As described above, the present invention has been described in detail based on the preferred embodiments. However, the present invention is not limited to these specific embodiments, and various forms without departing from the gist of the present invention are also included in the present invention. included. A part of the above-described embodiments may be appropriately combined.

(Other Examples)
The present invention supplies a program for realizing one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium, and one or more processors in a computer of the system or the apparatus read and execute the program. This processing can be realized. Further, it can also be realized by a circuit (for example, an ASIC) that realizes one or more functions.

Reference Signs List 100 Projector 110 Surrounding stereo camera 140 Projection surface 150 Display data

Claims (11)

Projecting means for projecting the display data on a projection surface;
Photographing means for photographing an image of the entire circumference in the horizontal direction of the projecting means,
Based on the image, based on the position of the projecting means, a user position specifying means for specifying a user position,
Based on the image, a designated position specifying unit that specifies a designated position on a projection plane that is a target of a designated operation performed using a manipulator by a user;
Passing the position of the projection means and the user position, and corresponding to the projection plane, the positional relationship between the position of the projection means and the user position on a predetermined plane, and the designated position by the user A tilt determining unit that determines the tilt of a first icon indicating the designated position, and indicating a relationship between the user position and the designated position, based on
Display control means for controlling the projection means to project the first icon whose inclination has been determined by the inclination determination means based on a positional relationship between the position of the projection means and the user position. Characteristic information processing device. The user position specifying means specifies the position of the user's face as the user position,
Based on the image, based on the position of the projection means, further comprising an operator position specifying means for specifying an operator position,
The specified position specifying means specifies a specified position on the projection plane based on the operator position and the position of the face,
The display control means controls the projection so as to project the first icon indicating the designated position specified by the designated position specifying means and indicating a relationship between the user position and the designated position. The information processing device according to claim 1. The plane is an information processing apparatus according to claim 1 or 2, characterized in that with respect to the vertical direction which is perpendicular to the plane. An entry position determination unit that determines an entry position of the first icon on the projection plane based on the positional relationship,
It said tilt deciding means, on the basis of the entry position and said instructed position, the information processing apparatus according to any one of claims 1 to 3, characterized by determining the slope of the first icon. The information processing apparatus according to claim 4 , wherein the display control unit controls to project an animation of the first icon moving from the approach position toward the designated position. 6. The information processing apparatus according to claim 5 , wherein the display control unit controls to hide the first icon after a lapse of a predetermined time after the first icon reaches the designated position. 7. apparatus. A length determining unit that determines a length of the first icon based on a distance between the pointing position and the operating element position;
The information processing apparatus according to claim 2, wherein the display control unit controls to project the first icon whose length is determined by the length determination unit. The projection means is movable;
The display control unit controls the projection unit to project a second icon indicating the direction of the designated position when the designated position is outside the projection plane due to movement of the projection unit. The information processing apparatus according to claim 2, wherein Recording means for recording the designated position in storage means,
The information processing apparatus according to claim 8 , wherein the display control unit controls to project the second icon based on the designated position recorded in the storage unit. An information processing method executed by an information processing apparatus,
A photographing step of photographing an image of the entire circumference in the horizontal direction of the projection means for projecting the display data on the projection surface;
Based on the image, based on the position of the projection means, a user position specifying step of specifying a user position,
Based on the image, a specified position specifying step of specifying a specified position on a projection plane to be a target of a specified operation performed by a user using an operator;
The positional relationship between the position of the projection unit and the user position on a predetermined plane, passing through the position of the projection unit and the user position, and corresponding to the projection plane, and the designated position by the user. A tilt determining step of determining the tilt of a first icon indicating the designated position and indicating a relationship between the user position and the designated position, based on
A display control step of controlling the projection means to project the first icon whose inclination has been determined in the inclination determination step based on a positional relationship between the position of the projection means and the user position. Characteristic information processing method. Computer
Shooting control means for controlling to shoot an image of the entire circumference in the horizontal direction of the projection means for projecting the display data on the projection surface,
Based on the image, based on the position of the projecting means, a user position specifying means for specifying a user position,
Based on the image, a designated position specifying unit that specifies a designated position on a projection plane that is a target of a designated operation performed using a manipulator by a user;
The positional relationship between the position of the projection unit and the user position on a predetermined plane, passing through the position of the projection unit and the user position, and corresponding to the projection plane, and the designated position by the user. , Based on the inclination position determining means for determining the inclination of a first icon indicating the designated position and indicating the relationship between the user position and the designated position,
A function of a display control unit that controls the projection unit to project the first icon whose inclination is determined by the inclination determination unit based on a positional relationship between the position of the projection unit and the user position. program.

Images