JP6435964B2 - Display device, program, and display method - Google Patents

Description

  The present invention relates to a display device, a program, and a display method.

  A technique is known in which a pointing operation is performed on a projection image by superimposing a pointer that moves in accordance with the movement of the projection device on the projection image displayed on the background (projection plane) (see, for example, Patent Document 1).

JP 2005-266471 A

  However, if the area onto which the projected image is projected does not fit on the flat projection plane, a part of the object (for example, an icon for operating the application) that is the object of the pointing operation is displayed outside the plane. There is. In such a case, it is difficult for the user to grasp the positional relationship between the objects, and it may be difficult to perform a pointing operation.

  On the other hand, it is troublesome for the user to manually adjust the display of each object to a position where the pointing operation can be easily performed.

  Therefore, in one aspect, an object of the present invention is to improve a pointing operation of an object displayed by projection.

  In one proposal, a display unit that projects and displays a pointer that operates a plurality of applications on a display object, a calculation unit that calculates a first distance to the display object on which the pointer is projected, Based on a first distance, in the display object, an extraction unit for extracting a region for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer, and the plurality of objects in the region A determination unit that determines a position not to be superimposed based on a relative position of each of the plurality of objects and the pointer, and the display unit displays the pointer and the plurality of objects at the determined position. A display device for projecting and displaying is provided.

  According to one aspect, it is possible to improve a pointing operation of an object displayed by projection.

The figure which shows the example of whole structure of the display system which concerns on 1st Embodiment. The figure which shows the example in case a projection image does not fit on one plane. The figure which shows the hardware structural example of the display system which concerns on 1st Embodiment. The figure which shows the function structural example of the display system which concerns on 1st Embodiment. 6 is a flowchart illustrating an example of a pointing operation start process according to the first embodiment. The figure which shows an example for selecting the plane which performs pointing operation. The figure which shows an example of the extracted area | region. The figure which shows an example of the size and position of an object. The figure which shows an example when a projection image is projected. 6 is a flowchart of an example of region extraction processing according to the first embodiment. The figure which shows an example of the bounding box surrounding a pointer. The flowchart of an example of the determination process of the expansion / contraction magnification which concerns on 1st Embodiment. The figure of an example explaining the area of the area | region in a captured image, and the area of a pointer. 6 is a flowchart of an example of position determination processing according to the first embodiment. The figure of an example explaining the repulsive force which an object receives from another charged particle. 6 is a flowchart of an example of a pointing operation process according to the first embodiment. The figure of an example explaining the variation | change_quantity of the position of the object by an attitude | position change. The figure which shows the function structural example of the display system which concerns on 2nd Embodiment. The flowchart of an example of the position determination process which concerns on 2nd Embodiment. The figure of an example explaining determination of the position of an object. The figure which shows the function structural example of the display system which concerns on 3rd Embodiment. The flowchart of an example of the determination process of the expansion / contraction magnification which concerns on 3rd Embodiment. The figure of an example explaining the distance for the determination of expansion / contraction magnification.

  Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings.

<First Embodiment>
[System overall configuration]
First, the configuration of the display system 1 according to the first embodiment will be described with reference to FIG. FIG. 1 is a diagram illustrating an example of the overall configuration of a display system according to the first embodiment. The display system 1 according to the present embodiment includes a pointing terminal 10 and a mobile terminal 20. The pointing terminal 10 and the portable terminal 20 are connected to be communicable by wire or wireless.

  The pointing terminal 10 is a wearable terminal as shown in FIG. 1, projects image data (projected image data) received from the mobile terminal 20, and displays the projected image S on the projection plane. In addition, as will be described later, the pointing terminal 10 can perform a pointing operation on each of the objects A to C displayed on the projected image S by the pointer P using posture change amount data obtained from the motion sensor. it can.

  Here, the object is, for example, an icon for operating an application installed in the mobile terminal 20. Therefore, the user can use an application corresponding to the operated object by performing a pointing operation on each of the objects A to C in the projection image S using the pointing terminal 10. However, the object is not limited to an icon or the like for operating the application. For example, each key of the software keyboard may be displayed as an object so that characters can be input.

  The pointing operation is an operation performed by the user, such as pointing the objects A to C with the pointer P in the projection image S or selecting and executing the application.

  The mobile terminal 20 is an information processing device that can be carried by the user, and stores projection source image data that is the original data of the projection image data projected by the various applications and the pointing terminal 10 described above. Further, as will be described later, the mobile terminal 20 determines the position and size (enlargement / reduction ratio) of each of the objects A to C in the projection image according to a user operation, and generates projection image data from the projection source image data. To do. More specifically, the mobile terminal 20 determines the position of each object, the enlargement / reduction ratio, and the like based on the display object projected by the pointing terminal 10 so that the user can easily perform the pointing operation according to the user's operation. Projection image data is generated from the projection source image data.

  1 shows an example in which the pointing terminal 10 and the mobile terminal 20 are different terminals, the present invention is not limited to this, and the pointing terminal 10 and the mobile terminal 20 may be realized by a single terminal. That is, the mobile terminal 20 may be included in the pointing terminal 10.

  Here, the case where it is necessary to determine the position and size (enlargement / reduction ratio) of each object described above so that the user can easily perform the pointing operation will be described with reference to FIG. FIG. 2 is a diagram illustrating an example when the projected image does not fit on one plane. In FIG. 2, among the objects A to D that are targets of the pointing operation by the pointer P in the projection image S, the objects A and B are projected on one plane L. On the other hand, the object C is projected onto a plane having a different angle from the plane L, and the object D partially projects from the plane L and is projected to the back of the plane L. In such a case, since it is difficult for the user to grasp the positional relationship between the object C and the object D, the user may not be able to accurately select the object C or the object D by operating the pointer P.

  Therefore, in the display system 1 according to the present embodiment, an area suitable for the pointing operation is extracted from the plane L on which the pointer P is projected, and the position and size (enlargement / reduction ratio) of the pointer P and the objects A to D are adjusted. Above, it projects so that it may be contained in the extracted area | region. Thereby, in the display system 1 which concerns on this embodiment, a user's pointing operation can be improved.

[Hardware configuration]
Next, the hardware configuration of the display system 1 according to the present embodiment will be described with reference to FIG. FIG. 3 is a diagram illustrating a hardware configuration example of the display system according to the first embodiment.

  The pointing terminal 10 includes a projection device 101, an imaging device 102, a motion sensor 103, an input device 104, an I / F device 105, a CPU (Central Processing Unit) 106, a RAM (Random Access Memory) 107, and a ROM (Read Only Memory). Etc. Each of these hardware is mutually connected by bus B1.

  The projection device 101 is a projector or the like, projects the projection image data received from the portable terminal 20 via the I / F device 105, and displays the projection image. The imaging device 102 is a camera (ranging camera) or the like that can measure a distance, images a predetermined range, and generates captured image data including the distance from the imaging device 102 to the imaging target. The captured image data generated by the imaging device 102 is transmitted to the mobile terminal 20 via the I / F device 105. Here, the captured image data is used when determining the size (enlargement / reduction ratio) and the like of each object in the mobile terminal 20, as will be described later. The motion sensor 103 measures the posture change amount of the pointing terminal 10 and outputs posture change amount data. Note that the distance measured by the imaging device 102 is hereinafter referred to as “first distance” for convenience.

  The input device 104 is, for example, various buttons and receives various input operations by the user. The I / F device 105 is an interface with the mobile terminal 20 and transmits and receives various data. The CPU 106 is an arithmetic device that reads programs and various data from the ROM 107 and the like onto the RAM 108 and executes various processes. The RAM 107 is a volatile semiconductor memory that temporarily stores programs and data. The ROM 108 is a nonvolatile semiconductor memory that can retain data even when the power is turned off.

  The portable terminal 20 includes a CPU 201, a RAM 202, a ROM 203, and an auxiliary storage device 204. The mobile terminal 20 includes an input device 205, an output device 206, an I / F device 207, a drive device 208, a communication device 209, and the like. Each of these hardware is mutually connected by bus B2.

  The CPU 201 is an arithmetic device that reads programs and various data from the auxiliary storage device 204 and the ROM 203 onto the RAM 202 and executes various processes. The RAM 202 is a volatile semiconductor memory that temporarily stores programs and data. The ROM 203 is a nonvolatile semiconductor memory that can retain data even when the power is turned off. The auxiliary storage device 204 is a non-volatile memory that stores programs and various data, and is, for example, an SSD (Solid State Drive). The stored programs and data include an OS (Operating System) that is basic software for controlling the entire mobile terminal 20 and various programs that operate on the OS.

  The input device 205 is a touch panel, for example, and accepts various input operations by the user. The output device 206 is a display, for example, and displays various processing results. The I / F device 207 is an interface with the pointing terminal 10 and transmits / receives various data. The drive device 208 is an interface with the recording medium 208a. Examples of the recording medium 208a include an SD memory card and a USB memory (Universal Serial Bus memory). The communication device 209 is an interface for connecting to a network.

  The display system 1 according to the present embodiment implements various processes described below with the hardware configuration illustrated in FIG. 3.

[Function configuration]
Next, the functional configuration of the display system 1 according to the present embodiment will be described with reference to FIG. FIG. 4 is a diagram illustrating a functional configuration example of the display system according to the first embodiment.

  The pointing terminal 10 includes an input unit 11, a projection unit 12, an imaging unit 13, and a change amount acquisition unit 14. Each of these units is realized by processing executed by the CPU 106 by a program installed in the pointing terminal 10.

  The input unit 11 receives an input operation for starting a pointing operation from the user. At this time, when receiving an input operation for starting a pointing operation from the user, the input unit 11 causes the projection unit 12 to project projection image data for projecting a pointer via the projection image generation unit 26.

  The projection unit 12 projects the projection image data received from the mobile terminal 20 and displays a projection image based on the projection image data.

  The imaging unit 13 captures a predetermined range including the pointer displayed by the projection unit 12 and generates captured image data. Here, the captured image data generated by the imaging unit 13 includes a first distance from the imaging device 102 to the imaging target. That is, the imaging unit 13 calculates a first distance from the imaging device 102 to the imaging target, and generates captured image data including the calculated first distance. The predetermined range may be a range including at least the projection image displayed by the projection unit 12.

  The change amount acquisition unit 14 acquires posture change amount data output from the motion sensor 103 in accordance with the posture change of the pointing terminal 10. Here, the posture change amount data is represented by an angle change amount before and after the posture change of the pointing terminal 10.

  The mobile terminal 20 includes a pointer specifying unit 21, a region extracting unit 22, an object determining unit 23, a scaling factor determining unit 24, a position determining unit 25, a projection image generating unit 26, and a change amount calculating unit 27. Each of these units is realized by processing executed by the CPU 201 by a program installed in the mobile terminal 20. The mobile terminal 20 includes an image data storage unit 28 and an object management data storage unit 29. Each of these storage units can be realized using the auxiliary storage device 204.

  The pointer specifying unit 21 specifies a pointer included in the captured image data generated by the imaging unit 13.

  The area extraction unit 22 extracts an area to be displayed by projecting the pointer and each object based on the captured image data generated by the imaging unit 13.

  The object determining unit 23 refers to the object management data storage unit 29 and determines an object to be projected and displayed on the area extracted by the area extracting unit 22.

  The enlargement / reduction ratio determination unit 24 determines the size (enlargement / reduction ratio) of each object and pointer to be projected and displayed on the region extracted by the region extraction unit 22.

  The position determination unit 25 determines the position of the pointer and each object to be projected and displayed on the region extracted by the region extraction unit 22. More specifically, the position determination unit 25 determines the position of the pointer and each object to be projected and displayed on the area based on the distance between the objects, the distance between the object and the pointer, and the distance between the object and the boundary of the area. To do. In order to distinguish the distance between the objects, the distance between the pointer and the object, and the distance between the object and the boundary of the region from the first distance described above, for the sake of convenience, the “second distance” is hereinafter referred to. Represent.

  The projection image generation unit 26 projects a pointer to be extracted and displayed on the extracted area based on the enlargement / reduction ratio determined by the enlargement / reduction ratio determination unit 24 and the position determined by the position determination unit 25, and the projection source of each object. Projection image data is generated from the image data. In addition, the projection image generation unit 26 generates projection image data based on the change amount of the position of the object calculated by the change amount calculation unit 27.

  The change amount calculation unit 27 calculates the amount by which the position of the object changes (position change amount) in accordance with the posture change of the pointing terminal 10 based on the posture change amount data acquired by the change amount acquisition unit 14. Based on the calculated position change amount, the projection image generation unit 26 generates projection image data in which the position change amount is canceled, so that the background (display object) is changed before and after the posture change of the pointing terminal 10. The object can be stationary.

  The image data storage unit 28 stores projection source image data. Here, the projection source image data stored in the image data storage unit 28 is, for example, image data (pointer image data) for displaying a projected image of a pointer or image data (for displaying a projected image of an object). Object image data). The projection image generation unit 26 described above acquires one or more projection source image data from the image data storage unit 28, and generates projection image data from the acquired projection source image data. The pointer image data and the object image data are associated with a position (initial position) in the projection image when displayed as a projection image. Here, in this embodiment, it is assumed that the initial position of the pointer image data is the center of the projection image.

  The object management data storage unit 29 stores object management data such as an operation history for the object. The object determination unit 23 described above acquires object management data from the object management data storage unit 29, and determines an object to be projected and displayed on the area extracted by the area extraction unit 22 based on the acquired object management data.

[Pointing operation start processing]
Next, the pointing operation start process of the display system 1 according to the present embodiment will be described with reference to FIG. FIG. 5 is a flowchart illustrating an example of a pointing operation start process according to the first embodiment. In the following, a process from when a user selects a display target for projecting an object or the like to perform a pointing operation until the pointer and each object are projected and displayed on the display target will be described.

First, the user performs an operation for starting a pointing operation via the input unit 11. Then, the projection image generation unit 26 acquires pointer image data from the image data storage unit 28, generates projection image data for displaying the pointer at the center of the projection image, and transmits the projection image data to the projection unit 12. Projection unit 12 displays by projecting the projected image S ₁ on the basis of the projection image data received from the projection image generation unit 26.

Then, as shown in FIG. 6, the user projects and displays the pointer P ₁ included in the projection image S ₁ on the display target object (here, the plane L) desired to perform the pointing operation (Step S ₁ ). S501).

Next, the imaging unit 13, while the pointer P ₁ is in a predetermined time, triggered by being displayed on the same location, or in response to an input operation via the input unit 11 by the user, the projected image S A predetermined range including ₁ is imaged to generate captured image data (step S502). At this time, the captured image data generated by the imaging unit 13 is associated with, for example, a first distance from the imaging device 102 to the imaging target for each pixel.

  Based on the captured image data, the pointer specifying unit 21 and the region extracting unit 22 extract a region for displaying the pointer and each object to be a pointing operation from the captured image data (step S503). That is, the pointer specifying unit 21 and the region extracting unit 22 extract a region R included in the plane L from the captured image W, for example, as illustrated in FIG. Here, the region R extracted by the region extraction unit 22 is, for example, a region where the first distance from the pointing terminal 10 to the display object is smoothly changed. Such region extraction processing for extracting the region R will be described later with reference to FIG.

  The object determining unit 23 refers to the object management data storage unit 29 and determines an object to be projected and displayed on the area extracted by the area extracting unit 22 (step S504). More specifically, the object determination unit 23 acquires object management data indicating an operation history and the like for the object from the object management data storage unit 29. Then, the object determination unit 23 determines an object to be projected and displayed on the region R from the object management data.

For example, when the objects A to D are projected and displayed by the pointing terminal 10 in the previous pointing operation, the object determination unit 23 determines to display the objects A to D in the region R. Further, for example, when the object A corresponding to the application A installed in the mobile terminal 20 is changed to the object A ′ by the previous user operation, the object determination unit 23 displays the object A ′ in the region R. And decide. Hereinafter, it is assumed that the object determination unit 23 determines to display the objects A _{1 to} D ₁ in the region R.

Next, the enlargement / reduction ratio determination unit 24 determines the size (enlargement / reduction ratio) of the pointer P ₁ and each of the objects A _{1 to} D ₁ that are projected and displayed on the region R extracted by the region extraction unit 22 (step S505). ). The position determination unit 25 determines the display position of the pointer P ₁ and each object A ₁ to D ₁ in the region R to be displayed by projecting a region R that has been extracted by the area extracting unit 22 (step S506) . That is, the enlargement / reduction ratio determination unit 24 and the position determination unit 25 make the position and size of the pointer P ₁ and the objects A _{1 to} D ₁ within the region R easy to perform a pointing operation as shown in FIG. To be determined. Such enlargement / reduction magnification determination processing and position determination processing will be described later with reference to FIGS. 12 and 14, respectively.

The position determination unit 25 in step S506, the display position in the region R of the pointer P _1, the pointer P ₁ in step S502 described above may be determined in a position that is displayed by projecting.

Next, the projection image generation unit 26 generates image data based on the enlargement / reduction magnification determined by the enlargement / reduction magnification determination unit 24 and the position of the pointer P ₁ and each of the objects A _{1 to} D ₁ determined by the position determination unit 25. Projection image data is generated from the projection source image data acquired from the storage unit. Then, the projection unit 12 displays a projection image based on the projection image data. Thus, the projected image S ₂ as shown in FIG. 9 is displayed.

In the projection image S ₂ shown in FIG. 9, the pointer P ₂ and the objects A _{2 to} D ₂ are the size and position determined by the enlargement / reduction ratio determination unit 24 for the size and position of the pointer P ₁ and the objects A _{1 to} D ₁ , respectively. This is adjusted at the position determined by the determination unit 25. That is, for example, the projection image generation unit 26 adjusts the pointer image data of the pointer P ₁ with the enlargement / reduction ratio determined by the enlargement / reduction ratio determination unit 24 and displays the projection image data to be displayed at the position determined by the position determination unit 25. Generate. Similarly, the projection image generation unit 26, for example, the object image data of an object A _1, adjusted by scaling factor determined by the scaling factor determining unit 24, projection image data to be displayed on the determined position by the position determination unit 25 Is generated. The same applies to the objects B _{1 to} D ₁ . Projection unit 12, by projecting by superimposing the respective projected image data generated in this manner, it is possible to display the projected image S ₂ as shown in FIG.

As described above, in the display system 1 according to the present embodiment, the region R is extracted from the plane L that the user desires to perform a pointing operation. Then, in the display system 1 according to the present embodiment, the extracted area R, by projecting the pointer _{P 2} and object _A 2 to D ₂ who adjust the size and position of the pointer _{P 1} and the object _A 1 to D ₁ Display. Thereby, a user's pointing operation can be improved.

[Area extraction processing]
Next, the region extraction processing in step S503 in FIG. 5 will be described with reference to FIG. FIG. 10 is a flowchart of an example of region extraction processing according to the first embodiment.

First, the pointer specifying unit 21 specifies a pointer from the captured image and surrounds it with a bounding box (step S1001). That is, the pointer identifying unit 21 acquires from the image data storage unit 28 the pointer image data of the pointer P _1, by using a method such as template matching on the captured image data by using the pointer image data, pointer P ₁ Is identified. Then, the pointer identifying unit 21, as shown in FIG. 11, the pointer _{P 1} identified, boxed (bounding box Q) containing pointers _{P 1.}

Next, the region extraction unit 22 acquires pixel values p ₁ ,..., P _n (n is an integer of 1 or more) in the captured image data corresponding to each side of the bounding box Q, and the acquired pixels A median value M ₁ of the values p ₁ ,..., _Pn is calculated (step S1002).

Next, the area extraction unit 22 has first distances d ₁ ,..., D _m (m is an integer equal to or greater than 1) associated with each pixel in the range surrounded by the bounding box Q in the captured image data. , And the median value M ₂ of the acquired first distances d ₁ ,..., _Dm is calculated (step S1003).

Subsequently, the region extracting unit 22 is a pixel whose difference from M ₁ is less than or equal to a preset first threshold in the captured image data, and which is less than or equal to a preset second threshold that is different from M _2. A pixel associated with the first distance is extracted (step S1004). The first threshold value and the second threshold value are determined in advance at the time of designing a program for realizing the present embodiment. By appropriately setting the first threshold value and the second threshold value, the color of the pixel in the vicinity of the pointer P ₁ does not change significantly, and the vicinity of the pointer P ₁ and the first distance change greatly. Unextracted pixels can be extracted.

  The region extraction unit 22 performs an opening process and a closing process on the pixels extracted in step S1004 (step S1005). By such opening processing and closing processing, it is possible to remove holes and protrusions when the pixels extracted in step S1004 are drawn. Thereby, the area extraction unit 22 can obtain one or more areas drawn by the pixels after the opening process and the closing process in the captured image drawn based on the captured image data.

Finally, the region extraction unit 22 extracts a region including the pointer P1 from _one or more regions obtained in step S1005 (step S1006). The region extracted by the region extraction unit 22 in this way is the region R shown in FIG.

Note that the region extraction unit 22 does not have to perform the process of step S1002. In this case, the region extraction unit 22 may be extracted pixels the difference between the median M ₂ in step S1004 is associated with the first distance equal to or less than the second threshold value. As described above, the region extraction unit 22 may extract the region R based on the first distance. However, in many cases, the region extracting unit 22 can extract a region more suitable for the pointing operation when the region is extracted based on the first distance and the pixel value as described with reference to FIG.

[Determination processing of enlargement / reduction ratio]
Next, the scaling factor determination process in step S505 in FIG. 5 will be described with reference to FIG. FIG. 12 is a flowchart of an example of a scaling factor determination process according to the first embodiment.

First, the scaling factor determining unit 24 calculates the area of the pointer in the captured image and the area of the region extracted by the region extracting unit 22 (step S1201). In other words, scaling factor determining section 24, as shown in FIG. 13, in the captured image W, and calculates an area of a region defined by the bounding box Q as the area Bp of the pointer P _1. The enlargement / reduction ratio determination unit 24 calculates the area Br of the region R extracted by the region extraction unit 22. Here, the area Bp may be calculated as the number of pixels included in the bounding box Q, and the area Br may be calculated as the number of pixels included in the region R.

Next, the scaling factor determination unit 24 calculates the area of the pointer and the area of the object in the projected image (step S1202). In other words, scaling factor determining section 24 obtains the pointer image data of the pointer P ₁ from the image data storage unit 28, the number of pixels of the pointer image data to the area Ap of the pointer P ₁ in the projection image. The enlargement / reduction ratio determination unit 24 acquires the object image data of each of the objects A _{1 to} D ₁ from the image data storage unit 28, and calculates the total number of pixels of each object image data as the area Ao of the objects A to D in the projection image. And

Finally, the enlargement / reduction ratio determination unit 24 calculates the enlargement / reduction ratio s by the following equations (1) to (3) (step S1203). Then, the enlargement / reduction ratio determination unit 24 projects the enlargement / reduction ratio s calculated in this manner onto the area extracted by the area extraction unit 22 and displays the size (enlargement / reduction) of the pointer P ₁ and each of the objects A _{1 to} D _1. Magnification). As described above, the projection image generation unit 26 adjusts the size of the pointer and the object displayed in the projection image by adjusting the projection source image data with the scaling factor s.

First, the enlargement / reduction ratio determination unit 24 is generated by imaging with the imaging unit 13 when the objects A _{1 to} D ₁ are projected and displayed without adjusting with the enlargement / reduction ratio s according to the following equation (1). The total Bo of the areas of the objects A _{1 to} D ₁ in the captured image data is calculated.

  Next, the ratio of the area Br obtained by scaling the sum of the area Bp and the area Bo by the square of the scaling ratio s and the area Br may be a preset area ratio k. The following equation (2) is derived. Note that the value of the area ratio k is determined in advance at the time of designing a program that realizes the present embodiment. By setting the area ratio k to a large value, the ratio of the pointer and the object to the area can be increased. On the other hand, by setting the area ratio k to a small value, the pointer and the object occupy the area. The ratio can be reduced.

  The enlargement / reduction ratio determination unit 24 calculates the enlargement / reduction ratio s by the following expression (3) using the expressions (1) and (2).

[Position determination process]
Next, the position determination process in step S506 in FIG. 5 will be described with reference to FIG. FIG. 14 is a flowchart of an example of position determination processing according to the first embodiment. In the position determination processing shown in FIG. 14, the pointer P ₁ and the objects A _{1 to} D ₁ are regarded as charged particles of the charge q, and the simulation is performed with the repulsive model, thereby performing the pointer P ₁ and the objects A _{1 to} D. The position of ₁ is determined.

  First, the position determination unit 25 arranges charged particles of the charge q at predetermined intervals on the boundary of the region extracted by the region extraction unit 22 (step S1401). That is, as shown in FIG. 15, charged particles of charge q are arranged on the boundary of the region R at predetermined intervals.

  Next, the position determination unit 25 determines whether or not the processing of steps S1403 to S1405 described later has been repeated a predetermined number of times (step S1402). If the position determination unit 25 repeats the processes in steps S1403 to S1405 a predetermined number of times, the position determination unit 25 proceeds to step S1406. On the other hand, if the position determination unit 25 has not repeated the processes of steps S1403 to S1405 a predetermined number of times, the position determination unit 25 proceeds to step S1403. Here, examples of the predetermined number of times include thousands to tens of thousands of times.

In step S1402, when it is determined that not repeated the predetermined number of times, the position determining unit 25, the pointer _{P 1} and the object _A 1 to D _1, and calculates the sum of the repulsive force received from other charged particles (Step S1402) .

For example, as shown in FIG. 15, the repulsive force F ₁ received from the pointer P ₁ , the repulsive forces F ₂ received from the other objects A _{1 to} B ₁ and D _1, and the charged particles at the boundary of the region R, for the object C _1. calculating the sum of the repulsive force F ₃ for receiving respectively from. Each such repulsive force F _n (n is an integer equal to or greater than 1) is expressed by the following equation (4), assuming that the second distance between charged particles is r and the Coulomb constant is k ₁ according to Coulomb's law. expressed. Accordingly, the sum F of the repulsive force F _n can be calculated by the sum of vectors of the repulsive force F _n.

The position determination unit 25 also calculates the sum of repulsive forces received from other charged particles for the pointer P ₁ and the other objects A _{1 to} B ₁ and D _{1 in the same} manner as described above.

Next, the position determination unit 25 calculates the acceleration of the pointer P ₁ and the objects A _{1 to} D _{1 at} the time point t (step S1404). Note that the time point t is a counter variable that is incremented by one when the position determination unit 25 repeats the processing of steps S1403 to S1405. Time t is incremented by one each time steps S1403 to S1405 are executed by the position determination unit 25 with an initial value of 0.

For example, for the object A ₁ , using the repulsive force sum F calculated in step S 1403 (where the repulsive force sum F at time _t is expressed as F _t ), the equation of motion is expressed by the following equation (5). Is done.

In the above formula (5), the mass m is a predetermined value for adjusting the acceleration a _t. The mass m may be the same value or a different value for each of the pointer P ₁ and the objects A _{1 to} D ₁ .

For the pointer P ₁ and the other objects B _{1 to} D ₁ , the position determination unit 25 calculates the acceleration at the time point t using the sum of the repulsive forces calculated in step S1403 as described above.

Subsequently, the position determination unit 25 calculates the position in each region R using the acceleration of the pointer P ₁ and the objects A _{1 to} D ₁ at the time point t calculated in step S1404 (step S1405).

For example, the object _{A 1,} using the acceleration _{a t} at the time t calculated in step S1404, the speed _{v t + 1} and the position _{x t + 1} at time t + 1 are represented by the respective following equations (6) and (7) The Here, Δt is a value set in advance as the time elapsed from time t to time t + 1.

In the equation (6), the speed v ₀ at the time point t = 0 may be, for example, v ₀ = 0. In Expression (7), the position x ₀ at the time point t = 0 may be the initial position of the pointer P ₁ or the objects A _{1 to} D ₁ . In this way, the position determination unit 25 calculates the position _{x t + 1} of the pointer _{P 1} and the object _A 1 to D ₁ from repulsion pointer _{P 1} and the object _A 1 to D ₁ receives at time t at time t + 1.

The position determination unit 25 also calculates the position at the time point t + 1 for the pointer P ₁ and the other objects B _{1 to} D ₁ using the acceleration calculated in step S1404 as described above.

In step S1402, if it is determined that repeated a predetermined number of times, the position determining unit 25, the position of the calculated pointer _{P 1} and the object _A 1 to D ₁ in step S1405, the pointer _{P 2} and object _A 2 to D ₂ (Step S1406).

Incidentally, used in the above step S1405, instead of formula (5), equation (5) preset add the one indicating the resistance using a predetermined constant k ₂ were the following equation with respect to (8) May be. Thereby, the positions of the pointer P ₁ and the objects A _{1 to} D ₁ may be determined at positions where the pointing operation can be performed more easily.

In FIG. 14, the charge of the pointer P ₁ and the objects A _{1 to} D _{1 and} the charge of the charged particles arranged at the boundary of the region R are all set to the same value q. It is good. For example, the charge of the pointer P ₁ may be q ₁ , the objects A _{1 to} D ₁ may be q ₂ , and the charge of charged particles arranged at the boundary of the region R may be q ₃ . Further, the charges of the objects A _{1 to} D ₁ may be different, and the charges of the charged particles arranged at the boundary of the region R may be different. For example, the positions of the pointer P ₁ and the objects A _{1 to} D ₁ can be determined at positions where the pointing operation can be more easily performed by changing the charges of the charged particles arranged at the boundary according to the shape of the region R. There is a case.

Further, in the above description, repeatedly performs the processes of steps S1403~ step S1405 with the pointer _{P 1} and the object _A 1 to D _1, the pointer _{P 1} be configured not to execute the processing of step S1403~ step S1405 Good. That is, the position determining unit 25, the position of the pointer P _1, the pointer P ₁ in step S502 of FIG. 5 may be determined to a position which is displayed by projecting.

[Pointing operation processing]
Next, processing for performing a pointing operation in the display system 1 according to the present embodiment will be described with reference to FIG. FIG. 16 is a flowchart of an example of a pointing operation process according to the first embodiment. As described above, in the display system 1 according to the present embodiment, a pointing operation is performed on an object with a pointer using posture change amount data obtained from the motion sensor 103 mounted on the pointing terminal 10. In the following, a description will be given assuming that the projected image S ₂ shown in FIG. 9 is displayed by pointing device 10.

  First, the change amount acquisition unit 14 determines whether or not the posture of the pointing terminal 10 has changed (step S1601). This may be determined based on whether or not the change amount acquisition unit 14 has acquired posture change amount data from the motion sensor 103. If the change amount acquisition unit 14 does not determine that the posture of the pointing terminal 10 has changed, nothing is done. On the other hand, if the change amount acquisition unit 14 determines that the posture of the pointing terminal 10 has changed, the process advances to step S1602.

In step S1601, when the change amount acquisition unit 14 determines that the posture of the pointing terminal 10 has changed, the change amount acquisition unit 14 transmits the posture change amount data acquired from the motion sensor 103 to the change amount calculation unit 27. . Then, the change amount calculation unit 27 calculates the change amount of the position of the objects A _{2 to} D ₂ accompanying the change in the posture of the pointing terminal 10 based on the posture change amount data (step S1602). For this, a case of calculating a change amount of the position of the object A _2, will be described with reference to FIG. 17. Hereinafter, the object A ₂ after the posture change is referred to as “object A ₂ ′”.

First, as shown in FIG. 17, in the projection image S ₂ before the posture change, as the projection surface and the horizontal x-axis defines the center as the origin. At this time, as shown in FIG. 17, half of the width in the x-axis direction of the projection image S ₂ is x _w (pixel), the position of the object A _{2 in} the x-axis direction is x _a (pixel), and the image of the pointing terminal 10 is displayed. Assume that the angle is θ _w , the angle of the object A _{2 from} the x-axis origin is θa, and the posture change amount data is Δθ. Further, the position in the x-axis direction of the object A ₂ ′ is assumed to be x _b (pixel). At this time, the following formula (9) is calculated from x _w : x _a = tan θ _w : tan θ _a .

Further, the following formula (10) is calculated from x _w : x _b = tan θ _w : tan (θ _a + Δθ).

Therefore, the change amount calculation unit 27 can calculate the change amount x _b −x _a of the position in the x-axis direction of the object A ₂ accompanying the posture change from the equations (9) and (10). Similarly, by performing the same calculation as described above for the y-axis orthogonal to the x-axis in the projection image S ₂ , the change amount calculation unit 27 changes the amount of change in the position of the object A _{2 in} the y-axis direction due to the posture change. y _b −y _a can be calculated. Thereby, the change amount calculation unit 27 can calculate the change amount of the position of the object A ₂ accompanying the posture change in the projection image S ₂ . Note that the amount of change in position is calculated for the other objects B _{2 to} D ₂ as well.

The projection image generation unit 26 generates projection image data based on the position change amount calculated in step S1602 so that the positions of the objects A _{2 to} D ₂ do not change due to the posture change. Then, the projection unit 12 projects the projection image data generated by the projection image generation unit 26 and displays the projection image (step S1603).

For example, the projection image generation unit 26 subtracts the change amount of the positions of the objects A _{2 to} D ₂ calculated above from the positions of the objects A ₂ ′ to D ₂ ′ of the projection image data after the posture change. Generate data. Thus, the projection image after the posture change is projected by the projection unit 12, the position of the pointer P ₂ While that appears to vary in accordance with the attitude change, the position of the object A ₂ to D ₂ is the posture change Regardless, it is displayed unchanged. Therefore, the user can perform a pointing operation on the objects A _{2 to} D ₂ with the pointer P ₂ using the pointing terminal 10.

<Second Embodiment>
Next, a display system 1 according to the second embodiment will be described. In the description of the second embodiment, differences from the first embodiment will be described, and components having the same functional configuration as the first embodiment will be denoted by the same reference numerals. Omit duplicate descriptions.

[Function configuration]
FIG. 18 is a diagram illustrating a functional configuration example of the display system according to the second embodiment. The position determination unit 25A included in the mobile terminal 20 of the display system 1 according to the second embodiment uses a method different from that of the first embodiment to project and display pointers displayed on the area extracted by the area extraction unit 22 and each of the pointers. Determine the position of the object. More specifically, the position determination unit 25A divides the bounding box surrounding the extracted area a predetermined number of times. Then, the predetermined position of the area obtained by the division is determined as the position of the pointer and each object.

[Position determination process]
Next, a second embodiment of the position determination process in step S506 of FIG. 5 will be described with reference to FIG. FIG. 19 is a flowchart of an example of position determination processing according to the second embodiment.

  First, the position determination unit 25A surrounds the region R ′ extracted by the region extraction unit 22 in the captured image with the bounding box Q ′ (step S1901). That is, the position determination unit 25A surrounds the region R ′ with the bounding box Q ′ as shown in FIG.

  Note that the position determining unit 25A may use a directed bounding box when surrounding the region R ′ with the bounding box Q ′. By using the effective bounding box, the position determination unit 25A can obtain a bounding box that is more suitable when surrounding the region R ′ (that is, there are many points circumscribing the region R ′).

  Next, the position determination unit 25A divides the range surrounded by the bounding box Q ′ so that the longitudinal direction of the bounding box Q ′ is equally divided (step S1902). That is, the position determination unit 25A performs the “first division” shown in FIG. Thereby, the range surrounded by the bounding box Q ′ is divided to obtain two quadrangles.

  When the bounding box Q ′ is a square, the position determination unit 25A may divide the predetermined one side equally.

  Subsequently, the position determination unit 25A determines whether or not the number of rectangles obtained by dividing the range surrounded by the bounding box Q ′ is equal to or greater than the number of pointers and objects to be projected and displayed (step S1903). If the number of rectangles obtained by the division is less than the number of pointers and objects to be projected and displayed, the process proceeds to step S1904. On the other hand, if the number of rectangles obtained by the division is equal to or greater than the number of pointers and objects to be projected and displayed, the process proceeds to step S1905.

  In step S1903, when the number of rectangles obtained by dividing is less than the number of pointers and objects to be projected and displayed, the position determination unit 25A further determines the rectangle obtained by dividing the rectangle in the longitudinal direction. Dividing into equal parts (step S1904). That is, the position determination unit 25A performs the “second division” shown in FIG. As described above, the position determination unit 25A repeatedly performs division until the number of quadrangles obtained by the division is equal to or greater than the number of pointers and objects to be projected and displayed.

In step S1903, when the number of rectangles obtained by division is equal to or greater than the number of pointers and objects to be projected and displayed, the position determination unit 25A determines the position of the center of gravity of each rectangle obtained by division. The position of the pointer and the object is determined (step S1905). That is, the position determination section 25A, as shown in FIG. 20 (b), the position of the center of gravity of each rectangle obtained by dividing, it is determined that the pointer _{P 2} and object _A 2 to D _2, respectively position.

Here, either the position of the center of gravity of which square the position of the pointer P _2, also, whether the position of the center of gravity of one square with any position of the object A ₂ to D _2, be determined as follows That's fine. That is, for example, the position of the center of gravity of the quadrangle closest to the center of the region R ′ or the position of the center of gravity is set as the position of the pointer P ₂ , and in a predetermined order (for example, clockwise) from the position of the pointer P ₂ The position of the center of gravity may be the positions of the objects A _{2 to} D ₂ .

  In step S1903, for example, when the position of the center of gravity of the rectangle is within a predetermined range from the boundary of the region R ′, the center of gravity may be excluded from the position where the pointer or the object is arranged. This is because when the position of the center of gravity of the rectangle is in the vicinity of the boundary of the region R ′, if the object or the like is placed at this position, the projected object may protrude outside the region R ′.

<Third Embodiment>
Next, a display system 1 according to the third embodiment will be described. In the description of the third embodiment, differences from the first embodiment will be described, and components having functional configurations similar to those of the first embodiment will be denoted by the same reference numerals. Omit duplicate descriptions.

[Function configuration]
FIG. 21 is a diagram illustrating a functional configuration example of the display system according to the third embodiment. The enlargement / reduction ratio determination unit 24A of the portable terminal 20 of the display system 1 according to the third embodiment has a size (enlargement / reduction ratio) after the position determination unit 25 determines the pointer to be projected and displayed and the position of each object. The point which determines is different from the first embodiment.

[Determination processing of enlargement / reduction ratio]
Next, the third embodiment will be described with reference to FIG. 21 regarding the scaling factor determination processing in step S505 of FIG. FIG. 22 is a flowchart of an example of a scaling factor determination process according to the third embodiment. Here, as described above, in the present embodiment, the process of step S506 is performed before the process of step S505 of FIG. That is, the scaling factor determination process described below is performed after the positions of the pointer P ₂ and the objects A _{2 to} D ₂ are determined by the position determination unit 25. Incidentally, the determination of the position of the pointer _{P 2} and object _A 2 to D ₂ may be performed by the position determination unit 25A of the second embodiment.

In the following, the case of determining the scaling factor s _a object A _1, will be described. Also in the case of determining the enlargement / reduction ratio of the pointer P ₁ and the objects B _{1 to} D ₁ , the processes in the subsequent steps S2201 to S2203 may be performed. Therefore, in the present embodiment, different scaling factors are determined for each of the pointer P ₁ and the objects A _{1 to} D ₁ .

First, scaling factor determining section 24A, when displayed by projecting the object A ₁ without adjustment scaling factor s _a, circumscribed objects A ₁ in the captured image data generated by imaging by the imaging unit 13 calculating the radius _{r a} of the circle (step S2201). Such radius r _a, by multiplying relative radius of the circumscribed circle of the object image data of the object A _1, the positive square root √ of Bp / Ap used in the equations (1) to (Bp / Ap) Can be calculated. Incidentally, the object image data of an object A _1, the length ratio of the object A ₁ in the captured image data, for example may be calculated as the ratio of the width of each of the bounding box.

Next, scaling factor determining section 24A calculates a second distance to other objects, such as when the pointer P ₁ and the object A ₁ to D ₁ is placed at a position determined by the position determination unit 25. That is, the enlargement / reduction ratio determination unit 24A includes the second distance T ₁ between the object A ₁ and the pointer P ₁ , the second distance T _{2 to} T ₄ between the object A ₁ and each of the other objects B _{1 to} D ₁ , And a second distance T ₅ between the object A ₁ and the boundary of the region R is calculated. Then, the scaling factor determination unit 24A calculates a minimum second distance among the calculated T _{1 to} T ₅ (step S2202). Here, scaling factor determining section 24A is assumed to have calculated T ₅ as the minimum of the second distances.

Finally, scaling factor determining section 24A uses the predetermined value _{k 3,} which is set in advance by the following equation (11) calculates the scaling factor _{s a} (step S2203). Accordingly, scaling factor _{s a} with respect to the object _{A 1} is calculated. In the present embodiment, the scaling factor is calculated for the pointer P ₁ and the objects A _{1 to} D ₁ as described above. The value of k ₃ is predetermined in the design or the like of the program to implement this embodiment.

  As described above, according to the display system 1 according to the first embodiment, the user can select a plane for performing a pointing operation. At this time, according to the display system 1 according to the first embodiment, a region in which the pointing operation is easily performed is extracted from the selected display object, and the pointer is provided in the extracted region so that the pointing operation can be easily performed. Project and display after adjusting the size and position of the object. Thereby, according to the display system 1 which concerns on 1st Embodiment, a pointing operation can be performed within the area | region extracted from the display target object which a user desires, and a user's pointing operation can be improved.

  In addition, according to the display system 1 according to the second embodiment, the position of the pointer and the object projected in the extracted area can be determined by a simple method.

  Furthermore, according to the display system 1 according to the third embodiment, the size of the pointer and the object projected on the extracted area content can be determined for each pointer and object. Thereby, according to the display system 1 which concerns on 3rd Embodiment, the size of the pointer made to project in the extracted area | region and an object can each be varied.

  As mentioned above, although the Example of this invention was explained in full detail, this invention is not limited to such specific embodiment, In the range of the summary of this invention described in the claim, various deformation | transformation・ Change is possible.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
A display unit that projects and displays a pointer for operating a plurality of applications on a display object;
A calculation unit that calculates a first distance to the display object on which the pointer is projected;
Based on the first distance, in the display object, an extraction unit that extracts a region for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer;
A determination unit that determines a position where the plurality of objects do not overlap in the region based on a relative position of each of the plurality of objects and the pointer;
The display unit
A display device that projects and displays the pointer and the plurality of objects at the determined position.
(Appendix 2)
An imaging unit that captures the display object and obtains an image;
The extraction unit includes:
The display device according to appendix 1, wherein the region is extracted from the display object based on a pixel of the display object and the first distance in the image.
(Appendix 3)
The determination unit is
Furthermore, the display device according to appendix 2, wherein scaling factors of the pointer and the plurality of objects are determined based on the image.
(Appendix 4)
The determination unit is
The position is determined based on a distance between the plurality of objects, a distance between the pointer and the plurality of objects, and a second distance indicating a distance between a boundary of the region and the plurality of objects. The display device according to any one of appendices 1 to 3.
(Appendix 5)
The determination unit is
The display device according to appendix 4, wherein the position is determined based on a repulsive force model using the second distance.
(Appendix 6)
Projecting and displaying pointers for operating multiple applications on the display object,
Calculating a first distance to the display object onto which the pointer is projected;
Based on the first distance, in the display object, an area for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer is extracted;
A position where the plurality of objects do not overlap in the region is determined based on a relative position of each of the plurality of objects and the pointer.
The processing to display is
A program for projecting and displaying the pointer and the plurality of objects at the determined position.
(Appendix 7)
Causing the display device to execute a process of capturing an image of the display object and obtaining an image;
The extraction process is:
The program according to appendix 6, wherein the region is extracted from the display object based on a pixel of the display object and the first distance in the image.
(Appendix 8)
The determination process is as follows:
Further, the program according to appendix 7, wherein scaling factors of the pointer and the plurality of objects are determined based on the image.
(Appendix 9)
The determination process is as follows:
The position is determined based on a distance between the plurality of objects, a distance between the pointer and the plurality of objects, and a second distance indicating a distance between a boundary of the region and the plurality of objects. The program according to any one of appendices 6 to 8.
(Appendix 10)
The determination process is as follows:
The program according to appendix 9, wherein the position is determined based on a repulsive force model using the second distance.
(Appendix 11)
Projecting and displaying pointers for operating multiple applications on the display object,
Calculating a first distance to the display object onto which the pointer is projected;
Based on the first distance, in the display object, an area for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer is extracted;
The display device executes a process of determining a position where the plurality of objects do not overlap in the region based on the relative positions of the plurality of objects and the pointer,
The processing to display is
A display method of projecting and displaying the pointer and the plurality of objects at the determined position.
(Appendix 12)
The display device executes a process of obtaining an image by imaging the display object,
The extraction process is:
The display method according to appendix 11, wherein the region is extracted from the display object based on a pixel of the display object and the first distance in the image.
(Appendix 13)
The determination process is as follows:
Furthermore, the display method according to appendix 12, wherein scaling factors of the pointer and the plurality of objects are determined based on the image.
(Appendix 14)
The determination process is as follows:
The position is determined based on a distance between the plurality of objects, a distance between the pointer and the plurality of objects, and a second distance indicating a distance between a boundary of the region and the plurality of objects. 14. The display method according to any one of appendices 11 to 13.
(Appendix 15)
The determination process is as follows:
The display method according to appendix 14, wherein the position is determined based on a repulsive force model using the second distance.

DESCRIPTION OF SYMBOLS 1: Display system 10: Pointing terminal 11: Input part 12: Projection part 13: Imaging part 14: Change amount acquisition part 20: Portable terminal 21: Pointer specific part 22: Area extraction part 23: Object determination part 24: Enlargement / reduction ratio determination Part 25: Position determining part:
26: Projection image generation unit 27: Change amount calculation unit 28: Image data storage unit 29: Object management data storage unit

Claims (7)

A display unit that projects and displays a pointer for operating a plurality of applications on a display object;
A calculation unit that calculates a first distance to the display object on which the pointer is projected;
Based on the first distance, in the display object, an extraction unit that extracts a region for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer;
A determination unit that determines a position where the plurality of objects do not overlap in the region based on a relative position of each of the plurality of objects and the pointer;
The display unit
A display device that projects and displays the pointer and the plurality of objects at the determined position. An imaging unit that captures the display object and obtains an image;
The extraction unit includes:
The display device according to claim 1, wherein the region is extracted from the display object based on a pixel of the display object and the first distance in the image. The determination unit is
The display device according to claim 2, further comprising: a scaling factor of the pointer and the plurality of objects is determined based on the image. The determination unit is
The position is determined based on a distance between the plurality of objects, a distance between the pointer and the plurality of objects, and a second distance indicating a distance between a boundary of the region and the plurality of objects. The display device according to any one of claims 1 to 3. The determination unit is
The display device according to claim 4, wherein the position is determined based on a repulsive force model using the second distance. Projecting and displaying pointers for operating multiple applications on the display object,
Calculating a first distance to the display object onto which the pointer is projected;
Based on the first distance, in the display object, an area for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer is extracted;
A position where the plurality of objects do not overlap in the region is determined based on a relative position of each of the plurality of objects and the pointer.
The processing to display is
A program for projecting and displaying the pointer and the plurality of objects at the determined position. Projecting and displaying pointers for operating multiple applications on the display object,
Calculating a first distance to the display object onto which the pointer is projected;
Based on the first distance, in the display object, an area for displaying a plurality of objects corresponding to the plurality of applications operated by the pointer is extracted;
The display device executes a process of determining a position where the plurality of objects do not overlap in the region based on the relative positions of the plurality of objects and the pointer,
The processing to display is
A display method of projecting and displaying the pointer and the plurality of objects at the determined position.

Images