WO2022215347A1

WO2022215347A1 - Content display control system

Info

Publication number: WO2022215347A1
Application number: PCT/JP2022/005304
Authority: WO
Inventors: 有希中村; 康夫森永; 望松本; 達哉西▲崎▼; 怜央水田; 弘行藤野
Original assignee: 株式会社Ｎｔｔドコモ
Priority date: 2021-04-08
Filing date: 2022-02-10
Publication date: 2022-10-13

Abstract

This content display control system 1 comprises: AR glasses 30 as a head-mounted display which is worn by a user and displays content in a virtual space; a detection unit 11 for detecting a display instruction operation that is reproduced by the user and is associated with target content; and a control unit 12 for controlling content display on the AR glasses 30 such that the target content associated with the detected display instruction operation is displayed at a position suitable for the user's visual recognition in a virtual space.

Description

Content display control system

The present disclosure relates to a content display control system that controls content display on a head-mounted display such as AR (Augmented Reality) glasses.

Conventionally, for users wearing head-mounted displays such as AR glasses, various contents (for example, browser screens, display screens for various information provided by cloud services, etc., image display screens, video display screens, etc. etc.) in a virtual space. For example, it is possible to arrange various contents in a virtual space centered on a user as a display area.

On the other hand, a method has been proposed in which the movement of the user's eyeballs is detected and the user easily selects a desired operation (see Patent Document 1).

Japanese Patent Application Laid-Open No. 2018-042004

However, there is no established method for the user to easily select desired content that the user wants to focus on and rearrange the content to a position in the display area that is easy to see (for example, in front of the user). Therefore, when there are many contents arranged in the display area, there is an inconvenience that it takes time and effort for the user to search for the desired contents. As a method for easily selecting desired content, it is conceivable that the user turns his face toward the desired content. It is often embarrassed for such reasons. Therefore, there is a great demand for a technique for rearranging desired content to a position where it is easy for the user to see it, as described above.

The present disclosure has been made to solve the above problems, and aims to arrange desired content at a position in the virtual space that is easy for the user to see without taking time and effort.

A content display control system according to the present disclosure includes a head-mounted display worn by a user to display content in a virtual space, a detection unit that detects a display instruction operation associated with target content reproduced by the user, and a control unit that controls content display on the head-mounted display such that the target content associated with the display instruction operation detected by the detection unit is displayed at the line-of-sight position of the user in the virtual space. . The ``position of the user's line of sight'' here means (1) the focal position of the user's line of sight, (2) the extension line passing through the reticle in the 360-degree omnidirectional virtual space centered on the user, and the omnispherical surface. and (3) a position pointed by a user using an operating device (controller).

In the content display control system described above, when the detection unit detects a display instruction action associated with target content reproduced by a user wearing a head-mounted display, the control unit controls the target content associated with the detected display instruction action. controls the display of content on the head-mounted display so that is displayed at the line-of-sight position of the user in the virtual space. As a result, desired content can be arranged at a position in the virtual space that is easy for the user to see without taking time and effort.

Further, a content display control system according to the present disclosure includes a head-mounted display worn by a user to display content in a virtual space, and a detection unit that detects a display instruction operation that is reproduced by the user and is associated with target content. a control unit that controls content display on the head-mounted display such that the target content associated with the display instruction operation detected by the detection unit is displayed at a position suitable for visual recognition by the user in a virtual space; , provided. In addition to the user's line-of-sight position described above, the "user's preferred viewing position" includes a position in front of the face that does not depend on the user's line of sight.

According to the present disclosure, it is possible to arrange desired content at a position in the virtual space that is easy for the user to see without taking time and effort.

(a) is a configuration diagram of a content display control system according to an embodiment of the invention, and (b) is a configuration diagram of a content display control system further provided with a correspondence table or the like in which content identification information is linked to content IDs and stored. is. (a) is a diagram showing an example of a controller, and (b) is a diagram showing an example of a user interface of a terminal substituting for the controller. FIG. 4 is a flow diagram showing a rough flow of processing executed in the content display control system; (a) is a diagram showing an example of tagging and registration of content in a virtual space, and (b) is a diagram showing an example of tagging and registration of content from a dedicated management system. It is a figure which shows the data example registered into the correspondence table. FIG. 10 is a flow diagram showing processing related to display control of target content at the line-of-sight position of the user; (a) is a diagram showing the initial state when the tag of the target content is uttered, (b) is a diagram showing a comparison between the speech recognition result and the correspondence table, and (c) is the state after rearrangement. FIG. 4 is a diagram showing; (a) is a diagram showing rearrangement pattern 1 for simple superimposition of target content, (b) is a diagram showing rearrangement pattern 2 for temporary placement of target content, and (c) is a diagram showing rearrangement pattern 2 for temporary placement of target content. FIG. 10D is a diagram showing a rearrangement pattern 3 for simple superimposition and holding of relative positional relationships of all contents in the azimuth direction, and FIG. It is a figure for demonstrating conversion between a rectangular coordinate and a three-dimensional polar coordinate (spherical surface coordinate). FIG. 4 is a diagram for explaining a first algorithm for rearranging target content to a line-of-sight position; FIG. 10 is a diagram showing a calculation example according to the first algorithm in rearrangement pattern 1; FIG. 10 is a diagram for explaining a second algorithm for rearranging target content to a line-of-sight position; FIG. 10 is a diagram for explaining an algorithm for maintaining a relative positional relationship in the azimuth direction in rearrangement pattern 3; FIG. 11 is a diagram for explaining an algorithm for maintaining relative positional relationships in rearrangement pattern 4; FIG. 11 is a diagram for explaining an algorithm for maintaining relative positional relationships in rearrangement pattern 4; FIG. 11 is a diagram for explaining an algorithm for maintaining relative positional relationships in rearrangement pattern 4; FIG. 11 is a diagram for explaining an algorithm for maintaining relative positional relationships in rearrangement pattern 4; (a) is a diagram showing semi-transparency of non-targeted content, (b) is a diagram showing sharpness adjustment of non-targeted content, (c) is a diagram showing outline glow adjustment of targeted content, (d) is a diagram showing target content display on a pop-up virtual screen. (a) is a diagram showing an example of comparison table data when target content is specified by pattern input using a controller, and (b) is a diagram showing an example of comparison table data when target content is specified by a hand gesture. It is a diagram. It is a figure which shows the hardware structural example of a terminal. FIG. 10 is a flow diagram showing processing related to display control of target content in front of a user's face. (a) is a diagram showing a state when the tag of the target content is uttered while directing the line of sight in a direction different from the direction of the face, and (b) is a diagram showing the state in which the target content is displayed at a display position in front of the face. FIG. 4C is a diagram showing a state, and (c) is a diagram showing a state when the line of sight is turned to the target content in front of the face.

A content display control system according to an embodiment of the invention will be described below with reference to the drawings.

(Configuration of content display control system)
As shown in FIG. 1(a), the content display control system 1 according to this embodiment includes a controller 20 operated by a user, a terminal 10 carried by the user, and AR glasses 30 worn by the user. Prepare.

The controller 20 has, for example, a configuration including buttons 21, a touch pad 22, a trigger 23 and a bumper 24 shown in FIG. 2(a). Note that the controller 20 is not an essential requirement in the content display control system 1, but can be replaced by the user interface of the terminal 10. FIG. For example, the user interface of the terminal 10 shown in FIG. 2(b) can be substituted by a user interface including a touch pad 16 consisting of a total of five buttons in the center, up, down, left, and right, and

buttons

17 and 18. FIG.

The AR glasses 30 correspond to a head-mounted display that displays content in virtual space, and displays content in a 360-degree omnidirectional virtual space centered on the user wearing it. In this embodiment, AR glasses are exemplified as a head-mounted display, but head-mounted displays other than AR glasses (for example, VR (Virtual Reality) goggles, etc.) may be employed.

The terminal 10 corresponds to, for example, a smart phone, a mobile phone, a notebook personal computer, etc., and includes a detection unit 11 and a control unit 12 shown in FIG. 1(a). The function, operation, etc. of each unit will be described below.

The detection unit 11 is a functional unit that detects a display instruction operation that is reproduced by the user and is associated with the target content. In the present embodiment, as an example, a display instruction operation of uttering a character string (tag) indicating target content while performing a predetermined operation using the controller 20 will be described. A predetermined operation using the controller 20 here will be exemplified later.

The control unit 12 is a functional unit that controls content display on the AR glasses 30 so that the target content associated with the display instruction operation detected by the detection unit 11 is displayed at the user's line-of-sight position in the virtual space. . In order to exhibit the above functions, the control unit 12 includes a specifying unit 12A and a display control unit 12B as described below.

The specifying unit 12A is a functional unit that specifies the target content associated with the display instruction operation detected by the detecting unit 11.

The display control unit 12B acquires the line-of-sight position of the user and controls content display on the AR glasses 30 so that the target content identified by the identification unit 12A is displayed at the line-of-sight position of the user in the virtual space. Department. As described above, the ``position of the user's line of sight'' includes (1) the focal position of the user's line of sight, (2) the extension line passing through the reticle in the 360-degree omnidirectional virtual space centered on the user, and the omnidirectional surface. and (3) the position pointed by the user using the controller 20. Of these, when (3) is adopted, the content using the controller 20 is used for the display instruction operation by the user. A pointing operation for the display position is added.

In this embodiment, as an example, a character string (tag) indicating the target content is linked in advance to the target content as "content specifying information" for specifying the target content. Further, in this embodiment, the user designates the target content by uttering a tag, which is a character string indicating the target content, while performing a predetermined operation described later using the controller 20 . Therefore, the detection unit 11 includes a microphone 11A that collects the user's utterance, and a voice recognition unit 11B that recognizes and converts the collected voice into text. Therefore, the detection unit 11 performs a display instruction operation such as "uttering a character string (tag) indicating the target content while performing a predetermined operation using the controller 20" (that is, a display instruction action) is detected, the specifying unit 12A specifies the target content linked to the content specifying information (character string (tag)) corresponding to the detected display instruction action.

Note that as one of the modifications described later, there is an example in which the target content is specified by a gesture of a part of the user's body (for example, the user's hand). and a gesture recognition unit 11D for recognizing gestures from moving image data obtained by imaging. Further, the predetermined actions using the controller 20 include, for example, the action of continuously pressing the button 21 or the bumper 24 n times, the action of long-pressing the button 21 or the bumper 24, the action of holding the trigger 23 while pulling it for a certain period of time, Examples include an action of tapping a specific portion of the touch pad 22 at the top, bottom, left, right, and center n times consecutively, and an action of pressing and holding a specific portion of the touch pad 22 at the top, bottom, left, right, and center. As for the "long press action" here, various variations can be provided by combining a plurality of actions as described above, such as starting a long press after tapping n times.

In this embodiment, as shown in FIG. 1(b), the terminal 10 stores a correspondence table 13 in which content identification information for various contents is linked to content IDs for identifying the contents, and a user ID. A registration unit 14 may further include a registration unit 14 for linking the content identification information to the content ID and registering it in the correspondence table 13 by performing a predetermined registration process. The correspondence table 13 is a content correspondence information database in which content identification information (for example, a tag consisting of a character string, a controller operation pattern, a hand gesture, etc.) is stored in association with a content ID, and an example is shown in FIG. As shown, tags associated with various content IDs are registered. In the following, processing operations, effects, etc. will be described assuming that the terminal 10 has the configuration shown in FIG. 1(b).

(Processing Executed in Content Display Control System)
As shown in FIG. 3, the processing executed in the content display control system 1 includes tagging registration to content (step S1) and display control of target content at the user's line of sight position (step S2).

Of the above, the tagging registration in step S1 is a process that is performed before the process in step S2, which is a feature of the present disclosure. For example, as shown in FIG. While the content is being displayed, the user operates the controller 20 to expand the menu of the content, select "tagging registration", and enter the tag (character string) of the content to tag the content. and register. For example, by entering "# ALPACA" for the alpaca (content ID: ID-XYZ1245) displayed in the virtual space, it is linked to the alpaca (content ID: ID-XYZ1245) as shown in Fig. 5 The tag "ALPACA" is registered in the correspondence table 13. It should be noted that inputting "#" before the tag when inputting the tag is an example of the tag input rule, and is not an essential requirement. As another mode, as shown in FIG. 4(b), the content may be tagged and registered from a dedicated management system running on a personal computer.

Next, using FIG. 6, the display control of the target content at the line-of-sight position of the user (step S2 in FIG. 3) will be described.

Execution of the processing in FIG. 6 is started when the user utters a tag, which is a character string indicating the target content, while performing a predetermined action using the controller 20 .

The detection unit 11 collects the user's utterance with the microphone 11A, and the speech recognition unit 11B performs speech recognition on the collected speech (step S21). The text data of the speech recognition result is transferred to the identification unit 12A.

Next, the identification unit 12A compares the speech recognition result with each record stored in the correspondence table 13 (step S22), and determines whether or not there is a record that matches the speech recognition result (step S23). If there is no record that matches the speech recognition result, the process is terminated. On the other hand, if there is a record that matches the voice recognition result, the specifying unit 12A specifies the content of the matching record as the target content, and transmits it to the display control unit 12B. Then, the display control unit 12B determines whether or not the target content is being displayed in the virtual space (step S24). Here, if the target content is being displayed, the display control unit 12B acquires the line-of-sight position of the user (step S25), and rearranges the content based on the line-of-sight position (step S25). S26).

On the other hand, if the target content is not being displayed in step S24, the display control unit 12B acquires the line-of-sight position of the user (step S27), renders the target content at the line-of-sight position, and displays the target content at the line-of-sight position. (Step S28).

Here, a specific example of the processing of FIG. 6 is shown using FIGS. 7(a) to (c). First, as shown in FIG. 7A, the initial state is a state in which four types of animal content are arranged with the user at the center in the virtual space. 6, the detection unit 11 recognizes the voice of the tag "FOX" by voice recognition, obtains the text "FOX" as a voice recognition result, and transfers it to the identification unit 12A. Then, in step S22, the identifying unit 12A compares the text "FOX" with the tag of each record stored in the correspondence table 13, and as shown in FIG. ID "ID-JSN3G49") is in the correspondence table 13 (YES in step S23), the content of the matching record (the content of the fox with the content ID "ID-JSN3G49") is specified as the target content, and the display is controlled. It is transmitted to the section 12B. Further, the display control unit 12B determines that the target content is being displayed in the virtual space (step S24), and sets the user's gaze position ((x _e , y _e , _ze ) shown in FIG. 7A) to Acquired (step S25), and rearrangement of the contents based on the line-of-sight position is executed according to a method described later (step S26). As an execution example of rearrangement, FIG. 7(c) shows an example in which only the fox content, which is the target content, is simply rearranged to the user's line-of-sight position (x _e , y _e , z _e ).

(description of different relocation patterns)
Next, various rearrangement patterns of contents in step S26 will be described with reference to FIGS. 8(a) to 8(d). It should be noted that the initial state before the rearrangement is a state in which four types of animal content are arranged around the user in the disguised space shown in FIG. 7A.

FIG. 8(a) shows a rearrangement pattern 1 that simply superimposes the target content. This rearrangement pattern 1 simply rearranges only the target content (fox content in this case) to the user's gaze position (x _e , y _e , z _e ) as in the example of FIG. 7(c). It's a pattern.

FIG. 8(b) shows rearrangement pattern 2 for temporarily arranging the target content. In this rearrangement pattern 2, only the target content (the fox content here) is temporarily moved to the user's gaze position (x _e , y _e , z _e ), a copy of the target content is generated, and the copy is moved. It is a pattern that is temporarily placed at the original position (x _t , y _t , z _t ). A copy of the target content is displayed, for example, translucent and cannot be operated. In addition, when the temporary placement ends, the target content displayed at the user's line-of-sight position (x _e , y _e , z _e ) is deleted, and the content displayed at the original position (x _t , y _t , z _t ) is restored. is returned to the original display form, and the operation details, changes, etc. to the target content displayed at the user's line-of-sight position (x _e , y _e , z _e ) are reflected.

FIG. 8(c) shows a rearrangement pattern 3 that simply superimposes the target content and maintains the relative positional relationship of all the content in the azimuth direction. In this rearrangement pattern 3, all of the displayed contents (hereinafter referred to as "display contents") (here, four types of animals) are moved while maintaining the relative positional relationship between the contents in the azimuth direction, Furthermore, it is a pattern in which only the target content (here, fox content) is translated to the user's gaze position (x _e , y _e , z _e ).

FIG. 8(d) shows a rearrangement pattern 4 that maintains the relative positional relationship between all displayed contents. In this rearrangement pattern 4, all the displayed contents are moved while maintaining the relative positional relationship between them, and the target contents (here, fox contents) are moved to the user's gaze position (x _e , y _e , z _e ). It is a pattern that rearranges to Note that the above four patterns are not essential requirements, and patterns other than these may be employed.

(Description of relocation of target content to line-of-sight position)
A point common to the various rearrangement patterns described above is that the target content is rearranged to the line-of-sight position of the user. Therefore, two algorithms relating to rearrangement of the target content to the line-of-sight position will be described below with reference to FIGS. 9 to 12. FIG. Note that the following algorithm is an example, and an algorithm other than the following may be adopted.

The first algorithm is a procedure that rotates the target content around the Z-axis in Cartesian coordinates and translates the target content to the user's line-of-sight position. As shown in FIG. 10, first, three-dimensional polar coordinates are obtained from rectangular coordinates for each of the target content and the user's line-of-sight position (process (1)). FIG. 9 shows a general relationship between orthogonal coordinates and three-dimensional polar coordinates (spherical coordinates). , Φ), the three-dimensional polar coordinates (r _t , θ _t , Φ _t ) of the target content and the three-dimensional polar coordinates (r _e , θ _e , Φ _e ).

It should be noted that only the azimuth angle of the three-dimensional polar coordinates may be obtained.

Next, the azimuth angle change amount is calculated by the following equation (2) (process (2)).
ΔΦ = _Φe - _Φt (2)

Then, the target content is rotated by ΔΦ around the Z-axis in the orthogonal coordinates according to the following equation (3) (process (3)). That is, the azimuth angle with respect to the user is adjusted. Note that the first term on the right side of Equation (3) is a rotation matrix about the Z-axis.

Furthermore, the target content is translated to the user's line-of-sight position according to the following equation (4) (process (4)). Note that the first term on the right side of Equation (4) is a translation matrix.

As described above, by applying the affine transformation matrix to all the points that make up the content (three-dimensional CG object), the rotation around the Z-axis and the translation to the user's gaze position are realized. .

Here, a specific example of calculation based on the above first algorithm in rearrangement pattern 1 will be described with reference to FIG.

For the first acquisition of the three-dimensional polar coordinates of the target content and the line-of-sight position, when the orthogonal coordinates (x _t , y _t , z _t ) of the target content are (1, -1, -1), the above formula ( 1), the following is obtained as the three-dimensional polar coordinates (r _t , θ _t , Φ _t ) of the target content (process (1)).

Also, the orthogonal coordinates (x _e , y _e , z _e ) of the user's gaze position are

, the following is obtained as the three-dimensional polar coordinates (r _e , θ _e , Φ _e ) of the line-of-sight position of the user by substituting into the above formula (1).

Next, using the above equation (2), the azimuth angle change amount ΔΦ is calculated as follows (process (2)).
ΔΦ＝Φ _e －Φ _t ＝(2π/3)-(-π/4)＝11π/12

Then, using the above formula (3), the target content is rotated by ΔΦ around the Z-axis in the orthogonal coordinates as follows (process (3)).

Furthermore, using the above equation (4), the target content is translated to the line-of-sight position of the user as follows (process (4)).

According to the above first algorithm, the Cartesian coordinates (x' _t , y' _t , z' _t ) of the target content are the Cartesian coordinates (x _e , y _e , _ze ) of the line-of-sight position of the user, that is,

becomes.

The second algorithm will be explained below. The second algorithm is a procedure that rotates the target content about the Z-axis in Cartesian coordinates, adjusts the polar angle with respect to the user, and translates the target content to the user's gaze position.

As shown in FIG. 12, first, three-dimensional polar coordinates are obtained from orthogonal coordinates for each of the target content and the user's line-of-sight position (process (1)). This process is similar to the first algorithm described above.

Next, in the three-dimensional polar coordinates of the target content and the three-dimensional polar coordinates of the line-of-sight position, the amount of change in each of the radius, azimuth angle, and polar angle is calculated as follows (process (2)).

Next, the target content is rotated by ΔΦ around the Z-axis in the orthogonal coordinates according to the following equation (5) (process (3)). That is, the azimuth angle with respect to the user is adjusted. Note that the first term on the right side of Equation (5) is a rotation matrix about the Z-axis.

Then, the target content is rotated by Δθ around the axis on the XY plane perpendicular to the line-of-sight position by the following equation (6) (process (4)). That is, the polar angle with respect to the user is adjusted. The first term on the right side of Equation (6) is the Rodriguez rotation matrix.

Furthermore, the target content is moved in parallel to the user's line-of-sight position according to the following equation (7) (process (5)). As a result, the orthogonal coordinates (x' _t , y' _t , z' _t ) of the target content become the orthogonal coordinates (x _e , y _e , z _e ) of the user's gaze position.

Note that the first term on the right side of Equation (7) corresponds to the following.

As described above, by applying the affine transformation matrix to all the points that make up the content (three-dimensional CG object), the rotation around the Z axis, the adjustment of the polar angle with respect to the user, and the user's line of sight Implement translation to position.

(Regarding rearrangement pattern 3)
An algorithm for maintaining the relative positional relationship in the azimuth angle direction unique to rearrangement pattern 3 in FIG. 8C will be described below with reference to FIG. As shown in FIG. 13, the three-dimensional polar coordinates (r _t , θ _t , Φ _t ) of the target content and the gaze position of the user are first calculated using the above-described formula (1), similarly to the first algorithm described above. and the three-dimensional polar coordinates (r _e , θ _e , Φ _e ) of are obtained (process (1)). It should be noted that only the azimuth angle of the three-dimensional polar coordinates may be obtained.

Next, the azimuth angle change amount is calculated using the above-described formula (2) (process (2)).

Then, each of all the display contents is rotated by ΔΦ around the Z-axis in the orthogonal coordinates according to the following formula (8) (process (3)). That is, the azimuth angle with respect to the user is adjusted for each display content. Note that n is a subscript for specifying each display content.

Furthermore, similar to the above-described first algorithm, the target content is translated to the user's line-of-sight position using Equation (4) described above, thereby relocating the target content to the user's line-of-sight position (processing ( Four)). It should be noted that the rearrangement method may employ the method of the second algorithm instead of the method of the first algorithm.

As described above, as shown in the upper right corner of FIG. 13, a rearrangement pattern 3 that simply superimposes the target content and maintains the relative positional relationship of all the content in the azimuth direction is realized.

(Regarding rearrangement pattern 4)
14 to 17, an algorithm for holding the relative positional relationship unique to rearrangement pattern 4 will be described below. As shown in FIG. 14, the three-dimensional polar coordinates (r _t , θ _t , Φ _t ) of the target content and the three-dimensional polar coordinates (r _e , θ _e , Φ _e ) are obtained (process (1)), and the amount of change in the radius, azimuth angle, and polar angle in the 3D polar coordinates of the target content and the 3D polar coordinates of the line-of-sight position are calculated as follows (process (2)).

Next, the coordinates obtained by rotating the target content by ΔΦ around the Z-axis in the orthogonal coordinates are calculated according to the following equation (9) (process (3)).

Next, the following equation (10) is used to provisionally calculate the post-rearrangement azimuth angle of the display content other than the target content (process (4)). Note that m is a subscript for specifying content other than the target content.

Next, the post-rearrangement polar angles of the contents other than the target contents among the displayed contents are tentatively calculated as follows (processing (5) shown in FIGS. 14 and 15). However, it is limited to the following range.

The tentative calculation of the polar angle after the rearrangement here is "when -π < Φ _e ≤ -π/2", "when -π/2 < Φ _e ≤ π/2", and "π/2 < Execution is divided into a total of four cases of Φ _e <π and Φ _e =π.

First, for the case of -π<Φ _e ≤ -π/2 (case (5)i in Fig. 14),
Azimuth angle Φ'm _(pre) after rearrangement is -π<Φ'm _(pre) _≤Φt +(π/2) or π-Arctan( _-xtz / _ytz ) _≤Φ'm If _(pre) ≤ π, the rearranged polar angle θ'm _(pre) is obtained by the following equation (11).
θ' _{m (pre)} = θ _m + Δθ (11)
Note that the condition "-π<Φ' _m(pre) ≤Φ _t +(π/2)" here means that the pole (θ _m ,Φ' _m(pre) )=(0,Φ _e +(π /2)) is removed, and the other condition ``π－Arctan(-x _tz /y _tz )≤Φ' _m(pre) ≤π'' has the pole (θ _m ,Φ' _m(pre) )= (0,π-Arctan(-x _tz /y _tz )) is excluded.
On the other hand, if the rearranged azimuth angle Φ′ _m(pre) is in a range other than the above range, the rearranged polar angle θ′ _m(pre) is obtained by the following equation (12).
θ' _{m (pre)} = θ _m - Δθ (12)

Next, for the case of -π/2<Φ _e ≤π/2 (case (5) ii in Fig. 14),
The azimuth angle Φ' _m(pre) after rearrangement is Φ _e -(π/2)<Φ' _m(pre) ≦Φ _e +(π/2) (however, the poles (θ _m , Φ' _{m( pre)} )=(0, Φ _e +(π/2))), the rearranged polar angle θ′ _m(pre) is obtained by the following equation (13).
θ' _{m (pre)} = θ _m + Δθ (13)
On the other hand, the azimuth angle Φ' _{m (pre)} after rearrangement is in a range other than the above (except for the pole (θ _m , Φ' _{m (pre)} ) = (0, Φ _e - (π/2))) , the rearranged polar angle θ′ _m(pre) is obtained by the following equation (14).
θ' _{m (pre)} = θ _m - Δθ (14)

Next, in the case of π/2<Φ _e <π (case of (5) iii in FIG. 15),
Azimuth angle Φ' _{m (pre)} after rearrangement is -π <Φ' _{m (pre)} ≤ -π + Arctan(-x _tz /y _tz ) or Arctan (-x _tz /y _tz ) ≤ Φ' _m If _(pre) ≤ π, the rearranged polar angle θ'm _(pre) is obtained by the following equation (15).
θ' _{m (pre)} = θ _m + Δθ (15)
On the other hand, the azimuth angle Φ' _{m (pre)} after rearrangement is in a range other than the above (however, the pole (θ _m , Φ' _{m (pre)} ) = (0, -π + Arctan (-x _tz /y _tz ) and the pole (θ _m , Φ' _m(pre) )=(0, Arctan(-x _tz /y _tz )))), the rearranged polar angle θ' _{m( pre)} .
θ' _{m (pre)} = θ _m - Δθ (16)

Furthermore, in the case of Φ _e =π (case of (5) iv in FIG. 15),
The azimuth angle Φ'm _(pre) after rearrangement is in the range of -π <Φ'm _(pre) ≤ -(π/2) or (π/2) ≤ Φ'm _(pre) ≤ π ( However, except for poles (θ _m ,Φ' _m(pre) )=(0,-π/2) and poles (θ _m ,Φ' _m(pre) )=(0,π/2)) obtains the rearranged polar angle θ′ _m(pre) by the following equation (17).
θ' _{m (pre)} = θ _m + Δθ (17)
On the other hand, if the rearranged azimuth angle Φ′ _m(pre) is in a range other than the above range, the rearranged polar angle θ′ _m(pre) is obtained by the following equation (18).
θ' _{m (pre)} = θ _m - Δθ (18)

Next, the polar coordinates after rearrangement of the contents other than the target contents among the displayed contents are determined as follows (process (6) shown in FIG. 15). Here, it is executed in two cases: "Polar angle θ' m(pre ₎ after rearrangement <0" and "Polar angle θ' _m(pre) after rearrangement >π". .

First, when the rearranged polar angle θ′ _m(pre) <0 (case (6)i in FIG. 15), the rearranged azimuth angle Φ′ _m(pre) is 0<Φ′ In the range of _m(pre) ≤ π, the rearranged polar coordinates (r' _m , θ' _m , Φ' _m ) are obtained by the following equation (19).
( _r'm , _θ'm , _Φ'm ) = (rm + Δr, _θ'm _(pre) , Φ'm _(pre) - π) (19)
On the other hand, if the rearranged azimuth angle Φ'm _(pre) is in the range of -π <Φ'm _(pre) ≤ 0, the rearranged polar coordinates ( _r'm ,θ' _m ,Φ' _m ).
( _r'm , _θ'm , _Φ'm ) = (rm + Δr, _θ'm _(pre) , Φ'm _(pre) + π) (20)

Next, when the rearranged polar angle θ′ _m(pre) >π (case (6)ii in FIG. 15), the rearranged azimuth angle Φ′ _m(pre) is 0<Φ In the range of ' _m _(pre) ≤ π, the rearranged polar coordinates ( _r'm , θ'm, _Φ'm ) are obtained by the following equation (21).
( _r'm , _θ'm , _Φ'm ) = (rm + Δr, 2π- _θ'm _(pre) , Φ'm _(pre) - π) (21)
On the other hand, if the rearranged azimuth angle Φ′ _m(pre) is in the range of −π<Φ′ _m(pre) ≦0, the rearranged polar coordinates (r′ _m ,θ' _m ,Φ' _m ).
( _r'm , _θ'm , _Φ'm ) = (rm + Δr, 2π- _θ'm _(pre) , Φ'm _(pre) + π) (22)

After determining the polar coordinates after rearrangement for content other than the target content as described above, the user presets TH (threshold value) as follows, and sets the radius r that defines the display area of the content in the depth direction. By adjusting, the adjusted radius r' _m may be obtained. In this case, it is possible to prevent the radius r from becoming extremely small and the radius r from becoming extremely large due to the amount of movement of the target content. However, care must be taken because it can be regarded as a process that creates a shift in the depth direction and destroys the relative position.

After that, "affine transformation of all contents other than the target content to coordinates after rearrangement (processing (7) in FIGS. 16 and 17)" and "affine transformation of target content to the line-of-sight position (processing in FIGS. 16 and 17 (processing (7) of FIG. 16 and FIG. 17)" 8))” are executed. The execution order of these processes is random, and they may be executed concurrently or in a predetermined order. Of the above, the latter "affine transformation of the target content to the line-of-sight position (processing (8) in FIGS. 16 and 17)" is executed by applying either the first algorithm or the second algorithm already explained. Since it is possible, redundant description is omitted here.

(Affine transformation of all contents other than the target contents to the coordinates after relocation)
In the affine transformation of all contents other than the target content to the rearranged coordinates (process (7) in FIGS. 16 and 17), is the target content affine transformed to the line-of-sight position by applying the first algorithm? , and the second algorithm is applied and executed.

When the affine transformation of the target content to the line-of-sight position is executed by applying the first algorithm, the affine transformation of all contents other than the target content to the coordinates after rearrangement is performed as shown in processing (7) in FIG. is executed.

First, the amount of change in the azimuth angle and the polar angle before and after rearrangement is calculated by the following equation (23) (process (7)i in FIG. 16).
_Δθm = _θ'm - _θm
ΔΦ _m =Φ' _m -Φ _m (23)

Next, the azimuth angle with respect to the user is adjusted by rotating ΔΦ _m around the z-axis according to the following equation (24) (process (7) ii).

Next, rectangular coordinates are acquired from the rearranged polar coordinates by the following equation (25) (process (7) iii).

Furthermore, the coordinates after the azimuth angle adjustment obtained by the above processing are translated by applying the following equation (26), and the orthogonal coordinates after the translation (x' _m , y' _m , z' _m ) is obtained (process (7) iv).

As described above, the Cartesian coordinates (x' _m , y' _m , z' _m ) after rearrangement of all contents other than the target contents are obtained.

On the other hand, when the target content is affine-transformed to the line-of-sight position by applying the second algorithm, all the contents other than the target content are affine-transformed to the coordinates after rearrangement in the process (7) of FIG. Executed as shown.

What is the first "calculation of the amount of change in the azimuth angle and polar angle before and after rearrangement (process (7) i in FIG. 17)" and the subsequent "ΔΦ _m rotation around the Z axis (process (7) ii)"? , is the same as the example of FIG. 16 described above, so redundant description will be omitted.

Next, the polar angle with respect to the user is adjusted by rotating Δθ _m around the axis on the xy plane according to the following equation (27) (process (7) iii).

Furthermore, the coordinates after adjustment of the azimuth angle and polar angle obtained by the above processing are translated by applying the following equation (28), and the orthogonal coordinates after translation (x' _m , y' _m ,z' _m ) is obtained (process (7) iv).

(Effect of the embodiment of the invention)
According to the embodiments of the invention described above, it is possible to arrange desired content at a position in the virtual space that is easy for the user to see without taking time and effort. Further, for the user, the processing of FIG. 6 can be started simply by uttering a tag indicating the target content while performing a predetermined operation using the controller, and the user can easily instruct the display of the desired content. be able to.

Further, when the initial state is "a state in which the target content is displayed in the virtual space", the target content in the virtual space is displayed according to a predetermined rearrangement pattern (any of the rearrangement patterns 1 to 4) based on the line-of-sight position. A rearrangement position of the content can be determined, and the target content can be rearranged to the determined rearrangement position. On the other hand, even if the initial state is "a state in which the target content is not displayed in the virtual space", by acquiring the user's line-of-sight position and rendering the target content at the acquired line-of-sight position, Target content can be displayed.

Also, as a rearrangement pattern from the state where the target content is displayed in the virtual space, a rearrangement pattern for rearranging all the display contents while maintaining the relative positional relationship in the azimuth direction for all the display contents. 3, it is possible to maintain the relative positional relationship in the azimuth angle direction for all display contents, and to avoid the layout that the user is particular about from collapsing significantly.

Further, as a rearrangement pattern from the state where the target content is displayed in the virtual space, the rearrangement pattern 4 for rearranging all the display contents while maintaining the relative positional relationship between all the display contents is included. , the relative positional relationship between all displayed contents can be maintained, and it is possible to avoid collapsing of the user's preferred layout.

(Modification)
18(a) to 18(d) will be used to explain how to deal with overlap in content display. If there is a possibility that the target content and other content (non-target content) overlap in the virtual space, for example, by taking the following measures, the overlap can be avoided and the inconvenience of the target content becoming difficult to see can be resolved. can.

That is, as shown in FIG. 18(a), the non-target content is translucent, as shown in FIG. A method of making the target content relatively easy to see by performing processing such as adjusting the value of r may be adopted. Alternatively, as shown in FIG. 18C, processing such as adjusting the brilliance of the outline of the target content may be performed to make the target content stand out and be easier to see. Furthermore, the target content may be displayed on a pop-up virtual screen as shown in FIG. 18(d). In this way, various techniques can be used to make target content relatively easy to see. The example of FIG. 18D is limited to rearrangement pattern 2, and the processing for adjusting the value of radius vector r in spherical coordinates is limited to rearrangement patterns 1-3.

Also, as a method for the user to specify the target content, it is not essential to utter the tag of the target content. For example, the user may specify the target content by inputting a pattern described later using a controller. , a hand gesture may be used to specify the target content. As an example of the former "pattern using a controller", as shown in FIG. , a pattern in which the button 21 is pressed n times consecutively, and a pattern in which a laser pointer function of the controller 20 draws a specific figure (for example, a circle) in the virtual space. In this case, the controller pattern as described above is registered in the correspondence table 13 in association with the content ID of the corresponding content. Hand gestures for designating target content by the latter hand gesture include, as shown in FIG. be done. In this case, the hand gesture as described above is registered in the correspondence table 13 in association with the content ID of the corresponding content. It should be noted that when target content is designated by a hand gesture, as described above, the detection unit 11 shown in FIGS. The configuration further includes a gesture recognition unit 11D for recognition.

As described above, as a modified example, instead of uttering the tag of the target content, the user can specify the target content and instruct to display it by a method such as pattern input using a controller or hand gesture. Display instructions of desired content can be easily given by various methods.

In the processing operation of the above embodiment, as shown in FIG. 1(b), the terminal 10 has a correspondence table 13 in which content specifying information (for example, a tag) is associated with a content ID and stored in advance, and the specifying unit 12A However, an example of specifying the target content from the content ID linked to the content specifying information corresponding to the display instruction operation detected by the detection unit 11 by referring to the correspondence table 13 has been shown. The correspondence table 13 is not an essential requirement. For example, by using the content ID itself as a tag that is content identification information, the correspondence table 13 is made unnecessary, and the identification unit 12A identifies content corresponding to the detected display instruction operation. The configuration shown in FIG. 1(a) may be adopted so that the target content can be specified immediately from the information. According to the configuration of FIG. 1(a), the simplification of the device configuration can be achieved.

In addition, in the configuration of FIGS. 1A and 1B, it is not essential that the terminal 10 includes all of the illustrated components (the detection unit 11, the identification unit 12A, the display control unit 12B, etc.). A part of the functional units may be installed in the server, and the target content may be specified by requesting the server from the terminal 10 to process. In this case, the content display control system according to the present disclosure is understood to have a configuration including the terminal 10 and the server.

(another embodiment)
Next, as another embodiment, a mode will be described in which content display control is performed so that target content is displayed in front of the user's face. Here, the display control unit 12B acquires the orientation of the user's face using an existing face recognition technology or the like, and the target content is displayed at a display position in front of the orientation of the user's face in the virtual space. , which controls the display of content on the head-mounted display. As the above-mentioned "display position in front of the face", for example, the position of the intersection of the extension line to the front of the face in the 360-degree omnidirectional virtual space centered on the user and the omnidirectional surface is adopted. Alternatively, a position at a predetermined distance from the face may be adopted on the forward extension of the face. It should be noted that the "predetermined distance" (distance from the face) can be appropriately adjusted by the user.

As the processing, the processing shown in FIG. 21 is executed instead of the "display control to the user's gaze position" in FIG. 6 described above. In FIG. 21, the same processing as in FIG. 6 is assigned the same number, and redundant description is omitted. If the target content is being displayed in the virtual space in step S24 of FIG. 21, the display control unit 12B acquires the orientation of the user's face using existing face recognition technology or the like (step S25A). Content rearrangement, which will be described later, is executed based on the display position of (step S26A). On the other hand, if the target content is not being displayed in step S24, the display control unit 12B acquires the orientation of the user's face (step S27A), and renders the target content at a display position in front of the face orientation to display the target content. The target content is displayed at the position (step S28A).

Here, a specific example of the processing of FIG. 21 will be shown using FIGS. 22(a) to (c). First, as shown in FIG. 22(a), the initial state is a state in which three types of animal content are arranged with the user at the center in the virtual space. When the tag "ALPACA" of the target content is uttered while viewing the target content "ALPACA" to be moved, the detection unit 11 recognizes the voice of the tag "ALPACA" in step S21 of FIG. The text "ALPACA" is obtained as a speech recognition result and transferred to the identification unit 12A. Then, in step S22, the specifying unit 12A compares the text "ALPACA" with the tag of each record stored in the correspondence table 13, and determines that there is a record matching the text "ALPACA" in the correspondence table 13 (step S22). YES in S23), the content of the matching record, Alpaca, is specified as the target content and transmitted to the display control unit 12B. Further, the display control unit 12B determines that the target content is being displayed in the virtual space (step S24), acquires the orientation of the user's face using existing face recognition technology or the like (step S25A), The content rearrangement is executed with reference to the forward display position ((x _e , y _e , z _e ) shown in FIG. 22(b)) (step S26A). FIG. 22B shows an example of rearranging only the target content (alpaca content) to the display position (x _e , y _e , z _e ) in front of the face direction. At this time, the user's line of sight is directed to the position before the movement of the target content, but the target content first moves to the display position in front of the face. After that, as shown in FIG. 22(c), the user can view the target content at a position that is easy for the user to see (here, in front of the face) by directing the line of sight forward. Note that when the target content is not being displayed in the virtual space, the display control unit 12B acquires the orientation of the user's face using an existing face authentication technology or the like (step S27A), and determines the display position in front of the orientation of the face. By rendering the target content in ((x _e , y _e , z _e ) shown in FIG. 22(b)), the target content is displayed in front of the face (step S28A). In this way, even when the target content is not being displayed, the target content can be displayed in front of the user's face regardless of the line of sight of the user. As described above, even in another embodiment, it is possible to place desired content at a position in the virtual space that is easy for the user to see without taking time and effort.

Also, the block diagrams used in the description of the above embodiments show blocks in units of functions. These functional blocks (components) are realized by any combination of at least one of hardware and software. Also, the method of implementing each functional block is not particularly limited. That is, each functional block may be implemented using one device that is physically or logically coupled, or directly or indirectly using two or more devices that are physically or logically separated (e.g. , wired, wireless, etc.) and may be implemented using these multiple devices. A functional block may be implemented by combining software in the one device or the plurality of devices.

Functions include judging, determining, determining, calculating, calculating, processing, deriving, investigating, searching, checking, receiving, transmitting, outputting, accessing, resolving, selecting, choosing, establishing, comparing, assuming, expecting, assuming, Broadcasting, notifying, communicating, forwarding, configuring, reconfiguring, allocating, mapping, assigning, etc. can't For example, a functional block (component) that makes transmission work is called a transmitting unit or transmitter. In either case, as described above, the implementation method is not particularly limited.

For example, the control unit in one embodiment of the present disclosure may function as a computer that performs processing in this embodiment. FIG. 20 is a diagram illustrating a hardware configuration example of terminal 10 according to an embodiment of the present disclosure. The terminal 10 described above may be physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, a bus 1007, and the like.

In the following explanation, the term "apparatus" can be read as a circuit, device, unit, or the like. The hardware configuration of the terminal 10 may be configured to include one or more of each device shown in the figure, or may be configured without some of the devices.

Each function of the terminal 10 is performed by causing the processor 1001 to perform calculations, controlling communication by the communication device 1004, controlling communication by the communication device 1004, and controlling the communication by the memory 1002 and the It is realized by controlling at least one of data reading and writing in the storage 1003 .

The processor 1001, for example, operates an operating system and controls the entire computer. The processor 1001 may be configured by a central processing unit (CPU) including interfaces with peripheral devices, terminals, arithmetic units, registers, and the like.

Also, the processor 1001 reads programs (program codes), software modules, data, etc. from at least one of the storage 1003 and the communication device 1004 to the memory 1002, and executes various processes according to them. As the program, a program that causes a computer to execute at least part of the operations described in the above embodiments is used. Although it has been explained that the above-described various processes are executed by one processor 1001, they may be executed simultaneously or sequentially by two or more processors 1001. FIG. Processor 1001 may be implemented by one or more chips. Note that the program may be transmitted from a network via an electric communication line.

The memory 1002 is a computer-readable recording medium, and is composed of at least one of, for example, ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), RAM (Random Access Memory), etc. may be The memory 1002 may also be called a register, cache, main memory (main storage device), or the like. The memory 1002 can store executable programs (program code), software modules, etc. for implementing a wireless communication method according to an embodiment of the present disclosure.

The storage 1003 is a computer-readable recording medium, for example, an optical disc such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disc, a magneto-optical disc (for example, a compact disc, a digital versatile disc, a Blu-ray disk), smart card, flash memory (eg, card, stick, key drive), floppy disk, magnetic strip, and/or the like. Storage 1003 may also be called an auxiliary storage device. The storage medium described above may be, for example, a database, server, or other suitable medium including at least one of memory 1002 and storage 1003 .

The communication device 1004 is hardware (transmitting/receiving device) for communicating between computers via at least one of a wired network and a wireless network, and is also called a network device, a network controller, a network card, a communication module, or the like.

The input device 1005 is an input device (for example, keyboard, mouse, microphone, switch, button, sensor, etc.) that receives input from the outside. The output device 1006 is an output device (eg, display, speaker, LED lamp, etc.) that outputs to the outside. Note that the input device 1005 and the output device 1006 may be integrated (for example, a touch panel). Each device such as the processor 1001 and the memory 1002 is connected by a bus 1007 for communicating information. The bus 1007 may be configured using a single bus, or may be configured using different buses between devices.

Each aspect/embodiment described in the present disclosure may be used alone, may be used in combination, or may be used by switching along with execution. In addition, the notification of predetermined information (for example, notification of “being X”) is not limited to being performed explicitly, but may be performed implicitly (for example, not notifying the predetermined information). good too.

Although the present disclosure has been described in detail above, it is clear to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be practiced with modifications and variations without departing from the spirit and scope of the present disclosure as defined by the claims. Accordingly, the description of the present disclosure is for illustrative purposes and is not meant to be limiting in any way.

The order of the processing procedures, sequences, flowcharts, etc. of each aspect/embodiment described in the present disclosure may be changed as long as there is no contradiction. For example, the methods described in this disclosure present elements of the various steps using a sample order and are not limited to the specific order presented.

Input/output information may be stored in a specific location (for example, memory) or managed using a management table. Input/output information and the like can be overwritten, updated, or appended. The output information and the like may be deleted. The entered information and the like may be transmitted to another device.

The term "based on" as used in this disclosure does not mean "based only on" unless otherwise specified. In other words, the phrase "based on" means both "based only on" and "based at least on."

Where "include," "including," and variations thereof are used in this disclosure, these terms are inclusive, as is the term "comprising." is intended. Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive OR.

In this disclosure, if articles are added by translation, such as a, an, and the in English, the disclosure may include that the nouns following these articles are plural.

In the present disclosure, the term "A and B are different" may mean "A and B are different from each other." The term may also mean that "A and B are different from C". Terms such as "separate," "coupled," etc. may also be interpreted in the same manner as "different."

DESCRIPTION OF SYMBOLS 1... Content display control system 10... Terminal 11... Detection part 11A... Microphone 11B... Voice recognition part 11C... Camera 11D... Gesture recognition part 12... Control part 12A... Specification part 12B... Display control Unit 13 Correspondence table 14 Registration unit 16

Touch pad

17, 18 Button 20 Controller 21 Button 22 Touch pad 23 Trigger 24 Bumper 30 AR glass (head mount display), 1001... processor, 1002... memory, 1003... storage, 1004... communication device, 1005... input device, 1006... output device, 1007... bus.

Claims

a head-mounted display that is worn by a user and displays content in a virtual space;
a detection unit that detects a display instruction action that is reproduced by the user and is associated with target content;
a control unit that controls content display on the head-mounted display such that the target content associated with the display instruction operation detected by the detection unit is displayed at a position suitable for viewing by the user in a virtual space;
A content display control system comprising:
a head-mounted display that is worn by a user and displays content in a virtual space;
a detection unit that detects a display instruction action that is reproduced by the user and is associated with target content;
a control unit that controls content display on the head-mounted display such that the target content associated with the display instruction operation detected by the detection unit is displayed at the line-of-sight position of the user in the virtual space;
A content display control system comprising:
The control unit
a specifying unit that specifies the target content associated with the display instruction operation detected by the detecting unit;
a display control unit that controls content display on the head-mounted display so that the target content specified by the specifying unit is displayed at the line-of-sight position of the user in virtual space;
including,
3. The content display control system according to claim 2.
The detection unit detects a display instruction operation reproduced by the user corresponding to content specifying information linked to the target content,
The identification unit identifies the target content linked to content identification information corresponding to the display instruction operation detected by the detection unit.
4. The content display control system according to claim 3.
The content display control system includes:
a content correspondence information database in which content identification information associated with content is associated with a content ID for identifying the content and stored;
further comprising
The specifying unit refers to the content correspondence information database and specifies the target content based on a content ID linked to content specifying information corresponding to the detected display instruction operation.
5. The content display control system according to claim 4.
When the detection unit detects a display instruction operation including an utterance linked to the target content by the user, the specifying unit specifies the target content linked to the voice recognition result of the utterance.
A content display control system according to any one of claims 3 to 5.
When the detection unit detects a display instruction operation including execution of an operation pattern by a user using a predetermined operation device or movement of a part of the body by the user, the identification unit detects the detected display identifying the target content linked to the instruction action;
A content display control system according to any one of claims 3 to 6.
The display control unit
When the target content specified by the specifying unit is displayed in a virtual space, a line-of-sight position of the user is acquired, and a virtual display is performed according to a predetermined rearrangement pattern based on the acquired line-of-sight position. Determining a rearrangement position of the target content in space, and rearranging the target content to the determined rearrangement position;
A content display control system according to any one of claims 3 to 7.
The rearrangement pattern is
a pattern for rearranging all displayed content while preserving relative azimuthal positional relationships for all displayed content; and
A pattern that rearranges all displayed content while maintaining the relative positional relationship between displayed content,
including at least one of
The content display control system according to claim 8.
The display control unit
When the target content specified by the specifying unit is not displayed in the virtual space, acquiring the line-of-sight position of the user and rendering the target content so that the target content is displayed at the acquired line-of-sight position. do,
A content display control system according to any one of claims 3 to 9.
The display control unit
changing a display mode of at least one of the target content and the other content so that the target content is easier to see when the target content displayed at the line-of-sight position overlaps with other displayed content;
A content display control system according to any one of claims 3 to 10.