JP2017068697A - Projection control device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a projection control device and a program which can display a display object such as characters on a position in a visual line direction, according to a position and a posture of a person, to a projected video, without attaching a projector by a viewer.SOLUTION: A reproduction control device 6 identifies a visual field range of a person according to a position of a head and a posture of the person who is in a prescribed three-dimensional space, and an effective visual field angle, then determines a position, a posture, and a size of a text object arranged in at least a partial area of a projection plane to which a video is projected in a visual field range of the person according to the position of the head and the posture of the person, and the effective visual field angle. The reproduction control device 6 combines the text object with the video projected by the projectors 5a-5c at the determined position, posture and size, and projects the video with which the text object is combined on the projection plane.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technical field of a projection system that projects an image on a projection surface.

  Conventionally, a technology is known in which a helmet equipped with a direction sensor and a projector is mounted on a person's head, the direction of the person's head is detected by the direction sensor, and a surrounding image in that direction is projected by the projector. (For example, Patent Document 1).

JP-A-6-167687

  By the way, when it is desired to project a character at a position in the gaze direction of the person as another image with respect to the video projected on the projection plane by the projector, as disclosed in Patent Document 1, the person who is a viewer uses the projector. When worn, characters are projected in accordance with the movement of the person's head, regardless of the person's state such as position and posture, and the visibility of the characters seen by the person is poor.

  The present invention has been made in view of the above points, and displays characters and the like at a position in a line-of-sight direction according to the position and posture of a person on a projected image without a projector being worn by a viewer. Provided are a projection control device and a program capable of displaying an object.

  In order to solve the above-described problem, the invention according to claim 1 is characterized in that the specifying means for specifying the visual field range of the person from the position, posture, and effective viewing angle of the person's head in a predetermined three-dimensional space; Arranged in at least a partial area of the projection surface on which the image is projected from the projection device within the field of view of the person identified from the position, posture, and effective viewing angle of the person's head identified by the identifying means Determining means for determining the position and size of the first display object; and combining means for combining the first display object on the image projected by the projection device at the position and size determined by the determining means; And a projection control means for projecting an image obtained by synthesizing the first display object onto the projection plane.

  According to a second aspect of the present invention, in the projection control apparatus according to the first aspect, after the video in which the first display object is synthesized is projected, the specifying unit is configured to determine the position and posture of the person's head. And the person's visual field range is newly specified from the effective viewing angle, and the determining means is the position of the person newly specified by the specifying means, the posture of the person, and the effective visual angle of the person specified from the effective viewing angle. Within the field of view, the position and size of the second display object arranged in at least a partial area of the projection surface on which the image is projected from the projection device are newly determined, and the combining means includes the first display unit A change amount of a value corresponding to at least one of the position, posture, and size of a display object is greater than or equal to a threshold value, or a change amount of a value corresponding to at least one of the position and posture of the person's head If it is equal to or greater than the threshold value, the second display object is synthesized on the video with the first display object synthesized at the position and size newly decided by the decision means, and the projection control means A video obtained by combining the first display object and the second display object is projected onto the projection plane.

  According to a third aspect of the present invention, in the projection control apparatus according to the first aspect, the specifying unit specifies the human visual field range at predetermined time intervals from the position, posture, and effective visual angle of the human head. The determining means includes a projection plane on which an image is projected from the projection device within the field of view of the person specified by the position, posture, and effective viewing angle of the person's head specified by the specifying means. The position and size of the first display object arranged in at least a part of the region are determined a plurality of times at predetermined time intervals, and the combining unit corresponds to the position and size of the plurality of times determined by the determining unit. The first display object is synthesized on the video projected by the projection device according to the average value of the values to be projected, and the projection control means is a video in which the first display object is synthesized. And characterized in that projected on the projection plane.

  According to a fourth aspect of the present invention, in the projection control apparatus according to the third aspect, after the video in which the first display object is synthesized is projected, the specifying unit is configured to determine the position and posture of the person's head. And the person's visual field range is newly specified from the effective viewing angle, and the synthesizing means is configured such that a change amount of a value corresponding to at least one of the position, posture, and size of the first display object is a threshold value. If it is above, or if the amount of change in the value corresponding to at least one of the head position and posture of the person is greater than or equal to a threshold value, the first position and size determined by the determining means are A second display object different from the first display object is synthesized on the video obtained by synthesizing the first display object, and the projection control means includes the first display object and the second display object. Characterized in that projecting the image and display object is synthesized on the projection plane.

  According to a fifth aspect of the present invention, in the projection control device according to the second or fourth aspect, a predetermined time after the video in which the first display object and the second display object are synthesized is projected, Deletion means for deleting the first display object from the video is further provided, and the projection control means projects the video in which the first display object is deleted and the second display object is synthesized. Projecting on a surface.

  According to a sixth aspect of the present invention, in the projection control device according to any one of the first to fourth aspects, the first display object is continuously projected from the time when it is combined with the video until it is deleted. Delete means for deleting the first display object from the video after the projection duration time after the video in which the time is associated and the first display object is synthesized is projected; and the first display object Is projected, and then the amount of change in the value corresponding to at least one of the position, posture, and size of the first display object, or the position of the head of the person in the three-dimensional space And when the amount of change in the value corresponding to at least one of the orientations is less than the threshold value for a predetermined time or longer, the projection duration associated with the first display object Characterized in that it comprises a and an extension means for extending the.

  According to a seventh aspect of the present invention, in the projection control apparatus according to any one of the first to sixth aspects, a projection timing on the projection plane is associated with the first display object, The synthesizing unit synthesizes the first display object on the video when the projection timing arrives, and the projection control unit generates a video obtained by synthesizing the first display object at the projection timing. Projecting on the projection plane is characterized.

  According to an eighth aspect of the present invention, there is provided a specifying step of specifying the field of view of the person from the position, posture, and effective viewing angle of the person in a predetermined three-dimensional space, and the specification of the person specified by the specifying step. The position of the first display object arranged in at least a partial area of the projection plane on which the image is projected from the projection device within the visual field range of the person specified from the position, posture, and effective viewing angle of the head, and A determination step of determining a size, a combining step of combining the first display object on a video projected by the projection device at the position and size determined by the determination step, and the first display object Causing the computer to execute a projection control step of projecting the synthesized image onto the projection plane.

  According to the first and eighth aspects of the present invention, it is possible to accurately project a display object at a position in the line-of-sight direction of a person in a projected image while maintaining visibility.

  According to the second aspect of the present invention, even if the visual field range of a person changes greatly in a short time, it is possible to quickly respond to the changed visual field range and the display object in the changed visual field range. Visibility can be improved.

  According to the invention described in claim 3, the visibility of the display object can be improved by using the moving average of the display object.

  According to the fourth aspect of the present invention, when the moving average of the display objects is used, even when the visual field range of the person changes greatly in a short time, the changed visual field range can be quickly dealt with. The visibility of the display object in the visual field range after the change can be improved.

  According to the fifth aspect of the present invention, a display object in a visual field range that is less likely to be visually recognized by a person can be quickly displayed even if the visual field range of the person changes greatly in a short time and then further changes. Can be deleted.

  According to the sixth aspect of the present invention, the visibility of the display object can be further improved in the visual field range in which the person's line of sight stays longer.

  According to the seventh aspect of the present invention, it is possible to effectively project a video in which the display object is synthesized on the projection plane at the projection timing associated with the display object.

(A) is a figure which shows the example of a schematic structure of the projection system S of this embodiment. (B) is a figure showing an example of the positional relationship of the apparatus arrange | positioned in the room seen from the upper direction of the room where the projection system S is utilized. It is a conceptual diagram which shows the visual field of a person, a visual field angle, and a visual field range. It is a conceptual diagram which shows the example of the visual field range of a person in the projection surface of virtual three-dimensional space. (A) is a conceptual diagram showing an example in which one lyric object is synthesized with a video projected on a projection plane in a virtual three-dimensional space. (B) is a conceptual diagram showing an example in which a video and a lyric object projected on a projection plane in a virtual three-dimensional space are projected and converted into a two-dimensional plane coordinate system based on the positions of the projectors 5a to 5c. is there. 5 is a flowchart illustrating an example of a projection control process executed by a control unit 53 in Embodiment 1. 10 is a flowchart illustrating an example of a projection control process executed by a control unit 53 in Embodiment 2. (A) is a conceptual diagram showing an example in which two lyrics objects are combined with an image projected on a projection plane in a virtual three-dimensional space. (B) is a conceptual diagram showing an example when one lyric object is deleted from two lyric objects synthesized with a video projected on a projection plane in a virtual three-dimensional space.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. Outline of Projection System Configuration and Operation]
First, with reference to FIG. 1 etc., the structure and operation | movement outline | summary of the projection system S of this embodiment are demonstrated. As shown in FIG. 1A, the projection system S includes a display / operation device 1, a music playback device 2, a camera 3, video playback devices 4a to 4c, projectors 5a to 5c, a playback control device 6, and the like. Composed. Here, the projectors 5a to 5c are examples of a projection device. The reproduction control device 6 is an example of a projection control device. The projection system S is used in a room in an amusement facility such as a karaoke store. The display / operation device 1, the music playback device 2, and the camera 3 are connected to the playback control device 6 via a wired or wireless connection, respectively. The video reproduction apparatuses 4a to 4c are connected to the reproduction control apparatus 6 via a wired LAN (Local Area Network) or a wireless LAN, a hub, and the like. The projectors 5a to 5c are respectively connected to the video reproduction devices 4a to 4c via wired or wireless. In addition, a microphone M and a speaker SP are connected to at least one of the music playback device 1 and the playback control device 6 via a wired or wireless connection.

  The display / operation device 1 is a device that receives an operation instruction from a user and outputs an instruction signal indicating the operation instruction to the reproduction control device 6. As the display / operation device 1, for example, a touch panel display including a display screen is used. The display / operation device 1 may be, for example, a remote controller of the music playback device 2. The display / operation device 1 displays, on the display screen, the titles of music that are candidates for selection of music to be played back by the music playback device 2 and the titles of videos that are candidates for selection of videos projected by the projectors 5a to 5c. The video includes a still image and a moving image. A moving image is displayed, for example, by switching image frames of a plurality of still images in time series. Examples of video include landscape video, artist promotional video, corporate or product advertising video, and the like. The display / operation device 1 transmits the music ID (identification information) and the reproduction instruction of the music selected according to the operation instruction from the user to the reproduction control device 6. The music ID and the reproduction instruction transmitted to the reproduction control device 6 are transmitted from the reproduction control device 6 to the music reproduction device 2, for example. Thus, the music instructed to be played is played back by the music playback device 2. The display / operation device 1 may transmit the music ID of the selected music to the music playback device 2, for example.

  Further, the display / operation device 1 transmits the video ID (identification information) and the projection instruction of the video selected according to the operation instruction from the user to the reproduction control device 6. The video data and the projection instruction indicated by the video ID transmitted to the reproduction control device 6 are transmitted from the reproduction control device 6 to the video reproduction devices 4a to 4c. In this way, the video instructed to be projected is played back by the video playback devices 4a to 4c and projected onto the projection plane by the projectors 5a to 5c. Note that the video data and the projection instruction corresponding to the music indicated by the music ID transmitted to the reproduction control device 6 may be transmitted from the reproduction control device 6 to the video reproduction devices 4a to 4c. When a music is being played back by the music playback device 2, a lyrics object (an example of a display object) of the music is combined with a video by the playback control device 6. As a result, the video combined with the lyrics object is played back by the video playback devices 4a to 4c and projected onto the projection plane by the projectors 5a to 5c. The lyrics object is, for example, an image of a lyrics telop that displays the lyrics of a song.

  The music playback device 2 is a device that plays back music such as karaoke. The music reproducing device 2 stores a plurality of music files in the storage unit. The music file stores music data of the music. Each music file is associated with a music ID of the music, for example. When the music reproducing device 2 receives the music ID and the reproduction instruction transmitted from the display / operation device 1 or the reproduction control device 6, the music reproducing device 2 reproduces the music instructed to reproduce and outputs the audio signal to the speaker SP. The user can sing with the microphone M while watching the lyrics object projected on the projection surface while the music playback device 2 plays the music. In this case, the music reproducing device 2 mixes the audio signal of the music and the user's singing voice signal collected by the microphone M, and outputs the mixed signal to the speaker SP.

  As the camera 3, for example, an RGB camera and a depth camera are used. The camera 3 continuously acquires RGB images by continuously capturing an imaging range that falls within the angle of view with the RGB camera. The depth camera is a depth sensor such as an infrared laser. The camera 3 continuously acquires depth images by continuously capturing an imaging range that falls within the angle of view with a depth camera. The depth image indicates the three-dimensional position coordinates (X, Y, Z) of each position on the projection plane in the three-dimensional space within the imaging range of the camera 3. For example, in the case of FIG. 1B, since a person (for example, a user singing while looking at a lyric object) exists in a three-dimensional space within the imaging range of the camera 3, the depth image is within the imaging range of the camera 3. The three-dimensional position coordinates of each position on the surface of the person in the three-dimensional space, and the three-dimensional position coordinates of each position on the wall surfaces a to c in the background of the person. In the case of the example of FIG. 1B, the camera 3 detects (recognizes) a person from the acquired depth image, and acquires part information indicating the position of each part that forms the outline and skeleton of the detected person. . The camera 3 continuously transmits the acquired RGB image, the acquired depth image, and the detected person part information to the reproduction control device 6. Note that the playback control device 6 may detect a person from the depth image and acquire part information of the detected person.

  The person part information is, for example, information in which three-dimensional position coordinates (X, Y, Z) are associated with each part forming the outline or skeleton of the person, and the depth to each position indicated in the depth image. It is possible to specify from Specifically, by comparing the depth to each position acquired by the depth camera with person position pattern data for determining the position of each part forming the person's outline or skeleton, Each part that forms is identified. The person position pattern data is used, for example, by reading out predetermined pattern data corresponding to the depth of each position indicated in the depth image from many types of person position patterns stored in advance in a predetermined database or the like. Examples of each part that forms the outline or skeleton of a person include the head, left shoulder, right shoulder, left elbow, right elbow, left hand, right hand, neck, trunk, waist, left buttocks, right Examples include a hip, a left knee, a right knee, a left foot, and a right foot. Such part information includes the rotation angle Θ of the part. The rotation angle Θ of a part is a rotation angle Θ based on another part connected to the part. Alternatively, the rotation angle Θ of a part may be a rotation angle Θ based on the position of a specific part indirectly connected to the part. For example, the rotation angle Θ of the head is a rotation angle Θ based on the neck connected to the head. The rotation angle Θ of the neck is a rotation angle Θ with respect to the body connected to the neck. The part information including the rotation angle Θ is expressed as (X, Y, Z, Θ). The part information of the person's head may indicate the center coordinates when the person's head is a sphere, or may indicate the position coordinates of each of a plurality of parts on the surface of the person's head.

  The video playback devices 4a to 4c are devices that play back the video to be projected. Receiving the video data and the projection instruction transmitted from the reproduction control device 6 respectively, the video reproduction apparatuses 4a to 4c reproduce the image instructed to project and output the image signal to the projectors 5a to 5c. The projectors 5a to 5c output images by emitting projection light corresponding to the image signals reproduced and output by the image reproducing devices 4a to 4c from the light emitting unit. As a result, the projectors 5a to 5c project an image on a projection surface within a projection range that falls within the projection angle. In the example of FIG. 1B, the projector 5a is disposed so as to project an image on a part of the wall surface b and a part of the wall surface c. The projector 5b is arranged to project an image on a part of the wall surface a, a wall surface b, and a part of the wall surface c. The projector 5c is disposed so as to project an image on a part of the wall surface a and a part of the wall surface b. For this reason, in the example of FIG. 1B, the person and the wall surfaces a to c are projection planes.

  In the example of FIG. 1B, a part of the projection surface on which the image is projected overlaps between the projectors 5a to 5c. Therefore, some of the images projected from the projectors 5a to 5c are overlapped. However, the projection direction or projection angle may be set so that the projection plane onto which the video is projected does not overlap between the projectors 5a to 5c. Further, each of the projectors 5a to 5c may project an image on one specific wall surface. The projection plane may be on the ceiling surface and floor surface of the room. In the example of FIG. 1B, three video playback devices 4a to 4c and three projectors 5a to 5c are shown. However, any number of video playback devices and projectors may be used. In the example of FIG. 1B, the video playback device and the projector correspond one-to-one, but may correspond to one-to-N (N is a natural number of 2 or more). For example, a plurality of projectors may be connected to one video reproduction device. When there is one video playback device, the video playback device may be incorporated in the playback control device 6 as video playback means (configured by hardware or software). The cameras 3 may be provided in a one-to-one correspondence with the respective projectors.

  The playback control device 6 is a device that performs music playback control, video projection control, and the like. The playback control device 6 includes IFs 61a to 61d, a storage unit 62, and a control unit 63, as shown in FIG. The IFs 61 a to 61 d, the storage unit 62, and the control unit 63 are connected to the bus 64. The IF 61 a is an interface between the music playback device 2 and the playback control device 6. The IF 61 b is an interface between the display / operation device 1 and the reproduction control device 6. The IF 61 c is an interface between the camera 3 and the playback control device 6. The IF 61d is an interface with the LAN. The storage unit 62 is configured by, for example, a hard disk drive. The storage unit 62 stores an OS (Operating System), the program of the present invention, and the like. The program of the present invention is a program that causes a CPU to execute a projection control process described later. The projection control process is also called a projection mapping process. The storage unit 62 stores a video file that stores original video data serving as original data of videos projected by the projectors 5a to 5c.

  Further, the storage unit 62 stores the lyric object data in association with the music ID for each music. The lyric object data includes, for example, lyric data (text data) indicating the lyrics of music and display timing data indicating the display timing of the lyric object (an example of projection timing). Since the lyrics of one song are composed of a plurality of phrases, the lyrics object data is stored for each phrase. One phrase is composed of a plurality of character strings. The lyric data may be image data of a lyric object that displays the lyrics of the music. The display timing indicated by the display timing data is represented, for example, by the elapsed time from the playback start point of the music. The lyric object data may include a color change timing that indicates the singing position of the lyric by changing the display color of the lyrics sentence.

  The control unit 63 includes a CPU (Central Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 63 is an example of a specifying unit, a determining unit, a combining unit, a deleting unit, an extending unit, and a projection control unit of the present invention. The control unit 63 executes projection control processing according to the program of the present invention. In particular. The control unit 63 specifies the position, posture, and effective viewing angle of the person's head in a predetermined three-dimensional space. Here, the predetermined three-dimensional space is a virtual three-dimensional space recognized by the control unit 63, and is specified from the three-dimensional position coordinates indicated by the depth image acquired from the camera 3. In other words, this three-dimensional space is a three-dimensional space in a range in which an image on which a person is projected can be observed. The position of the person's head is specified by the three-dimensional position coordinates indicated by “part information of the person's head” acquired from the camera 3. The effective viewing angle of the person is specified from a general effective viewing angle stored in advance in the storage unit 62, for example.

  The posture of the person's head is specified by, for example, the person's line-of-sight direction and the rotation angle ψ when the head rotates around the center axis when the axis extending in the person's line-of-sight direction is the central axis. This rotation angle ψ is taken into consideration when the person tilts his / her neck. This is because even if the viewing direction of the person is the same, if the rotation angle ψ is different, the viewing angle described later changes and the viewing range changes. However, there is little change in the visual field range when the viewing direction of the person is the same and the rotation angle ψ changes, so the rotation angle ψ does not have to be considered. In this case, the posture of the person's head is specified only by the person's line-of-sight direction, for example. The line-of-sight direction of the person can be a normal vector extending vertically from the position of the person's eyes (for example, perpendicular to the face surface). This normal vector is calculated from the rotation angle Θ of the head, for example. Here, the rotation angle Θ of the head is calculated with reference to the rotation angle Θ of the neck connected to the head. The rotation angle Θ of the neck is a rotation angle based on the rotation angle Θ of the body i. That is, the rotation angle Θ of the head is calculated by a predetermined calculation based on the rotation angle Θ of other parts. For this reason, if more part information (for example, from the left foot part and the right foot part to the head part) can be obtained, a highly accurate rotation angle Θ can be obtained, and as a result, the accuracy of the normal vector can be obtained. Will also improve. The position of the person's eyes can be estimated from the person's head. Alternatively, when the part information of the person's head indicates the three-dimensional position coordinates of each of a plurality of parts on the surface of the person's head, and the position of a specific part of the face such as the nose can be specified, The eye position can be estimated from a specific part of the face such as the nose. In this case, the rotation angle ψ described above can also be estimated. Alternatively, the position of the person's eyes can be estimated by detecting the face of the person from the RGB image acquired from the camera 3. For detecting a human face, for example, a Viola-Jones detection algorithm, an algorithm such as face part (eyes, nose, mouth) detection or skin color detection is used. Alternatively, the position of the person's eyes may be estimated from the position of the personal belongings such as the microphone M. It should be noted that a vector between the position of the head and the position of the left shoulder and a vector between the position of the head and the position of the right shoulder using the part information of the head, the left shoulder, and the right shoulder. The normal vector may be calculated by obtaining the outer product of.

  Next, the control unit 63 specifies the field of view of the person from the position of the head of the specified person, the posture of the person, and the effective viewing angle. Here, the visual field of a person is determined from a horizontal visual field and a vertical visual field. As shown in FIGS. 2A and 2B, the horizontal field of view and the vertical field of view each have a stable field of view determined by a stable field of view and an effective field of view determined by an effective field of view with respect to the direction of the line of sight of the person. And a discrimination field determined by the discrimination field angle. In general, the effective viewing angle is preferably 30 degrees for a horizontal field of view and 20 degrees for a vertical field of view, for example. A person's field of view changes depending on the posture of the person's head. For example, when the person looks up upward as shown in FIG. 2C, when the person looks down downward as shown in FIG. 2D, when the person faces left or right in the horizontal direction, Changes when the person tilts his head. Then, as shown in FIG. 2F, the visual field range projected on the projection plane is a range that fits in the horizontal visual field and vertical visual field of the person. In the present embodiment, the control unit 63 determines the distance from the position of the person's head placed in the virtual three-dimensional space to the projection plane based on the part information of the person's head, the posture of the person, and the effective viewing angle. From this, the visual field range of the person on the projection plane in the virtual three-dimensional space is specified. The visual field range of such a person becomes wider as the distance from the position of the head of the person arranged in the virtual three-dimensional space to the projection plane becomes longer. In the example of FIG. 3A, the position P1 of the person's head has a longer distance to the projection plane than the position P2 of the person's head. Therefore, the projection range H1 corresponding to the position P1 is wider than the projection range H2 corresponding to the position P2. In addition, the visual field range of the person changes according to the change in the gaze direction of the person. In the example of FIG. 3B, the line-of-sight direction from the person position P1 changes to the line-of-sight direction from the person position P3. For this reason, the projection range H1 corresponding to the position P1 has moved to the projection range H3 corresponding to the position P3.

  Next, the control unit 63 positions, postures, and sizes (scales) of the lyric objects arranged in at least a partial area of the projection plane on which the images are projected from the projectors 5a to 5c within the specified visual field range of the person. ). Here, the projection surfaces on which the images are projected from the projectors 5a to 5c are recognized by the control unit 63 as projection surfaces in a virtual three-dimensional space. The position of the lyric object is determined as, for example, the three-dimensional position coordinate of the center of the visual field range specified on the projection plane as described above. The posture of the lyric object is determined, for example, as a normal vector extending vertically from the surface of the lyric object (that is, the projection surface on which the lyric object is arranged). Note that the posture of the lyrics object is not necessarily determined. In other words, only the position and size of the lyrics object may be determined. The size of the lyric object differs depending on the shape of the lyric object. The shape of the lyrics object is not particularly limited, but is set in advance to, for example, a rectangle, a circle, an ellipse, or the like. For example, when the shape of the lyric object is rectangular, the width and height of the lyric object are determined as the size so that the lyric object is within the visual field range. Further, for example, when the shape of the lyric object is circular, the radius of the lyric object is determined as the size so that the lyric object is within the visual field range. Note that the lyrics object does not have to be completely within the visual field range so that the lyrics object can be within the visual field range, and the lyrics displayed in the lyrics object need only be within the visual field range.

  Next, the control unit 63 synthesizes the lyrics object on the video projected by the projectors 5a to 5c according to the determined position, posture, and size. Here, the images projected by the projectors 5a to 5c are images instructed to be projected. For example, as shown in FIG. 4A, the control unit 63 virtually projects the video V onto the projection plane in the virtual three-dimensional space, and the determined position and orientation on the projected video V. The lyric object is synthesized on the video V by arranging (for example, superimposing) the lyric object OB1 of the size. In this composition, the lyric object may be deformed according to the video V.

  Next, the control unit 63 transmits the video data of the video V combined with the lyric object OB1 and the projection instruction to the video playback devices 4a to 4c, so that such video V is actually projected by the projectors 5a to 5c. Project onto a surface. As a result, it is possible to accurately project (display) the lyric object at a position in the gaze direction of the person in the projected image while maintaining visibility. In the present embodiment, the video V is projected by the three projectors 5a to 5c. Therefore, the control unit 63 generates three videos projected by the projectors 5a to 5c from the video V in which the lyrics object OB1 is synthesized. In addition, the video V in which the lyric object OB1 is synthesized is three-dimensional data represented by a three-dimensional coordinate system. For this reason, as shown in FIG. 4B, the control unit 63 converts the three images (three-dimensional data) corresponding to the projectors 5a to 5c to the positions of the projectors 5a to 5c (that is, in the three-dimensional space). Then, the image data of each image is transmitted to each of the image reproducing devices 4a to 4c. For example, the control unit 23 projects and converts the coordinates of the video in the three-dimensional coordinate system into the coordinates of the two-dimensional planar coordinate system of the projectors 5a to 5c using, for example, a matrix for projective transformation.

  By the way, the visual field range of a person changes according to a change in the position of the person's head or the posture of the person. Therefore, the control unit 63 may specify the person's visual field range at the predetermined time interval from the position of the person's head, the posture of the person, and the effective viewing angle specified at the predetermined time interval. In this case, each time the person's visual field range is specified, the control unit 63 newly determines the position, posture, and size of the lyrics object within the newly specified person's visual field range. Then, the control unit 63 synthesizes the lyric object on the video V with the newly determined position, posture, and size instead of the already synthesized lyric object OB1. Thereby, the projection position of the lyric object projected by the projectors 5a to 5c changes according to the change in the visual field range of the person. That is, the lyric object moves on the image projected on the projection plane. However, depending on the person, the head may always be moved. In this case, if the projected position of the projected lyric object changes according to the change in the visual field range of the person, that is, if the lyric object moves, the person is moved. On the contrary, it becomes difficult to see the lyrics. For this reason, it is preferable to use a moving average of the lyric object so that a slight movement of the person is not reflected in the lyric object. In this case, as will be described later, the control unit 63 arranges the lyric object on the video V in accordance with the average value of the values corresponding to the position, posture, and size of the lyric object for a plurality of times determined at predetermined time intervals. become. Thereby, the visibility of the lyrics can be improved by using the moving average of the lyrics object.

[2. Projection Control Processing of Reproduction Control Device 6]
Next, the projection control process of the reproduction control apparatus 6 of the present embodiment will be described separately for Example 1 and Example 2. In the following description, before or after the start of the projection control process, the playback control device 6 receives from the display / operation device 1 the music ID and playback instruction of the music selected in accordance with the operation instruction from the user. The received music ID and playback instruction are transmitted to the music playback device 2 so that the music playback device 2 plays back the music for which playback has been instructed.

Example 1
First, the projection control process of the first embodiment will be described with reference to FIG. When the control unit 63 of the reproduction control device 6 receives, for example, the video ID and the projection instruction of the video V selected according to the selection instruction from the user who entered the room from the display / operation device 1, the control unit 63 illustrated in FIG. The projection control process is started. In this way, the video selected by the user is the projection target. Alternatively, the video V to be projected may be a video corresponding to a music for which playback is instructed.

  When the projection control process shown in FIG. 5 is started, a queue for holding the position, posture and size of the lyrics object is generated in a predetermined area of the RAM. Here, the queue means a data structure (in other words, a table) that holds N (N is a natural number of 2 or more) pieces of data in a first-in first-out (First In First Out) system. In the case of the present embodiment, a set of the position, orientation and size of the lyrics object is held in the queue as one data.

  Next, the control unit 63 acquires a depth image in the three-dimensional space within the imaging range of the camera 3 from the camera 3, and the video V to be projected on the projection plane in the virtual three-dimensional space based on the acquired depth image. Is projected (step S1). Next, the control unit 63 determines whether or not to end the projection control process (step S2). For example, when receiving an end instruction from the display / operation device 1, the control unit 63 determines to end the projection control process (step S <b> 2: YES), and ends the projection control process illustrated in FIG. 5. On the other hand, when it determines with the control part 31 not complete | finishing a projection control process (step S2: NO), it progresses to step S3.

  In step S <b> 3, the control unit 63 acquires human part information in a three-dimensional space within the imaging range of the camera 3. Next, as described above, the control unit 63 acquires the position and posture of the person's head from the part information and the like of the person acquired in step S3 (step S4). Next, the control unit 63 converts the position and posture of the person's head acquired in step S4 into values in the virtual three-dimensional space, thereby making the person's head (for example, the eyes in the virtual three-dimensional space). ) And the above-mentioned effective viewing angle (step S5). And the control part 63 specifies a visual field range from the position and attitude | position of a person's head, and an effective viewing angle.

  Next, the control unit 63 determines whether there is a lyrics to be displayed at the current time (step S6). For example, the control unit 63 determines whether or not the music is being played by the music playback device 2. If the music is being played, the control unit 63 acquires the lyrics object data corresponding to the lyrics of the music being played from the storage unit 62. The control unit 63 refers to the display timing data of the acquired lyrics object data, and determines that there is a lyrics to be displayed at the current time when the current time is the display timing indicated by the display timing data (step S6: YES). The process proceeds to step S7. That is, when the display timing associated with the lyrics object OB1 that displays the lyrics indicated by the lyrics data of the acquired lyrics object data has arrived, the control unit 63 combines the lyrics object OB1 on the video V in the subsequent processing. Then, the video V synthesized with the lyrics object OB1 is projected onto the projection plane. Thereby, the video V in which the lyrics object OB1 is synthesized can be effectively projected onto the projection plane at the display timing associated with the lyrics object OB1. On the other hand, when it is determined that there is no lyrics to be displayed at the current time (step S6: NO), the control unit 63 proceeds to step S13.

  In step S7, the control unit 63 creates a virtual three-dimensional space within the human field of view specified by the position of the person's head (for example, eyes), the posture of the person specified in step S5, and the effective viewing angle. The position, posture, and size (scale) of the lyric object arranged on the projection plane in are determined. That is, the position, posture, and size of the lyric object that are most appropriate for the position are determined in a virtual three-dimensional space from the position of the person's head (for example, the eyes), the posture of the person, and the effective viewing angle. Next, the control unit 63 adds the set of the position, posture, and size determined in step S7 to the queue as one data (step S8).

  Next, the control unit 63 determines whether or not the number of data held in the queue is greater than N (step S9). When it is determined that the number of data held in the queue is greater than N (step S9: YES), the control unit 63 deletes the oldest data from the queue (step S10), and proceeds to step S11. On the other hand, when it is determined that the number of data held in the queue is not greater than N (step S9: NO), the control unit 63 proceeds to step S11.

  In step S11, the control unit 63 determines the average value of the data held in the queue as the current position, posture, and size of the lyrics object. The average value of data held in the queue is an average value of values corresponding to the position, posture, and size in the queue. That is, the moving average of the lyrics object is calculated. Next, the control unit 63 generates a lyrics object OB1 (for example, an image) that displays the lyrics indicated by the lyrics data of the acquired lyrics object data at the position, posture, and size determined in step S11. As shown, the generated lyric object OB1 is synthesized on the video V projected on the projection plane in the virtual three-dimensional space (step S12), and the process proceeds to step S16.

  On the other hand, in step S13, the control unit 63 determines whether there is a lyrics object OB1 arranged on the video V projected on the projection plane in the virtual three-dimensional space. When it is determined that there is the lyrics object OB1 arranged on the video V (step S13: YES), the control unit 63 deletes the lyrics object OB1 arranged on the video V from the virtual three-dimensional space ( The process proceeds to step S14) and step S15. On the other hand, when the control unit 63 determines that there is no lyrics object OB1 arranged on the video V (step S13: NO), the control unit 63 proceeds to step S15. In step S15, the control unit 63 empties the queue by deleting all data held in the queue, and proceeds to step S16.

  In step S16, the control unit 63 projects and converts the video V projected in the virtual three-dimensional space into a two-dimensional plane coordinate system based on the positions of the projectors 5a to 5c, and converts the video data of each video. The video data is transmitted to each of the video reproduction devices 4a to 4c, and the process returns to step S2. When the lyrics object OB1 is synthesized with the video V in step S12, the video V with the lyrics object OB1 synthesized is projected onto the projection plane by the projectors 5a to 5c.

  According to the first embodiment, the control unit 63 specifies the person's head position, posture, and effective viewing angle at a predetermined time interval (that is, a time interval based on the loop period of steps S2 to S16), and specifies the specified person. The position, posture and size of the lyrics object OB1 are determined at predetermined time intervals (determined in step S7) within the visual field range of the person specified from the head position, posture, and effective viewing angle. Then, the control unit 63 synthesizes the lyric object OB1 on the video V in accordance with the average value of the values corresponding to the determined positions, postures, and sizes, and the video V in which the lyric object OB1 is synthesized is the projector 5a. Project to the projection surface by ~ 5c. Thus, the visibility of the lyrics can be improved by using the moving average of the lyrics object OB1.

(Example 2)
Next, a projection control process according to the second embodiment will be described with reference to FIG. In the first embodiment, the visibility of lyrics can be improved by using a moving average of lyrics objects. However, in the projection control process of the first embodiment, when the visual field range of the person changes greatly in a short time (for example, within 1 second), the lyrics object cannot follow a sudden movement, and the movement of the lyrics object becomes slow. End up. On the other hand, when the person's field of view changes greatly in a short time, if the display position of the lyric object is switched immediately, the lyric object is displayed and hidden even if the person moves his gaze for a moment. On the other hand, the visibility of the lyrics may be reduced.

  In order to improve such a point, in the second embodiment, when the change amount of the value corresponding to at least one of the position, posture, and size of the lyrics object is equal to or larger than the threshold value, or the position and posture of the person When the amount of change in the value corresponding to at least one of the values is equal to or greater than the threshold value, a new lyric object is generated, and the generated new lyric object is synthesized on the video together with the already generated lyric object. That is, in the second embodiment, a plurality of cues described in the first embodiment can be generated, and lyric objects are generated and projected by the number of queues. Further, each generated lyric object is associated with a life parameter that is a projection continuation time from the synthesis to the video until the deletion. In the projection control process described below, a change amount of a value corresponding to at least one of the position, posture, and size of the lyrics object will be described as an example.

  The projection control process shown in FIG. 6 is started when the video ID of video V and the projection instruction are received from the display / operation device 1 as in the first embodiment. When the projection control process shown in FIG. 6 is started, the control unit 63 projects the video V to be projected onto the projection plane in the virtual three-dimensional space (step S21). Next, the control unit 63 determines whether or not to end the projection control process (step S22). If it is determined not to end the projection control process (step S22: NO), the control unit 63 proceeds to step S23.

  In step S <b> 23, the control unit 63 acquires human part information in a three-dimensional space within the imaging range of the camera 3. Next, the control unit 63 acquires the position and posture of the person's head from the part information and the like of the person acquired in step S23 (step S24). Next, the control unit 63 converts the position and posture of the person's head acquired in step S24 into the values in the virtual three-dimensional space, so that the position and posture of the person's head in the virtual three-dimensional space. And the above-mentioned effective viewing angle is specified (step S25). And the control part 63 specifies a visual field range from the position and attitude | position of a person's head, and an effective viewing angle.

  Next, the control unit 63 determines whether or not there is a lyrics object arranged on the video V in the previous process (step S26). When it is determined that there is a lyrics object arranged on the video V in the previous process (step S26: YES), the control unit 63 proceeds to step S27. On the other hand, when it is determined that there is no lyrics object arranged on the video V in the previous process (step S26: NO), the control unit 63 proceeds to step S28. In step S27, the control unit 63 deletes all the lyrics objects arranged on the video V in the previous process from the virtual three-dimensional space, and proceeds to step S28.

  In step S28, the control unit 63 determines whether there is a lyrics to be displayed at the current time. When it is determined that there is lyrics to be displayed at the current time (step S28: YES), the control unit 63 proceeds to step S29. On the other hand, when it is determined that there is no lyrics to be displayed at the current time (step S28: NO), the control unit 63 proceeds to step S47.

  In step S29, the control unit 63 creates a virtual three-dimensional space within the field of view of the person identified from the position of the person's head (for example, eyes) identified in step S25, the posture of the person, and the effective viewing angle. The position, posture, and size (scale) of the lyric object arranged on the projection plane in are determined. The position of the lyrics object is determined from the coordinates of the person's head, for example. The posture of the lyrics object is determined from the posture of the person, for example. The size (scale) of the lyrics object is determined from, for example, the effective viewing angle of the person. These may be determined in advance by a table in which the position, posture, and size (scale) of the lyrics object are determined in accordance with the position of the person's head, the posture of the person, and the effective viewing angle.

  Next, the control unit 63 determines whether there are one or more queues (step S30). If it is determined that there is one or more queue (step S30: YES), the control unit 63 proceeds to step S31. On the other hand, if it is determined that there is no queue (step S30: NO), the control unit 63 proceeds to step S34.

  In step S31, the control unit 63 calculates the average value of data held in the queue for each queue, and proceeds to step S32. The average value of data held in the queue is an average value of values corresponding to the position, posture and size in the queue. Note that the average value of the data held in the queue may be an average value with a value corresponding to at least one of the position, posture, and size in the queue.

  In step S32, the control unit 63 calculates, for each queue, an error between a value corresponding to at least one of the position, posture, and size determined in step S29 and the average value calculated in step S31. This error is the amount of change in the position, posture, and size of the lyrics object since the previous process. For example, the control unit 63 uses the square root of the sum of the square of the value difference corresponding to the position, the square of the value difference corresponding to the posture, and the square of the value difference corresponding to the size as the error. It is good to calculate.

  Next, the control unit 63 determines whether there is a queue in which the error calculated in step S32 is less than a threshold value (or may be equal to or less than the threshold value) (step S33). When it is determined that there is no queue whose error is less than the threshold (step S33: NO), the control unit 63 proceeds to step S34. That is, if there is a queue whose error is greater than or equal to the threshold, the process proceeds to step S34. Here, the error being equal to or greater than the threshold means that the amount of change from the previous process of the value corresponding to at least one of the position, orientation, and size of the lyrics object is equal to or greater than the threshold. In other words, the error being equal to or greater than the threshold means that the amount of change in the value corresponding to at least one of the position and posture of the person is equal to or greater than the threshold. In such a case, for example, as shown in FIG. 7A, in addition to the lyrics object OB1 (an example of the first display object) already arranged on the video V, a new lyrics object OB2 (second Of the display object) is generated in the processing after step S34, and the generated new lyrics object OB2 is synthesized on the video V together with the already generated lyrics object OB1. Accordingly, even when the visual field range of a person changes greatly in a short time, it is possible to quickly cope with the changed visual field range and improve the lyrics visibility in the changed visual field range. On the other hand, when the control unit 63 determines that there is a queue whose error is equal to or smaller than the threshold (step S33: YES), the control unit 63 proceeds to step S36.

  In step S34, the control unit 63 generates a new queue and sets an initial value based on the current time (for example, a time several seconds after the current time) in the lifetime parameter of the generated queue. Here, the generated queue is the first queue or the second and subsequent queues. Next, the control unit 63 sets the queue generated in step S34 as a queue to be processed this time (step S35), and proceeds to step S39.

  On the other hand, in step S36, the control unit 63 determines whether or not there are a plurality of queues in which the error calculated in step S32 is less than a threshold value (may be equal to or less than the threshold value). If the control unit 63 determines that there are not a plurality of queues with errors less than the threshold (that is, there is only one) (step S36: NO), the control unit 63 proceeds to step S37. On the other hand, if the control unit 63 determines that there are a plurality of queues whose error is less than the threshold (step S36: YES), the control unit 63 proceeds to step S38.

  In step S37, the control unit 63 sets a queue whose error is less than the threshold value as a queue to be processed this time, and proceeds to step S39. In step S38, the control unit 63 sets the queue with the smallest error among the plurality of queues with the error less than the threshold as the queue to be processed this time, deletes queues other than the queue with the smallest error, and proceeds to step S39. move on.

  In step S39, the control unit 63 adds the set of the position, posture, and size determined in step S29 as one data to the current processing target queue. Next, the control unit 63 determines whether or not the number of data held in the queue to be processed this time is greater than N (step S40). When it is determined that the number of data held in the current processing target queue is greater than N (step S40: YES), the control unit 63 deletes the oldest data from the current processing target queue (step S41), and the step Proceed to S42. On the other hand, if the control unit 63 determines that the number of data held in the queue to be processed this time is not greater than N (step S40: NO), the control unit 63 proceeds to step S42.

  In step S42, the control unit 63 extends the lifetime parameter of the queue to be processed this time by a certain time. Here, the lifetime parameter of the queue to be processed this time is a lifetime parameter associated with the lyrics object generated according to the data in the queue to be processed this time. Note that it is desirable that the queues whose life parameter is to be extended are limited to queues whose error determined in step S33 continues for a predetermined time or more and is less than the threshold value. As a result, the longer the visual field range in which the person's line of sight stays, the longer the lyric object can be displayed in this visual field range, so that the visibility of the lyrics can be further improved. In other words, as the visual field range in which the person's line of sight stays is shorter, the lyrics object in this visual field range can be quickly deleted by the processing in step S44 described later. Further, the certain time is set according to one processing time from step S22 to step S48 in the processing shown in FIG. 6, for example. For example, when the processing time for one time is 0.02 seconds, the certain time is set to about 0.02 seconds.

  Next, the control unit 63 determines whether there is any queue other than the current processing target queue that has reached the time indicated by the lifetime parameter (step S43). When it is determined that there is a queue at which the time indicated by the life parameter has arrived (step S43: YES), the control unit 63 deletes the queue at which the time indicated by the life parameter has arrived (step S44), and proceeds to step S45. That is, for example, as shown in FIG. 7A, the control unit 63, as shown in FIG. 7B, is a predetermined time after the video V in which the lyrics object OB1 and the lyrics object OB2 are combined is projected. Then, the lyrics object OB1 is deleted from the video. As a result, even if the person's visual field range changes greatly in a short time and then further changes, the lyrics object in the visual field range that is less likely to be visually recognized by the person can be quickly deleted. Moreover, even if the number of lyrics objects increases due to frequent changes in the field of view of the person, it can be appropriately reduced over time by the life parameter. On the other hand, when the control unit 63 determines that there is no queue at which the time indicated by the lifetime parameter has arrived (step S43: NO), the control unit 63 proceeds to step S45.

  In step S45, the control unit 63 determines the average value of the data held in the queue for each queue as the current position, orientation, and size of the lyrics object. That is, the position, posture, and size of the current lyrics object are determined for each of all the cues that exist at this time. Next, the control unit 63 generates a lyric object for each cue (that is, as many as the number of cues) with the position, posture, and size determined in step S45, and the image projected on the projection plane in the virtual three-dimensional space. The generated lyric object is synthesized on V (step S46), and the process proceeds to step S48.

  On the other hand, in step S47, the control unit 63 deletes all the cues and the corresponding lyric objects, and proceeds to step S48. In step S48, the control unit 63 projects and converts the video V projected in the virtual three-dimensional space into a two-dimensional plane coordinate system based on the positions of the projectors 5a to 5c, and converts the video data of each video. The video is transmitted to each of the video reproduction devices 4a to 4c, and the process returns to step S22. If the lyrics object is combined with the video V in step S46, the video V with the combined lyrics object is projected onto the projection plane by the projectors 5a to 5c.

  According to the second embodiment, when the amount of change in the value corresponding to at least one of the position, posture, and size of the lyrics object OB1 is equal to or greater than the threshold value, or at least one of the position and posture of the person's head When the change amount of the value corresponding to is equal to or greater than the threshold, the control unit 63 combines the lyrics object OB1 with the lyrics object OB2 different from the lyrics object OB1 at the latest position and size determined in step S29. The video V synthesized on the video V and the lyrics object OB1 and the lyrics object OB2 synthesized is projected onto the projection plane by the projectors 5a to 5c. For this reason, when the moving average of the lyrics object is used, even if the person's field of view changes greatly in a short time, it can quickly respond to the changed field of view, and in the changed field of view The visibility of the lyrics object can be improved.

  In Example 2, after the position, posture, and effective viewing angle of the person's head are specified in step S25, the position, posture, and size of the lyrics object are determined in step S29, and then stored in the queue. It is determined whether or not there is a queue whose error is less than a threshold value from the stored data, and processing for generating a new queue is performed. However, after the position, posture and effective viewing angle of the person's head are specified, a new cue is created from the data stored in the queue without determining the position, posture and size of the lyrics object in step S29. It may be determined whether or not to generate, and a process of generating a new queue according to the determination result may be performed.

  As described above, according to the above embodiment, the playback control device 6 identifies and identifies the person's visual field range from the position, posture, and effective viewing angle of the person's head in the predetermined three-dimensional space. Determine the position, posture and size of the lyrics object placed in at least a part of the projection plane on which the image is projected within the human field of view from the position of the person's head, the posture of the person, and the effective viewing angle. . Then, the playback control device 6 is configured to synthesize the lyric object on the image projected by the projectors 5a to 5c with the determined position, posture and size, and to project the synthesized image of the lyric object on the projection plane. As a result, a person who is a viewer can accurately display a lyric object at a position in the line-of-sight direction according to the position and posture of the person on the projected image without wearing a projector. Can do. Further, in this embodiment, since the viewer does not need to wear the projector, for example, it is possible to prevent characters and the like from being difficult to see due to blurring due to the movement of the viewer's head.

  In the above embodiment, the lyric object has been described as an example of the display object. However, the present invention can be applied to a display object other than the lyric object, such as a message or a comment.

DESCRIPTION OF SYMBOLS 1 Display / operation device 2 Music playback device 3 Cameras 4a-4e Video playback devices 5a-5e Projector 6 Playback control device S Projection system

Claims (8)

A specifying means for specifying the field of view of the person from the position, posture, and effective viewing angle of the person's head in a predetermined three-dimensional space;
The first unit is arranged in at least a partial region of the projection plane onto which an image is projected from the projection device within the field of view of the person identified from the position, posture, and effective viewing angle of the person's head by the identifying unit. Determining means for determining the position and size of the display object;
Synthesizing means for synthesizing the first display object on the image projected by the projection device at the position and size determined by the determining means;
Projection control means for projecting an image in which the first display object is combined onto the projection plane;
A projection control apparatus comprising: After the video in which the first display object is synthesized is projected, the specifying unit newly specifies the human visual field range from the position, posture, and effective visual angle of the human head,
The determination means includes a projection plane on which an image is projected from the projection device within the field of view of the person specified from the position, posture, and effective viewing angle of the person newly specified by the specification means. Determining a new position and size of the second display object arranged in at least a part of the area;
The synthesizing unit may be configured such that a change amount of a value corresponding to at least one of the position, posture, and size of the first display object is a threshold value or more, or at least one of the position and posture of the person's head If the change amount of the value corresponding to one of the two is greater than or equal to a threshold value, the second display object is combined with the first display object at the position and size newly determined by the determination means. Synthesized on top,
The projection control apparatus according to claim 1, wherein the projection control unit projects an image obtained by combining the first display object and the second display object onto the projection plane. The specifying means specifies the person's visual field range from the position, posture, and effective viewing angle of the person at predetermined time intervals,
The determining means includes at least one of projection surfaces onto which an image is projected from the projection device within the field of view of the person identified from the position, posture, and effective viewing angle of the person's head identified by the identifying means. Determining a position and a size of the first display object arranged in a region of a plurality of times at predetermined time intervals,
The synthesizing unit synthesizes the first display object on an image projected by the projection device according to an average value of values corresponding to the position and size for a plurality of times determined by the determining unit,
The projection control apparatus according to claim 1, wherein the projection control unit projects an image obtained by combining the first display objects onto the projection plane. After the video in which the first display object is synthesized is projected, the specifying unit newly specifies the human visual field range from the position, posture, and effective visual angle of the human head,
The synthesizing unit may be configured such that a change amount of a value corresponding to at least one of the position, posture, and size of the first display object is a threshold value or more, or at least one of the position and posture of the person's head When the amount of change in the value corresponding to one of the values is equal to or greater than a threshold value, the second display object different from the first display object at the latest position and size determined by the determination unit is the first display object. The display object is synthesized on the synthesized video,
The projection control apparatus according to claim 3, wherein the projection control unit projects an image obtained by combining the first display object and the second display object onto the projection plane. A deletion unit that deletes the first display object from the video after a predetermined time from the projection of the video in which the first display object and the second display object are combined;
5. The projection according to claim 2, wherein the projection control unit causes the projection surface to project an image in which the first display object is deleted and the second display object is combined. 6. Control device. The first display object is associated with a projection duration from when it is combined with the video to when it is deleted,
Deletion means for deleting the first display object from the video after the projection duration time after the video in which the first display object is synthesized is projected;
After an image in which the first display object is synthesized is projected, a change amount of a value corresponding to at least one of the position, posture, and size of the first display object or the three-dimensional space The projection duration associated with the first display object when the amount of change in the value corresponding to at least one of the head position and posture of the person is less than the threshold for a predetermined time or longer. Extending means for extending
The projection control apparatus according to claim 1, further comprising: Projection timing onto the projection plane is associated with the first display object,
The synthesizing unit synthesizes the first display object on the video when the projection timing arrives,
The projection control apparatus according to claim 1, wherein the projection control unit projects an image in which the first display object is synthesized at the projection timing onto the projection plane. A specific step of identifying the person's visual field range from the position, posture, and effective visual angle of the person's head in a predetermined three-dimensional space;
Arranged in at least a partial area of the projection plane on which the image is projected from the projection device within the visual field range of the human specified by the position, posture, and effective viewing angle of the human head specified by the specifying step. A determining step for determining the position and size of the first display object;
Combining the first display object on the image projected by the projection device at the position and size determined by the determining step;
A projection control step of projecting an image on which the first display object is synthesized on the projection plane;
A program that causes a computer to execute.