JP2013037420A - Content display system, content display control method, content display control device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a content display system and the like capable of preventing human errors and achieving display control having high entertainment property.SOLUTION: A control device 5 stores content components, locus information, and control commands in association with each other in a control command locus DB 51. Then the control device 5 performs the steps of: initializing each DB (S21); acquiring a taken image from an imaging device 3 (S22); extracting a specific area and projecting it on an image to be processed (S23); extracting a coordinate of a bright spot in the image to be processed (S24); extracting a locus of light of a laser light oscillation device 2 (S25); and checking whether the extracted locus corresponds to any of the control command by referring to the control command locus DB 51. When the locus corresponds to a control command (YES in S26), the control device 5 executes display control of content according to the corresponding control command (S27).

Description

  The present invention relates to a content display system and the like in which an operator instructs display control of content by irradiating an object in real space with light from a laser light oscillation device.

  Conventionally, in a presentation using a computer and a projector, an operator uses a laser light oscillation device such as a laser pointer to indicate a specific position in content (for example, a moving image or a still image) projected by the projector. Has been. Since the laser beam oscillation device serves only as an indicator rod, the operator must separately operate an input device connected to a computer in order to control content display. In order to avoid such a complicated operation, a system capable of instructing display control of content by a laser light oscillation device is desired.

  For example, the interactive presentation control system disclosed in Patent Document 1 includes a display computer, a computer-controlled image projector, a laser pointer, a projection screen, a digital camera and a control module, and the display computer generates an electronic image. The image is projected as a presentation image on the projection screen. The lecturer draws a predetermined gesture space pattern on the presentation image using a laser pointer, the digital camera acquires these presentation images, and the processing unit in the control module analyzes the image to identify the gesture space pattern. . It is compared to a predetermined set of gesture space patterns, a match is found, and then the display command associated with it is selected.

  Further, in the projector system disclosed in Patent Document 2, an arbitrary position on the projector screen is designated by laser light irradiation with a light pen. The personal computer detects that the designation at the same position continues for a certain time or longer based on the image taken by the light receiver. After a certain period of time has elapsed, the personal computer generates an ON flag signal corresponding to a mouse click button operation.

  Patent Document 3 also discloses a laser pointer for aiming at a part of an exhibit and forming a laser figure, an imaging means for photographing the exhibit on which the laser figure is formed, and generating photographed image data, and photographing. Matching image data with data for graphic recognition, recognizing a laser graphic and creating attribute information of the laser graphic, graphic coordinate calculating means for calculating the aiming coordinates of the laser graphic, aiming coordinates, With the figure attribute information as a clue, from the exhibit information, provided information acquisition means for reading the explanation information of the exhibit part associated with the laser figure, and the information explaining the part of the exhibit are superimposed on the exhibit image, An exhibit information providing system comprising an information management device comprising provision information display means for displaying is disclosed.

Japanese Patent Laid-Open No. 2001-125738 JP 2008-225556 A JP 2010-224646 A

  However, in Patent Document 1, since the gesture space pattern and the display command are associated with each other regardless of the content, when the presentation is performed, unintended display control can be performed due to a human error. High nature. For example, if the operator unintentionally shakes the hand holding the laser pointer, even if the operator is irradiating the laser pointer light only to point to a specific position, some gesture space It is determined that the pattern matches, and unintended display control (for example, “go to the next image”) is performed.

  In Patent Document 2, only an operation of pressing a mouse click button and an operation of releasing it are realized. Therefore, the dynamic operation that the viewer is surprised cannot be realized, and the entertainment property is poor. Further, similarly to Patent Document 1, since a flag signal corresponding to a mouse click button operation is generated regardless of the content, there is a high possibility that unintended display control is performed due to a human error.

  On the other hand, in Patent Document 3, since the information to be displayed is associated with the aiming area information indicating a specific part of the exhibit and the graphic attribute information of a different laser figure for each laser pointer, if the laser pointer is correctly selected, The possibility of unintended display control is low. However, as the operation of the operator, the aiming area information is merely irradiated with the light of the laser pointer, and the dynamic operation that is surprising to the viewer cannot be realized, and the entertainment property is poor.

  The present invention has been made in view of the above-described problems, and the object of the present invention is to provide a content display system that can suppress human errors and realize display control rich in entertainment properties. Is to provide.

  In order to achieve the above-mentioned object, a first invention is a content display system in which an operator instructs display control of content by irradiating an object in real space with light of a laser light oscillation device. The laser light oscillation device, a photographing device for photographing a region irradiated with light from the laser light oscillation device, a content display device for displaying the content, a control device for controlling the photographing device and the content display device, The control device is associated with trajectory information for identifying the trajectory of light of the laser light oscillation device for each content component constituting the content, and a control command for realizing specific display control. First storage means for storing; acquisition means for acquiring a photographed image photographed by the photographing apparatus; An extraction means for extracting a light trajectory of the optical oscillation device, a determination means for determining the corresponding control command with reference to the first storage means for the light trajectory extracted by the extraction means, and the determination A content display system comprising: content display control means for performing display control of the content based on the control command determined by the means. According to the first invention, appropriate display control according to the content can be performed, human error can be suppressed, and display control rich in entertainment can be realized.

  The trajectory information in the first invention comprises a set of a determination time and a determination region, and the determination means is configured such that an irradiation time and an irradiation position of a bright spot constituting a light trajectory extracted by the extraction means is the trajectory. The control command is determined in accordance with the determination time in the information and whether it is included in the determination area, and the control device superimposes an image indicating the position of the determination area on the image of the content component. It is desirable to further include trajectory information display control means for displaying on any display device including the content display device. As a result, the content producer and operator can visually confirm the determination area during the display control test, and the content production work and confirmation work are facilitated.

  The control device according to the first aspect of the invention preferably further comprises trajectory information update means for accepting a change in the position of the determination area and updating the contents of the first storage means. As a result, the content producer or operator can easily correct the trajectory information determined to be prone to human error as a result of the display control test. In particular, by visually displaying and adjusting the trajectory determination under the execution environment, it is possible to make adjustments in accordance with the operation of the laser light oscillation device by the actual operator.

  Further, the control device according to the first aspect of the invention is a second storage means for storing the coordinate information of the corresponding point in the coordinate space in the internal processing of the control device in association with the coordinate information of the feature point in the captured image. , Generating a specific area image from the captured image, extracting feature points from the specific area image, and mapping the feature points to corresponding points in the coordinate space in the internal processing of the control device with reference to the second storage unit Projecting means for projecting the specific area image to generate a processed image, and the extracting means extracts the coordinates of the bright spot from the plurality of processed images, and extracts the coordinates of the extracted coordinates. It is desirable that the set be a locus of light of the laser light oscillation device. Thus, the shooting direction of the shooting device and the projection direction of the content display device are not restricted, and the content display system can be applied to real spaces of various environments.

  The control device according to the first aspect of the present invention further comprises third storage means for storing shooting parameters of the shooting apparatus in accordance with an execution environment, and the acquisition means refers to the third storage means. It is desirable to acquire the captured image. This eliminates the need for some image processing procedures by the control device and improves the processing efficiency of the control device. As a result, the response performance of the content display system can be improved.

  According to a second aspect of the present invention, in the content display system in which the operator instructs the display control of the content by irradiating the object in the real space with the light of the laser light oscillation device, the light of the laser light oscillation device is irradiated. Control the content display control device that controls the image capturing device that captures the area to be displayed and the content display device that displays the content, and identifies the trajectory of the laser light oscillation device for each content component constituting the content A content display control method by a control device including a first storage unit that stores a control command for realizing specific display control in association with trajectory information to be acquired, and acquiring a photographed image photographed by the photographing device An extraction step of extracting a light trajectory of the laser light oscillation device from the captured image; With respect to the light trajectory extracted in the exiting step, the first storage means is referred to, a determination step for determining the corresponding control command, and the display of the content based on the control command determined in the determination step And a content display control step for performing control. According to the second invention, it is possible to perform appropriate display control according to the content, suppress human error, and realize display control rich in entertainment properties.

  According to a third aspect of the present invention, in the content display system in which the operator instructs the display control of the content by irradiating the object in the real space with the light of the laser light oscillation device, the light of the laser light oscillation device is irradiated. A content display control device that controls a photographing device that captures a region to be displayed and a content display device that displays the content, and identifies a light locus of the laser light oscillation device for each content component constituting the content First storage means for storing a control command for realizing specific display control in association with the trajectory information, acquisition means for acquiring a photographed image photographed by the photographing apparatus, and laser light oscillation from the photographed image An extraction means for extracting a light trajectory of the apparatus, and the light path extracted by the extraction means with respect to the first memory , And a content display control unit that performs display control of the content based on the control command determined by the determination unit. It is a content display control device. According to the third invention, the content display system of the first invention can be configured, and the content display method of the second invention can be executed.

  A fourth invention is a program for causing a computer to function as the content display control device of the third invention. The content display control apparatus of the fourth invention can be obtained by installing the program of the fourth invention on a general-purpose computer.

  With the content display system of the present invention, human error can be suppressed and display control rich in entertainment can be realized.

Configuration diagram of content display system Schematic perspective view of content display system Sequence diagram during execution Flow chart showing process flow of control device Diagram explaining specific area extraction / projection function The figure which shows an example of the template of a control command locus image The figure explaining registration processing of control command locus DB Example of data registered in control command locus DB The figure explaining the update process of control command locus DB Schematic diagram showing data correspondence The figure explaining the 1st display control example The figure explaining the 2nd example of display control The figure explaining the 2nd example of display control

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment of the present invention, “content” includes moving image data, still image data (including all images, pictures, tables, documents, etc. displayed on a computer screen), text data, audio data, music. Includes data. As a typical example of the content in the embodiment of the present invention, a plurality of still image data and moving image data are collected based on one scenario, and particularly perceived visually.

  Hereinafter, data such as still image data and moving image data constituting the content is referred to as a “content component”. In the case of still image data, typically, each piece is a “scene” and is a different content component. In the case of moving image data, typically, each “chapter” in which a plurality of frames are collected is a different content component. In the content display system according to the embodiment of the present invention, various display controls are realized for each content component.

  First, an outline of the content display system will be described with reference to FIGS.

  FIG. 1 is a configuration diagram of a content display system. As shown in FIG. 1, the content display system 1 includes a laser light oscillation device 2, a photographing device 3, a content display device 4, a control device 5, and the like. In the content display system 1, an object 7 in the real space 6 is assumed. The object 7 may be a three-dimensional object such as a sphere as shown in FIG. 1, or may be a two-dimensional object such as a wall surface or a floor as shown in FIG.

  The laser light oscillation device 2 is, for example, a laser pointer. The photographing device 3 is, for example, a digital video camera. The content display device 4 is, for example, a projector. The control device 5 is, for example, a computer.

  The control device 5 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), a storage device (hard disk, etc.), a display device (liquid crystal display, etc.), an input device (mouse, keyboard, etc.), etc. Is provided. A program for realizing the image processing unit 11 and the content processing unit 13 is stored in the storage device of the control device 5. In the storage device of the control device 5, a database system (or a simple file group) for realizing the image processing information storage unit 12 and the content information storage unit 14 is constructed.

  The image processing unit 11 includes a captured image acquisition function 21, a specific area extraction / projection function 22, a bright spot extraction function 23, a locus extraction function 24, and the like. The image processing information storage unit 12 includes a captured image extraction database (hereinafter abbreviated as “DB”) 31, a specific area extraction DB 32, a bright spot extraction DB 33, a locus extraction DB 34, and the like. The content processing unit 13 includes a control command trajectory determination function 41, a content display control function 42, and the like. The content information storage unit 14 includes a control command locus DB 51, a content DB 52, and the like.

  FIG. 2 is a schematic perspective view of the content display system. As shown in FIG. 2, the operator 8 holds the laser light oscillation device 2. The imaging device 3 captures the captured image area 61 (a dashed-dotted rectangle) and transmits the captured image to the control device 5. The content display device 4 displays the content in the content display area 62 (dotted rectangle) according to the control from the control device 5. In the content display system 1, when the operator 8 irradiates the object 7 in the real space 6 with the light of the laser light oscillation device 2, the control device 5 displays content corresponding to the light trajectory of the laser light oscillation device 2. Take control. The control device 5 identifies the light trajectory as a point, a line, a figure, or the like.

  In the example of FIG. 2, the captured image area 61 includes a content display area 62. In this case, if the operator 8 irradiates the light of the laser light oscillation device 2 inside the content display region 62, the locus of the light of the laser light oscillation device 2 can be extracted by the processing of the control device 5 described later.

  Further, for example, by matching the photographed image area 61 and the content display area 62, it is not necessary to align the photographed image area 61 and the content display area 62 by the control device 5, and the processing content can be reduced. As a result, the response performance of content display control can be improved. Even when the photographed image area 61 and the content display area 62 coincide with each other, as will be described later, the control device 5 causes distortion caused by a shift between the photographing direction of the photographing apparatus 3 and the projection direction of the content display apparatus 4. Is corrected.

  In addition, for example, by making the photographed image area 61 and the content display area 62 different places or a plurality of areas, the entertainment property can be improved. For example, when another content display device 4 is added from the state of FIG. 2 and another content display area 62 is provided on the floor surface, display control is performed as if the content displayed on the wall surface is skipped to the floor surface. realizable. That is, when the operator 8 irradiates the content displayed on the wall surface with the light of the laser light oscillation device 2 and moves the light of the laser light oscillation device 2 in the direction of the floor surface, the content is displayed on the wall surface at a certain timing. Display control can be realized in which the displayed content disappears and the same content is displayed on the floor.

  In the following, for easy understanding, it is assumed that the photographing device 3 and the content display device 4 are each one, the object 7 is a wall surface, and the photographed image region 61 is a plane region including the content display region 62. However, the processing of the control device 5 described below is performed when the number of the photographing devices 3 and the content display devices 4 is plural, when the object 7 is a three-dimensional object, and when the photographed image region 61 and the content display region 62 are different. This can also be applied.

  FIG. 3 is a sequence diagram at the time of execution. In addition, about the process of S4-S9 shown in FIG. 3, the control apparatus 5 which is hardware performs according to each function of the software shown in FIG. In the following, for ease of explanation, the software functions are mainly described for convenience.

  As shown in FIG. 3, the operator 8 holds the laser light oscillation device 2 (S 1) and irradiates the object 7 with the laser light. On the other hand, the imaging device 3 captures the captured image area 61 of the real space 6 including the object 7 at a predetermined frame rate (for example, 30 frames / second) (S3).

  The photographed image acquisition function 21 refers to the photographed image extraction DB 31 and obtains a photographed image from the photographing device 3 (S4). The specific area extraction / projection function 22 refers to the specific area extraction DB 32, extracts a specific area from the captured image, and projects it onto the processed image (S5). The bright spot extraction function 23 refers to the bright spot extraction DB 33 and extracts the position of the bright spot in the processed image (S6). The trajectory extraction function 24 refers to the trajectory extraction DB 34 and extracts the locus of bright spots (S7).

  The control command track determination function 41 determines a control command with reference to the control command track DB 51 (S8). The content display control function 42 performs display control of the content stored in the content DB 52 according to the control command (S9).

  The content display device 4 displays content on the object 7 in accordance with display control by the content display control function 42 (S10). The operator 8 or the viewer views the content displayed on the object 7 (S11).

  The operator 8 may or may not irradiate the object 7 with laser light. On the other hand, the control device 5 repeats the process according to the frame rate of the photographing device 3. When the control device 5 newly determines a control command in S8, the content displayed on the content display device 4 is changed. Then, the content displayed on the object 7 by the content display device 4 changes.

  Next, details of the processing of the control device 5 will be described with reference to FIGS.

  FIG. 4 is a flowchart showing the flow of processing of the control device. After the photographing device 3 and the content display device 4 are set at predetermined positions, before the sequence shown in FIG. 3 is executed, the control device 5 stores the calibration result of the photographing device 3 in accordance with the execution environment in each DB. This is reflected and each DB is initialized (S21).

  The captured image area 61 is illuminated with the illumination light of the real space 6 and the projection light of the content display device 4 superimposed on each other. In order to accurately extract the bright spot of the laser beam from such a photographed image region 61, the control device 5 sets the photographing parameters of the photographing device 3 to optimum values according to the execution environment, and the photographed image extraction DB 31. Is stored as numerical information.

  The control device 5 adjusts the shooting parameter so that the bright spot of the laser beam can be easily extracted according to the execution environment, and uses the shot image shot according to the shooting parameter as input data. This eliminates the need for some image processing procedures (such as preprocessing for extracting bright spots according to input data) by the control device 5 and improves the processing efficiency of the control device 5. As a result, the response time from when the operator 8 operates the laser beam transmission device 2 until the desired display control is realized can be shortened, and the response performance of the content display system 1 can be improved.

  In general, the coordinate space of the captured image does not match the coordinate space of the content stored in the control device 5. However, as will be described later, in order to switch the display control of the content depending on which position of the content component is irradiated with the light of the laser light oscillation device 2, the coordinate space of both is matched in the internal processing of the control device 5. It is necessary to let Therefore, the control device 5 acquires a captured image in advance, associates the coordinate information of the feature point in the captured image with the coordinate information of the corresponding point in the coordinate space in the internal processing of the control device 5, and extracts the specific region extraction DB 32. Is stored as numerical information.

  In general, the light attribute information (luminance, color, shape, etc.) of the laser light oscillator 2 is various. Therefore, in order to accurately extract the bright spot of the laser light, for example, the light of the laser light oscillation device 2 irradiated on the photographed image area 61 is photographed in advance, so that the light attribute of the laser light oscillation device 2 is obtained. Information is obtained and stored in the bright spot extraction DB 33 as numerical information.

  When the execution of the sequence shown in FIG. 3 is started, the captured image acquisition function 21 of the control device 5 refers to the captured image extraction DB 31 and acquires a captured image from the imaging device 3 (S22). The control device 5 assumes that the input data from the photographing device 3 has been photographed according to the photographing parameters stored in the photographed image extraction DB 31, and performs image processing as necessary. For example, in order to easily extract the bright spot of the laser beam, the control device 5 depends on only the photographing parameters stored in the photographed image extraction DB 31 regardless of the input data, and the brightness and contrast of the input data. Conversion processing and the like are performed to obtain a captured image. Normally, it is necessary to determine the parameters of the conversion process from the input data, but in the embodiment of the present invention, the parameters of the conversion process are determined based only on the shooting parameters stored in the shot image extraction DB 31. Therefore, processing can be executed at high speed.

  Next, the specific area extraction / projection function 22 of the control device 5 refers to the specific area extraction DB 32 and projects the specific area onto the extracted / processed image (S23).

  FIG. 5 is a diagram for explaining the specific area extraction / projection function. The specific area extraction / projection function 22 extracts a specific area from the captured image 63 that is image data of the captured image area 61. The specific area is an area where the content creator has allowed the laser light oscillation device 2 to irradiate light. In the example of FIG. 5, the specific area is the content display area 62. Note that, since the shooting direction of the shooting device 3 and the projection direction of the content display device 4 do not coincide with each other, as shown in FIG. .

  As described above, the specific area extraction DB 32 stores the coordinate information of the corresponding points in the coordinate space in the internal processing of the control device 5 in association with the coordinate information of the feature points in the captured image 63. Therefore, the specific area extraction / projection function 22 generates a specific area image 64 from the captured image 63, extracts a plurality of feature points from the specific area image 64, and refers to the specific area extraction DB 32 to perform internal processing of the control device 5. The specific area image 64 is projected by mapping to a corresponding point in the coordinate space at, and a processed image 65 is generated. For example, the control device 5 calculates a projection matrix for projecting an image in the coordinate space of the captured image 63 onto the coordinate space in the internal processing from a set of a plurality of feature points and corresponding points, and based on the calculated projection matrix. Thus, all the pixels of the specific area image 64 are projected onto the coordinate space in the internal processing to obtain a processed image 65.

  Even if the object 7 is a three-dimensional object, the specific region extraction / projection function 22 performs a three-dimensional extension of the projective transformation or an extension by synchronizing the plurality of imaging devices 3 and the content display device 4. Similarly, the processed image 65 can be generated.

  Returning to the description of FIG. The bright spot extraction function 23 refers to the bright spot extraction DB 33 and extracts the coordinates of the bright spot in the processed image 65 (S24). As described above, since the attribute information of the light of the laser light oscillation device 2 is stored in the bright spot extraction DB 33, the coordinates of the bright spot can be extracted with high accuracy.

  Next, the trajectory extraction function 24 refers to the trajectory extraction DB 34 and extracts the light trajectory of the laser light oscillation device 2 from the bright spot coordinate group extracted in S24 (S25).

  In the trajectory extraction DB 34, the trajectory existence determination time (the unit may be seconds or the number of frames) is stored as numerical information. For example, if the frame rate of the photographing device 3 is 30 frames / second, 30 frames are input to the control device 5 per second. That is, the specific area extraction / projection function 22 generates 30 processed images 65 per second. When the trajectory existence determination time is 1 second (= 30 frames), the trajectory extraction function 24 checks 30 processed images 65 for each trajectory existence determination time, and whether or not each contains a bright spot. To check. If there is no processed image 65 including a bright spot, the trajectory extraction function 24 sets a trajectory (a so-called NULL value) that no bright spot exists. On the other hand, when there is a processed image 65 including a bright spot, the trajectory extraction function 24 sets the bright spot included in the processed image 65 at the earliest time to the processed image 65 at the latest time. The included bright point is the end point, a vector having a length and direction is calculated from the coordinate information of the start point and the end point, and the calculated vector is used as a locus.

  Next, the control command trajectory determination function 41 refers to the control command trajectory DB 51 and confirms whether the trajectory extracted in S25 corresponds to any control command (S26). The process of S26 will be described later with reference to FIGS.

  When the locus does not correspond to the control command (NO in S26), the process is repeated from S22. On the other hand, when the trajectory corresponds to the control command (YES in S26), the content display control function 42 refers to the content DB 52, performs content display control (S27), and repeats the processing from S22. Note that the control command trajectory DB 51 and the content DB 52 are associated with each other by using content component identification information as a key.

  By the way, when the object 7 moves in the space 6, it is necessary to change the shooting direction of the shooting device 3 and the projection direction of the content display device 4. That is, the position of the object 7, the shooting direction of the shooting device 3, and the projection direction of the content display device 4 change. Therefore, it is necessary to initialize each DB again. Therefore, when the object 7 moves in the space 6, after determining NO in S26 and after the process of S27, the process is repeated from S21 as shown by the dotted line in FIG.

  Hereinafter, the process of FIG. 29 will be described with reference to FIGS. In addition, a registration process and an update process of the control command trajectory DB 52, which are the premise of the process of FIG.

  FIG. 6 is a diagram illustrating an example of a template for a control command trajectory image. The control command is a program that realizes specific display control. The control command locus image 66 is an image for associating the control command with the light locus of the laser light oscillation device 2.

  The control command locus image 66 is the same size as the processed image 65 and the content component image, and a plurality of divided areas are defined. In the control command locus image 66a shown in FIG. 6A, 2 rows × 4 columns = 8 divided regions A to H are defined. In addition, in the control command locus image 66b shown in FIG. 6B, 3 rows × 3 columns = 9 divided regions A to H are defined.

  FIG. 7 is a diagram for explaining control command locus DB registration processing. FIG. 8 is an example of data registered in the control command locus DB 51. The registration process of the control command trajectory DB 51 is performed by interactive processing between the content producer and the control device 5.

  First, a content producer displays content components on a display device (including the content display device 4) connected to the control device 5. An image 67 shown in FIG. 7A is an example of a content component.

  Next, the content producer causes the display device connected to the control device 5 to display an image in which the content component and the control command locus image 66 are superimposed. An image 68 shown in FIG. 7B is an image obtained by superimposing the image 67 and the control command locus image 66b.

  Next, the content creator selects one control command from a list (not shown) of control commands defined in advance. The control command list is displayed on the display device by the control device 5 when the start of the registration process is instructed, for example. In the example shown in FIGS. 7 and 8, the content creator selects “display next scene” as the control command.

  Next, the content producer selects the divided area of the control command locus image 66 via the input device of the control device 5 and inputs the locus information. FIG. 7C shows an example in the case of inputting trajectory information related to an operation of flicking light from the laser light oscillation device 2 from left to right.

  When the content producer selects the region D, the control device 5 recognizes that the region D is set as the start point region (shown as the start point of the hatching 71 and the arrow 73 in FIG. 7C). Next, when the content producer selects the region F, the control device 5 recognizes that the region F is set as the end point region (shown as the end point of hatching 72 and arrow 73 in FIG. 7C). . Further, the content creator also inputs a determination time (see FIG. 8) for giving a certain clearance to the flick speed. That is, the control device 5 determines that the operation of flicking the light of the laser light oscillation device 2 from the left to the right is performed if the bright spot of the laser light moves from the start point region to the end point region within the determination time. Become.

  Then, the control device 5 associates the content component, the trajectory information, and the control command, and stores them in the control command trajectory DB 51. The control device 5 stores in the control command locus DB 51 that the region D is the start point region and the region F is the end point region as a determination region (see FIG. 8). The control device 5 also stores the determination time in the control command locus DB 51.

  In the data 76 shown in FIG. 8, the content component is “scene 1”, the determination time is “a to b seconds”, the determination area is “D → F”, and the control command is “display next scene”. More precisely, for example, the upper left coordinates and the lower right coordinates of each area are stored in the determination area. That is, the determination area stores the upper left coordinates (x1, y1) and lower right coordinates (x2, y2) of the start point area, and the upper left coordinates (x3, y3) and lower right coordinates (x4, y4) of the end point area. The More strictly, the control command stores, for example, identification information of a program for realizing “display next scene”.

  In the example of the data 76, if the locus of the light of the laser beam oscillation device 2 with respect to the content component of “scene 1” moves from the divided areas D to F during a to b seconds, “Display next scene” is executed as a control command to be executed. Note that the image of the content component of “Scene 1” is an image 67 shown in FIG.

  FIG. 9 is a diagram illustrating the update process of the control command locus DB. When the operator 8 tries the display control registered in the control command locus DB 51, unintended display control may be performed. For example, in the example shown in FIGS. 7 and 8, the operator 8 irradiates the central portion of the image 67 with the light of the laser light oscillation device 2 in order to prompt the viewer to pay attention to the central portion of the image 67. Even if the operator 8 does not intend, an error that the next scene is displayed may occur. This is a human error caused by the divided area D that is the start point area and the divided area F that is the end point area being too large. Therefore, in the embodiment of the present invention, a screen interface capable of easily changing the size of the divided area is provided.

  In FIG. 9A, the same trajectory information as in FIG. 7C is set. When the content producer moves the boundary line 74 of each region in the direction of the arrow 75 via the input device of the control device 5, the control device 5 can display each region as shown in FIG. 9B, for example. Change the size of. In the example shown in FIG. 9B, the four boundary lines are interlocked and moved outward from the central portion in accordance with the movement amount of the boundary line 74. The divided areas A to D and F to I are reduced, and the divided area E is increased.

  Then, the control device 5 updates the control command trajectory DB 51 based on the trajectory information set in FIG. 9B. Specifically, the control device 5 determines the upper left coordinates (x1, y1) and lower right coordinates (x2, y2) of the start point region, and the upper left coordinates (x3, y3) and lower right coordinates (x4, y4) of the end point region. ). As described above, by performing the update process, it is possible to avoid the above-described human error caused by the divided area D and the divided area F being too large.

  FIG. 10 is a schematic diagram showing the correspondence of data. FIG. 10 shows data in the control command trajectory DB 51 and the content DB 52 for the content components “scene 1” and “scene 3”.

  FIG. 10 shows “image 67” as the content component of “scene 1” and “image 77” as the content component of “scene 3”. In addition, FIG. 10 shows data 76 associated with the content component “scene 1” and data 78 associated with the content component “scene 3”.

  For example, when the content producer or operator 8 instructs the display of the control command trajectory image 66 associated with the content component “scene 1”, the control device 5 displays the control command trajectory image 66c from the data 76. Generate and display on the display device. Alternatively, the control device 5 may generate the control command trajectory image 66c when storing the data 76 and store it in the storage device, read the control command trajectory image 66c from the storage device, and display it on the display device. good.

  In the example shown in FIG. 10, one control command is associated with one content component, but a plurality of different control commands may be associated. When associating a plurality of different control commands, the control device 5 performs control so that determination times and determination regions do not overlap in the registration process.

  The control command trajectory determination function 41 of the control device 5 refers to data as shown in FIG. 10 and confirms whether the trajectory extracted by the trajectory extraction function 24 corresponds to any control command. For example, when an image of the content component “scene 1” is displayed on the object 7 by the content display device 4, the control command locus determination function 41 refers to the data 76 associated with “scene 1”. If the trajectory extracted by the trajectory extraction function 24 is a trajectory in which the bright spot of the laser beam moves from the start point region D to the end point region F within the determination time [a to b seconds], “display next scene” It is determined that it corresponds to the control command.

  FIG. 11 is a diagram illustrating a first display control example. The first display control example shown in FIG. 11 is a display control example according to data 76 (= “display command for next scene”) for the content component of “scene 1” shown in FIG. is there.

  A thick line 81 shown in FIG. 11A indicates that the operator 8 irradiates the light of the laser light oscillation device 2 with respect to the image 67 projected on the object 7 (= the image of the content component of “Scene 1”). The trajectory is shown. An arrow 82 indicates the direction of the locus. That is, the locus of the thick line 81 is substantially a straight line from the left to the right.

  In the state of FIG. 11A, the photographing device 3 photographs the photographed image area 61 including the image 67 and transmits it to the control device 5. The image processing unit 11 of the control device 5 extracts a trajectory for the image 67, and the content processing unit 13 of the control device 5 determines that the control command corresponding to the trajectory of the thick line 81 is “display next scene”. Then, display control is performed so that the image 83 of the next scene (= the image of the content component of “scene 2”) is displayed on the content display device 4. Then, the content display device 4 projects an image 83 onto the object 7 in accordance with the display control of the control device 5 as shown in FIG.

  As shown in the first display control example, the operator 8 can display the next scene by operating only the laser light oscillation device 2 without operating the input device of the control device 5 or the like. . That is, in the embodiment of the present invention, display control that does not place a burden on the operator 8 can be realized.

  12 and 13 are diagrams illustrating a second display control example. In the second display control example shown in FIGS. 12 and 13, data 78 (= “control command for displaying content inside the trajectory”) is applied to the content component of “scene 3” shown in FIG. It is an example of display control. In the case of the control command “display content inside the locus”, a mask image is associated with the image of the content component in addition to the image 77. The mask image is a higher layer than the image 77 and covers the entire image 77 in the initial state. In the mask image 91 shown in FIG. 12A, all the pixels have the same color so that it can be easily understood that the image 77 is masked.

  A thick line 92 shown in FIG. 12A indicates a trajectory of the operator 8 irradiating the light of the laser light oscillator 2 with respect to the mask image 91 projected on the object 7. An arrow 93 indicates the direction of the locus. That is, the locus of the thick line 92 is substantially elliptical in the clockwise direction.

  In the state of FIG. 12A, the imaging device 3 captures the captured image area 61 including the mask image 91 and transmits it to the control device 5. The image processing unit 11 of the control device 5 extracts a trajectory for the mask image 91, and the content processing unit 13 of the control device 5 indicates that the control command corresponding to the trajectory of the thick line 92 is “display content inside the trajectory”. The display control is performed so that the image 77 (= the image of the content component of “Scene 3”) is displayed on the content display device 4 inside the locus of the thick line 92. More specifically, the content processing unit 13 of the control device 5 performs image processing on the mask image 91 to cut an area inside the locus of the thick line 92. Since the image 77 is arranged in the layer below the mask image 91, the image 77 is displayed in the shaved area. Then, as shown in FIG. 12B, the content display device 4 projects an image on the object 7 in which the mask image 91 and the image 77 in which the region inside the trajectory is superposed is superimposed according to the display control of the control device 5. .

  FIG. 13 shows a state in which the operator 8 further operates the laser light oscillation device 2 and gives an instruction to display the image 77 in a wider range from the state of FIG.

  A thick line 94 shown in FIG. 13A indicates a locus on which the operator 8 irradiates the light of the laser light oscillation device 2 with respect to the mask image 91 projected on the object 7. An arrow 95 indicates the direction of the locus. That is, the locus of the thick line 94 is substantially elliptical in the clockwise direction, and is located outside the thick line 92.

  In the state of FIG. 13A, the imaging device 3 captures the captured image area 61 including the mask image 91 and the image 77 and transmits it to the control device 5. The image processing unit 11 of the control device 5 extracts a trajectory for the mask image 91, and the content processing unit 13 of the control device 5 indicates that the control command corresponding to the trajectory of the thick line 94 is “display content inside the trajectory”. And the display control is performed so that the image 77 (= the image of the content component of “Scene 3”) is displayed on the content display device 4 inside the locus of the thick line 94. More specifically, the content processing unit 13 of the control device 5 performs image processing on the mask image 91 to cut an area inside the locus of the thick line 94. Note that the area inside the locus of the thick line 92 is treated as already cut. Then, as shown in FIG. 13B, the content display device 4 projects a superimposed image of the mask image 91 and the image 77 in which the area inside the trajectory is cut out on the object 7 according to the display control of the control device 5.

  As shown in the second display control example, the operator 8 operates only the laser light oscillation device 2 without operating the input device or the like of the control device 5, thereby tracing the light trajectory of the laser light oscillation device 2. It is possible to realize display control in which content is gradually displayed inside. That is, in the embodiment of the present invention, it is possible to realize display control rich in entertainment for both the operator 8 and the viewer.

  As described above, the control device 5 in the content display system 1 performs control for realizing specific display control in association with the trajectory information for identifying the light trajectory of the laser light oscillation device 2 associated with each content component. The command is stored in the control command locus DB 51. And the control apparatus 5 acquires the picked-up image 63 image | photographed with the imaging device 3, extracts the locus | trajectory of the light of the laser beam oscillation apparatus 2 from the captured image 63, and is memorize | stored about the extracted locus | trajectory of light. By referring to the information, the corresponding control command is determined, and content display control is performed based on the determined control command. As a result, the content display system 1 can perform appropriate display control according to the content, suppress human errors, and realize display control rich in entertainment properties.

  In addition, the trajectory information stored in the control command trajectory DB 51 includes a set of determination time and determination area, and the control device 5 indicates that the irradiation time and irradiation position of each bright spot constituting the extracted light trajectory is the trajectory. The corresponding control command is determined depending on the determination time in the information and whether it is included in the determination area. Then, the control device 5 superimposes an image indicating the position of the determination area on the image of the content component, and displays it on an arbitrary display device including the content display device 4. As a result, the content producer or operator 8 can visually confirm the determination area during the display control test, and the content production work and confirmation work are facilitated.

  In addition, the control device 5 accepts a change in the position of the determination area and updates the content of the control command locus DB 51. As a result, the content producer and the operator 8 can easily correct the trajectory information determined to be prone to human error as a result of the display control test.

That is, the control device 5 has a function of visually displaying and adjusting the trajectory determination under the execution environment. Such an adjustment function has a remarkable effect that the operator 8 can operate the laser light oscillation device 2 by a natural operation (speed or distance). For example, when content is displayed by a projector, the physical projection size (content position and size) varies depending on the execution environment. If the above-described adjustment function is not provided and, for example, the stored determination area (time or range) is fixed at the time of design, the following problems may occur.
・ If an unreasonable human action occurs, it may cause errors such as lost or excessive detection of bright spots.
-Skill of the operator 8 is required, and a load that the operator 8 adjusts to the system is generated, which is not desirable.
・ Repeated and repetitive work of experiment and design during development occurs.
-Redevelopment is required when reusing the same system under different execution environments.
-At the development site, it is only possible to confirm the appearance and operation of the control device 5 on the display, and it is difficult to imagine the operation at the site.
Therefore, by using the function of visually displaying and adjusting the trajectory determination under the execution environment and updating the contents of the control command trajectory DB 51, the actual operation of the laser light oscillation device 2 by the operator 8 is adjusted. Adjustment is possible, and the above-mentioned problems can be solved.

  The control device 5 stores the coordinate information of the corresponding points in the coordinate space in the internal processing in the specific area extraction DB 32 in association with the coordinate information of the feature points in the captured image 63. And the control apparatus 5 produces | generates the specific area image 64 from the picked-up image 63, extracts a feature point from the specific area image 64, refers to specific area extraction DB32, and makes a feature point the corresponding point of the coordinate space in internal processing. The specific area image 64 is projected by mapping, and a processed image 65 is generated. Further, the control device 5 extracts the coordinates of the bright spots for the plurality of processed images 65, and uses the set of the extracted coordinates as the light locus of the laser light oscillation device 2. Thus, the shooting direction of the shooting device 3 and the projection direction of the content display device 4 are not restricted, and the content display system 1 can be applied to the real space 6 in various environments.

  Further, the control device 5 stores the shooting parameters of the shooting device 3 in accordance with the execution environment in the shot image extraction DB 31. And the control apparatus 5 acquires the picked-up image 63 with reference to picked-up image extraction DB31. This eliminates the need for some image processing procedures by the control device 5 and improves the processing efficiency of the control device 5. As a result, the response performance of the content display system 1 can be improved.

  In the above description, two types of “display next scene” and “display content inside the trajectory” have been described as examples of control commands. However, the present invention is not limited to these examples. For example, “start current scene”, “display previous scene”, “switch scene”, “animation in scene”, and the like can be realized as control commands. Further, as described above, by adding another content display device 4 and providing another content display area 62 on the floor surface, a more dynamic control command such as “image skipping from the wall surface to the floor surface” is also realized. be able to.

  In the above description, as an example of the human error that can be avoided by the present invention, the human error caused by the divided area D and the divided area F in FIG. 7C being too large has been described. Human errors that can be made are not limited to this example.

  For example, after all the scenes have been explained, they may be returned to the first scene for the next explanation. At this time, if the laser light oscillation device 2 is accidentally irradiated with light, some display control (for example, “display next scene”) is performed according to the control command. Therefore, when displaying the first scene after displaying all the scenes, it is conceivable to avoid human error by not performing display control at all until a certain time elapses. Specifically, in the control command trajectory DB 51, data with a determination time “x seconds from the start of display”, a determination area “all”, and a control command “do nothing” is stored in the content component of the first scene. May be stored in association with each other.

  The preferred embodiments of the content display system and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood.

DESCRIPTION OF SYMBOLS 1 ......... Content display system 2 ......... Laser light oscillation device 3 ......... Shooting device 4 ......... Content display device 5 ......... Control device 6 ......... Real space 7 ......... Object 8 ......... Operation 11 ......... Image processing unit 12 ......... Image processing information storage unit 13 ......... Content processing unit 14 ......... Content information storage unit 21 ......... Captured image acquisition function 22 ......... Specific area extraction / projection function 23 ......... Bright spot extraction function 24 ......... Track extraction function 31 ... …… Photograph image extraction DB
32 ... Specific area extraction DB
33 ..... Bright spot extraction DB
34 .... Trajectory extraction DB
41... Control command locus determination function 42... Content display control function 51.
52 ……… Content DB
61... Imaged image area 62... Content display area 63... Imaged image 64... Specific area image 65 ... Processed image 66, 66 a, 66 b, 66 c, 66 d ......

Claims (8)

A content display system in which an operator instructs display control of content by irradiating an object in real space with light of a laser light oscillation device,
The laser light oscillation device;
A photographing device for photographing a region irradiated with light from the laser light oscillator;
A content display device for displaying the content;
A control device for controlling the photographing device and the content display device;
Composed by
The controller is
First storage means for storing a control command for realizing specific display control in association with trajectory information for identifying a trajectory of light of the laser light oscillation device for each content component constituting the content;
Obtaining means for obtaining a photographed image photographed by the photographing device;
Extraction means for extracting a light locus of the laser light oscillation device from the photographed image;
A determination unit that determines the corresponding control command with reference to the first storage unit with respect to the locus of light extracted by the extraction unit;
Content display control means for performing display control of the content based on the control command determined by the determination means;
A content display system comprising: The trajectory information consists of a set of determination time and determination area,
The determination means corresponds to whether or not the irradiation time and irradiation position of the bright spot constituting the light trajectory extracted by the extraction means are included in the determination time and the determination region in the trajectory information. Determine the control command,
The controller is
Trajectory information display control means for superimposing an image indicating the position of the determination region on the image of the content component element and displaying it on an arbitrary display device including the content display device;
The content display system according to claim 1, further comprising: The controller is
Trajectory information update means for receiving a change in the position of the determination area and updating the contents of the first storage means,
The content display system according to claim 2, further comprising: The controller is
Second storage means for storing the coordinate information of the corresponding point in the coordinate space in the internal processing of the control device in association with the coordinate information of the feature point in the captured image;
Generating a specific area image from the captured image, extracting a feature point from the specific area image, and mapping the feature point to a corresponding point in a coordinate space in the internal processing of the control device with reference to the second storage unit Projecting means for projecting the specific area image and generating a processed image,
Further comprising
2. The extraction unit according to claim 1, wherein the extraction unit extracts the coordinates of a bright spot from a plurality of the processed images, and sets a set of the extracted coordinates as a light locus of the laser light oscillation device. Item 4. The content display system according to any one of Items 3 to 4. The controller is
Third storage means for storing imaging parameters of the imaging apparatus in accordance with the execution environment;
Further comprising
The content display system according to claim 1, wherein the acquisition unit acquires the captured image with reference to the third storage unit. In a content display system in which an operator directs display control of content by irradiating light in a laser beam to an object in real space, shooting is performed to shoot a region irradiated with light from the laser light oscillator The content display control device that controls the device and the content display device that displays the content is associated with trajectory information that identifies the light trajectory of the laser light oscillation device for each content component constituting the content A content display control method by a control device comprising first storage means for storing a control command for realizing specific display control,
An obtaining step of obtaining a photographed image photographed by the photographing device;
An extraction step of extracting a light trajectory of the laser light oscillation device from the captured image;
A determination step of determining the corresponding control command with reference to the first storage unit for the light trajectory extracted by the extraction step;
A content display control step for performing display control of the content based on the control command determined by the determination step;
A content display control method comprising: In a content display system in which an operator directs display control of content by irradiating light in a laser beam to an object in real space, shooting is performed to shoot a region irradiated with light from the laser light oscillator A content display control device for controlling a device and a content display device for displaying the content,
First storage means for storing a control command for realizing specific display control in association with trajectory information for identifying a trajectory of light of the laser light oscillation device for each content component constituting the content;
Obtaining means for obtaining a photographed image photographed by the photographing device;
Extraction means for extracting a light locus of the laser light oscillation device from the photographed image;
A determination unit that determines the corresponding control command with reference to the first storage unit with respect to the locus of light extracted by the extraction unit;
Content display control means for performing display control of the content based on the control command determined by the determination means;
A content display control device comprising:   A program for causing a computer to function as the content display control device according to claim 7.

Images