JP2016197220A - Interaction execution method and device employing interaction execution method, as well as program employing interaction execution method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an interaction execution method that can provide an interactive property to a spatial image to be formed via a half mirror.SOLUTION: A method is adapted to: position a display instrument building in imaging means in the vicinity of an outer peripheral edge of a picture display area in a half mirror so as not to shield an imaging range owned by the imaging means; cause a picture to be displayed in the picture display area of the display instrument; cause the picture to be formed above the half mirror via the half mirror as a spatial picture; and execute an interaction with respect to the spatial picture. After causing the picture to be displayed in the picture display area, whether a pointing body pointing out the spatial picture in the imaging area of the imaging means appears is determined in accordance with an image to be shot by the imaging means, and when it is determined that the pointing body appears in the imaging area, a new picture relevant to the picture is caused to be formed as an action continuous to the spatial picture by causing the new picture to be displayed in the picture display area.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an interaction execution method, an interaction execution apparatus employing the method, and a program for executing the interaction.

  In recent years, as a new video display method, a video display device for forming a video image in the air has been proposed. This is to form an image of the original display screen in a different space from the display screen by various optical methods, and the apparatus is generally composed of a lens array, a corner cube array, or the like. A well-known technique is employed in which the image is optically imaged at a plane-symmetrical position with respect to the half mirror using a retroreflective sheet and a half mirror (Patent Documents 1 and 2).

  Further, an image display device that realizes user interaction that detects an approach position of a viewer's finger with a sensor built in the image display surface and changes a two-dimensional image displayed on the image display surface according to the approach position. Has been proposed (Patent Document 3).

  On the other hand, the present applicant has filed a patent application (Japanese Patent Application No. 2015-30040) for the spatial image forming apparatus 1 shown in FIG. The apparatus 1 includes a support pedestal 3 for placing the display device 2, a retroreflecting unit 4, a half mirror unit 5, and a side plate unit 6 for fixing each unit with a specific angular relationship. The display device 2 has a simple box structure in which an insertion port 7 for inserting the display device 2 into the support base 3 is provided.

  Then, the display device 2 such as a smart device is inserted through the insertion port 7 and attached to the support pedestal 3, and the video 9 is displayed in the video display area 8 as shown in FIG. After the light output from the video display area 8 of the display device 2 placed on 3 travels along the optical path a, part of the light is reflected by the half mirror unit 5, travels along the optical path b, and enters the retroreflective unit 4. The light incident on the retroreflecting unit 4 is reflected so as to pass through the same optical path c, is incident on the half mirror unit 5 again, and a part of the light incident on the half mirror unit 5 is transmitted through the optical path d. The aerial image 10 is imaged at a plane-symmetrical position with the half mirror 5 as a symmetry plane.

JP-A-56-158320 JP 09-506717 A International Publication No. 2007/013215

  The present invention has a technical problem of providing an interactive property to a spatial image formed through a half mirror, and has studied the conventional image display device.

  The simplest method of directly inputting the image displayed on the display screen of the image display device is a method of directly inputting the image displayed on the display screen with a finger or a stylus pen. However, this method has a problem that a series of operations at the time of operation interrupts the screen display and consequently affects a three-dimensional spatial image.

  Further, if a wireless input device such as Bluetooth or a wired input device connected via USB is connected and remotely operated, the first problem can be avoided, but there is a problem that the interactivity with the spatial image is impaired.

  Furthermore, the image display device disclosed in Patent Document 3 has a problem that a special image display means is required in addition to the first problem.

  Therefore, the present invention pays attention to the fact that a commercially available smart device such as a smartphone or a tablet has a built-in camera, and if data captured by the built-in camera is used, even a commercially available smart device can interactively display a spatial image. The above technical problem has been achieved by obtaining the knowledge that it can be imparted with the property.

  The technical problem can be solved by the present invention as follows.

  The interaction execution method according to the present invention displays a video in a video display area of a display device, forms the video as a spatial video above the half mirror via a half mirror, and executes the interaction on the spatial video The display device includes an imaging unit in the vicinity of an outer periphery of the video display area, and the half mirror is arranged so as not to block an imaging range of the imaging unit, and the video display area It is determined from an image captured by the imaging unit whether or not an indicator pointing to the spatial video has appeared in the imaging range of the imaging unit after the video is displayed on the imaging unit, and the indicator appears in the imaging range. When it is determined that a new image related to the image is displayed in the image display area, the image is formed as a continuous operation on the spatial image. .

  Further, the interaction execution device according to the present invention is provided with a display device including a video display region for displaying video and an imaging unit built in the vicinity of the outer periphery of the video display region, and an angle with respect to the display device. And a half mirror arranged so as not to obstruct an imaging range of the imaging means, displaying an image in an image display area of the display device, and displaying the image via the half mirror above the half mirror An interaction execution device that forms an image as a spatial video and executes an interaction with the spatial video, wherein the first display means displays the video in the video display area, and the video is displayed by the first display means. Determination means for determining whether or not an indicator pointing to the spatial image has appeared in the imaging range of the imaging means after displaying the video in the display area A second display means for displaying a new video related to the video in the video display area when the judgment means judges that the indicator has appeared in the imaging range. Is.

  Further, the present invention provides the interaction executing device, wherein the video display area is formed in a rectangular shape and an imaging means is built in near one edge of the outer peripheral edge of the rectangular shape, and is a long convex lens having a bowl-shaped cross section that descends in both longitudinal directions. However, the long convex side of the long convex lens is arranged along the one edge so as to close the imaging range of the imaging means so as not to block the video display area.

  According to the present invention, in the interaction execution device, the long convex lens is exposed above the half mirror surface.

  Further, the program according to the present invention displays a video in a video display area, a computer mounted on a display device having a video display area for displaying video and an imaging means built in the vicinity of the outer periphery of the video display area. An image display range of the image pickup means is a first display means to be displayed, and an indicator that indicates a spatial image formed above the half mirror by the image after the image is displayed in the image display area by the first display means. And a means for displaying a new video related to the video in the video display area when the determination means determines that the indicator has appeared in the imaging range. It functions as a second display means.

  Furthermore, the program according to the present invention displays a video on a display device including a video display area for displaying a video and an imaging unit built in the vicinity of the outer periphery of the video display area. An image display range of the image pickup means is a first display means to be displayed, and an indicator that indicates a spatial image formed above the half mirror by the image after the image is displayed in the image display area by the first display means. And calculating the vertical and / or horizontal position of the indicator in the video display area when it is determined by the determining means that the indicator appears in the imaging range. A calculation means and a second display means for selecting a new video based on the calculation result of the calculation means and displaying the new video in the video display area. It is intended.

  According to the present invention, a spatial image is imaged using a display device incorporating an imaging means in the vicinity of the outer periphery of the video display area, and an indicator pointing to the spatial video appears in the imaging range of the imaging means. When it is determined that a new video related to the video is displayed and imaged as a continuous operation on the spatial video, the spatial video can be made interactive.

  It is a perspective view which shows the interaction execution apparatus which concerns on this invention.   It is a perspective view which shows the new space image imaged in the interaction execution apparatus shown in FIG.   FIG. 2 is a schematic block diagram of the display device illustrated in FIG. 1.   It is a flowchart figure explaining an example of an interaction execution program.   The camera position is a schematic diagram at the center of one edge, showing the difference in the captured image depending on the position of the camera of the display device. The upper diagram (a) is the captured image, and the lower diagram (b) is the display device.   FIG. 2 is a schematic diagram showing the difference in the captured image depending on the position of the camera of the display device, where the camera position is to the right of one edge, the upper diagram (a) is the captured image, and the lower diagram (b) is the display device.   FIG. 2 is a schematic diagram showing a difference in the captured image depending on the position of the camera of the display device, with the camera position on the left side of one edge, the upper diagram (a) being a captured image, and the lower diagram (b) being a display device.   It is a schematic diagram explaining the indication position of a pointer.   It is a flowchart figure explaining an example of an interaction execution program.   It is a perspective view which shows a new space image | video.   It is a perspective view which shows a new space image | video.   It is a perspective view which shows the interaction execution apparatus which concerns on this invention.   FIG. 13 is a perspective view of a convex lens illustrated in FIG. 12.   It is a schematic diagram which shows the relationship between the visual field area | region of a built-in camera position, and the optical system of a convex lens.   It is a perspective view which shows an example of a spatial image imaging device.   FIG. 16 is a schematic diagram illustrating an optical system in the spatial image forming apparatus illustrated in FIG. 15.

  Embodiment 1 FIG.

  Hereinafter, embodiments of the present invention will be described with reference to FIGS. In these drawings, the same reference numerals as those in FIGS. 15 and 16 denote the same or corresponding parts.

  In FIG. 1, reference numeral 11 denotes an interaction execution device, which includes a rectangular display screen (video display area) 8 for displaying video 9 and a camera (imaging means) 14 built in the vicinity of one edge 13 of the outer periphery 12 of the display screen 8. A portable display device (so-called smart device) 2 equipped with a communication function and a box-type spatial image forming apparatus 1 shown in FIG. 15 to which the display device 2 is attached. Whether or not the indicator 15 (see FIG. 2) pointing to the spatial video 10 appears in the imaging range of the camera 14 after the video 9 is displayed on the screen 8 is determined based on the image captured by the camera 14, and the imaging range is included in the imaging range. When it is determined that the indicator 15 has appeared, the display screen 8 has a function of displaying a new video 16 a related to the video 9. Reference numeral 17a denotes a new spatial video for the new video 16a.

  The box-type spatial imaging device 1 is supported with an angle with respect to a flat rectangular support pedestal portion 3 on which the display device 2 is mounted and a display screen 8 of the display device 2 mounted on the support pedestal portion 3. The flat rectangular half mirror part 5 arranged above the pedestal part 3 and the half mirror part 5 and the support pedestal part 3 are arranged at positions facing each other with respect to the support pedestal part 3. The half mirror portion is formed by an optical system formed by the flat rectangular retroreflecting portion 4 that is bent substantially continuously at a right angle, the display device 2 that is attached to the support base portion 3, the half mirror portion 5, and the retroreflecting portion 4. 5 and the side plate portion 6 for fixing the respective portions with a specific angular relationship so as to form a spatial image 10 above, and the display device 2 includes one edge 13 of the rectangular outer peripheral edge 12. Camera 1 on the half mirror 5 side The are placed on the support pedestal 3 positioned so as to be exposed from the upper surface of the half mirror unit 5, the half mirror unit 5 is disposed so as not to block the imaging range is a field area that the camera 14 has.

  As shown in FIG. 3, the display device 2 includes the display screen 8 that displays the video 9 and the new video 16a, and a speaker 18 that outputs music and audio related to the video 9 and the new video 16a. A camera 14 that captures the indicator 15, and a communication unit 19 that constructs WiFi, a wireless LAN, a telephone function, and the like that can download the interaction execution program shown in FIG. An I / O interface 20 for connecting an external device such as a USB for storing the interaction execution program and the image data, etc., a first storage unit 21 for storing the image data such as the video 9, and the video 9 A second storage unit 22 for storing image data such as a new video 16a and the interaction execution program are stored. An external storage unit 24 provided with a third storage unit 23 is configured to include a central control unit (CPU) 25 for controlling the various devices to be mounted on the display device 2, a.

  Next, the interaction execution program illustrated in FIG. 4 will be described.

  After the display device 2 is mounted on the box-type spatial image forming apparatus 1, the interaction execution program is read from the program storage unit 23 and executed by the control of the CPU 25.

  Under the control of the CPU 25, the image data of the video 9 is read from the first storage unit 21, and the video 9 is displayed on the display screen 8 (first display means: step s01). As a result, a spatial image 10 appears above the half mirror unit 5 as shown in FIG. Next, whether or not the indicator 15 pointing to the spatial image 10 has appeared in the imaging range of the camera 14 is determined by recognition of an imaging element (not shown) built in the camera 14 (determination means: step s02), and the CPU 25 If the image pickup device recognizes the indicator 15 under the control of, the CPU 25 then reads new image data representing the movement of the video 9 from the second storage unit 22 under the control of the CPU 25 and displays it on the display screen 8. The new image 16a is displayed (second display means: step s03). As a result, as shown in FIG. 2, a new spatial image 17 a that moves continuously from the image 9 appears above the half mirror unit 5.

  In the present embodiment, when the indicator 15 appears in the imaging range of the camera 14, a new video 16 a is displayed regardless of the position on one edge of the camera 14.

  By the way, the image 9 displayed on the display screen 8 forms a spatial image 10 of the image 9 while maintaining its positional relationship. Therefore, as shown in FIG. 5, in the case of the display device 2 in which the camera 14 is built in substantially the center of the one side edge 13, the photographed image of the indicator 15 photographed by the camera 14 is on the display screen 8. The central position is the imaging range, and the positional relationship is consistent with the image 9 displayed on the display screen 8. On the other hand, as shown in FIG. 6, in the case of the display device 2 in which the camera 14 is built on the right side of the one edge 13, the captured image of the indicator 15 captured by the camera 14 is on the display screen 8. Since the right side position is the imaging range, the position of the captured image of the indicator 15 is corrected to the left side so as to be aligned with the image 9 displayed on the display screen 8, and the camera 14 has one edge as shown in FIG. 7. In the case of the display device 2 built in the left side of 13, the photographed image of the indicator 15 taken by the camera 14 is displayed on the display screen 8 because the left side position is an imaging range with respect to the display screen 8. If the position of the captured image of the indicator 15 is corrected to the right in order to match with the image 9 being displayed, the recognition accuracy of the indicator 15 can be further increased.

  As an example of another correction method, a method of using a camera position database for each model of the display device 2 having a different camera position in advance and referencing the database at the time of recognition to correct the camera position variation by software Examples include a correction method in which a hand or the like is photographed at a specific position with respect to the display device 2 at the start, a calibration amount is calculated from a deviation from an expected reference position, and reflected in subsequent input operations. be able to.

  Therefore, the camera position may be on the right side or the left side from the center of the one edge 13, and may be near the right end or near the left end.

  The videos 9 and 16a include still images as well as moving images. Further, the new video 16a may be returned to the original video 9 by not recognizing the indicator 15, or may be continuously displayed until it is recognized again, and it is recognized again. By doing so, a third new video may be displayed. The indicator 15 may be a finger or a pointing stick.

  According to the present embodiment, if a smart device such as a smartphone or a tablet with a built-in camera is used as the display device 2, a hand movement or an object having a specific figure / color is photographed with the built-in camera. Since it can be reflected in the display video through the image analysis process, it is not necessary to prepare additional equipment for input operation.

  Embodiment 2. FIG.

  The present embodiment forms a new spatial image that differs depending on the position of the indicator 15 that points to the spatial image 10 in the first embodiment, and in FIGS. The same reference numerals as those in FIGS. 15 and 16 denote the same or corresponding parts, which will be described with reference to FIGS.

  In the CPU 25 according to the present embodiment, the position in the spatial image 10 pointed to by the indicator 15 is determined based on the position of the indicator 15 that appears in the imaging range of the camera 14, the upper / lower positions, left / right The position is calculated to correspond to the image 9 of the spatial image 10, and the position of the indicator 15 on the display screen 8 is divided into four parts by dividing the display screen 8 vertically and horizontally as shown in FIG. The divided area is converted into data corresponding to the imaging area of the image sensor of the camera 14, and in which imaging area the indicator 15 is imaged is calculated and replaced as the position of the indicator 15 on the display screen 8. To do. Then, new image data corresponding to the divided area is read from the second storage unit 22 and displayed on the display screen 8. Therefore, new image data corresponding to the position of the indicator 15 with respect to the display screen 8 is stored in the second storage unit 22, and the indicator 15 pointing to the spatial video 10 is the left area 26 on the display screen 8. Image data of the video 16a (see FIG. 2) newly displayed in the case, image data of the video 16b (see FIG. 10) newly displayed in the upper area 27 on the display screen 8, and the display screen 8 Image data of a video 16c (see FIG. 11) that is newly displayed in the lower area 28 in FIG. In the program storage unit 23, the interaction execution program shown in FIG. 9 according to the present embodiment, the imaging area data of the display screen 8 divided into the left area 26, the upper area 27, and the lower area 28, and Is stored.

  In the interaction execution program, as shown in FIG. 9, under the control of the CPU 25, the image data of the video 9 is read from the first storage unit 21 and displayed on the display screen 8 (first display means: step s04), the spatial image 10 shown in FIG. 1 appears above the half mirror unit 5, and then the camera 14 incorporates whether or not the indicator 15 pointing to the spatial image 10 has appeared in the imaging range of the camera 14. Judgment is made by recognition of an image sensor (not shown) (determination means: step s05), and if the image sensor recognizes the indicator 15 under the control of the CPU 25, then the display screen 8 is controlled under the control of the CPU 25. Is calculated by the above-described method (calculation means: step s06), and if the result of the calculation is the left area 26, The image data of the new video 16a expressing the movement with respect to the video 9 is read from the storage unit 22 and the new video 16a is displayed on the display screen 8 as shown in FIG. A new video 16b (see FIG. 10) is displayed on the display screen 8, and if it is the lower area 28, a new video 16c (see FIG. 11) is similarly displayed on the display screen 8 (second display means: Step s07).

  As a result, as shown in FIG. 2, a new spatial video 17a that moves continuously to the video 9 appears above the half mirror unit 5 due to the display of the new video 16a, and the new video 16b is displayed. As shown in FIG. 10, a new space image 17b appears in the same manner, and a new space image 17c appears in the same manner as shown in FIG. 11 by the display of the new image 16c. Note that another new video 16b or video 16c may be continuously displayed by recognizing the indicator 15 again while the new video 16a is displayed.

  Since the center axis of the imaging range of the camera and the display screen of the spatial image formed in the space are substantially parallel, the vertical direction on the captured image of the camera and the vertical direction of the display screen of the spatial video are the same. First, since the vertical direction on the display surface of the spatial image is the depth direction on the image captured by the camera, the depth direction position of the indicator 15 can be captured based on the change in the size of the indicator 15. If, for example, the size of the indicator 15 at the center of the display screen 8 is set in advance, the vertical coordinate of the display screen 8 is calculated from the difference from the reference. Can be sought.

  Embodiment 3 FIG.

  In the interaction execution apparatus 11 according to the first embodiment or the second embodiment, the present embodiment is configured by positioning the convex lens 29 on the camera 14. In FIGS. 12 to 14, FIGS. 15 and 16 indicate the same or corresponding parts.

  As shown in FIGS. 12 and 13, the display screen 8 of the display device 2 is formed in a rectangular shape, and a camera 14 is built in the vicinity of one edge 13 corresponding to the upper edge of the rectangular outer peripheral edge 12 of the display screen 8. A convex lens 29 is arranged so as to close the visual field region (imaging range) 30 of the image.

  The convex lens 29 is a long lens having a bowl-shaped cross section that descends in both longitudinal directions, and the long side 31 of the convex lens 29 is displayed along the upper side (one side edge) 13 of the display device 2 so as not to block the display screen 8. The convex lens 29 is located above the upper surface of the half mirror unit 5 (see FIG. 12). Furthermore, since the shape of the convex lens 29 is bilaterally symmetric, the optical axis of the convex lens 29 is positioned at the center that bisects the upper side 13.

  Thereby, even if the position of the camera 14 is different within the range of the upper side 13, as shown in FIG. 14, the visual field region 30 in the center position camera 14, the left position camera 14 ', and the right position camera 14' '. , 30 ′, 30 ″ are refracted through the convex lens 29 based on the optical characteristics of the convex lens 29 and pass through the focal point 32, so that the field of view range of the cameras 14, 14 ′, 14 ″ The indicator 15 appearing at can be recognized as appearing at the same position.

  Further, since the indicator 15 existing at the position of the focal point 32 is always photographed so as to be in the center of the imaging range of the camera, if the appearance position of the spatial image 10 is made equal to the focal length of the convex lens 29, Furthermore, the indicator 15 can be recognized with high accuracy.

  The convex lens 29 may be a biconvex lens or a plano-convex lens, or a Fresnel lens. Since the convex lens 29 has only to have a shape in which the long side 31 extends along the upper side 13, the convex lens 29 may have a rectangular shape in a top view, or may have a long side on one side along the upper side 13 and an arc shape on the other side. Moreover, the length of the convex lens 29 should just be the length which can cover the different camera built-in position which changes with models. The convex lens 29 may be bonded and fixed to the half mirror part 5 via an auxiliary member. Further, since the long convex lens 29 having a rectangular shape in a top view is convex only in one axial direction, a smartphone whose camera position variation is mainly only in the horizontal width direction of the display screen is optimal as the display device 2.

  According to the present embodiment, since the convex lens 29 is arranged, in addition to the effects of the first embodiment, a plurality of display devices 2 in which the built-in position of the camera 14 varies within the range of the one edge 13. In addition, it is possible to recognize the indicator 15 without considering the variation in the camera position for each display device 2.

  DESCRIPTION OF SYMBOLS 1 Spatial image formation apparatus, 2 Display apparatus (smart device), 3 Support base part, 4 Retroreflective part, 5 Half mirror part, 6 Side plate part, 7 Insertion slot, 8 Video display area (display screen), 9 video, DESCRIPTION OF SYMBOLS 10 Spatial image, 11 Interaction execution apparatus, 12 Outer periphery, 13 One edge, 14 Imaging means (camera), 15 Indicator, 16a, 16b, 16c New image, 17a, 17b, 17c New spatial image, 18 Speaker, 19 communication unit, 20 I / O interface, 21 first storage unit, 22 second storage unit, 23 program storage unit, 24 external storage unit, 25 central control unit (CPU), 26 left area, 27 upper area, 28 lower Area, 29 convex lens, 30 field of view (imaging range), 31 long side, 32 focus

Claims (6)

  A method of displaying an image in a video display area of a display device, forming the image as a spatial image above the half mirror via a half mirror, and executing an interaction with the spatial image, the display device Incorporates imaging means in the vicinity of the outer peripheral edge of the video display area, and the half mirror is arranged so as not to block the imaging range of the imaging means, and the video is displayed after the video is displayed in the video display area. Whether or not an indicator pointing the spatial video has appeared in the imaging range of the means based on an image captured by the imaging means, and the video display when it is determined that the indicator has appeared in the imaging range An interaction execution method, wherein a new video related to the video is displayed in an area and formed as an operation that is continuous with the spatial video.   A display device including a video display area for displaying video and an imaging unit built in the vicinity of the outer periphery of the video display region, and an angle with respect to the display device so as not to block an imaging range of the imaging unit A half-mirror disposed on the display device, displaying a video on a video display area of the display device, and forming the video as a spatial video above the half-mirror via the half-mirror. An interaction execution device for performing an interaction with the first display means for displaying an image in the image display area, and the imaging after the image is displayed in the image display area by the first display means. Determining means for determining whether or not an indicator pointing to the spatial image has appeared in the imaging range of the means; and the indicator in the imaging range by the determining means Appearance to that fact that comprises a second display means for the displaying the new image associated with the video on the video display area when it is determined, a an interaction execution apparatus according to claim.   An image display area is formed in a rectangular shape, and an image pickup means is built in the vicinity of one edge of the outer periphery of the rectangle. The interaction execution device according to claim 2, wherein the image capturing area of the image capturing unit is closed along the edge so as not to block the image display area.   4. The interaction execution device according to claim 3, wherein the long convex lens is exposed above the half mirror surface. A computer mounted on a display device comprising a video display area for displaying video and an imaging means built in the vicinity of the outer periphery of the video display area;
First display means for displaying video in the video display area;
After an image is displayed on the image display area by the first display means, it is determined whether or not an indicator indicating a spatial image formed above the half mirror by the image has appeared in the imaging range of the imaging means. Means to
A program that functions as a second display unit that displays a new video related to the video in the video display area when the determination unit determines that the indicator appears in the imaging range. A computer mounted on a display device comprising a video display area for displaying video and an imaging means built in the vicinity of the outer periphery of the video display area;
First display means for displaying video in the video display area;
After an image is displayed on the image display area by the first display means, it is determined whether or not an indicator indicating a spatial image formed above the half mirror by the image has appeared in the imaging range of the imaging means. Means to
A calculating means for calculating the vertical and / or horizontal position of the indicator in the video display area when the determining means determines that the indicator has appeared in the imaging range;
A program that functions as a second display unit that selects a new video based on a calculation result of the calculation unit and displays the new video in the video display area.

Images