WO2018042963A1 - 注目対象物に関する情報提供システムおよび情報提供方法 - Google Patents
注目対象物に関する情報提供システムおよび情報提供方法 Download PDFInfo
- Publication number
- WO2018042963A1 WO2018042963A1 PCT/JP2017/027221 JP2017027221W WO2018042963A1 WO 2018042963 A1 WO2018042963 A1 WO 2018042963A1 JP 2017027221 W JP2017027221 W JP 2017027221W WO 2018042963 A1 WO2018042963 A1 WO 2018042963A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- image
- unit
- dimensional image
- information
- viewer
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Ceased
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4126—The peripheral being portable, e.g. PDAs or mobile phones
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/20—Scenes; Scene-specific elements in augmented reality scenes
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/017—Head mounted
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
-
- G—PHYSICS
- G06—COMPUTING OR CALCULATING; COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating three-dimensional [3D] models or images for computer graphics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/218—Source of audio or video content, e.g. local disk arrays
- H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/4104—Peripherals receiving signals from specially adapted client devices
- H04N21/4122—Peripherals receiving signals from specially adapted client devices additional display device, e.g. video projector
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
- H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
- H04N21/44204—Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/47—End-user applications
- H04N21/472—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content
- H04N21/4722—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content
- H04N21/4725—End-user interface for requesting content, additional data or services; End-user interface for interacting with content, e.g. for content reservation or setting reminders, for requesting event notification, for manipulating displayed content for requesting additional data associated with the content using interactive regions of the image, e.g. hot spots
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
- H04N21/81—Monomedia components thereof
- H04N21/816—Monomedia components thereof involving special video data, e.g 3D video
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
- G02B27/01—Head-up displays
- G02B27/0101—Head-up displays characterised by optical features
- G02B2027/014—Head-up displays characterised by optical features comprising information/image processing systems
Definitions
- the present invention relates to an information providing system and an information providing method related to an object of interest, and particularly suitable for use in a system that provides incidental information related to an object noted by a user among objects displayed in a three-dimensional image. It is.
- VR virtual reality
- VR head mounted display
- goggles and freely walks around the three-dimensional space drawn on the HMD by the computer so that the user can virtually In general, it is possible to have various experiences.
- the VR can also present the user with a world that exceeds the actual constraints of time and space.
- Patent Document 1 useful information corresponding to a target portion is given to the viewer by detecting the target portion of the viewer who is viewing an image that changes over time in a three-dimensional image distributed to the viewer. It is disclosed to provide. Specifically, a spatio-temporal marker is set by adding the accompanying information of the object to the spatio-temporal volume, and the spatio-temporal marker of the object that the viewer is interested in is identified based on the detection result of the target part. And providing the viewer with accompanying information in the spatiotemporal marker.
- Patent Document 1 also integrates and aggregates line-of-sight data, performs predetermined statistical processing, obtains data related to the attention level of product information of each spatiotemporal marker in all the aggregated viewers, and stores the data It is also described that it is provided to a person's computer.
- attribute information of an object arranged in a graphic represented in a three-dimensional space is stored, and search information input from input means includes viewpoint and gazing point movement information. It is described that the importance of an object is determined based on search information, and an object for displaying attribute information is determined based on the determined importance of the object.
- JP 2016-46730 A Japanese Patent No. 3025173
- the present invention has been made to solve such a problem, and an object thereof is to make it possible to use a three-dimensional image as a more useful information transmission medium.
- a three-dimensional image formed by associating associated information with one or more objects included in an image that changes over time is reproduced and displayed on a display. Then, one or more objects that are viewed by the viewer are detected in the three-dimensional image, and are ranked in descending order of the degree of attention.
- the accompanying information corresponding to one or more objects of interest is provided to the viewer through a medium different from the display that displayed the three-dimensional image according to the ranking. I have to.
- the accompanying information corresponding to the target object detected when the 3D image is displayed on the display such as the HMD is displayed after the reproduction of the 3D image is stopped or stopped. Later, it is provided to the viewer through a medium different from the above display. Therefore, more useful accompanying information can be provided to the viewer without being restricted by the amount of information, the type of information, the information provision time, and the like. Then, the viewer can carefully confirm the useful accompanying information as described above provided about the object noticed during viewing of the 3D image after viewing the 3D image.
- the accompanying information related to one or more objects of interest is provided in a state of ranking in descending order of the degree of attention, it is easy for the viewer to confirm the accompanying information preferentially from the objects of high interest. it can.
- a three-dimensional image can be used as a more useful information transmission medium.
- FIG. 1 is a diagram illustrating an example of the overall configuration of an information providing system (hereinafter simply referred to as an information providing system) related to a target object according to the present embodiment.
- the information providing system includes a playback device 100, an HMD 200, a server device 300, and a user terminal 400.
- the playback device 100, the HMD 200, and the server device 300 may be configured as separate units as illustrated, or may be configured such that the playback device 100 and the HMD 200 are provided as one computer. Moreover, the structure by which all the reproducing
- the playback device 100 plays back a three-dimensional image and provides accompanying information associated with an object in the three-dimensional image to a viewer, and also functions as an information providing device.
- the three-dimensional image is a parallax image for creating a VR three-dimensional space, and the three-dimensional image reproduced by the reproduction apparatus 100 is displayed on the HMD 200 worn by the viewer.
- the HMD 200 is an example of a display described in the claims.
- the server device 300 receives the accompanying information transmitted from the playback device 100 and transmits the accompanying information to the user terminal 400 of the viewer designated in advance.
- the user terminal 400 is a terminal used by a viewer, and for example, a mobile terminal such as a smartphone or a tablet terminal can be used.
- the playback apparatus 100 detects which target in the image is being watched by a viewer who is wearing the HMD 200 and watching a three-dimensional image, and at the same time, the target target has a high degree of attention. Ranking in order. Then, after the reproduction of the three-dimensional image ends or stops, the accompanying information regarding the target object is provided from the reproduction device 100 to the user terminal 400 via the server device 300 according to the ranking.
- the address of the user terminal 400 (for example, an e-mail address, a short message address, a communication application or SNS application address, etc.) is registered in advance in the server apparatus 300 in association with the user ID or the like. deep.
- the playback device 100 provides accompanying information regarding the target object to the server device 300 together with the user ID and the like.
- the server device 300 provides accompanying information to the user terminal 400 according to the address of the user terminal 400 registered in advance in association with the user ID sent from the playback device 100.
- FIG. 2 is a block diagram illustrating a functional configuration example of the information providing system according to the present embodiment.
- the playback device 100 of this embodiment includes an image playback unit 11, a target object detection unit 12, a ranking unit 13, and an accompanying information transmission unit 14 as its functional configuration.
- the playback device 100 according to the present embodiment includes an image storage unit 15 as a storage medium.
- the functional blocks 11 to 14 can be configured by any of hardware, DSP (Digital Signal Processor), and software.
- each of the functional blocks 11 to 14 actually includes a CPU, RAM, ROM, etc. of a computer, and is stored in a recording medium such as RAM, ROM, hard disk, or semiconductor memory. Is realized by operating.
- the server apparatus 300 includes an accompanying information receiving unit 31, an address specifying unit 32, and an accompanying information transmitting unit 33 as its functional configuration.
- the server device 300 also includes an address storage unit 34 as a storage medium.
- Each of the functional blocks 31 to 33 can also be configured by any of hardware, DSP, and software.
- the user terminal 400 includes an accompanying information receiving unit 41 and an accompanying information providing unit 42 as its functional configuration.
- Each of these functional blocks 41 to 42 can also be configured by any of hardware, DSP, and software. When configured by software, it may be a dedicated application for receiving information. Alternatively, general-purpose applications (communication application, short message application, web browser, SNS application, etc.) can be used.
- the image storage unit 15 of the playback device 100 stores the above-described three-dimensional image.
- This three-dimensional image is an image whose contents change over time, and one or more objects are included in the three-dimensional image.
- the accompanying information is associated with each object.
- the target object is an object to which the accompanying information is added, and any object that exists in the image can be set as the target object. In other words, the object to which the accompanying information is added becomes the target.
- Accompanying information is information related to the object, and its content is arbitrary.
- the accompanying information is information for explaining the object in detail, and can be any one or a combination of text information, audio information, image information, and moving image information.
- the URL Uniform ⁇ ⁇ ⁇ Resource Locator
- the URL of the introduction site, advertisement information, and the like can be associated as accompanying information.
- the association of the accompanying information with the object in the image is manually performed by the creator of the three-dimensional image, for example. Specifically, while reproducing an original three-dimensional image to which no accompanying information is added, a desired object in the image is designated and identification information is given to the object. Furthermore, desired accompanying information is associated with the identification information and stored in the image storage unit 15.
- the incidental information only needs to be associated with the identification information of the object in the three-dimensional image, and may be stored separately from the three-dimensional image.
- the incidental information is stored in association with the object, what has been described here is an example, and the present invention is not limited to this.
- a three-dimensional image is an image (moving image) whose contents change over time
- the same object is usually continuously displayed over a plurality of frames.
- a three-dimensional image is stored in the image storage unit 15 in a state where identification information of the object is given to each frame for a plurality of frames in which the same object exists. Identification of the same object in each frame can be performed using known image processing techniques relating to image recognition and object tracking.
- the image reproduction unit 11 reproduces the 3D image stored in the image storage unit 15 and displays it on the HMD 200.
- the image playback unit 11 starts playback of the three-dimensional image in response to pressing of the playback button, and automatically ends playback when all playback is completed.
- the image playback unit 11 stops playback of the three-dimensional image at that time.
- the image reproduction unit 11 reproduces a three-dimensional image composed of parallax images for creating a VR three-dimensional space.
- the HMD 200 on which this three-dimensional image is displayed is equipped with a gyro sensor and an acceleration sensor, and can detect the movement of the viewer's head. Then, the image reproduction unit 11 reproduces the three-dimensional image so that the three-dimensional space realized on the display of the HMD 200 dynamically changes according to the movement of the viewer's head detected by the sensor of the HMD 200. Control.
- the image reproduction unit 11 reproduces a three-dimensional image that expands the front three-dimensional space when the viewer faces the front, and a three-dimensional image that expands the right three-dimensional space when the viewer turns to the right.
- a three-dimensional image in which the left three-dimensional space is expanded is reproduced.
- the attention object detection unit 12 detects one or more objects that are viewed by the viewer in the three-dimensional image displayed on the HMD 200 while the image reproduction unit 11 is reproducing the three-dimensional image. As described above, since the three-dimensional image in the direction in which the viewer is facing is always displayed on the HMD 200, the target displayed on the HMD 200 can be defined as the target target.
- an object displayed in a predetermined area (hereinafter referred to as a screen center area 51) set in the center of the display screen is detected as an attention object.
- a screen center area 51 an object displayed in a predetermined area set in the center of the display screen.
- an object that is continuously displayed in the screen center area 51 for a predetermined time or longer is detected as an attention object. In this way, it is possible to prevent the object displayed in the center area 51 of the screen from being accidentally detected as the target object while the viewer is changing his head.
- the ranking unit 13 specifies the degree of attention by the viewer for one or more target objects detected by the target object detection unit 12 when the reproduction of the three-dimensional image by the image reproduction unit 11 ends or stops, Ranking in descending order of attention.
- the degree of attention is evaluated by the length of time when attention is paid to the object. The longer the time of attention, the higher the degree of attention.
- the length of time is equivalent to the number of frames in which the target object is displayed in the screen center area 51. Since each frame is provided with identification information for each object (object), it is possible to detect how many frames of identification information the object has been continuously displayed in the screen center area 51. It is.
- the accompanying information transmission unit 14 displays the accompanying information corresponding to one or more attention objects detected by the attention object detection unit 12 as the ranking unit 13. Are transmitted to the server apparatus 300 together with the user ID and the like according to the ranking specified by the above. That is, the accompanying information transmission unit 14 reads the accompanying information corresponding to the identification information of the target object from the image storage unit 15 and transmits it to the server device 300.
- the accompanying information receiving unit 31 of the server device 300 receives the user ID and the accompanying information of the target object sent from the playback device 100.
- the address specifying unit 32 refers to the address storage unit 34 using the user ID received by the accompanying information reception unit 31 as a key, so that the user terminal 400 stored in the address storage unit 34 in advance is associated with the user ID. Identify the address.
- the accompanying information transmitting unit 33 transmits the accompanying information received by the accompanying information receiving unit 31 to the user terminal 400 according to the address specified by the address specifying unit 32.
- the accompanying information receiving unit 41 of the user terminal 400 receives the accompanying information of the target object sent from the playback device 100.
- the accompanying information providing unit 42 provides the viewer with the following information received by the accompanying information receiving unit 41 according to the ranking.
- the playback apparatus 100 sequentially transmits to the server apparatus 300, and the server apparatus 300 to the user terminal 400, and provides it to the viewer through a medium different from the HMD 200 displaying the three-dimensional image.
- the medium different from the HMD 200 is the user terminal 400.
- the accompanying information transmitting unit 14 of the playback device 100 the accompanying information receiving unit 31, the address specifying unit 32, and the accompanying information transmitting unit 33 of the server device 300, the accompanying information receiving unit 41 of the user terminal 400, and the accompanying information providing unit 42.
- the accompanying information providing unit of the present invention is configured.
- various embodiments can be applied as a form for providing the accompanying information according to the ranking.
- one or more accompanying information can be displayed in a list in ascending order of ranking.
- a name 61 representing the target object, outline information 62, and a link 63 for accessing detailed information are displayed.
- by clicking the link 63 it is possible to receive provision of detailed accompanying information (text information, audio information, image information, video information, etc.).
- the accompanying information is composed of information that can be displayed on the screen such as text information and image information, one or more accompanying information may be displayed as a list in ascending order of ranking.
- FIG. 5 to 7 are diagrams showing an operation example of the information providing system according to the present embodiment configured as described above.
- FIG. 5 shows an operation example of the playback device 100
- FIG. 6 shows an operation example of the server device 300
- FIG. 7 shows an operation example of the user terminal 400.
- the flowchart of the playback device 100 shown in FIG. 5 starts when the playback button of the 3D image is pressed.
- the image reproduction unit 11 reproduces a three-dimensional image stored in the image storage unit 15 and displays it on the HMD 200 (step S1).
- the image reproduction unit 11 reproduces the three-dimensional image so that the three-dimensional space realized on the display of the HMD 200 dynamically changes according to the movement of the viewer's head detected by the sensor of the HMD 200. To control.
- the target object detection unit 12 determines whether or not the target object to which the identification information is added in the image continues to be displayed in the screen center area 51 for a predetermined time or longer. It is determined whether or not the viewer has seen the object for a predetermined time (step S2). If there is no such object, the process returns to step S1. On the other hand, when it is determined that there is an object as described above, the target object detection unit 12 detects the target object as the target object (step S3).
- the ranking unit 13 determines the length of time that the target object detected by the target object detection unit 12 continues to be displayed in the screen center area 51, that is, the length of time that the viewer is paying attention to the target object. Is measured as a degree of attention by the viewer and stored (step S4). That is, the ranking unit 13 stores the identification information of the target object and the length of time (degree of attention) that the viewer has focused on the target in association with each other.
- step S5 the image reproduction unit 11 determines whether or not the reproduction of the three-dimensional image is finished or stopped (step S5), and if not finished or stopped, the process returns to step S1 and continues the processing.
- the attention time degree of attention
- the ranking unit 13 determines the degree of attention of the one or more objects of interest based on the one or more objects of interest and the attention time stored during the reproduction. Ranking in descending order (step S6).
- the accompanying information transmitting unit 14 stores the accompanying information corresponding to the one or more target objects detected by the target object detecting unit 12 during the reproduction of the three-dimensional image, using the identification information of the target object as a key.
- the data is read from the unit 15 and transmitted to the server device 300 according to the ranking specified by the ranking unit 13 (step S7). Thereby, the process of the reproducing apparatus 100 shown in FIG.
- the flowchart of the server apparatus 300 shown in FIG. 6 is repeatedly executed at all times while the power of the server apparatus 300 is on.
- the accompanying information receiving unit 31 determines whether or not the accompanying information of the target object has been received from the playback device 100 (step S11). If not received, the determination in step S11 is repeated.
- the address specifying unit 32 refers to the address storage unit 34 using the user ID received together with the accompanying information as a key.
- the address of the user terminal 400 corresponding to the user ID is specified (step S12).
- the accompanying information transmitting unit 33 transmits the accompanying information received by the accompanying information receiving unit 31 to the user terminal 400 according to the address specified by the address specifying unit 32 (step S13). Thereafter, the processing returns to step S11 and the above processing is continued.
- the flowchart of the user terminal 400 shown in FIG. 7 starts when a dedicated application is started, for example.
- the accompanying information receiving unit 41 determines whether or not the accompanying information of the target object has been received from the server device 300 (step S21). If not received, the determination in step S21 is repeated.
- the accompanying information receiving unit 41 receives the accompanying information of the target object transmitted from the server device 300, the accompanying information providing unit 42, for example, as shown in FIG. Are displayed in a list (step S22). Then, the accompanying information providing unit 42 determines whether any link 63 being displayed in the list is clicked (Step S23).
- the accompanying information providing unit 42 provides the viewer with the accompanying information received by the accompanying information receiving unit 41 (step S24). That is, if the accompanying information is text information, image information, video information, etc., they are displayed on the display of the user terminal 400. If the accompanying information is audio information, it is output from the speaker of the user terminal 400. Thereby, the process of the flowchart shown in FIG. 7 is completed.
- a 3D image in which incidental information is associated with each of one or more objects included in an image that changes over time is reproduced and displayed on the HMD 200.
- the three-dimensional image one or more objects that are viewed by the viewer are detected, and are ranked in descending order of the degree of attention.
- accompanying information corresponding to one or more objects of interest is provided to the viewer through the user terminal 400 different from the HMD 200 according to the ranking.
- the incidental information corresponding to the target object detected when the three-dimensional image is displayed on the HMD 200 is displayed after the end or stop of the reproduction of the three-dimensional image. It is provided to the viewer through the terminal 400. Therefore, more useful accompanying information can be provided to the viewer without being restricted by the amount of information, the type of information, the information provision time, and the like.
- the viewer After the viewer removes the HMD 200, the viewer carefully uses the user terminal 400 to obtain the useful accompanying information provided regarding the target object in the virtual space by the three-dimensional image displayed on the HMD 200. It can be confirmed. Moreover, since the accompanying information related to one or more objects of interest is provided in a state of ranking in descending order of the degree of attention, it is easy for the viewer to confirm the accompanying information preferentially from the objects of high interest. it can.
- a three-dimensional VR image can be used as a more useful information transmission medium.
- FIG. 8 is a block diagram illustrating a functional configuration example of the playback apparatus 100 in this case.
- an accompanying information transmission unit 14 ′ and an image storage unit 15 ′ are provided instead of the accompanying information transmission unit 14 and the image storage unit 15.
- the other image reproduction unit 11, target object detection unit 12 and ranking unit 13 have the same functions as those shown in FIG.
- the image storage unit 15 ′ stores a non-parallax image corresponding to the three-dimensional parallax image in addition to the three-dimensional parallax image for creating a VR three-dimensional space on the HMD 200.
- the non-parallax image is a three-dimensional image or a two-dimensional image that can be displayed in a normal form on the display of the user terminal 400.
- the accompanying information transmission unit 14 ′ transmits to the server device 300 a reproduction scene image of a three-dimensional image when the target object is displayed on the HMD 200 together with the accompanying information of the target object.
- the reproduction scene image referred to here is a non-parallax image corresponding to the three-dimensional parallax image when the target object is displayed on the HMD 200.
- the three-dimensional parallax image when the target object is displayed on the HMD 200 can be specified by, for example, a frame identifier such as a frame number and information indicating the display direction of the image according to the movement of the viewer's head. Is possible.
- the non-parallax image used as the reproduction scene image can be specified by the frame identifier of the non-parallax image corresponding to the frame identifier of the three-dimensional parallax image and information indicating the display direction of the image.
- the accompanying information and the reproduction scene image transmitted from the reproduction apparatus 100 to the server apparatus 300 are transmitted to the user terminal 400 and provided to the viewer.
- the viewer can confirm which object is the accompanying information while remembering the scene viewed in the three-dimensional space of the HMD 200. Can be convenient.
- FIG. 9 is a diagram illustrating a configuration example of the information providing system in this case.
- the information providing system includes a playback device 100 ′, an HMD 200 on which a 3D image played back by the playback device 100 ′ is displayed, and a user terminal 400 ′ used by a viewer.
- the playback device 100 ′ and the user terminal 400 ′ are connected by, for example, a wireless network such as a wireless LAN or Bluetooth (registered trademark) or a wired network such as USB (Universal Serial) Bus).
- the accompanying information transmission unit (not shown) of the playback device 100 ′ transmits the accompanying information to the user terminal 400 ′ according to the address of the user terminal 400 ′ registered in advance in the playback device 100 ′.
- the user terminal 400 ′ provides accompanying information received from the playback device 100 ′ to the viewer using the display or speaker of the user terminal 400 according to the ranking.
- the playback device 100 ′ transmits the accompanying information to the user terminal 400 ′, and the user terminal 400 ′ receives the accompanying information and provides it to the viewer.
- the accompanying information providing unit of the invention is configured.
- the accompanying information may be provided to the viewer using the second display 201 connected to the playback apparatus 100 ′′ as a medium separately from the HMD 200.
- the second display 201 The non-parallax image can be displayed in a normal form.
- the accompanying information providing unit of the present invention is configured by the function of the playback device 100 ′′ displaying the accompanying information on the second display 201. Is done.
- a visual line detection sensor may be installed in the HMD 200 to detect the viewer's actual visual line, and the target displayed in the direction of the visual line may be detected as the target object.
- a camera that captures the viewer's eyes may be provided, and the viewer's line of sight may be detected by performing image processing on a captured image of the eyes.
- the means for detecting the viewer's line of sight is not limited to this example, and other known means can also be used.
- the line of sight detection function is used with priority, and when the HMD 300 not having the line of sight detection function is connected to the playback apparatus 110.
- the target displayed in the screen center area 51 may be detected as the target target.
- Whether the HMD 300 has a line-of-sight detection function may be specified by the user operating the playback device 110, or may be automatically determined by the playback device 110 detecting the identification information of the HMD 300. It may be.
- the accompanying information is displayed on the three-dimensional image even during reproduction of the three-dimensional image, and the accompanying information is provided to the user terminals 400, 400 ′ or the second display 201 even after the reproduction is finished or stopped.
- the content of the accompanying information displayed during the reproduction of the three-dimensional image may be different from the content of the accompanying information provided after the reproduction is finished or stopped.
- a mode in which simple accompanying information is displayed during reproduction of a three-dimensional image, and detailed accompanying information is provided to the viewer after the reproduction is finished or stopped.
- the playback device 100, 100 ′, 100 ′′ is provided with the ranking unit 13 has been described.
- the server device 300 or the user terminal 400 may be provided.
- 100 ′, 100 ′′ transmits the accompanying information of one or more objects of interest and the attention time for each object of interest to the server device 300 or the user terminal 400.
- a mobile terminal such as a smartphone or a tablet terminal has been described as an example of the user terminal 400, 400 ', but the present invention is not limited to this.
- the user terminals 400 and 400 ′ do not necessarily have to be mobile terminals, and can be used as long as they are terminals used by viewers and can be connected to the playback apparatus 100 ′ or the server apparatus 300.
- the playback devices 100, 100 ′, and 100 ′′ are also information providing devices.
- the playback device for 3D images and the information providing device for accompanying information are configured separately.
- the playback device may include the image playback unit 11 and the image storage unit 15 of Fig. 2, and the information providing device may include the target object detection unit 12, the ranking unit 13, and the accompanying information transmission unit 14. Good.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Human Computer Interaction (AREA)
- Computer Hardware Design (AREA)
- Computer Graphics (AREA)
- Software Systems (AREA)
- Computer Networks & Wireless Communication (AREA)
- Optics & Photonics (AREA)
- User Interface Of Digital Computer (AREA)
- Management, Administration, Business Operations System, And Electronic Commerce (AREA)
- Processing Or Creating Images (AREA)
- Information Transfer Between Computers (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Priority Applications (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| EP17845972.3A EP3493553A4 (en) | 2016-08-30 | 2017-07-27 | SYSTEM FOR PROVIDING INFORMATION RELATING TO AN ATTENTION MATERIAL AND INFORMATION PROCESSING METHOD |
| CN201780051919.5A CN109644293A (zh) | 2016-08-30 | 2017-07-27 | 与关注对象物相关的信息提供系统以及信息提供方法 |
| US16/328,215 US10452917B2 (en) | 2016-08-30 | 2017-07-27 | Attention object-related information providing system and information providing method |
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2016-167511 | 2016-08-30 | ||
| JP2016167511A JP6694637B2 (ja) | 2016-08-30 | 2016-08-30 | 注目対象物に関する情報提供システムおよび情報提供方法 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| WO2018042963A1 true WO2018042963A1 (ja) | 2018-03-08 |
Family
ID=61300583
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| PCT/JP2017/027221 Ceased WO2018042963A1 (ja) | 2016-08-30 | 2017-07-27 | 注目対象物に関する情報提供システムおよび情報提供方法 |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US10452917B2 (cg-RX-API-DMAC7.html) |
| EP (1) | EP3493553A4 (cg-RX-API-DMAC7.html) |
| JP (1) | JP6694637B2 (cg-RX-API-DMAC7.html) |
| CN (1) | CN109644293A (cg-RX-API-DMAC7.html) |
| WO (1) | WO2018042963A1 (cg-RX-API-DMAC7.html) |
Families Citing this family (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2009059248A1 (en) | 2007-10-31 | 2009-05-07 | Emsense Corporation | Systems and methods providing distributed collection and centralized processing of physiological responses from viewers |
| US20110106750A1 (en) * | 2009-10-29 | 2011-05-05 | Neurofocus, Inc. | Generating ratings predictions using neuro-response data |
| JP6361714B2 (ja) * | 2015-09-30 | 2018-07-25 | キヤノンマーケティングジャパン株式会社 | 情報処理装置、情報処理システム、その制御方法及びプログラム |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3025173B2 (ja) | 1995-04-13 | 2000-03-27 | シャープ株式会社 | データベース検索システム |
| JP2004180272A (ja) * | 2002-08-21 | 2004-06-24 | Nippon Telegr & Teleph Corp <Ntt> | 情報提供装置及び情報提供方法並びに情報提供プログラム及び情報提供プログラムを記録した記録媒体 |
| JP2012238232A (ja) * | 2011-05-12 | 2012-12-06 | Nippon Hoso Kyokai <Nhk> | 興味区間検出装置、視聴者興味情報提示装置、および興味区間検出プログラム |
| JP2015049637A (ja) * | 2013-08-30 | 2015-03-16 | 日本放送協会 | 興味内容推定装置及び興味内容推定プログラム |
| JP2016046730A (ja) | 2014-08-25 | 2016-04-04 | 学校法人早稲田大学 | 視聴者注目情報提供システム、時空間マーカ設定装置及びそのプログラム、並びに、情報提供装置及びそのプログラム |
Family Cites Families (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2005038008A (ja) * | 2003-07-15 | 2005-02-10 | Canon Inc | 画像処理方法、画像処理装置 |
| US20070156540A1 (en) * | 2006-01-05 | 2007-07-05 | Yoram Koren | Method and apparatus for re-configurable vehicle interior design and business transaction |
| JP5609021B2 (ja) * | 2009-06-16 | 2014-10-22 | ソニー株式会社 | コンテンツ再生装置、コンテンツ提供装置及びコンテンツ配信システム |
| JP5700820B2 (ja) * | 2011-06-01 | 2015-04-15 | 株式会社日本総合研究所 | 映画館で鑑賞した映画を再視聴させる映画再視聴提供システム及び映画再視聴サービス提供方法 |
| JP6028351B2 (ja) * | 2012-03-16 | 2016-11-16 | ソニー株式会社 | 制御装置、電子機器、制御方法、及びプログラム |
| JP5935640B2 (ja) * | 2012-10-01 | 2016-06-15 | ソニー株式会社 | 情報処理装置、表示制御方法及びプログラム |
| US9448404B2 (en) * | 2012-11-13 | 2016-09-20 | Qualcomm Incorporated | Modifying virtual object display properties to increase power performance of augmented reality devices |
| US10133342B2 (en) | 2013-02-14 | 2018-11-20 | Qualcomm Incorporated | Human-body-gesture-based region and volume selection for HMD |
| US9452354B2 (en) * | 2013-06-07 | 2016-09-27 | Sony Interactive Entertainment Inc. | Sharing three-dimensional gameplay |
| JP5845211B2 (ja) * | 2013-06-24 | 2016-01-20 | キヤノン株式会社 | 画像処理装置、画像処理方法 |
| WO2015190138A1 (ja) * | 2014-06-11 | 2015-12-17 | ソニー株式会社 | 表示制御装置、表示制御方法、およびプログラム |
| US9767614B2 (en) * | 2015-03-24 | 2017-09-19 | Nokia Technologies Oy | Augmented reality |
| JP5961736B1 (ja) * | 2015-08-17 | 2016-08-02 | 株式会社コロプラ | ヘッドマウントディスプレイシステムを制御する方法、および、プログラム |
| JP6433862B2 (ja) * | 2015-08-18 | 2018-12-05 | 株式会社コロプラ | ヘッドマウントディスプレイシステム制御プログラム、コンピュータ及び方法 |
-
2016
- 2016-08-30 JP JP2016167511A patent/JP6694637B2/ja active Active
-
2017
- 2017-07-27 WO PCT/JP2017/027221 patent/WO2018042963A1/ja not_active Ceased
- 2017-07-27 EP EP17845972.3A patent/EP3493553A4/en not_active Ceased
- 2017-07-27 US US16/328,215 patent/US10452917B2/en active Active
- 2017-07-27 CN CN201780051919.5A patent/CN109644293A/zh active Pending
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP3025173B2 (ja) | 1995-04-13 | 2000-03-27 | シャープ株式会社 | データベース検索システム |
| JP2004180272A (ja) * | 2002-08-21 | 2004-06-24 | Nippon Telegr & Teleph Corp <Ntt> | 情報提供装置及び情報提供方法並びに情報提供プログラム及び情報提供プログラムを記録した記録媒体 |
| JP2012238232A (ja) * | 2011-05-12 | 2012-12-06 | Nippon Hoso Kyokai <Nhk> | 興味区間検出装置、視聴者興味情報提示装置、および興味区間検出プログラム |
| JP2015049637A (ja) * | 2013-08-30 | 2015-03-16 | 日本放送協会 | 興味内容推定装置及び興味内容推定プログラム |
| JP2016046730A (ja) | 2014-08-25 | 2016-04-04 | 学校法人早稲田大学 | 視聴者注目情報提供システム、時空間マーカ設定装置及びそのプログラム、並びに、情報提供装置及びそのプログラム |
Non-Patent Citations (1)
| Title |
|---|
| See also references of EP3493553A4 |
Also Published As
| Publication number | Publication date |
|---|---|
| JP2018037755A (ja) | 2018-03-08 |
| CN109644293A (zh) | 2019-04-16 |
| US10452917B2 (en) | 2019-10-22 |
| EP3493553A1 (en) | 2019-06-05 |
| US20190244025A1 (en) | 2019-08-08 |
| EP3493553A4 (en) | 2019-06-05 |
| JP6694637B2 (ja) | 2020-05-20 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US12134041B2 (en) | Image processing program, image processing method, and image processing apparatus | |
| WO2018036456A1 (zh) | 追踪识别视频图像中的商品并展示商品信息的方法和装置 | |
| US10560724B2 (en) | Video content distribution system and content management server | |
| CN112995759A (zh) | 互动业务处理方法、系统、装置、设备及存储介质 | |
| KR20150026367A (ko) | 화면 미러링을 이용한 서비스 제공 방법 및 그 장치 | |
| CN104811814A (zh) | 基于视频播放的信息处理方法和系统、客户端和服务器 | |
| WO2019105274A1 (zh) | 媒体内容的展示方法、装置、计算设备及存储介质 | |
| WO2016035748A1 (ja) | サーバ装置、動画配信システム、それに用いられる制御方法及びコンピュータプログラム | |
| US20200322678A1 (en) | Video play method and apparatus, and device | |
| JP2019003367A (ja) | 広告情報処理システム、広告表示領域評価方法および広告情報処理用プログラム | |
| KR20150025122A (ko) | 헤드 마운트 디스플레이 디바이스 및 그 제어 방법 | |
| KR20150133496A (ko) | 네트워크를 통해 헤드마운트형 디스플레이 장치를 포함하는 수신기에 영상을 전송하는 방법과, 이를 위한 송신기, 중계 서버 및 수신기 | |
| JP2017146980A (ja) | コンテンツ再生装置および方法、並びにコンテンツ提供装置および方法 | |
| CN112969093A (zh) | 互动业务处理方法、装置、设备及存储介质 | |
| JP6694637B2 (ja) | 注目対象物に関する情報提供システムおよび情報提供方法 | |
| JP2021140761A (ja) | 配信システム、視聴装置、動画生成装置、情報処理方法、および動画生成方法 | |
| CN106303601A (zh) | 多媒体文件的播放方法和装置 | |
| US12586304B2 (en) | Program, information processing method, and information processing system | |
| JP6804968B2 (ja) | 情報配信装置、情報配信方法および情報配信プログラム | |
| WO2016167160A1 (ja) | データ生成装置および再生装置 | |
| US11962743B2 (en) | 3D display system and 3D display method | |
| KR102084970B1 (ko) | 가상현실 관람 방법 및 가상현실 관람 시스템 | |
| JP2019213196A (ja) | 三次元360°映像データの伝送方法とそのためのディスプレイ装置及び映像保存装置 | |
| CN104618376B (zh) | 播放媒体内容的方法、服务器和显示装置 | |
| JP2019205182A (ja) | 注目対象物に関する情報提供システムおよび情報提供方法 |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
| 121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 17845972 Country of ref document: EP Kind code of ref document: A1 |
|
| NENP | Non-entry into the national phase |
Ref country code: DE |
|
| ENP | Entry into the national phase |
Ref document number: 2017845972 Country of ref document: EP Effective date: 20190226 |