WO2020246237A1

WO2020246237A1 - Information processing system

Info

Publication number: WO2020246237A1
Application number: PCT/JP2020/019899
Authority: WO
Inventors: 道生小林
Original assignee: パロニム株式会社
Priority date: 2019-06-05
Filing date: 2020-05-20
Publication date: 2020-12-10
Also published as: JP2022066293A; JP7554503B2; JP2023054039A; JP7229601B2; JPWO2020246237A1

Abstract

The present invention addresses the problem of making it possible to easily obtain predetermined information relating to an object included in a live image at a timing during viewing of the live image or after viewing of the live image. An association information management unit 121 associates association information for providing object information to a viewer W with an object J designated by the viewer W in a live image L, and manages the association information, the associating being performed at a predetermined timing prior to the designation. A presentation control unit 122 presents the live image L to the viewer W. A designation acceptance unit 123 accepts the designation via a tap operation performed on a TIG area A by the viewer W of the live image L. An acquisition unit 124 acquires the association information associated with the object J for which the designation has been accepted. A provision control unit 125 provides the object information to the viewer W on the basis of the acquired association information.

Description

Information processing system

The present invention relates to an information processing device system.

Conventionally, there has been a technique of providing a link area in a still image or a moving image, accepting an operation on the link area, and displaying predetermined information.
For example, in Patent Document 1, when an operation is performed on a link area displayed as "GO to ticket reservation" on the page of an electronic book of a travel guidebook, a WEB page dedicated to ticket reservation as a link destination is displayed. The technology is described.
Further, for example, the applicant has already applied for a technique that enables acquisition of predetermined information about an object included in the electronic book at a timing different from that of browsing the page of the electronic book including a moving image. (Patent Document 2).

Japanese Patent No. 5537724 Japanese Patent Application No. 2019-047406 Specifications and drawings

However, conventional techniques including Patent Document 1 and Patent Document 2 are techniques for making it possible to acquire predetermined information about an object included in the still image or moving image data generated by imaging.
That is, in the conventional technique, it is assumed that predetermined information about an object included in the data of an existing image is acquired, but a live image such as a so-called live broadcast is not assumed.
However, there are many users who desire to acquire predetermined information about an object included in the live image during viewing of the live image or at a timing after viewing the live image.
Conventional techniques including Patent Document 1 and Patent Document 2 have not been able to sufficiently meet such requests from users.

The present invention has been made in view of such a situation, so that predetermined information regarding an object included in the live image can be easily acquired during viewing of the live image or at a timing after viewing the live image. The purpose is to provide a method of making.

In order to achieve the above object, the information processing system of one aspect of the present invention is
Information for providing predetermined information to the first user for an object existing in a predetermined area in a predetermined time zone, which is designated by the first user in the displayed live image, is specified. A management means that manages the information as linked information by linking at a predetermined timing before
A presentation control means that executes control for presenting the live image to the first user, and
When the first user who is presented with the live image performs a predetermined operation on the predetermined area, the first reception means that recognizes that the designation of the object existing in the predetermined area has been made and accepts the designation. When,
An acquisition means for acquiring the association information managed by being associated with the object for which the designation has been accepted, and
A provision control means that executes control to provide the predetermined information to the first user based on the association information acquired by the acquisition means.
To be equipped.

According to the present invention, predetermined information regarding an object included in the live image can be easily acquired during viewing of the live image or at a timing after viewing the live image.

It is a figure which shows the outline of an example of the service which can be realized by the information processing system which concerns on one Embodiment of this invention. It is a figure which shows the structure of the information processing system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the hardware configuration of the management server in the information processing system of FIG. It is a functional block diagram which shows the functional configuration for executing the association information setting support process and the association information provision process among the functional configurations of the management server of FIG. It is a figure which shows an example of GUI (Graphical User Interface) displayed on the setter terminal when the setter sets an object before the live image is displayed on the setter terminal. It is a figure which shows an example of the GUI displayed on the setter terminal when the set person sets the time zone in which the TIG area becomes effective before the live image is displayed on the setter terminal. It is a figure which shows an example of the GUI displayed on the setter terminal when the set person sets the time zone in which the TIG area becomes effective before the live image is displayed on the setter terminal. Displayed on the setter terminal when the setter sets the position of the TIG area and the time zone during which the TIG area is valid while the live image is being displayed on the setter terminal. It is a figure which shows an example of GUI. Displayed on the setter terminal when the setter sets the position of the TIG area and the time zone during which the TIG area is valid while the live image is being displayed on the setter terminal. It is a figure which shows an example of GUI. The GUI displayed on the setter terminal when the position of the TIG area and the time zone in which the TIG area is enabled are automatically set while the live image is being displayed on the setter terminal. It is a figure which shows an example. The GUI displayed on the setter terminal when the position of the TIG area and the time zone in which the TIG area is enabled are automatically set while the live image is being displayed on the setter terminal. It is a figure which shows an example. It is a figure which shows the example which this service is applied when a clerk of a clothing store tries on a product and distributes a live image. It is a figure which shows the example different from FIG. 9 among the examples to which this service is applied when a clerk of a clothing store tries on a product and distributes a live image. It is a figure which shows the example different from FIG. 9 and FIG. 10 among the examples to which this service is applied when a clerk of a clothing store tries on a product and distributes a live image. It is a figure which shows the example of the mode in which the tag is read based on the operation of a viewer. It is a figure which shows the example of another aspect in which a reader reads a tag. It is a figure which shows the example of another aspect in which a reader reads a tag. It is a figure which shows the example of another aspect in which a reader reads a tag. It is a figure which shows the example of another aspect in which a reader reads a tag. It is a figure which shows the delay which can occur in the information processing system of FIG. It is a figure which shows the example of the method of synchronizing by using the delay shown in FIG. 14A. As a method different from FIG. 14B, it is a figure which shows an example of synchronizing by directly embedding TIG metadata in a live image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

First, with reference to FIG. 1, the outline of this service to which the information processing system of FIG. 2 described later is applied will be described.

FIG. 1 is a diagram showing an outline of an example of a service (hereinafter, referred to as “the service”) that can be realized by the information processing system according to the embodiment of the present invention.

This service is an example of a service provided by a service provider (not shown).
According to this service, information about an object (hereinafter referred to as "object information") can be easily associated with an object displayed in a live image.

Here, the "live image" is a concept that includes all so-called live broadcasts, including those that are displayed with a delay due to the influence of, for example, the communication environment.

Here, the "object" includes an area such as an image or a character showing a person or an object appearing as a subject in the live image, or an image (still image or moving image) such as a telop displayed superimposed on the live image. ) Etc. are included.

"Object information" includes all information related to objects, but in this service, it is assumed that the information cannot be viewed with the live image itself (information obtained from others).
The viewer specifies a desired object among one or more objects displayed on the live image while viewing the live image displayed on a terminal such as a smartphone (viewer terminal 3 as shown in FIG. 2 described later). Do the operation to do.
As a result, the viewer can easily acquire the object information associated with the object specified by the viewer.

In this service, live images are to be distributed to viewers. Further, in the present embodiment, the live image is viewed using the dedicated application software (hereinafter referred to as "viewing application") for viewing the live image, which is pre-installed on the viewer's terminal. ..
However, the method of viewing the live image is not particularly limited, and for example, a method of viewing the live image by the browser function may be adopted.

In FIG. 1, a live image L showing a live broadcast by a local reporter in Hawaii is drawn. This live image L is displayed on the viewer's terminal. That is, the viewer can watch the live broadcast, for example, by watching the live image L displayed on the terminal.

One or more objects are displayed in the live image L.
Specifically, for example, the live image L includes, for example, an object J1 indicating a local reporter (person) and an object J2 indicating a telop described as “local reporter ○○○○”.
In addition to this, the live image L displays clothes (products), hotels (facilities), palm trees (animals and plants), the sea (natural objects), etc. of female talents, all of which correspond to objects. ..
Here, when it is not necessary to individually distinguish various objects such as objects J1 and J2, these are collectively referred to as "object J".

While viewing the live image L, for example, when the viewer is interested in the local reporter appearing in the live image L, the viewer performs an operation of designating the object J1 or the object J2 indicating the local reporter.

The object information of the object J1 and the object J2 is, for example, information about the local reporter, specifically, for example, when the local reporter is a female talent, the name (art name), the name of the production office to which the local reporter belongs, and the disclosure. Profiles etc. that have been created are included. In addition to that, information on clothes worn by local reporters, specifically, for example, brand names, product names, available retailers (including EC (Electronic Commerce) sites), etc. are included. May be good.
As a result, the viewer can obtain the object information of the object J1 or the object J2 specified by himself / herself at an arbitrary timing.
Here, the "arbitrary timing" means not only the timing of viewing the live image, but also the timing of viewing a live image different from the live image, after viewing the live image itself, and the like. ..

Specifically, in this service, in a live image, information for providing the object information to the viewer is linked and managed for each one or more objects J that can be specified based on the operation of the viewer. ing.
The information associated with the object J in this way and for providing the object information to the viewer is hereinafter referred to as "association information".

"Linked information" includes all information for providing object information to the viewer.
Specifically, for example, the following can be adopted as the linking information of the object J1 indicating the local reporter (person) and the object J2 indicating the telop described as "local reporter ○○○○".
That is, information about the local reporter, specifically, for example, when the local reporter is a female talent, the name (stage name), the name of the production office to which the local reporter belongs, the public profile, etc. (object information) are posted. The URL (Uniform Resource Locator) of the created WEB page can be adopted as the association information. In addition to that, the URL of the WEB page containing information on clothes worn by local reporters, specifically, for example, brand names, product names, available stores, etc. (object information), is linked. It can also be adopted as information.

Furthermore, adopting a URL as "association information" is only an example.
For example, various keywords related to the object J can be adopted as the association information. In this case, the viewer's terminal, server, or the like can perform a search on a predetermined search WEB site using the keyword, and provide the search result to the viewer as object information.

Further, the number of linked information linked to one object J is not limited to one, and may be plural.
For example, a URL that is legitimate in this service can be adopted as the first link information. In addition to the URL, a URL separately extracted by this service (for example, automatically recommended and displayed) can be adopted as the second linking information. Then, the viewer may be able to select a desired one of the first association information and the second association information by operating the terminal. In this case, the WEB site existing at the selected URL is provided to the viewer as the object information of the object J.
In other words, the object information provided to the viewer for one object J is not limited to one, and may be plural.

The viewer can receive the object information of the object J by performing an operation on the terminal using the association information associated with the predetermined object J.
For example, when the linked information is a URL, the operation of using the linked information is an operation of instructing to access the URL. In this case, the WEB site existing at the URL is displayed on the viewer's terminal, so that the object information posted on the WEB site is provided to the viewer.
Here, the service provider (not shown) determines the number of accesses when the URL is accessed by the viewer by operating the URL when the link information is a URL. You may count and set various billing according to the number of accesses.

Here, the timing of the operation using the association information associated with the predetermined object J is not particularly limited during viewing of the live image including the object J, and the other live image is being viewed or the live image is displayed. It is possible to set any timing such as after the viewing is finished.
In this way, in this service, when the operation specified by the object J is performed, the object J can be operated at an arbitrary timing by using the association information associated with the predetermined object J. The associative information associated with is stored in a predetermined place (for example, a predetermined storage place in the viewing application).
As a result, the viewer can perform an operation using the desired linked information among the one or more linked information stored in the predetermined location at an arbitrary timing.

Here, the "operation of designating the object J (operation of storing the association information in a predetermined place)" by the viewer is specifically referred to as a "TIG area" in which the object J can exist in the live image. An operation that specifies an area to be called.
Specifically, for example, the object J1 existing in the TIG area A1 is designated by performing the operation of designating the displayed TIG area A1 in the live image L of FIG. 1.
Further, for example, the object J2 existing in the TIG area A2 is designated by performing the operation of designating the displayed TIG area A2 in the live image L of FIG. 1.
Here, when it is not necessary to individually distinguish various TIG areas such as TIG areas A1 and A2, these are collectively referred to as "TIG area A".

In this service, it is assumed that the operation of tapping the TIG area A in which the object J exists is adopted as the operation of designating the object J. That is, by tapping the TIG area A, the viewer can specify the object J corresponding to the TIG area A and store the association information of the object J in a predetermined place.
Specifically, for example, in the example of FIG. 1, when the TIG area A1 is tapped, the association information of the object J1 corresponding to the TIG area A1 (URL of the WEB page on which the object information is posted, etc.) is stored in a predetermined place. Will be done.

Here, in this service, a place called "TIG stock" is adopted as a predetermined place where the association information is stored.
That is, the association information of the object J1 designated by the tap of the TIG area A1 by the viewer is stored and managed in the "TIG stock" as stock information.
Here, as the object information provided to the viewer based on the association information, any form such as text, still image, and moving image can be adopted regardless of the form of the corresponding object J. Therefore, the "TIG stock" can be grasped as a common stock page that straddles a plurality of live images.
Specifically, for example, although not shown, the viewer can display a list of one or more object information stored in the "TIG stock" by tapping a predetermined button displayed on the terminal. ..

In the live image, if the object J tapped by the viewer is not associated with the object J's association information, the association information cannot be provided to the viewer. In such a case, even if the viewer taps the object J, no reaction may be made, or the terminal may display that the tapped object J is not associated with the association information. Good.

As described above, according to this service, the viewer of the live image can easily acquire the object information of the object J by simply tapping the TIG area A in which the object J displayed in the live image exists. Can be done.
In addition, this service can be realized only when various settings are made by the setter for the displayed live image.
Hereinafter, a specific method that makes it possible to easily perform various settings for realizing this service will be described.

FIG. 2 is a diagram showing a configuration of an information processing system according to an embodiment of the present invention.

The information processing system shown in FIG. 2 is configured to include a management server 1, a setter terminal 2, a viewer terminal 3, an external server 4, and a camera CAM.
The management server 1, the setter terminal 2, the viewer terminal 3, and the external server 4 are each connected to each other via a predetermined network N such as the Internet.

The management server 1 is managed by a service provider (not shown), and executes various processes for realizing this service while appropriately communicating with the setter terminal 2 and the viewer terminal 3.

The setter terminal 2 is an information processing device operated by the setter C, and is composed of, for example, a personal computer, a smartphone, a tablet, or the like.

The viewer terminal 3 is an information processing device operated by the viewer W, and is composed of, for example, a personal computer, a smartphone, a tablet, or the like.

The external server 4 manages object information that can be provided by the association information, for example, when the association information is a URL, various WEB sites (WEB sites on which the object information is posted) existing at the URL.

The camera CAM is a camera used for photographing a subject. The mode of the camera CAM is not particularly limited. For example, a camera generally used for surveillance, a Web camera, a camera of a smartphone or a tablet, or the like can be adopted.

FIG. 3 is a block diagram showing an example of the hardware configuration of the management server in the information processing system of FIG.

The management server 1 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a bus 14, an input / output interface 15, an input unit 16, and an output unit 17. A storage unit 18, a communication unit 19, and a drive 20 are provided.

The CPU 11 executes various processes according to the program recorded in the ROM 12 or the program loaded from the storage unit 18 into the RAM 13.
Data and the like necessary for the CPU 11 to execute various processes are also appropriately stored in the RAM 13.

The CPU 11, ROM 12 and RAM 13 are connected to each other via the bus 14. An input / output interface 15 is also connected to the bus 14. An input unit 16, an output unit 17, a storage unit 18, a communication unit 19, and a drive 20 are connected to the input / output interface 15.

The input unit 16 is composed of, for example, a keyboard or the like, and outputs various information.
The output unit 17 is composed of a display such as a liquid crystal display, a speaker, or the like, and outputs various information as images or sounds.
The storage unit 18 is composed of a DRAM (Dynamic Random Access Memory) or the like, and stores various data.
The communication unit 19 communicates with other devices (for example, the setter terminal 2, the viewer terminal 3, the external server 4, etc.) via the network N including the Internet.

A removable medium 30 made of a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is appropriately mounted on the drive 20. The program read from the removable media 30 by the drive 20 is installed in the storage unit 18 as needed.
In addition, the removable media 30 can also store various data stored in the storage unit 18 in the same manner as the storage unit 18.

Although not shown, the setter terminal 2, the viewer terminal 3, and the external server 4 in FIG. 2 can have basically the same configuration as the hardware configuration shown in FIG. Therefore, the description of the hardware configuration of the setter terminal 2, the viewer terminal 3, and the external server 4 will be omitted.

By collaborating with various hardware and various software of the management server 1 of FIG. 3 as described above, it becomes possible to execute various processes including the link information setting support process and the link information providing process on the management server 1. As a result, the service provider (not shown) can provide the above-mentioned service.
The “linking information setting support process” means a process that enables the setter C to easily set the linking information for the object J that may exist in the live image L.
The “linking information providing process” refers to a process that enables the viewer W to easily acquire the linking information associated with the object J that may exist in the live image L.
Hereinafter, the functional configuration for executing the association information setting support processing and the association information provision processing executed in the management server 1 will be described.

FIG. 4 is a functional block diagram showing a functional configuration for executing the association information setting support processing and the association information provision processing among the functional configurations of the management server of FIG.

As shown in FIG. 4, in the CPU 11 of the management server 1, the setting unit 101 functions when the execution of the association information setting support process is controlled. In the setting unit 101, the object setting unit 111, the position setting unit 112, the time zone setting unit 113, and the recognition unit 114 function.
Further, in the CPU 11, the viewing unit 102 functions when the execution of the association information providing process is controlled. In the viewing unit 102, the association information management unit 121, the presentation control unit 122, the designated reception unit 123, the acquisition unit 124, the provision control unit 125, and the stock management unit 126 function.

The object setting unit 111 of the setting unit 101 sets the object J at any timing before the timing at which the association information is associated with the object J.
Here, "setting the object J" means managing the object J that can be displayed on the live image L by adding information (for example, an ID) that can uniquely identify the object J.
Further, "before the timing at which the linking information is linked" means before the linking information is linked to the object J by the linking information management unit 121, which will be described later. For example, the timing before the live image is displayed on the setter terminal 2 and the timing when the object J is recognized by the recognition unit 114 described later after the live image is displayed on the setter terminal 2 are "linked. It corresponds to "before being struck".
The object J set by the object setting unit 111 is stored and managed in the object DB 181 as the object J registered in this service.

The position setting unit 112 sets at least a part of the display area displayed as the live image L as the TIG area A in which the operation of designating the object J by the viewer W is performed at a predetermined timing.
Here, the position setting unit 112 can set the position of the TIG area A in the live image L, for example, at the following timing.

That is, the position setting unit 112 can set the position of the TIG area A in the live image L before the live image L is displayed on the setter terminal 2.
As a result, the setter C can set the position of the TIG area A in the live image L in advance, and then display the live image L on the viewer terminal 3.

Further, the position setting unit 112 is the position of the TIG area A in the live image L based on the input operation for designating the position by the setter C while the live image L is being displayed on the setter terminal 2. Can be set.
As a result, the setter C can set the position of the TIG area A by himself / herself while checking the live image L without setting the position of the TIG area A in the live image L in advance.

Further, the position setting unit 112 can set the position of the TIG area A in the live image L based on the recognition result of the object J by the recognition unit 114 described later.
As a result, the setter C can automatically recognize the object J and automatically set the position of the TIG area A without checking the live image L by himself / herself.

The time zone setting unit 113 sets a time zone in which the TIG area A becomes effective while the live image L is being displayed.
That is, the TIG area A is set as one area of the live image L by the position setting unit 112, but according to the time zone setting unit 113, the TIG area A in which the position is set becomes effective. And the time zone when it becomes invalid can be set.
Here, "valid" means a state in which the viewer can acquire the association information by performing an operation (tap operation) for designating the object J. "Invalid" means a state in which the association information cannot be acquired even if the viewer performs an operation (tap operation) for designating the object J.

Here, the time zone setting unit 113 can set the time zone in which the TIG area A is valid in the live image L, for example, at the following timing.

That is, the time zone setting unit 113 can set the time zone in which the TIG area A is valid in the live image L before the live image L is displayed on the setter terminal 2.
As a result, the setter C can display the live image L on the viewer terminal 3 after setting the time zone in which the TIG area A is valid in the live image L in advance.

Further, the time zone setting unit 113 is in the TIG area A in the live image L based on an input operation for designating the time zone by the setter C while the live image L is being displayed on the setter terminal 2. You can set the time zone when is valid.
As a result, the setter C sets the time zone in which the TIG area A is valid by himself / herself while checking the live image L without setting the time zone in which the TIG area A is valid in the live image L in advance. can do.

Further, the time zone setting unit 113 can set a time zone in which the TIG area A is valid in the live image L based on the recognition result of the object J by the recognition unit 114 described later.
As a result, the setter C can automatically recognize the object J and automatically set the time zone in which the TIG area A is valid without checking the live image L by itself.

While the live image L is being displayed, the recognition unit 114 recognizes the object J existing in the predetermined area in the predetermined time zone.
Specifically, the recognition unit 114 automatically recognizes the object J existing in the live image L displayed on the setter terminal 2. Here, the specific method for the recognition unit 114 to recognize the object J is not particularly limited, and for example, a technique such as image recognition can be used.

By functioning the setting unit 101 having the above functional configuration, it is possible to realize the association information setting support processing in the management server 1.

The association information management unit 121 of the viewing unit 102 refers to the object J existing in the TIG area A in the predetermined time zone, which is designated by the viewer W as the first user in the displayed live image L. The linking information for providing the object information to the viewer W is linked and managed at a predetermined timing before the designation by the viewer W is made.
Specifically, the association information management unit 121 performs the following processing before the operation for designating the object J displayed on the live image L (for example, tap operation) is performed. That is, the association information management unit 121 associates and manages the association information with each of one or more objects J stored and managed in the object DB 181 as the object J that can be displayed on the live image L.

The presentation control unit 122 executes control for presenting a live image L on which one or more objects J can be displayed to the viewer terminal 3. For example, the presentation control unit 122 executes control for presenting the live image L of FIG. 1 to the viewer terminal 3.

Further, the presentation control unit 122 executes control for presenting the stock information (association information) to the viewer W by superimposing it on the live image L or making it independent of the live image L.
Specifically, the presentation control unit 122 superimposes a list of thumbnail images of one or more objects J (for example, “TIG stock”) stored as stock information on the live image L, or sets the live image L as the live image L. Executes the control presented to the viewer W as an independent page.

Further, the presentation control unit 122 controls to superimpose a moving image (for example, animation) indicating that the object J has been added to the stock information on the live image L and present it to the viewer when the object J is designated. To execute.

When the viewer W presenting the live image L taps the TIG area A, the designated reception unit 123 recognizes that the object J existing in the TIG area A has been designated and specifies the TIG area A. Accept.
Specifically, for example, the designated reception unit 123 accepts the designation when the viewer W performs an operation (for example, a tap operation) to specify the object J for the live image L displayed on the viewer terminal 3. ..

The acquisition unit 124 acquires the association information managed by being associated with the object J whose designation has been accepted.
Specifically, for example, the acquisition unit 124 acquires the association information that is associated and managed with the object J that has been designated by the viewer W.

The provision control unit 125 executes control to provide the object information to the viewer terminal 3 based on the association information acquired by the acquisition unit 124.

The stock management unit 126 stores and manages the associated information of one or more objects J designated by the viewer W as stock information, for example, in the above-mentioned "TIG stock". Further, the stock management unit 126 classifies and manages one or more stock information based on a predetermined classification standard and a predetermined classification method.

By functioning the viewing unit 102 having the above functional configuration, it is possible to realize the associated information providing process in the management server 1.

Next, a specific example of the association information setting support process will be described with reference to FIGS. 5 to 8.
FIG. 5 is a diagram showing an example of a GUI displayed on the setter terminal when the setter sets the object before the live image is displayed on the setter terminal.

When the setter C sets the object J before the live image L is displayed on the setter terminal, the setter C uses the GUI shown in FIG. 5 (hereinafter, referred to as “object preset screen”).
In this embodiment, various settings are made using the dedicated application software (hereinafter referred to as "setting application") installed in advance on the setter terminal 2.
However, the method for making various settings is not particularly limited, and for example, a method for making various settings using a browser function may be adopted.

The object preset screen shown in FIG. 5 is configured to include display areas F1 and F2. According to the object preset screen, one or more objects J can be selected from one or more objects J that are managed by being associated with the association information in the object DB181.
In the GUI shown in FIGS. 5, 6A, 6B, 8A and 8B, the object J is expressed as an "item", but both have the same meaning.
Of these, the display area F1 includes a time counter M, a button B1 labeled "TIG edit", a button labeled "heat map", a button labeled "analytics", and a "copy". A button labeled "Preview" and a button labeled "Preview" are displayed.

The time counter M displays the time when the live image L is displayed.
The button B1 described as "TIG edit" is a button that is pressed when the position of the TIG area A and the time zone in which the TIG area A is valid are set. When this button B1 is pressed, the screen transitions to the setting screen shown in FIGS. 6A and 6B.
The button labeled "heat map" is pressed when displaying in the heat map which area of the live image L each of the one or more viewers W viewing the live image L is tapping. Button.
The button labeled "Analytics" is a button that is pressed when various analyzes are performed on the live image L.
The button described as "copy" is a button that is pressed when copying the URL indicating the distribution source of the live image L.
The button described as "preview" is a button that is pressed when previewing a WEB page specified by a URL indicating the distribution source of the live image L.

Further, in the display area F2, a list H of objects J (hereinafter referred to as "object list H") as candidates for setting the TIG area A and a button B2 described as "add from registered items" are displayed. And the button B3 described as "new upload" and the button B1 described as "TIG edit" are displayed.

The object list H is a list of thumbnail images showing each of the objects J1 to J8 to be set. Here, as shown in FIG. 5, a part of the association information may be written together in each of the objects J1 to J8 displayed in the object list H.

By pressing the button B2 labeled "Add from registered items", the object J registered in this service can be added to the object list H.
That is, by pressing the button B2, the setter C selects one or more objects J from the one or more objects J stored and managed in the object DB 181 and adds them to the object list H. be able to.

By pressing the button B3 labeled "new upload", an object J that has not yet been registered in this service can be registered as an object J of this service and then added to the object list H.
That is, by pressing the button B3, the setter C can store and manage the new object J in the object DB 181 and add it to the object list H.

The button B1 labeled "TIG edit" is the same as the button B1 displayed in the display area F1. Therefore, the description of the button B1 displayed in the display area F2 will be omitted.

6A and 6B are examples of GUIs displayed on the setter terminal when the setter sets the time zone in which the TIG area is valid before the live image is displayed on the setter terminal. It is a figure which shows.

When the setter C sets the time zone in which the TIG area A is valid before the live image L is displayed on the setter terminal, the GUI shown in FIGS. 6A and 6B (hereinafter, "time zone preset") Called "screen").

The time counter M and the object list H are displayed on the time zone preset screen shown in FIG. 6A.
As described above, the time counter M displays the time when the live image L is displayed.
As described above, the object list H is a list of thumbnail images showing each of the objects J1 to J8 to be set.

When the setter C presses the button B11, the counting of the time displayed on the time counter M is started. After the time count is started, the setter C performs an operation of selecting a desired object J from the object list H at an arbitrary timing. In the present embodiment, the setter C selects the object J by tapping the desired object J.

For example, when the setter C performs an operation (tap operation) for selecting the object J8, the object list H disappears from the time zone preset screen and includes a thumbnail image showing the object J8, as shown in FIG. 6B. The TIG area A is displayed.

The example of FIG. 6B shows that the TIG area A in which the object J8 can exist becomes effective from the timing when the time of 2 seconds 56 elapses after the time counting by the time counter is started.
Then, the setter C sets a time zone in which the TIG area A in which the object J8 can exist becomes effective while visually recognizing the time displayed on the time counter M.
In addition, the setter C can switch the display / non-display of the TIG area A by pressing the button B12 displayed at the lower right of the time zone preset screen.

Further, when adding any of the objects J2 to J8 other than the object J1, the setter C presses the button B13 marked with "+ item" at the upper right of the time zone preset screen. As a result, the object list shown in FIG. 6A is displayed again, and the setter C selects an arbitrary object J by the same operation as when the object J1 is selected.
The setter C can easily set the time zone in which the TIG area A is valid for each object J by performing the above operations on the time zone preset screen.

7A and 7B show a case where the setter sets the position of the TIG area and the time zone in which the TIG area is valid while the live image is being displayed on the setter terminal. It is a figure which shows an example of the GUI displayed on the setting person terminal.

When the setter C sets the position of the TIG area A and the time zone in which the TIG area A is valid while the live image L is being displayed on the setter terminal, FIGS. 7A and 7A and FIGS. The GUI shown in 7B (hereinafter referred to as "custom setting screen") is used.

The custom setting screens shown in FIGS. 7A and 7B are configured to include display areas F3 and F4.

The live image L is displayed in the display area F3. The setter C can freely set the position of the TIG area A and the time zone in which the TIG area A is valid while viewing the live image L displayed in the display area F3.

The object list H is displayed in the display area F4. The setter C selects an arbitrary object J from one or more objects J displayed as thumbnails in the object list H, and sets the TIG area A at an arbitrary position in an arbitrary time zone of the live image L. be able to.
Here, there are the following two methods for the setter C to set the position of the TIG area A. The first method is the method shown in FIG. 7A, which can be freely set for any position of the live image L (hereinafter, referred to as “free position setting”). The second method is the method shown in FIG. 7B, in which the live image L is divided into a plurality of blocks and the TIG area A is set in any of the blocks (hereinafter, referred to as “block position setting”). Is.

Here, when the free position setting is adopted, the setter C selects an arbitrary object J from the object list H and sets the TIG area A at an arbitrary position.
Specifically, for example, as shown in FIG. 7A, the setter C selects the object J5 from the object list H and sets the TIG area A11 so as to be superimposed on the object J5 appearing in the live image L. Further, the setter C selects the object J7 from the object list H, and sets the TIG area A12 so as to be superimposed on the object J7 appearing in the live image L.
As a result, the setter C can set the positions of the TIG areas A11 and A12 in the live image L, respectively.

Further, each of the TIG areas A11 and A12 for which the position has been set is provided with a switching button B21 capable of switching between valid (ON) and invalid (OFF) at an arbitrary timing.
As a result, the setter C can switch between valid (ON) and invalid (OFF) at an arbitrary timing by pressing the switching button B21.
Further, the setter C makes various settings before the object J appears in the live image L, and simply presses the switching button B21 at the timing when the object J appears in the live image L to enable the TIG area A11. Invalidation can be easily set.
In the example shown in FIG. 7A, the TIG area A11 is valid (ON) and the TIG area A12 is invalid (OFF).

When the block position setting is adopted, the setter C selects an arbitrary object J from the object list H and sets the TIG area A at any position of the blocks T1 to T9.
Specifically, for example, as shown in FIG. 7B, the setter C selects the object J8 from the object list H and also selects the position of the TIG area A13. The position of the TIG area A13 can be selected by pressing the pull-down button B22 provided on the right side of the object J8 displayed as a thumbnail in the object list H. In the example of FIG. 7B, "4" (block T4) is selected by pressing the pull-down button B22.
As a result, the setter C can set the TIG area A13 so as to be superimposed on the object J8 appearing in the live image L at a predetermined timing.

Further, similarly to the free position setting shown in FIG. 7A, the TIG area A13 in which the position is set is provided with a switching button B21 capable of switching between valid (ON) and invalid (OFF) at an arbitrary timing. ing.
As a result, the setter C can switch between valid (ON) and invalid (OFF) at an arbitrary timing by pressing the switching button B21.
In the example shown in FIG. 7B, the TIG area A13 is invalid (OFF).

8A and 8B show the setting when the position of the TIG area and the time zone in which the TIG area is valid are automatically set while the live image is being displayed on the setter terminal. It is a figure which shows an example of the GUI displayed on the person terminal.

When the setting of the position of the TIG area A and the setting of the time zone in which the TIG area A is valid are automatically performed while the live image L is being displayed on the setter terminal, the setter C is shown in the figure. The GUI shown in 8A and FIG. 8B (hereinafter referred to as “recognition object preset screen”) is used.

Here, there are the following two types of recognition object preset screens.
The first type of setting screen is the setting screen shown in FIG. 8A, which is a setting screen used when a method of recognizing an object J existing in the live image L at a specific position without following is adopted.
The second type of setting screen is the setting screen shown in FIG. 8B, which is a setting screen used when a method of following and recognizing an object J existing in the live image L is adopted.
The recognition object preset screen shown in FIGS. 8A and 8B is a setting screen used when a plurality of objects J having common association information are grouped into one group and these are set as one object J.
Before the live image L is displayed on the setter terminal 2, the setter C edits the recognition object preset screen and presses the button B31 labeled "update" to set the object J. be able to.

Of these, the recognition object preset screen shown in FIG. 8A is set as one object J11 that shares the association information K for the three types of objects J111 to J113.
As a result, if any of the objects J111 to J113 is displayed at a specific position of the live image L while the live image L is being displayed on the setter terminal 2, the object J is recognized and TIG. Area A is automatically set.
For example, as shown in FIG. 8A, when the object J113 is displayed at a specific position of the live image L displayed on the setter terminal 2, the object J113 is automatically recognized and the TIG area A21 is automatically set. ..

Further, the recognition object preset screen shown in FIG. 8B is set as one object J12 having the same association information K for n types of objects J12-1 to J12-n (n is an integer value of 1 or more).
As a result, if any of the objects J12-1 to J12-n is displayed at any position of the live image L while the live image L is being displayed on the setter terminal 2, the object J is displayed. It is recognized and the TIG area A is automatically set.
For example, as shown in FIG. 8B, when the object J12-r (r is an integer value of 1 or more and n or less) is displayed on the live image L displayed on the setter terminal 2, the object J12-r is automatically recognized. Then, the TIG area A22 is automatically set.

Further, although not shown, it is possible to automatically recognize all the objects J existing in the live image L and automatically associate all the association information with the objects J.
In this case, the object J may be automatically recognized, for example, by performing various machine learning using the pre-teacher data.

Next, an example in which this service is applied when a clerk CL of a clothing store distributes a live image L introducing a product will be described with reference to FIGS. 9 to 11.
FIG. 9 is a diagram showing an example in which this service is applied when a clerk CL of a clothing store tries on a product I and distributes a live image L.
Specifically, for example, the clerk CL introduces the desired product I in the store while trying it on. The appearance of the clerk CL introducing the product I while trying it on is captured by the camera CAM and distributed as a live image L to the customer (viewer W) via the management server 1. At this time, the setter terminal 2 can acquire various information from the camera CAM, the reader R, and the like to generate the live image L.
Hereinafter, the distribution of the live image L, in which the clerk CL introduces the product I in the store while trying it on, will be described in more detail.

As shown in FIG. 9, the clerk CL tries on the product I in the store and reads the tag D attached to the product I by using the reader R provided at a predetermined position in the store.
The mode of the tag D read by the reader R is not particularly limited. For example, identifiers such as RFID (Radio Frequency Identification) tags, QR codes (registered trademarks), and barcodes can be adopted.

In addition, the clerk CL introduces the product I in front of the camera CAM installed at a predetermined position in the store. As a result, the camera CAM captures an image of the product I from which the tag D has been read and the clerk CL in the state where the product I is tried on.

As described above, the setter terminal 2 acquires the data of the captured image of the clerk CL in the state of trying on the product I from the camera CAM. Further, the setter terminal 2 acquires the information of the tag D attached to the product I from the reader R. The captured image of the clerk CL who tried on the product I is displayed on the setter terminal 2 as a part or all of the live image L. At this time, the product I tried on by the clerk CL is recognized as an object J having object information, and is separated from the image of the clerk CL trying on the product I and displayed as a unique image on the live image L. The object. Specifically, for example, as shown in FIG. 9, the captured image of the clerk CL who tried on the product I and the object J indicating the single product I independent of the captured image are displayed on the setter terminal 2 as the live image L. To.

Further, the live image L is distributed and displayed on the viewer terminal 3 owned by the customer (viewer W) via the management server 1.
The customer (viewer W) can decide whether or not to purchase the product I while observing the appearance of the clerk CL trying on the product I displayed as the live image L on the viewer terminal 3. At this time, the customer (viewer W) accesses the EC (Electronic Customer) site S on which the object information is posted by performing an operation of tapping the product I displayed as the object J on the viewer terminal 3. Can be done.
When the customer (viewer W) purchases the product I, he / she may purchase the product I online from the EC site S or directly at the clothing store (actual store). For example, when the customer (viewer W) purchases the product I on the EC site S, the customer (viewer W) can immediately purchase the product I that he / she is interested in by viewing the live image L. Further, for example, when the customer (viewer W) purchases the product I directly at the clothing store (actual store), the customer (viewer W) may decide to purchase the product I after actually trying it on. At this time, the customer (viewer W) can also obtain (confirm) the information of the product I selected by himself / herself at the clothing store (actual store).

That is, as described with reference to FIG. 5 and the like, the setter C needs to set the position of the TIG area A and the time zone in which the TIG area A is valid while viewing the live image L. there were. However, in the case of the example of FIG. 9, as described above, the clerk CL who performs the live distribution can perform the operation of reading the tag D of the product I with the reader R. As a result, the service provider and the clerk CL can deliver the live image L without setting by the setter C.

In addition, the management server 1 can store and archive the live image L. That is, the management server 1 can make the viewer W view the live image L by distributing the archive of the live image L to the viewer terminal 3 based on the request from the viewer terminal 3.
Further, the setter terminal 2 may be able to acquire a comment from the viewer W who views the live image L via the management server 1. That is, the clerk CL may deliver the interactive live image L while confirming the comment from the viewer W. Specifically, for example, the clerk CL can deliver the live image L while responding to the comment "What kind of tops should I match with?" From the viewer W.

FIG. 10 is a diagram showing an example different from FIG. 9 among the examples in which this service is applied when a clerk of a clothing store tries on a product and distributes a live image.

FIG. 10 shows an example of a mode in which the fitting appearance of the clerk CL is projected on a large display DSP as a full-length display installed in the store of the clothing store.
In the above-mentioned example of FIG. 9, the clerk CL delivered while confirming his / her fitting appearance imaged by the camera CAM as a live image L displayed on the setter terminal 2. That is, the fitting appearance of the clerk CL imaged by the camera CAM was displayed as a live image L on the setter terminal 2 and the viewer terminal 3.
On the other hand, in the example of FIG. 10, the fitting figure of the clerk CL is further projected on the large display DSP installed in the store. As a result, the clerk CL can distribute the live image L by using the display DSP as a full-length mirror.

As shown in FIG. 10, the clerk CL of the clothing store tries on the desired product I in the store, and uses the reader R provided at a predetermined position in the store to attach the tag D to the product I. To read.

In addition, the clerk CL introduces the product I in front of the camera CAM installed in the vicinity of the display DSP in the store. As a result, an captured image is taken with the clerk CL in a state where the tag D is read and the product I is tried on.
The mode of the display DSP installed in the store is not particularly limited. For example, it may be a touch panel type display capable of accepting touch operations, which has a built-in OS (Operating System).
Further, mirror signage may be adopted as the display DSP. In this case, the appearance of the clerk CL is projected on the display DSP as a mirror, and various information and the like as a result of reading the tag D are superimposed and displayed on the appearance of the clerk CL.

The captured image of the clerk CL who tried on the product I captured by the camera CAM is displayed as a live image L on the setter terminal 2 and the viewer terminal 3 as in the example of FIG. Then, the product I is recognized as the object J having the object information and is displayed on the live image L as an image different from the image of the clerk CL (in the state where the product I is tried on) as in the example of FIG. The object.

Here, when the display DSP is a touch panel type, the clerk CL can perform an operation of tapping or dragging the product I (object J) included in the live image L displayed on the display DSP. .. In this case, the clerk CL can change the display arrangement of the object J, switch between "valid" and "invalid" of the object J, and perform operations.

Further, the live image L is distributed and displayed on the viewer terminal 3 owned by the customer (viewer W) via the management server 1.
The customer (viewer W) decides whether or not to purchase the product I while observing the fitting appearance of the clerk CL displayed as the live image L on the viewer terminal 3. Here, as in the example of FIG. 9, when the operation of tapping the product I displayed as the object J on the viewer terminal 3 is performed, the customer (viewer W) is posted the object information of the product I. You can access the EC site S. As a result, the customer (viewer W) can also purchase the product I online from the EC site S, as in the example of FIG. 9, and the clothing store (actual store). ) Will also be available for direct purchase.

In the example in which the service described with reference to FIGS. 9 and 10 is applied, the setter terminal 2 has been described as acquiring various information from the camera CAM and the reader R and displaying the live image L. However, the setter terminal 2 and the camera CAM or reader R do not have to be separate hardware, and may be provided as, for example, a smartphone or tablet terminal.
When a smartphone or tablet terminal including a camera CAM or reader R is provided as the setter terminal 2, the clerk CL uses dedicated application software for distributing live images (hereinafter referred to as "distribution application"). Live images can be distributed. The distribution application provides functions related to distribution including various functions included in the above-mentioned setting application.
In this case, the clerk CL can easily deliver the live image with a single smartphone or the like. In addition, as a clothing store, live images can be distributed using a popular smartphone or the like, so that the cost related to the equipment for distribution can be reduced.

FIG. 11 is a diagram showing an example different from FIGS. 9 and 10 in the example in which this service is applied when a clerk of a clothing store tries on a product and distributes a live image.
In the examples of FIGS. 9 and 10 described above, the captured image (live image L) of the clerk CL who tried on the product I is transmitted from the setter terminal 2 to the management server 1. As shown in the example, there is also a mode in which the OS is built in the display DSP and the reader R and the camera CAM are also built in the display DSP. In this aspect, it is only necessary to install the display DSP (OS, reader R, camera CAM built-in type) at a predetermined position in the store, and it is possible to save the trouble of installing each of the camera CAM and the reader R at a predetermined position in the store. .. In addition, the display DSP (OS, reader R, camera CAM built-in type) that is easy to install can be sold as a set product, and the display DSP (OS, reader R, camera CAM built-in type) can be sold independently. You can also do it.

In an example of this service applicable to the clothing store shown in FIGS. 9 to 11 described above, the clerk CL reads the tag D using the reader R installed in the store. However, this embodiment is merely an example, and there is also an embodiment in which the tag D is read based on the operation of the viewer W without using, for example, the reader R.
FIG. 12 is a diagram showing an example of a mode in which the tag is read based on the operation of the viewer.

In the example of FIG. 12, when the viewer W selects the tag D of the object J displayed on the viewer terminal 3 (for example, a tap operation), the process of recognizing the selected tag D is executed. validate.
This eliminates the need for setting work by the setter C. In addition, information (hereinafter, referred to as "TIG metadata") indicating which position and at which timing of the live image the one or more objects J having the association information are displayed is stored in the management server 1. There is no need. Further, it is not necessary to prepare a reader R in the store.

In an example of this service applicable to the clothing store shown in FIGS. 9 to 11 described above, the clerk CL reads the tag D using the reader R installed in the store. However, this aspect is merely an example, and there are other aspects in which the reader R reads the tag D. 13A to 13D show examples of other embodiments in which the reader R reads the tag D.
13A to 13D are diagrams showing another mode in which the reader R reads the tag D.

13A and 13B show an example in which the tag D is attached to the clothing worn by the model appearing in the fashion show in advance.
In the case of the example of FIG. 13A, the leader R is installed above (for example, the ceiling) near the end of the stage (runway) on which the model walks (near the position where the model turns).
As a result, when the reader R approaches the end of the stage (runway) where the model walks (near the position where the model turns), the reader R can read the tag D previously attached to the clothing of the model. Then, the object J indicating the clothing of the model is displayed in a part of the live image L displayed on the setter terminal 2 (and the viewer terminal 3 (not shown)).
In the case of the example of FIG. 13B, the leader R is embedded near the end of the stage (runway) on which the model walks (near the position where the model turns).
As a result, as in the case of FIG. 13A, when the leader R approaches the end of the stage (runway) where the model walks (near the position where the model turns), the tag D attached to the model's clothing in advance Can read. Then, the object J indicating the clothing of the model is displayed in a part of the live image L displayed on the setter terminal 2 (and the viewer terminal 3 (not shown)).

FIG. 13C shows an example in which a leader R and a camera CAM are installed on a small flying object V such as a drone. In this case, for example, the tag D is arranged at each place (for example, a famous place) on the ground (for example, a tourist spot).
As a result, the small aircraft V captures the ground (for example, a tourist spot) from the sky using the camera CAM, and reads the tags D at various locations by the reader R. Then, although not shown, an object J indicating various places (for example, a famous place) on the ground (for example, a tourist spot) is displayed on a part of the live image L displayed on the setter terminal 2 and the viewer terminal 3. The small flying object V can be used not only outdoors but also indoors (for example, an event of a large-scale indoor facility).

FIG. 13D shows an example in which the reader R is installed on the wall.
In this case, for example, it is possible to attach a tag D to the clothing worn by the talent who participates in the event held indoors in advance.
As a result, the leader R reads the tag D previously attached to the clothing of the personality during the event. Then, the object J indicating the clothing of the model is displayed in a part of the live image L displayed on the setter terminal 2 (and the viewer terminal 3) (not shown).

Next, a method of synchronizing the data displayed on the viewer terminal 3 will be described with reference to FIGS. 14A to 14C.
FIG. 14A is a diagram showing delays that can occur in the information processing system.
The solid arrow in FIG. 14A indicates the data flow and delay when the TIG metadata TM is not embedded in the live image L. Further, the dashed arrow in FIG. 14A indicates the data flow and delay when the TIG metadata TM is directly embedded in the live image L.

First, the data flow and delay when the TIG metadata TM is not embedded in the live image L will be described.
As shown in FIG. 14A, the live image L captured by the camera CAM is transmitted to the setter terminal 2 and the external server 4, respectively. Here, a slight delay D1 occurs before the setter terminal 2 receives the live image L transmitted from the camera CAM. Further, when the setting work on the setter terminal 2 is completed, the TIG metadata TM is transmitted from the setter terminal 2 to the management server 1. Here, a slight delay D2 occurs before the management server 1 receives the TIG metadata TM transmitted from the setter terminal 2. After that, the TIG metadata TM is transmitted from the management server 1 to the viewer terminal 3. Here, a slight delay D3 occurs before the viewer terminal 3 receives the TIG metadata TM transmitted from the management server 1.
That is, the live image L captured by the camera CAM is received by the setter terminal 2, and then the TIG metadata TM is received from the setter terminal 2 via the management server 1 to the viewer terminal 3. Delays D1 to D3 occur.

Further, a delay D4 due to transcoding (compression) or the like occurs before the external server 4 receives the live image L transmitted from the camera CAM. Further, a delay D5 due to decoding (restoration) or the like occurs before the viewer terminal 3 receives the live image L transmitted from the external server 4.
That is, delays D4 and D5 may occur before the live image L captured by the camera CAM is received by the viewer terminal 3 via the external server 4.

FIG. 14B is a diagram showing an example of a method of synchronizing using the delay shown in FIG. 14A.

In FIG. 14B, the live image L, the TIG metadata TM, or the TIG meta offset, which are input or output by the camera CAM, the setter terminal 2, the management server 1, the external server 4, and the viewer terminal 3, respectively. It is shown on the time axis. The rectangular blocks arranged in parallel with the time axis show a video frame showing a length of 1 second. Also, the elliptical blocks arranged in parallel with the time code indicate the TIG metadata TM indicating the length of 1 second. The numbers in the elliptical block indicate the elapsed time since the object J became effective.
Further, the "TIG meta offset" means an offset used to delay the TIG metadata TM. "Offset" means operating different devices with a time lag. By using offset, it is possible to synchronize related data recorded on different timecodes.
That is, the object J becomes effective at the timing of "05:06" (5 minutes and 6 seconds) in the time code of the live image L. Therefore, the TIG meta offset is "+05:06" (plus 5 minutes and 6 seconds), and the elapsed time of the TIG metadata TM is the timing of "05:06" (5 minutes and 6 seconds) in the time code of the live image L. Will be "0" (0 seconds).

Here, when the camera CAM and the setter terminal 2 are compared, the delay D1 occurs in the setter terminal 2.
Further, when the setter terminal 2 and the management server 1 are compared, a delay D2 occurs in the management server 1 for the TIG metadata TM. That is, delays D1 and D2 occur between the camera CAM and the management server 1.
Further, when the management server 1 and the viewer terminal 3 are compared, the delay D3 occurs in the viewer terminal 3. That is, delays D1 to D3 occur between the camera CAM, the setter terminal 2, the management server 1, and the viewer terminal 3.

Next, when the camera CAM and the external server 4 are compared, the delay D4 occurs in the viewer terminal 3.
Further, when the external server 4 and the viewer terminal 3 are compared, the delay D5 occurs in the viewer terminal 3. That is, delays D4 and D5 occur between the camera CAM, the external server 4, and the viewer terminal 3.

Here, the video frame and the TIG metadata TM are compared with respect to the total time of the delay occurring between the camera CAM and the viewer terminal 3. Then, in the example shown in FIG. 14B, the total delay time (delays D4 and D5) of the video frame is longer than the total delay time (delays D1 to D3) of the TIG metadata TM. Therefore, the viewer terminal 3 intentionally displays the TIG metadata TM in a delayed state by using the TIG meta offset.
Specifically, the TIG metadata TM having the TIG meta offset of "+05:06" (plus 5 minutes and 6 seconds) should be aligned with the video frame having the time code of "05:06" (5 minutes and 6 seconds). Delay to. As a result, the video frame and the TIG metadata TM can be synchronized.

In addition to the example shown in FIG. 14B, the total delay time (delays D4 and D5) of the video frame may be shorter than the total delay time (delays D1 to D3) of the TIG metadata TM. In such a case, the display of the live image L is delayed. As a result, the video frame and the TIG metadata TM can be synchronized.

Further, the viewer W may start viewing from the middle of the live image L in which the object J is already valid. In such a case, for example, as shown in FIG. 14B, if the total delay time (delays D4 and D5') of the video frame is longer than the total delay time (delays D1 to D3') of the TIG metadata TM. , The viewer terminal 3 uses the TIG meta offset to delay and display the TIG metadata TM.
Specifically, the elapsed time of the TIG metadata TM having the TIG meta offset of "+05:06" (plus 5 minutes and 6 seconds) in the video frame having the time code of "05:09" (5 minutes and 9 seconds). Delay to match (3 seconds). As a result, the video frame and the TIG metadata TM can be synchronized.

Next, the data flow and delay when the TIG metadata TM is directly embedded in the live image L will be described.
Returning to FIG. 14A, the setter terminal 2 transmits the TIG metadata TM directly embedded in the live image L to the external server 4 as the metadata of the live image L. Then, the external server 4 delivers the live image L in which the TIG metadata TM is embedded to the viewer terminal 3.

In this case, when the camera CAM and the setter terminal 2 are compared, a delay D1 occurs in the setter terminal 2 as in the case where the TIG metadata TM is not directly embedded in the live image L.
Further, when the setter terminal 2 and the external server 4 are compared, the delay D6 occurs in the external server 4.
Further, when the external server 4 and the viewer terminal 3 are compared, the delay D7 occurs in the viewer terminal 3. That is, delays D1, D6, and D7 occur between the camera CAM, the setter terminal 2, the external server 4, and the viewer terminal 3.

FIG. 14C is a diagram showing an example of synchronizing by directly embedding TIG metadata in a live image as a method different from that of FIG. 14B.
As shown in FIG. 14C, when the TIG metadata TM is embedded in the live image L, since the video frame and the TIG metadata TM are set, the video frame and the TIG metadata TM are synchronized.
That is, when the viewer terminal 3 receives the live image L in which the TIG metadata TM is embedded, when decoding the live image L, the viewer terminal 3 extracts the TIG metadata TM from the metadata of the live image L and displays the live image L. Synchronize when you do.
Further, as described above, the viewer W may start viewing from the middle of the live image L in which the object J is already valid. Even in such a case, since the video frame and the TIG metadata TM are set, the video frame and the TIG metadata TM can be synchronized from the timing when the viewing is started.
Further, although not shown, the association information is transmitted from the management server 1 to the viewer terminal 3 in response to the operation of selecting the object J by the viewer W.
Further, although not shown, a TIG metadata TM for management purposes or reuse in an archive may be separately transmitted from the setter terminal 2 to the management server 1 as needed.

Although one embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and modifications, improvements, and the like within the range in which the object of the present invention can be achieved are included in the present invention. Is.

For example, in the above-described embodiment, what is specified as the object J is only an example.
That is, the live image L includes innumerable candidates that can correspond to the object J. Therefore, as described above, the convenience of the viewer W can be further improved by linking the linking information to as many objects J as possible in advance.

Further, in the above-described embodiment, when the viewer W specifies, the associated information of the object J is stored in the "TIG stock" as stock information, but this is only an example. The association information of the object J designated by the viewer W may be configured not to be saved as stock information.

Further, for example, although only one setter C is drawn in FIG. 2, this is only an example, and there may be a plurality of setters C. In this case, a plurality of setters C may perform various setting operations at the same time for the same live image L. As a result, parallel work can be realized by a plurality of setters C, so that the setting work can be made more efficient.

Further, in the example shown in FIG. 8A, a baseball player is described as an example of the object J, but the present invention is not limited to this, and for example, an advertisement such as a signboard of a baseball field is automatically recognized as the object J and the TIG area A may be set automatically.

Also, for example, the shape of the area where the object J can be specified is not limited to a quadrangle, and can be a free rectangle.

Further, for example, the live image L shown in FIGS. 1, 7A, 7B, 8A and 8B is merely an example, and may have other configurations.

Further, according to the present invention, in addition to the above-mentioned services and effects, the following services and effects can be realized.
That is, the viewer W obtains information on the spot while operating the viewer terminal 3 about the clothes and accessories worn by the person appearing in the live image L (for example, the local reporter in FIG. 1). Or can be purchased.
In addition, the viewer W simply performs an operation of designating the telop displayed on the live image L (for example, the telop indicated by the object J2 in FIG. 1), and the performer indicated by the telop (for example, the local reporter in FIG. 1). You can get detailed information about players and players.
In addition, the viewer W can acquire detailed information according to the angle of the camera (not shown) that is capturing the live image L.
Further, for example, when the live image L is a live broadcast of sports, the viewer W simply performs an operation of designating the object J displayed on the live image L to time supplementary information such as techniques and rules. Can be obtained by Lee.
In addition, the viewer W can also perform so-called "throwing money" by dividing the area in the live image L. That is, the viewer W can make a "throwing money" while distinguishing where (or who) the live image L is.
Further, it is also possible to transmit a predetermined signal (hereinafter referred to as "display trigger") from the TIG area A side to an external device (not shown) that switches the camera that captures the live image L. .. As a result, the object J and the TIG area A can be easily synchronized. Further, each scene of the live image L and the object J can be easily synchronized.
Further, the telop (object J) and the TIG area A can be easily synchronized based on the display trigger transmitted from the external device (not shown) that displays the telop (object J) on the live image L. .. Further, each scene of the live image L and the object J can be easily synchronized.
In addition, by touching (tapping) the live image L, the setter C aggregates in real time which (who) of the live image L is popular and reflects it in the subsequent camera work and event progress. Can be done.
Further, if the live image L is being displayed, the TIG area A can be set in the CM (commercial). Since the time and content of the CM (commercial) to be displayed are predetermined, the position of the TIG area A and the time zone in which the TIG area A is effective can be preset.
The service can also be applied to live commercials, such as those performed at recording sites.
Further, the TIG area A may be the entire live image L. Thereby, for example, in the live image L of the concert, the object information of the music can be acquired by tapping any position of the live image L during the performance of the music.
It is also possible to give feedback to the setter C and the person who provides the live image L about how much the tap operation on the TIG area A by the viewer W has been performed.
In addition, since it is expected that the degree of setting the TIG area A will change from the setter C, it is expected that the number of setters C as professionals with advanced skills will increase.
Further, although the conventional sponsors use CM (commercial) as an advertising medium, it can be expected that the number of sponsors using live image L as an advertising medium will increase.
In addition, this service can be provided for live images L of business exhibitions and the like.
In addition, this service can be applied to live images L such as news and news programs. As a result, supplementary information for the viewer W and information that explains difficult information in an easy-to-understand manner can be used as object information.
It is also possible for the viewer W to set the TIG area A in the live video as the setter C. That is, it is also possible to enable the viewer W to use the above-mentioned setting application.
Further, the viewer W can also enable the viewer W to distribute the live image L in which the TIG area A is set by himself / herself.
Further, by accumulating a certain number of objects J (stock information) in the "TIG stock", a predetermined event can be made to occur.
Further, when the TIG area A corresponding to one object J is tapped, a plurality of link destinations (jump destinations) can be selected.
Further, when an area other than the TIG area A is tapped, the animation can be automatically played. As a result, the convenience of the viewer W can be enhanced by using the visual expression.
The "TIG stock" can also be customized. In addition, the effect can be changed according to the specific mode of the operation for designating the object J. Specifically, for example, the effect can be changed by the operation angle, tapping time, double tapping, and the like. More specifically, for example, when the first association information and the second association information are associated with the object J, the first association information is saved when the deep pressing operation is performed, while the object J is touched twice (double tap). ) And the second association information can be saved.
It is also possible to make the object J stockable by the movement of the eyes of the viewer W.

Further, in the examples of FIGS. 9 to 11, when the tag D is read by the reader R, the object with the tag D (for example, clothing) is recognized as the object J and displayed in the live image L. It is not particularly limited to. For example, the camera CAM may recognize the tag D by image processing without reading the tag using the reader R. Further, for example, the object (for example, clothing) itself may be recognized by the image processing of the camera CAM and displayed as the object J in the live image L.

Further, the system configuration shown in FIG. 2 and the hardware configuration of the management server 1 shown in FIG. 3 are merely examples for achieving the object of the present invention, and are not particularly limited.

Further, the functional block diagram shown in FIG. 4 is merely an example and is not particularly limited. That is, it suffices if the information processing system is provided with a function capable of executing the above-mentioned series of processes as a whole, and what kind of functional block is used to realize this function is not particularly limited to the example of FIG. ..

Further, the location of the functional block is not limited to FIG. 4, and may be arbitrary.
For example, in the example of FIG. 4, the functional block required for executing the linking information providing process and the linking information setting support process is configured to be provided on the management server 1 side, but this is only an example. For example, by installing a dedicated application on the setter terminal 2 or the viewer terminal 3, at least a part of these functional blocks may be provided on the setter terminal 2 or the viewer terminal 3.
Further, one functional block may be configured by a single piece of hardware, a single piece of software, or a combination thereof.

When the processing of each functional block is executed by software, the programs constituting the software are installed on a computer or the like from a network or a recording medium.
The computer may be a computer embedded in dedicated hardware. Further, the computer may be a computer capable of executing various functions by installing various programs, for example, a general-purpose smartphone or a personal computer in addition to a server.

The recording medium containing such a program is not only composed of removable media, which is distributed separately from the device main body to provide the program to each user, but also is preliminarily incorporated in the device main body to each user. It is composed of the provided recording media and the like.

In the present specification, the steps for describing a program recorded on a recording medium are not necessarily processed in chronological order according to the order, but are not necessarily processed in chronological order, but in parallel or individually. It also includes the processing to be executed.

Further, in the present specification, the term of the system means an overall device composed of a plurality of devices, a plurality of means, and the like.

Summarizing the above, the information processing system to which the present invention is applied suffices to have the following configuration, and various various embodiments can be taken.
That is, the information processing system to which the present invention is applied is
In the displayed live image (for example, the live image L in FIG. 1), a predetermined time zone (for example, the local reporter in FIG. 1) designated by the first user (for example, the viewer W in FIG. 2) becomes the live image. For an object (for example, an object J1 indicating a local reporter (person) in FIG. 1) existing in a predetermined area (for example, TIG area A1 in FIG. 1) of a predetermined time zone (appearing time zone), predetermined information (for example) is given to the first user. Information for providing (object information) is linked at a predetermined timing before the designation is made (for example, a timing before the live image is displayed on the setter terminal 2), and the information is managed as linked information. Management means (for example, the linked information management unit 121 in FIG. 4) and
A presentation control means (for example, a presentation control unit 122 in FIG. 4) that executes control for presenting the live image to the first user, and
When the first user performs a predetermined operation (for example, a tap operation) on the predetermined area on which the live image is presented (for example, displayed on the viewer terminal 3 in FIG. 2 or the like), the live image is present in the predetermined area. The first receiving means (for example, the designated receiving unit 123 in FIG. 4) that recognizes that the designation of the object is made and accepts the designation.
An acquisition means (for example, acquisition unit 124 in FIG. 4) for acquiring the association information managed by being associated with the object to which the designation is accepted, and
Providing control means (for example, FIG. 4) that executes control for providing the predetermined information to the first user (for example, displaying it on the viewer terminal 3 of FIG. 2 or the like) based on the association information acquired by the acquisition means. Provided control unit 125) and
To be equipped.

As a result, the management means associates the associated information with the object and manages it, and the presentation control means presents the live image to the viewer. Then, the first reception means accepts the designation of the object by the viewer, the acquisition means acquires the association information associated with the object, and the presentation control means provides the viewer with predetermined information based on the association information. To do.
As a result, the viewer can easily acquire predetermined information about the object included in the live image at the timing of viewing the live image or after viewing the live image.
Specifically, for example, the viewer W in the above-described embodiment can easily acquire the object information related to the object J included in the live image L at the timing during viewing of the live image L or after viewing the live image L. can do.

In addition, the first setting means (for example, the object setting unit 111 of FIG. 4) for setting the object (for example, registering an item) first setting the object before the predetermined timing (for example, before the association information is associated with the object J). )When,
A second setting means (for example, FIG.) for performing a second setting of setting at least a part of the display area displayed as the live image as the predetermined area in which the predetermined operation for the designation of the object is performed. Position setting unit 112) of 4 and
Can be further prepared.

As a result, the object can be set before the first setting means associates the association information with the object.

In addition, the first setting means performs the first setting before the live image is displayed.
The second setting means can perform the second setting before the live image is displayed.

As a result, the first setting means sets the object before the live image is displayed, and the second setting means performs a predetermined operation for designating the object before the live image is displayed. Set the position of.
As a result, the setter can set the position of a predetermined area in the live image in advance and then present the live image to the viewer.
Specifically, for example, the setter C in the above-described embodiment can set the position of the TIG area A in the live image L in advance and then present the live image L to the viewer W.

Further, a third setting means (for example, the time zone setting unit 113 in FIG. 4) for setting the predetermined time zone in which the predetermined area becomes effective while the live image is displayed is further provided. be able to.

As a result, the third setting means sets a predetermined time zone in which the predetermined area becomes effective while the live image is being displayed.
As a result, the setter can set a time zone in which the predetermined area is valid in the live image.
Specifically, for example, the setter C in the above-described embodiment can set a time zone in which the TIG area A is effective in the live image L.

Further, the third setting means can perform the third setting at a time before the live image is displayed.

As a result, the third setting means sets a predetermined time zone in which the predetermined area becomes effective while the live image is being displayed at a time before the live image is displayed.
As a result, the setter can present the live image to the viewer after setting the time zone in which the predetermined area is valid in the live image in advance.
Specifically, for example, the setter C in the above-described embodiment may set a time zone in which the TIG area A is effective in the live image L in advance, and then present the live image L to the viewer W. it can.

In addition, the second setting means performs the second setting while the live image is being displayed.
The third setting means can perform the third setting while the live image is being displayed.

As a result, the second setting means sets the position of the predetermined area in which the predetermined operation for designating the object is performed before the live image is displayed, and the third setting means is the most displayed live image. Set a predetermined time zone during which the predetermined area becomes effective.
As a result, the setter can set the time zone in which the predetermined area becomes effective without setting the time zone in which the predetermined area becomes effective in the live image in advance.
Specifically, for example, the setter C in the above-described embodiment sets the time zone in which the TIG area A is valid in the live image L without setting the time zone in which the TIG area A is valid. be able to.

Further, the second setting means performs the second setting based on the designation by the second user while the live image is being displayed.
The third setting means can perform the third setting based on the designation by the second user while the live image is being displayed.

As a result, the second setting means sets the position of the predetermined area in which the predetermined operation for designating the object is performed based on the designation by the setter before the live image is displayed. Further, the third setting means sets a predetermined time zone in which the predetermined area becomes effective based on the designation by the setter while the live image is being displayed.
As a result, the setter can set the time zone in which the predetermined area becomes effective by himself / herself while checking the live image without setting the time zone in which the predetermined area becomes effective in the live image in advance.
Specifically, for example, the setter C in the above-described embodiment does not set in advance the time zone in which the TIG area A is valid in the live image L, and while checking the live image L, the TIG area A himself You can set the time zone when is valid.

Further, a recognition means (for example, the recognition unit 114 in FIG. 4) for recognizing the object existing in the predetermined area in the predetermined time zone while the live image is being displayed can be further provided.
In this case, the second setting means performs the second setting on the object recognized by the recognition means.
The third setting means can make the third setting for the object recognized by the recognition means.

As a result, the recognition means recognizes an object existing in a predetermined area in a predetermined time zone while the live image is being displayed.
As a result, the setter can automatically recognize the object and set the position of the predetermined area without checking the live image by himself / herself.
Specifically, for example, the setter C in the above-described embodiment can automatically recognize the object J and set the position of the TIG area A without checking the live image L by itself.

Further, the first setting means can set the object when the object is recognized by the recognition means.

As a result, the first setting means sets the object when the recognition means recognizes the object.
As a result, the setter can automatically recognize all the objects existing in the live image and automatically associate all the association information with the objects.
Specifically, for example, the setter C in the above-described embodiment can automatically recognize all the objects J existing in the live image L and automatically associate all the association information with the objects J.

1: Management server 2: Setting person terminal 3: Viewer terminal 4: External server, 11: CPU, 12: ROM, 13: RAM, 14: Bus, 15: Input / output interface, 16: Input unit, 17 : Output unit, 18: Storage unit, 19: Communication unit, 20: Drive, 30: Removable media, 101: Setting unit, 111: Registration unit, 112: Position setting unit, 113: Time zone setting unit, 114: Recognition unit , 121: Linked information management unit, 122: Presentation control unit, 123: Designated reception unit, 124: Acquisition unit, 125: Provision unit, 126: Stock management unit, 181: Object DB, C: Setter, W: Viewer , L: Live image, J: Object, A: TIG area, B: Button, M: Time counter, H: Object list, F: Display area, K: Link information, Q: Object information, T: Block, N: Network, D: Tag, DSP: Display, CAM: Camera, R: Tag, V: Small Aircraft

Claims

Information for providing predetermined information to the first user for an object existing in a predetermined area in a predetermined time zone, which is designated by the first user in the displayed live image, is specified. A management means that manages the information as linked information by linking at a predetermined timing before
A presentation control means that executes control for presenting the live image to the first user, and
When the first user who is presented with the live image performs a predetermined operation on the predetermined area, the first reception means that recognizes that the designation of the object existing in the predetermined area has been made and accepts the designation. When,
An acquisition means for acquiring the association information managed by being associated with the object for which the designation has been accepted, and
A provision control means that executes control to provide the predetermined information to the first user based on the association information acquired by the acquisition means.
Information processing system equipped with.
A first setting means for performing the first setting for setting the object before the predetermined timing, and
A second setting means for performing a second setting in which at least a part of the display area displayed as the live image is set as the predetermined area in which the predetermined operation for the designation of the object is performed.
Further prepare
The information processing system according to claim 1.
The first setting means performs the first setting before the live image is displayed.
The second setting means performs the second setting before the live image is displayed.
The information processing system according to claim 2.
A third setting means for setting a predetermined time zone in which the predetermined area becomes effective while the live image is displayed is further provided.
The information processing system according to claim 2 or 3.
The third setting means performs the third setting before the live image is displayed.
The information processing system according to claim 4.
The second setting means performs the second setting while the live image is being displayed.
The third setting means performs the third setting while the live image is being displayed.
The information processing system according to claim 4 or 5.
The second setting means performs the second setting based on the designation by the second user while the live image is being displayed.
The third setting means performs the third setting based on the designation by the second user while the live image is being displayed.
The information processing system according to claim 6.
Further provided with a recognition means for recognizing the object existing in the predetermined area in the predetermined time zone while the live image is displayed.
The second setting means performs the second setting with respect to the object recognized by the recognition means.
The third setting means makes the third setting for the object recognized by the recognition means.
The information processing system according to claim 6.
The first setting means sets the object when the object is recognized by the recognition means.
The information processing system according to claim 8.