JP7316584B2 - Augmentation image display method and augmentation image display system - Google Patents

Description

The present disclosure relates to an augmentation image display method and an augmentation image display system.

Japanese Unexamined Patent Application Publication No. 2002-200001 discloses an image processing apparatus that automatically generates an additional image for virtually decorating an object to be imaged, reflecting the user's situation. This image processing apparatus recognizes the imaged object based on the imaged image, and performs additions for decorating the object according to the user situation of the user visually recognizing the object and the recognition result of the object. Generate an image. The user situation indicates the user's biometric information, emotion information, preference information, action situation, or environmental information.

JP 2014-203377 A

According to Patent Document 1, for example, an additional image for performing decoration according to the user's preference, physical condition, mood, behavior, etc. can be generated and superimposed on the object and displayed. improvement is possible. However, in Patent Literature 1, an image (for example, a No consideration is given to decoratively superimposing an augmentation image) on the screen. For this reason, there is room for improvement compared to the conventional technology in terms of providing a technique for the user, who is the viewer, to conceptually grasp the feelings of the person appearing in the TV program described above through the screen.

The present disclosure has been devised in view of the above-described conventional circumstances, and enables viewers to conceptually grasp the feelings of a person appearing in a TV program such as a sports game through a screen, thereby improving the viewer's convenience. An object of the present invention is to provide an augmentation image display method and an augmentation image display system that contribute to the improvement of

The present disclosure includes a step of estimating an emotion of at least one person displayed in video data, a step of superimposing image data showing an incarnation of the person on the video data, and generating augmentation image data corresponding to the emotion of the person from the data, displaying the generated augmentation image data in association with the person , and displaying the augmentation image data. and displaying the augmented image data overlaid on the person .

Further, the present disclosure is an augmentation image display system including a broadcasting station device that broadcasts video data , and a receiver that is operated by a user and receives and displays the video data , wherein the receiver: estimating the emotion of at least one person displayed in video data ; superimposing image data showing the incarnation of the person on the video data; and using the video data to derive the emotion of the person from the image data. Provided is an augmentation image display system that generates augmentation image data according to the person and displays the generated augmentation image data in association with the person.

Further, the present disclosure is an augmentation image display system including a broadcasting station device that broadcasts video data , and a receiver that is operated by a user and receives and displays the video data , wherein the receiver: A request for purchase processing of image data showing the avatar of at least one person displayed in video data is sent to the broadcasting station device, and the broadcasting station device purchases the avatar of the person in response to the purchase processing request. purchasing the image data shown, estimating the emotion of the person based on the video data, and generating augmented image data corresponding to the emotion of the person from the image data using the video data. and generating television program data in which the generated augmentation image data is associated with the person , and delivering the television program data to the receiver.

Advantageous Effects of Invention According to the present disclosure, it is possible for a viewer to conceptually grasp the feelings of a person appearing in a TV program such as a sports game through a screen, which contributes to improving convenience for the viewer.

1 is a diagram showing a system configuration example of an augmentation image display system according to Embodiment 1; FIG. A diagram showing an example of viewing a television program on which an augmented image is superimposed via a mobile terminal. A diagram showing an example of how a television program on which an augmentation image is superimposed is viewed through a television receiver. Block diagram showing a hardware configuration example of a broadcasting station device Block diagram showing a hardware configuration example of a mobile terminal Block diagram showing a hardware configuration example of a television receiver Diagram showing an example of user purchase data in table format Diagram showing example display of augmented images in chronological order A diagram showing a display example of an augmentation image Flowchart showing an example of the operation procedure of a series of augmentation image processing

(Circumstances leading to the embodiment according to the present disclosure)
Stream broadcasting services for sports programs continue to demand the development of virtual overlay technology. Such overlay technology can be an intuitive and attractive method when used to provide viewers with information on various statistical data of a game, or to improve entertainment or advertising value. As an overlay technology, Augmented Reality (AR) technology may be used to provide virtual value to broadcast content, such that virtual elements are incorporated over real elements. Incorporation of such AR technology into sports programs increases the possibility that virtual reality (VR) technology and AR technology will be used more and more when broadcasting sports programs.

In systems using the VR and AR technologies described above, tracking parts of a person's body is a fundamental task. A solution is needed. Similar problems are being tackled in computer vision, where raw image processing technologies are enabling applications that track the human body, from individual position tracking to accurate posture assessment. Although it depends on the required specifications, many problems remain to be solved in order to follow the body parts of the human body. Tracking of the central part of an individual can be achieved based on motion tracking using captured video. Such video processing follows or replaces the entire human body, including the face and head. Recently, it has become possible to accurately evaluate the posture of a human body using a plurality of video frames.

Online games, such as those that offer pure cosmetic changes, have also remained a major attraction for microtransactions, with a market of over $3 million in April 2018 alone. Players of online games refine their gaming experience and buy new clothes, new emotions (e.g. some emotions reflected in animations), new pets, new dances for their virtual avatars. do. The key to the success of such microtransactions is that they have no impact on game play. This enhances the player's experience, for example, so that the player is willing to make small purchases to effect a visual change in the game.

Based on the above, the following points can be cited as problems of the conventional technology leading to the following embodiments. First, viewers of sports programs do not have the means to visually see the emotions that the players are feeling. I couldn't figure out what was going on. Second, virtual marketplaces that do not affect gameplay have proven to be very valuable, but have not been applied to broadcast sports programming. Thirdly, if there is an opportunity for advertisements or product introductions during the broadcast of a sports program, it may interfere with viewing or result in failure. This is thought to be because the viewer focuses on content (for example, movement of the player) that is different from the content of advertisements and product introductions in the broadcast stream.

Therefore, in the following embodiment, an augmentation that enables viewers to conceptually grasp the feelings of a person appearing in a TV program such as a sports game through a screen and contributes to improvement of convenience for viewers. An example of an image display method and an augmentation image display system will be described.

(Embodiment)
Hereinafter, embodiments specifically disclosing an augmentation image display method and an augmentation image display system according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, more detailed description than necessary may be omitted. For example, detailed descriptions of well-known matters and redundant descriptions of substantially the same configurations may be omitted. This is to avoid unnecessary verbosity in the following description and to facilitate understanding by those skilled in the art. It should be noted that the accompanying drawings and the following description are provided for a thorough understanding of the present disclosure by those skilled in the art and are not intended to limit the claimed subject matter.

In addition, the term "unit" or "device" used in the embodiments is not limited to a physical configuration simply realized by hardware, and also includes those whose functions are realized by software such as a program. Also, the function of one configuration may be implemented by two or more physical configurations, or the functions of two or more configurations may be implemented by, for example, one physical configuration.

As a first use case according to the embodiment, a scene is assumed in which a family of three members consisting of a father P1, a mother P2, and a daughter P3 as an example of users watches a TV program (for example, a relay program of a sports game) at home. do. Father P1 and mother P2 are very interested in the game of the sport (eg baseball), but daughter P3 has no interest in the game of the sport (eg baseball) (see FIGS. 2 and 3). In the following description, baseball is exemplified as a sport, but the sport is not limited to baseball, and may be any of other sports such as soccer, rugby, golf, and basketball.

<Configuration of Augmentation Image Display System>
FIG. 1 is a diagram showing a system configuration example of an augmentation image display system 1 according to Embodiment 1. As shown in FIG. The augmentation image display system 1 includes at least a broadcasting station device 10 and a mobile terminal 20 or a television receiver 30 . Here, the mobile terminal 20 or the television receiver 30 means including at least one of the mobile terminal 20 and the television receiver 30, and may include both. The broadcasting station device 10 and the mobile terminal 20 or the television receiver 30 are connected to communicate with each other via the network NW1. The network NW1 is composed of at least wired or wireless lines, or may be a line combining wired and wireless lines.

The broadcasting station device 10 is installed in the broadcasting station and receives, via the network NW2, image data captured by a high-performance camera CM1 for broadcast shooting, which is installed, for example, at a predetermined location in a baseball stadium. The network NW2 includes at least a wired line such as an optical fiber line or a wireless line. A wireless line is, for example, a local area network (LAN) or a wide area network (WAN). Broadcasting station device 10 uses video data from camera CM1 to generate a broadcast stream of a baseball game being held at a baseball field, and transmits the broadcast stream to mobile terminal 20 and television receiver 30 via network NW1. To deliver. The broadcast stream is, for example, a stream of broadcast content suitable for terrestrial digital television broadcasting compatible with one-segment (see below) or full-segment (see below) (that is, a stream of video data showing the content of sports games such as baseball). , and hereinafter may be referred to as “player video data”. Details of the internal configuration of the broadcasting station apparatus 10 will be described later with reference to FIG.

A mobile terminal 20 as an example of a receiver is movably arranged in the home of a user (for example, the above-described family of three) (see FIG. 2), and exchanges various signals (for example, user data, control signals) can be transmitted and received. The mobile terminal 20 is, for example, a smart phone or a tablet terminal. The mobile terminal 20 receives and displays, for example, a broadcast stream (player video data) compatible with so-called One Seg (officially, One Segment) or Full Seg (officially, Full Segment) via the network NW1. FIG. 2 is a diagram showing an example of how a television program on which an augmentation image is superimposed is viewed via the mobile terminal 20. As shown in FIG. Therefore, users (specifically, father P1, mother P2, and daughter P3) can view the broadcast stream delivered from the broadcasting station device 10 via the mobile terminal 20. FIG. In addition, the mobile terminal 20 can receive a user's operation, and a skin pack image representing a personification of a player (for example, a professional baseball player) participating in a baseball game (for example, one form of appearance) by the user's operation. (see later) can be purchased through online shopping. Although the illustration is simplified in FIG. 1, an access point (not shown) corresponding to a wireless LAN or the like is installed in the user's home, and the broadcast stream distributed via the network NW1 such as the Internet line is the above-mentioned It may be sent to the mobile terminal 20 via an access point. Details of the internal configuration of the mobile terminal 20 will be described later with reference to FIG.

A television receiver 30, which is an example of a receiver, is fixedly placed in a predetermined location such as a rack in the home of a user (for example, the above-described family of three) (see FIG. 3), and is adapted for digital terrestrial television broadcasting. A stream (player video data) is received via the network NW1 and displayed. FIG. 3 is a diagram showing an example of how a television program on which an augmentation image is superimposed is viewed through the television receiver 30. As shown in FIG. Therefore, users (specifically, father P1, mother P2, and daughter P3) can view the broadcast stream distributed from the broadcasting station device 10 via the television receiver 30. FIG. In addition, the television receiver 30 can accept an operation via a user's remote control (formally, a remote controller), and a player (for example, a form of appearance) participating in a baseball game (for example, For example, a skin pack image (see below) representing an incarnation of a professional baseball player can be purchased through online shopping. Also, the television receiver 30 may be communicably connected to the camera CM2 installed in the user's home via a wired cable LNC, or the camera CM2 may be omitted. Details of the internal configuration of the television receiver 30 will be described later with reference to FIG.

Note that the augmentation image display system 1 may be configured to further include one or more cameras CM1 . . . and a camera CM2.

The camera CM1 is placed at a predetermined location in the baseball stadium (for example, a predetermined location suitable for photographing various players, managers, and mascot characters participating in the baseball game), and photographs the baseball game for broadcasting. The video data (player video data) obtained by shooting is sent to the broadcasting station device 10 via the network NW2. Although only one camera CM1 is shown in FIG. 1, a plurality of cameras CM1 may be provided.

The camera CM2 is placed at a predetermined location in the user's home (for example, a living room where the television receiver 30 is installed), and captures a broadcast stream (for example, player video data) displayed on the television receiver 30. The camera CM2 sends the imaged player video data to the television receiver 30 via the wired cable LNC.

FIG. 4 is a block diagram showing a hardware configuration example of the broadcasting station apparatus 10. As shown in FIG. The broadcasting station device 10 includes a memory 11 , a recording device 12 , a processor 13 , a communication section 14 and an augmentation image processing section 15 . The memory 11, the recording device 12, the processor 13, the communication unit 14, and the augmentation image processing unit 15 are connected by an internal bus so that data or information can be input/output to each other. Note that the augmentation image processing unit 15 may be omitted from the broadcasting station device 10 when the portable terminal 20 is provided with the augmentation image processing unit 25 . Similarly, the augmentation image processing unit 15 may be omitted from the broadcasting station device 10 when the television receiver 30 is provided with the augmentation image processing unit 35 .

The memory 11 is configured using a primary storage device (for example, RAM (Random Access Memory), ROM (Read Only Memory)), and stores programs necessary for executing the operation of the broadcasting station apparatus 10, and furthermore, programs generated during operation. Temporarily store data or information. The RAM is a work memory used when the processor 13 operates, for example. The ROM stores in advance a program for controlling the processor 13, for example.

The recording device 12 is configured using a secondary storage device (for example, HDD (Hard Disk Drive) or SSD (Solid State Drive)) or a tertiary storage device (for example, optical disk, SD card). The recording device 12 may store the player video data sent from the cameras CM1. In addition, the recording device 12 stores user purchase data (FIG. 7) indicating the history of skin pack image purchase processing based on the user's (for example, daughter P3 who shows no interest in baseball games) operation to an online shopping site (not shown). ) can be saved. In addition, the recording device 12 stores emotion characteristic data (not shown) that indicates the correspondence relationship between the heart rate variation characteristic of a person (for example, a player participating in a baseball game) for a certain period of time and the person's emotion. You can

Details of the user purchase data will now be described with reference to FIG.

FIG. 7 is a diagram showing an example of user purchase data PRCS1 in a table format. The user purchase data PRCS1 has a configuration in which items such as a user ID, a skin pack ID, a purchase date, a purchase amount, and remarks, and substantive information for each item are associated with each other. In FIG. 7, the user purchase data PRCS1 is shown in a table format for easy understanding. For example, a user with user ID "001" has a history of purchasing a skin pack image with skin pack ID "1010" worth "¥300" on "July 3, 2019". Similarly, a history is recorded in which the user with the user ID “002” purchased the skin pack image with the skin pack ID “2121” worth “¥500” on “June 30, 2019”. .

In this way, the recording device 12 can hold history information indicating what kind of skin pack image the user has purchased for each user who is a viewer of the broadcast stream. As will be described later, this user purchase data may also be stored in the mobile terminal 20 or the television receiver 30 . In this case, only the user-purchased data related to skin pack images purchased by the user of the portable terminal 20 or the television receiver 30 is stored. It can be reduced compared to the case where user purchase data of all users are stored.

The processor 13 is configured using, for example, a CPU (Central Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field Programmable Gate Array). The processor 13 functions as a control unit of the broadcasting station device 10, and performs control processing for overall control of the operation of each unit of the broadcasting station device 10, input/output processing of data to and from each unit of the broadcasting station device 10, Performs data arithmetic processing and data storage processing. Processor 13 operates according to a program stored in memory 11 .

The processor 13 uses the player video data received and acquired from one or more cameras CM1 . Then, the program video PES and program audio PES are packetized, encoded, multiplexed, and modulated to generate a broadcast stream. Details of the process of generating a broadcast stream using the player video data received and acquired by the processor 13 from one or more cameras CM1 are disclosed, for example, in Japanese Unexamined Patent Application Publication No. 2010-21790. detailed description is omitted.

In addition, the processor 13 performs purchase processing of the augmentation source images (in other words, skin pack images) registered in the augmentation database 155 according to the user's operation. Data (see below) is stored in the recording device 12 . When the processor 13 or the augmentation image processing unit 15 determines that the settlement process for the purchase process is completed, the augmentation source image (skin pack image) is displayed in a predetermined area (for example, the left end or right end) of the broadcast stream. may As a result, when the user (for example, daughter P3) watches a baseball game on the portable terminal 20 or the television receiver 30, even if he/she does not show interest in the baseball game, he/she can apply his/her personal information to the baseball player. You can use your favorite character image data as a skin pack image. In other words, the mobile terminal 20 or the television receiver 30 superimposes the skin pack image purchased by the user in light of the player's emotion or movement, thereby gradually increasing the daughter P3's interest in the baseball game. be able to.

The communication unit 14 is capable of transmitting and receiving data or information with one or more cameras CM1 through the network NW2, and is capable of transmitting data or information to and from the mobile terminal 20 or the television receiver 30 through the network NW1. It consists of a communication circuit capable of transmitting and receiving data or information. As the communication method by the communication unit 14, for example, a wired communication method such as WAN, LAN, and power line communication corresponding to each of the networks NW1 and NW2, and a wireless communication method such as WiFi (registered trademark) and mobile communication for mobile phones and the like are exemplified.

The augmentation image processing unit 15, like the processor 13, is configured using, for example, a CPU, DSP, or FPGA. The augmentation image processing unit 15 has an emotion determination unit 151, an extension processing unit 152, a follow-up processing unit 153, a correction processing unit 154, and an augmentation database 155 as functional configurations. Augmentation image processing, which is a generic term for the processing that is performed. At least one of the emotion determination unit 151, the extension processing unit 152, the follow-up processing unit 153, and the correction processing unit 154 in the augmentation image processing unit 15 may be executed by the processor 13, and the augmentation image processing unit 15 may be executed by the processor 13. Augmentation database 155 of 15 may be configured within recording device 12 .

The emotion determination unit 151 uses the player video data (for example, one form of image data) saved in the recording device 12 to determine the heart rate of the player and the heart rate of the player shown in each image that constitutes the player video data. Analyze fluctuations over a certain period of time (for example, several seconds). Specifically, the emotion determination unit 151 detects the face of the player appearing in the player video data, and detects the time series of pixel values of specific components (for example, G (green) component) of each pixel of the face region (skin color region). Generate data. The emotion determination unit 151 performs predetermined filtering processing on the time-series data, detects a pulse wave from which high-frequency noise components and low-frequency fluctuation components are removed, and derives a heart rate (pulse rate). The details of the processing for analyzing the heart rate of the player using the player video data are disclosed, for example, in Japanese Patent No. 6323809 or Japanese Patent No. 6358506, so detailed description thereof will be omitted.

The emotion determination unit 151 also determines the emotion corresponding to the heart rate variation characteristics of the player based on the emotion characteristic data stored in the recording device 12 and the analysis result of the heart rate variation in a certain period of time. Determine and output as feelings (feelings).

The augmentation processing unit 152 selects an augmentation source image corresponding to the player's emotion determined by the emotion determination unit 151 from the augmentation database 155 as an augmentation image. The augmentation processing unit 152 may generate an augmentation image corresponding to the player's emotion determined by the emotion determination unit 151 using an augmentation source image selected from the augmentation database 155 . The augmentation processor 152 superimposes the selected or generated augmentation image for application to the player on the broadcast stream. Specific means for this superimposition may be means for overlaying directly above the player (see FIG. 8) or means for displaying around the player (see FIG. 9). Details of the augmentation image will be described later with reference to FIGS. 8 and 9. FIG.

The follow-up processing unit 153 analyzes the player video data stored in the recording device 12 and determines whether or not the player has moved. If the follow-up processing unit 153 determines that the player has moved, it analyzes the player video data to determine the player's posture after the movement (for example, the player's limbs, physique, and other body shape), the player's player video, and so on. Identify each location (coordinates and area) in the data. In addition to the player video data, the follow-up processing unit 153 also uses sensing results obtained by other sensors and the like to determine the player's posture after movement (eg, the player's limbs, physique, and other whole-body shape), the player's Each position (coordinates and area) in the player video data may be specified.

When the follow-up processing unit 153 determines that the player has moved, the correction processing unit 154 corrects the posture of the player after movement (for example, the shape of the player's whole body such as limbs and physique), the position of the player in the player video data, and the position of the player in the player video data. Each of the (coordinates and regions) identification results are used to change (modify) properties (eg, shape, color, size, or a combination thereof) of the augmented image currently in use. Note that, when a modification permission condition is assigned to the augmentation source image, the modification processing unit 154 may modify the augmented image within the range indicated by the modification permission condition. In other words, depending on the augmentation image, the correction processing unit 154 corrects the augmentation image on the premise of some restrictions when the player moves. As for the correction permission condition, for example, when the emotion is determined to be uplifting, the color (eg, blue) may be modified step by step to a brighter color (eg, blue→green→yellow→orange→red). The color may be changed to a dark color if it is determined to be a setback, but the use of black is not permitted in order to ensure minimum visibility. Needless to say, the modification permission condition is not limited to the contents described above.

The augmentation database 155 holds augmentation source image data, which is the source image of the augmented image to be superimposed to apply to the player on the broadcast stream. In the accompanying drawings, the augmentation database is abbreviated as "Aug DB". The augmented image may be the same as the original augmented image. The original augmentation image is an object (object) directly overlaid on the player, such as clothes, hats, etc., having a three-dimensional shape, and indicates the player's emotion. Also, the augmentation source image may be an image or indicator having a two-dimensional shape, and the image or indicator indicates the player's emotion. Also, the augmentation source image may be an image of the player's virtual followers (eg, fans, supporters, imitators), and the image indicates the player's emotions.

The augmentation source image or the augmentation image can be modified by the modification processing unit 154 according to the player's feelings. The possibilities for modification may vary according to modification permission conditions pre-assigned to the augmentation source image. For example, clothing color, size, and shape may be modified. However, the augmentation image is changed (corrected) only when it is detected (determined) that the player's emotion has changed. Note that the original augmentation image is not limited to image data registered in the augmentation database 155 in advance, and is provided by the system administrator of the augmentation image display system 1 or a third party (for example, a sponsor). Image data may be added or deleted as appropriate. For example, when a television program broadcasting station in which the broadcasting station apparatus 10 is installed and a sponsor company such as a movie production company conclude a sponsorship contract for a new movie, one or more character image data appearing in the new movie are augmented. It may be additionally registered as a reference source image.

For example, when the original image for augmentation is a blue bowler hat, the correction processing unit 154 gradually enlarges the bowler hat as the emotion determination unit 151 determines that the player's emotion is in an excited state. Fixed.

FIG. 5 is a block diagram showing a hardware configuration example of the mobile terminal 20. As shown in FIG. The mobile terminal 20 includes a memory 21 , a recording device 22 , a processor 23 , a communication section 24 , an augmentation image processing section 25 and an input display section 26 . The memory 21, the recording device 22, the processor 23, the communication section 24, and the augmentation image processing section 25 are connected by an internal bus so that data or information can be input/output to each other. Note that the augmentation image processing unit 25 may be omitted from the portable terminal 20 when the broadcasting station device 10 is provided with the augmentation image processing unit 15 .

The memory 21 is configured using a primary storage device (eg, RAM, ROM), and temporarily stores programs necessary for executing the operation of the mobile terminal 20 as well as data or information generated during operation. The RAM is a work memory used when the processor 23 operates, for example. The ROM stores in advance a program for controlling the processor 23, for example.

The recording device 22 is configured using a secondary storage device (eg, HDD or SSD) or a tertiary storage device (eg, optical disk, SD card). The recording device 22 may store the broadcast stream (player video data) distributed from the broadcasting station device 10 . In addition, the recording device 22 stores user purchase data (FIG. 7) that indicates the history of skin pack image purchase processing based on the user's (for example, daughter P3 who shows no interest in baseball games) operation to an online shopping site (not shown). ) can be saved. In addition, the recording device 22 stores emotion characteristic data (not shown) that indicates the correspondence relationship between the heart rate fluctuation characteristics of a person (for example, a player participating in a baseball game) during a certain period of time and the person's emotion. You can

The processor 23 is configured using, for example, a CPU, DSP or FPGA. The processor 23 functions as a control unit of the mobile terminal 20, and performs control processing for overall control of the operation of each unit of the mobile terminal 20, data input/output processing with each unit of the mobile terminal 20, and data calculation. Processing and data storage. Processor 23 operates according to a program stored in memory 21 .

In addition, the processor 23 performs purchase processing of the augmentation source images (in other words, skin pack images) registered in the augmentation database 255 according to the user's operation. The data (see FIG. 7) are stored in the recording device 22 .

The communication unit 24 is configured by a communication circuit capable of transmitting/receiving data or information to/from the broadcasting station apparatus 10 via the network NW1. The communication method by the communication unit 24 includes, for example, a wired communication method such as WAN, LAN, and power line communication corresponding to the network NW1, and a wireless communication method such as WiFi (registered trademark) and mobile communication for mobile phones. are exemplified.

The augmentation image processing unit 25, like the processor 23, is configured using, for example, a CPU, DSP, or FPGA. The augmentation image processing unit 25 has an emotion determination unit 251, an extension processing unit 252, a follow-up processing unit 253, a correction processing unit 254, and an augmentation database 255 as functional configurations. Augmentation image processing, which is a generic term for the processing that is performed. Note that at least one of the emotion determination unit 251, the extension processing unit 252, the follow-up processing unit 253, and the correction processing unit 254 of the augmentation image processing unit 25 may be executed by the processor 23, and the augmentation image processing unit Augmentation database 255 of 25 may be configured within recording device 22 .

The emotion determination unit 251 uses the player video data (for example, one form of image data) stored in the recording device 22 to determine the heart rate of the player and the heart rate of the player shown in each image that constitutes the player video data. Analyze fluctuations over a certain period of time (for example, several seconds). Specifically, the emotion determination unit 251 detects the face of the player appearing in the player video data, and detects the time series of pixel values of specific components (for example, G (green) component) of each pixel of the face area (flesh color area). Generate data. The emotion determination unit 251 performs predetermined filtering processing on the time-series data, detects a pulse wave from which high-frequency noise components and low-frequency fluctuation components are removed, and derives a heart rate (pulse rate).

In addition, the emotion determination unit 251 determines the emotion corresponding to the heart rate variation characteristic of the player based on the emotion characteristic data stored in the recording device 22 and the analysis result of the heart rate variation in a certain period of time. Determine and output as feelings (feelings).

The augmentation processing unit 252 selects the augmentation source image corresponding to the player's emotion determined by the emotion determination unit 251 from the augmentation database 255 as the augmentation image. The augmentation processing unit 252 may generate an augmentation image corresponding to the player's emotion determined by the emotion determination unit 251 using an augmentation source image selected from the augmentation database 255 . The augmentation processor 252 superimposes the selected or generated augmentation image for application to the player on the broadcast stream (see above).

The follow-up processing unit 253 analyzes the player video data stored in the recording device 22 and determines whether or not the player has moved. If the follow-up processing unit 253 determines that the player has moved, it analyzes the player video data to determine the player's posture after the movement (for example, the player's limbs, physique, and other body shape), the player's player video, and so on. Identify each location (coordinates and area) in the data. In addition to the player image data, the follow-up processing unit 253 uses sensing results obtained by other sensors and the like to determine the player's posture after movement (for example, the player's limbs, physique, and other whole-body shape), the player's Each position (coordinates and area) in the player video data may be specified.

When the follow-up processing unit 253 determines that the player has moved, the correction processing unit 254 corrects the posture of the player after movement (for example, the shape of the player's whole body such as limbs and physique), the position of the player in the player video data, and the position of the player in the player video data. Each of the (coordinates and regions) identification results are used to change (modify) properties (eg, shape, color, size, or a combination thereof) of the augmented image currently in use. Note that, if a modification permission condition is assigned to the augmentation source image, the modification processing unit 254 may modify the augmented image within a range indicated by the modification permission condition. In other words, depending on the augmentation image, the correction processing unit 254 corrects the augmentation image on the premise of some restrictions when the player moves.

The augmentation database 255 holds augmentation source image data, which is the source image of the augmented image to be superimposed to apply to the player on the broadcast stream.

The input display unit 26 is configured using, for example, a touch panel. The input display unit 26 can accept a user's operation, and is user data for display generated by the processor 23 or the augmentation image processing unit 25 (for example, player video data, or an image on which an augmentation image is superimposed). player video data). For example, the input display unit 26 receives a user's operation and outputs it to the processor 23 at the time of purchase processing of the augmentation source image (skin pack image). The processor 23 may display the augmentation source image (skin pack image) in a predetermined area of the input display section 26 when determining the completion of the payment process for the purchase process. As a result, the user (for example, daughter P3) can use his or her favorite character image data as a skin pack image as if it were applied to a baseball player, even if the user does not show interest in a baseball game. In other words, the portable terminal 20 superimposes the skin pack image purchased by the user on the player's emotion or movement, thereby gradually increasing the interest of the daughter P3 in the baseball game.

FIG. 6 is a block diagram showing a hardware configuration example of the television receiver 30. As shown in FIG. The television receiver 30 includes a memory 31 , a recording device 32 , a processor 33 , a communication section 34 , an augmentation image processing section 35 and a display panel 36 . The memory 31, the recording device 32, the processor 33, the communication section 34, the augmentation image processing section 35, and the display panel 36 are connected by an internal bus so that data or information can be input/output to each other. Note that the augmentation image processing unit 35 may be omitted from the television receiver 30 when the broadcasting station device 10 is provided with the augmentation image processing unit 15 .

The memory 31 is configured using a primary storage device (eg, RAM, ROM), and temporarily stores programs necessary for executing the operation of the television receiver 30 as well as data or information generated during operation. do. The RAM is a work memory used when the processor 33 operates, for example. The ROM stores in advance a program for controlling the processor 33, for example.

The recording device 32 is configured using a secondary storage device (eg, HDD or SSD) or a tertiary storage device (eg, optical disc, SD card). The recording device 32 may store the broadcast stream (player video data) distributed from the broadcasting station device 10, and the captured video data of the player video data sent from the camera CM2. In addition, the recording device 32 stores a history of skin pack image purchase processing based on the operation of the user (for example, daughter P3 who shows no interest in baseball games) to an online shopping site (not shown) via the remote control (see above). User purchase data (see FIG. 7) indicating the may be stored. In addition, the recording device 32 stores emotion characteristic data (not shown) that indicates the correspondence relationship between the heart rate fluctuation characteristics of a person (for example, a player participating in a baseball game) during a certain period of time and the person's emotion. You can

The processor 33 is configured using, for example, a CPU, DSP or FPGA. The processor 33 functions as a control unit of the television receiver 30, performs control processing for overall control of the operation of each unit of the television receiver 30, and performs data input to and from each unit of the television receiver 30. It performs output processing, data arithmetic processing, and data storage processing. Processor 33 operates according to a program stored in memory 31 .

The processor 33 demodulates, decodes, extracts, selects, and synthesizes the broadcast stream received and acquired from the broadcast station device 10, thereby producing program video data including commercial video for viewing and commercial audio. Player video data including program audio data is output via the input display unit 26 and a speaker (not shown). Details of the process of generating player video data for viewing (for display) using the broadcast stream received and acquired by the processor 33 from the broadcasting station device 10 are disclosed in, for example, Japanese Unexamined Patent Application Publication No. 2010-21790. Therefore, detailed description is omitted.

In addition, the processor 33 performs purchase processing of the augmentation source images (in other words, skin pack images) registered in the augmentation database 355 in accordance with the user's operation. The data (see FIG. 7) are stored in the recording device 32 .

The communication unit 34 is configured by a communication circuit capable of transmitting/receiving data or information to/from the broadcasting station apparatus 10 via the network NW1. The communication method by the communication unit 34 includes, for example, a wired communication method such as WAN, LAN, and power line communication corresponding to the network NW1, and a wireless communication method such as WiFi (registered trademark) and mobile communication for mobile phones. are exemplified. Further, the communication unit 34 receives and accepts the user's operation using the remote control (see above) during the purchase process of the augmentation source image (skin pack image), and outputs the operation to the processor 33 . The processor 33 may display the augmentation source image (skin pack image) in a predetermined area of the display panel 36 when determining the completion of the payment process for the purchase process. As a result, the user (for example, daughter P3) can use his or her favorite character image data as a skin pack image as if it were applied to a baseball player, even if the user does not show interest in a baseball game. In other words, the television receiver 30 superimposes the skin pack image purchased by the user in light of the player's emotion or movement, thereby gradually increasing the daughter P3's interest in the baseball game.

The augmentation image processing unit 35, like the processor 33, is configured using, for example, a CPU, DSP, or FPGA. The augmentation image processing unit 35 has an emotion determination unit 351, an extension processing unit 352, a follow-up processing unit 353, a correction processing unit 354, and an augmentation database 355 as functional configurations. Augmentation image processing, which is a generic term for the processing that is performed. At least one of the emotion determination unit 351, the extension processing unit 352, the tracking processing unit 353, and the correction processing unit 354 of the augmentation image processing unit 35 may be executed by the processor 33, and the augmentation image processing unit 35 may be executed by the processor 33. Augmentation database 355 of 35 may be configured in recording device 32 .

The emotion determining unit 351 uses the player video data (for example, one form of image data) saved in the recording device 32 or the imaged video data thereof to calculate the heart rate of the player shown in each image forming the player video data. and the fluctuation of the heart rate over a certain period (for example, several seconds). Specifically, the emotion determination unit 351 detects the face of the player appearing in the player video data, and detects the time series of pixel values of specific components (for example, G (green) components) of each pixel of the face region (skin color region). Generate data. The emotion determination unit 351 performs predetermined filtering processing on the time-series data, detects a pulse wave from which high-frequency noise components and low-frequency fluctuation components are removed, and derives a heart rate (pulse rate).

In addition, the emotion determination unit 351 determines the emotion corresponding to the characteristics of the heart rate variation based on the emotion characteristic data stored in the recording device 32 and the analysis result of the heart rate variation in a certain period of time. Determine and output as feelings (feelings).

The augmentation processing unit 352 selects from the augmentation database 355 the augmentation source image corresponding to the player's emotion determined by the emotion determination unit 351 as the augmentation image. The augmentation processing unit 352 may generate an augmentation image corresponding to the player's emotion determined by the emotion determination unit 351 using the augmentation source image selected from the augmentation database 355 . The augmentation processor 352 superimposes the selected or generated augmentation image for application to the player on the broadcast stream (see above).

The follow-up processing unit 353 analyzes the player video data stored in the recording device 32 and determines whether or not the player has moved. If the follow-up processing unit 353 determines that the player has moved, it analyzes the player video data to determine the player's posture after the movement (for example, the player's limbs, physique, and other body shape), the player's player video, and so on. Identify each location (coordinates and area) in the data. In addition to the player video data, the follow-up processing unit 353 uses sensing results obtained by other sensors and the like to determine the player's posture after movement (for example, the player's limbs, physique, and other whole-body shapes), the player's Each position (coordinates and area) in the player video data may be specified.

When the follow-up processing unit 353 determines that the player has moved, the correction processing unit 354 corrects the posture of the player after the movement (for example, the shape of the player's whole body such as limbs and physique), the position of the player in the player video data, and the position of the player in the player video data. Each of the (coordinates and regions) identification results are used to change (modify) properties (eg, shape, color, size, or a combination thereof) of the augmented image currently in use. Note that, if a modification permission condition is assigned to the augmentation source image, the modification processing unit 354 may modify the augmented image within the range indicated by the modification permission condition. In other words, the correction processing unit 354 corrects the augmentation image, depending on the augmentation image, assuming some restrictions when the player moves.

The augmentation database 355 holds augmentation source image data, which is the source image of the augmented image to be superimposed to apply to the player on the broadcast stream.

The display panel 36 is configured using, for example, a liquid crystal (LCD) panel or an organic EL (electroluminescence) panel. The display panel 36 displays user data for display generated by the processor 33 or the augmentation image processing unit 35 (for example, player image data or player image data superimposed with an augmentation image).

<Example of superimposed augmentation image>
Next, an example of superimposing the player video data of the augmentation image executed by the augmentation image display system 1 according to the first embodiment will be described with reference to FIGS. 8 and 9. FIG. FIG. 8 is a diagram showing a display example of an augmentation image in chronological order. FIG. 9 is a diagram showing a display example of an augmentation image. 8 and 9, for the sake of clarity, the augmentation image processing unit 25 or the augmentation image processing unit 35 analyzes the player's emotions based on the player video data and creates an augmentation image. Although it is assumed that various processes of generation and display are executed, the augmentation image processing unit 15 may execute them.

In FIG. 8, when the user is viewing a baseball game on the mobile terminal 20 or the television receiver 30, a player PL0 (for example, a father P1 and a mother P2) participating in the baseball game at time T=t0 Favorite player) is displayed on the input display unit 26 or the display panel 36 . At this time, it is assumed that daughter P3 purchases the hairstyle and clothes of a hero that daughter P3 likes as a skin pack image (augmentation image) that is an incarnation of player PL0 by the operation of daughter P3.

Then, at time T=t1 after time T=t0, portable terminal 20 or television receiver 30 adapts the physique and posture of player PL0 using the skin pack image that has undergone purchase processing as an augmentation image. (see virtual player PL1a). As a result, the image of the favorite hero (character) of the daughter P3 becomes the player PL1 superimposed on the player PL0, and the daughter P3 begins to take an interest in the baseball game. It is more likely that P2 will also be able to comfortably watch the baseball game.

At time T=t2 after time T=t1, portable terminal 20 or television receiver 30 determines that player PL0 in the player video data has moved, such as player PL0 entering the batter's box. Suppose At this time, the mobile terminal 20 or the television receiver 30 superimposes the currently displayed augmentation image so as to match the position and orientation of the player PL0 after movement so as to follow the movement of the player PL0. (see virtual player PL1b). It is also assumed that mobile terminal 20 or television receiver 30 determines that player PL0, who has entered the batter's box, is gradually getting excited.

At time T=t3 thereafter, the portable terminal 20 or the television receiver 30 corrects the augmentation image step by step so as to correspond to the gradual heightened emotions of the player PL0. A mention image is superimposed and displayed (see virtual player PL1c). As a result, the viewing user can intuitively and visually recognize that the baseball player is in an excited state with gradually increasing emotions through the virtual players PL1b and PL1c displayed on the mobile terminal 20 or the television receiver 30. can be easily grasped.

In FIG. 9, as in FIG. 8, when the user is watching a baseball game on the mobile terminal 20 or the television receiver 30, players PL0 (for example, father P1 and mother P2) participating in the baseball game favorite player) is displayed on the input display section 26 or the display panel 36 . At this time, it is assumed that daughter P3 purchases a mascot character MSC1 having the face of an animal (for example, a cat) that daughter P3 likes as a skin pack image (augmentation image) that is an incarnation of player PL0. .

Thereafter, portable terminal 20 or television receiver 30 uses the purchased skin pack image of mascot character MSC1 as an augmentation image to fit the physique and posture of player PL0 ( For example, it is superimposed on the right side) and displayed. As a result, the image of the mascot character MSC1 that the daughter P3 likes shows the emotions of the player PL0. The father P1 and the mother P2 watching together are more likely to be able to comfortably watch the baseball game.

When mobile terminal 20 or television receiver 30 detects that player PL0 is feeling frustrated, portable terminal 20 or television receiver 30 uses the same skin pack image that has undergone purchase processing to cause player PL0 to feel frustrated. A mascot character MSC2 indicating the presence of the player PL0 is superimposed and displayed around the player PL0 (for example, to the right). As a result, the daughter P3 can visually and intuitively grasp the current regret or sadness of the player PL0 by seeing the sad mascot character MSC2 reflected next to the player PL0. You can expect to start to take an interest in the game little by little.

When mobile terminal 20 or television receiver 30 detects that player PL0 is happy, portable terminal 20 or television receiver 30 uses the same skin pack image that has undergone purchase processing to make player PL0 happy. A mascot character MSC3 indicating the presence of the player PL0 is superimposed and displayed around (for example, right next to) the player PL0. As a result, the daughter P3 can visually and intuitively grasp the happy situation of the player PL0 by seeing the happy mascot character MSC3 displayed next to the player PL0. You can expect them to start to take an interest.

<Operation of Augmentation Image Display System>
Next, an operation example of the augmentation image display system 1 according to Embodiment 1 will be described with reference to FIG. FIG. 10 is a flow chart showing an example of an operation procedure of a series of augmentation image processing. In the description of FIG. 10, for ease of understanding, the augmentation image processing unit 25 or the augmentation image processing unit 35 analyzes the player's emotion based on the player video data, generates and displays the augmentation image, and performs the following operations. , the augmentation image processing unit 15 may also perform the various processes.

In FIG. 10, the mobile terminal 20 or the television receiver 30 purchases a skin pack image (original augmentation image) selected by the user according to the user's operation through online transactions such as online shopping. (St1). Based on the recorded player video data, the mobile terminal 20 or the television receiver 30 detects the player's heart rate (for example, one form of vital information) and the heart rate displayed on each image constituting the player video data. Calculate and analyze the variation (for example, one form of vital information) in a certain period of time (for example, about several seconds) (St2).

The mobile terminal 20 or the television receiver 30, based on the emotional characteristic data stored in the recording device 22 and the analysis result of the heart rate fluctuation in a certain period obtained in step St2, determines the characteristics of the fluctuation of the heart rate. is determined as the player's sentiment (emotion) (St3). The mobile terminal 20 or the television receiver 30 uses the skin pack image (original augmentation image) purchased in step St1 to use the augmentation image (for example, A form of superimposition) is generated or selected (St4). The mobile terminal 20 or the television receiver 30 superimposes and displays the augmentation image corresponding to the player's emotion so as to apply it to the player on the player video data (broadcast stream) (St4).

Here, if the portable terminal 20 or the television receiver 30 has not detected that the player has moved (St5, NO), steps St2 to St4 are repeated until the player's movement is detected. is repeated.

On the other hand, if the mobile terminal 20 or the television receiver 30 detects that the player has moved (St5, YES), the mobile terminal 20 or the television receiver 30 receives the player video data or other sensor data. Using the sensing results obtained in step 6, the posture of the player after movement (for example, the shape of the player's whole body such as limbs and physique) and the position (coordinates and area) of the player in the player image data are specified (St6). .

The mobile terminal 20 or the television receiver 30 uses the result of specifying the posture of the player after movement (for example, the shape of the player's whole body such as limbs and physique) and the position (coordinates and area) of the player in the player video data. to change (correct) the characteristics of the currently used augmentation image (for example, shape, color, size, or a combination thereof) (St7). In particular, when the modification permitting condition is assigned to the augmentation source image, the portable terminal 20 or the television receiver 30 modifies the augmented image within the range indicated by the modification permitting condition (St7 ).

After step St7, unless the application (that is, application for superimposed display of the augmentation image) is terminated by the user's operation (St8, NO), the processes of steps St2 to St7 are repeated until the application is terminated. On the other hand, after step St7, if the application (that is, the application for superimposed display of the augmentation image) is terminated by the user's operation (St8, YES), the operation of the augmentation image display system 1 is terminated.

As described above, the augmentation image display system 1 according to Embodiment 1 can perform at least one person (for example, a player) appearing (for example, participating in) a television program (for example, a baseball game) according to a user's operation. purchase processing of image data (for example, augmentation source image data, which is a skin pack image) showing the incarnation of . The augmentation image display system 1 estimates a player's emotion based on video data of a television program (for example, player video data showing a player participating in a baseball game). The augmentation image display system 1 uses the original augmentation image data to generate augmentation image data corresponding to the emotion of the player, and associates the generated augmentation image data with the player of the television program. indicate. The above-described augmentation image processing may be executed in any of the broadcasting station device 10, the mobile terminal 20, or the television receiver 30. Display processing of the broadcast stream is executed by the mobile terminal 20 or the television receiver 30 .

As a result, the augmentation image display system 1 can display the feelings (feelings) of a person (for example, a player participating in the game) appearing in a television program such as a sports game, for example, on the mobile terminal 20 or on the television receiver. The viewer can conceptually grasp the information through the display screen of the device 30, which contributes to improving the viewer's convenience. Therefore, the augmentation image display system 1 can appropriately display an augmentation image that matches the player's emotion in accordance with the player's shape such as position and posture, thereby appropriately improving the entertainment value of watching TV programs. can.

Further, the augmentation image display system 1 overlays the augmentation image data on a person (for example, a player) when displaying the augmentation image data. As a result, by viewing the augmented image overlaid on the player, the user can directly and concretely grasp the emotion of the player, which is difficult to grasp only through the screen at first glance. can be enjoyed more.

Also, the augmentation image display system 1 displays the augmentation image data around a person (for example, a player) when displaying the augmentation image data. As a result, the user can properly express the emotions of the player without degrading the visibility of the TV program, because the cause of deterioration of the visibility caused by the overlay display on the player is properly eliminated. You can enjoy watching TV programs more by grasping them.

Further, when a person (for example, a player) moves in a television program, the augmentation image display system 1 follows the movement of the person and displays at least the shape of the augmentation image data according to the movement of the person. to fix. When displaying the augmentation image data, the augmentation image display system 1 displays the augmentation image data after at least the shape is corrected. Thereby, the user can view the augmented image displayed so as to match the latest position and posture of the player via the mobile terminal 20 or the television receiver 30 . Therefore, the augmentation image display system 1 can follow and appropriately display the augmentation image even if the position of the player changes, and can continue to improve the entertainment value of watching the TV program.

Further, when the augmentation image display system 1 detects that the person's emotion is gradually elevated when generating the augmentation image data, the augmentation image display system 1 responds to the gradual elevation of the person's emotion. Generate stepwise different augmentation image data. As a result, the user can intuitively and visually grasp how the person's emotions are gradually heightened and excited as the augmented image is changed (corrected) step by step. can enjoy

Further, the augmentation image display system 1 stores image data (for example, an augmentation source image that is a skin pack image) that can be purchased based on a user's operation in a database (for example, the augmentation databases 155, 255, and 355). registered and managed. As a result, since the augmentation image display system 1 can register and manage a wide variety of original augmentation images as a database, the degree of freedom in selecting skin pack images that are of interest to users of a wide range of ages is greatly improved. can improve.

Further, the augmentation image display system 1 can be used for image data (for example, an original augmentation image that is a skin pack image) when a movie production company concludes a sponsorship contract for a new movie with a TV program broadcasting station. ), at least one character image data appearing in the new movie is registered and managed in the augmentation databases 155 , 255 , and 355 . As a result, the user can watch a TV program superimposed according to the player's emotions, movements, etc. using, for example, popular and famous character image data appearing in a new movie as a skin pack image. You can watch the program.

Further, when the augmentation image display system 1 detects a change in the player's emotion when generating the augmentation image data, the augmentation image display system 1 corrects the color of the augmentation image data according to the changed player's emotion. . As a result, the user can easily grasp that the player's feelings have changed through the portable terminal 20 or the television receiver 30 from the change in the color of the augmentation image, thereby gaining a deeper understanding of the contents of the television program. be able to take an interest.

Also, the receiver is the mobile terminal 20 carried by the user. As a result, the user can enjoy watching a television program on which an augmentation image matching the player's emotion is superimposed and displayed, even while carrying the portable terminal 20 and moving around.

Also, the receiver is the television receiver 30 placed at the place where the user is located. As a result, the user can enjoy viewing a television program superimposed with an augmentation image that matches the player's emotion through the television receiver 30 placed in a relaxing space (for example, in the user's home). .

Various embodiments have been described above with reference to the drawings, but it goes without saying that the present disclosure is not limited to such examples. It is obvious that a person skilled in the art can conceive of various modifications, modifications, substitutions, additions, deletions, and equivalents within the scope of the claims. Naturally, it is understood that it belongs to the technical scope of the present disclosure. In addition, the constituent elements of the various embodiments described above may be combined arbitrarily without departing from the gist of the invention.

The present disclosure provides an augmentation image display system and an augmentation image display system that allow viewers to conceptually grasp the emotions of a person appearing in a television program such as a sports game through a screen, thereby contributing to improved convenience for viewers. It is useful as a method of displaying a reference image.

1 Augmentation Image Display System 10 Broadcast Station Devices 11, 21, 31 Memories 12, 22, 32 Recording Devices 13, 23, 33 Processors 14, 24, 34 Communication Units 15, 25, 35 Augmentation Image Processing Unit 20 Mobile Phone terminal 26 input display unit 30 television receiver 36 display panel 151, 251, 351 emotion determination unit 152, 252, 352 extension processing unit 153, 253, 353 follow-up processing unit 154, 254, 354 correction processing unit 155, 255, 355 Augmentation database CM1, CM2 camera

Claims (11)

estimating the emotion of at least one person displayed in the video data;
a step of superimposing image data representing an incarnation of the person on the video data;
using the video data to generate augmentation image data corresponding to the emotion of the person from the image data, and displaying the generated augmentation image data in association with the person. ,
displaying the augmented image data, displaying the augmented image data overlaid on the person ;
Augmentation image display method. estimating the emotion of at least one person displayed in the video data;
a step of superimposing image data representing an incarnation of the person on the video data;
using the video data to generate augmentation image data corresponding to the emotion of the person from the image data, and displaying the generated augmentation image data in association with the person. ,
displaying the augmentation image data around the person ;
Augmentation image display method. estimating the emotion of at least one person displayed in the video data;
a step of superimposing image data representing an incarnation of the person on the video data;
a step of generating augmented image data corresponding to the emotion of the person from the image data using the video data, and displaying the generated augmented image data in association with the person;
following the movement of the person when the person moves within the video data;
modifying at least the shape of the augmented image data to follow the movement of the person;
displaying the augmented image data in the step of displaying the augmented image data ;
Augmentation image display method. In the step of generating the augmentation image data, when it is detected that the emotion of the person is gradually enhanced, the augmentation image is stepwise different corresponding to the stepwise enhancement of the emotion of the person. generate data,
The augmentation image display method according to claim 1. estimating the emotion of at least one person displayed in the video data;
a step of superimposing image data representing an incarnation of the person on the video data;
a step of generating augmented image data corresponding to the emotion of the person from the image data using the video data, and displaying the generated augmented image data in association with the person;
a step of purchasing image data showing the incarnation of the person displayed in the video data according to a user operation;
registering and managing the image data that can be purchased in a database ;
Augmentation image display method. The video data is video data of a television program,
When the movie production company concludes a sponsorship contract for a new movie with the broadcasting station of the television program, in the step of registering and managing the image data, the image data of at least one character appearing in the new movie is added to the above-described image data. registered and managed in the database,
The augmentation image display method according to claim 5 . In the step of generating the augmentation image data, when a change in the emotion of the person is detected, modifying the color of the augmentation image data according to the changed emotion of the person.
The augmentation image display method according to claim 1. An augmentation image display system including a broadcasting station device that broadcasts video data and a receiver that is operated by a user and receives and displays the video data,
The receiver is
estimating the emotion of at least one person displayed in the video data;
superimposing image data representing an incarnation of the person on the video data;
generating augmented image data corresponding to the emotion of the person from the image data using the video data, and displaying the generated augmented image data in association with the person;
Augmentation image display system. The receiver is a mobile terminal carried by the user,
9. The augmentation image display system according to claim 8 . wherein the receiver is a television set placed where the user is located;
9. The augmentation image display system according to claim 8 . An augmentation image display system including a broadcasting station device that broadcasts video data and a receiver that is operated by a user and receives and displays the video data,
The receiver is
sending a request for purchase processing of image data showing the incarnation of at least one person displayed in the video data to the broadcasting station device;
The broadcasting station device
Purchasing the image data showing the incarnation of the person in response to the purchase processing request,
estimating the emotion of the person based on the video data;
Augmentation image data corresponding to the emotion of the person is generated from the image data using the video data, TV program data is generated by associating the generated augmentation image data with the person, and deliver to the receiver,
Augmentation image display system.

Images