JP2022120646A - Imaging apparatus and imaging system - Google Patents

Abstract

To provide an imaging apparatus and an imaging system capable of imaging a desirable facial expression of a user.SOLUTION: An imaging apparatus 110 comprises: image display means 111 for displaying an image included in a content; imaging means 112 for imaging a user; content reproduction means 113 for reproducing the content; imaging control means 116 which controls imaging timing by the imaging means in accordance with the reproduction time of the content by the content reproduction means; and image recording means 115 which records a photographic image obtained by imaging by the imaging means.SELECTED DRAWING: Figure 2

Description

TECHNICAL FIELD The present disclosure relates to techniques for shooting content viewers.

2. Description of the Related Art Conventionally, in theme parks and sightseeing spots, photography services have been provided in which customers are photographed and the photographed photographs are sold. In such a photographing service, there are cases in which a photograph is taken by a camera that is visible to the user, who is a customer, and in which the photograph is taken without the user's knowledge while riding an attraction (see Patent Document 1).

JP 2020-71519 A

In the case of shooting with a camera at a position where the user can see, such as shooting by a shooting box or staff, the user makes a facial expression for shooting, which may result in an unnatural facial expression. When the image is captured without the user's knowledge, the user's natural expression can be captured, but since the user does not express his/her intention to take the image in advance, the user may feel uncomfortable. Therefore, there is a need for a mechanism that allows the user to voluntarily photograph and to photograph natural facial expressions.

Accordingly, an object of the present disclosure is to provide a photographing device and a photographing system capable of photographing a user's preferred facial expression.

In order to solve the above problems, in the present disclosure,
image display means for displaying an image included in content;
a photographing means for photographing a user;
content reproduction means for reproducing the content;
a photographing control means for controlling timing of photographing by the photographing means in accordance with reproduction of the content by the content reproducing means;
an image recording means for recording a photographed image acquired by photographing by the photographing means;
To provide an imaging device having

Further, the imaging device of the present disclosure is
The photographing means may photograph at regular time intervals during the reproduction of the content by the content reproducing means.

Further, the imaging device of the present disclosure is
The photographing means may photograph in a predetermined time period determined according to the content.

Further, the imaging device of the present disclosure is
The apparatus may further comprise facial expression recognition photographing means for recognizing the facial expression of the subject, wherein the photographing means may photograph the subject according to the facial expression of the subject photographed by the facial expression recognition photographing means.

Further, the imaging device of the present disclosure is
further comprising facial expression recognition means for recognizing facial expressions of a person;
The shooting means shoots moving images continuously during playback of the content,
The facial expression recognition means determines a range to be extracted according to the facial expression of the subject in the captured moving image,
The image recording means may record the determined moving image.

Also, in this disclosure:
the photographing device;
a management server connected to the imaging device;
The management server is
a content supplying means for supplying the content reproduced by the content reproducing means to the photographing device;
a photographed image obtaining means for obtaining the photographed image from the photographing device;
user state analysis means for analyzing the user state of the captured image;
To provide an imaging system having

Further, the imaging system of the present disclosure is
further comprising an advertisement server connected to the photographing device;
The ad server
content transmission means for transmitting the content to the management server;
a user state data acquisition unit for acquiring user state data, which is data of the user state analyzed by the user state analysis unit, from the management server.

According to the present disclosure, it is possible to provide a photographing device and a photographing system capable of photographing a user's natural expression.

1 is a configuration diagram of an imaging system according to an embodiment of the present disclosure; FIG. 2 is a diagram showing details of an imaging device 110. FIG. 3 is a diagram showing details of a management server 120; FIG. FIG. 3 is a diagram showing details of an advertisement server 140; FIG. 2 is a hardware configuration diagram of a user terminal 150. FIG. 4 is a sequence diagram showing processing operations of the imaging system; FIG. It is a figure which shows the imaging|photography mode of the imaging means which concerns on this embodiment. FIG. 10 is a diagram showing the relationship between a photographing device and a user at the time of photographing; FIG. 10 is a diagram showing a screen of a display device 110f immediately after content reproduction is finished; 10A and 10B are diagrams showing how the screen of the display device 110f of the photographing device 110 changes. FIG. FIG. 10 is a diagram showing how an image of content having a plurality of stories that can be branched changes.

Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings.
<1. System configuration>
FIG. 1 is a configuration diagram of an imaging system according to an embodiment of the present disclosure. In FIG. 1, 110 is a photographing device, 120 is a management server, 130 is a content publishing server, 140 is an advertising server, 150 is a user terminal, and 160 is a network. The photographing device 110, management server 120, content publishing server 130, advertisement server 140, and user terminal 150 are connected to a public network 160 such as the Internet, and can exchange data with each other. In the example of FIG. 1, for convenience of explanation, only one photographing device 110, content publishing server 130, advertising server 140, and user terminal 150 are shown, but a plurality of each may be connected. Actually, the photographing device 110 is installed for each photographing spot, and the number of user terminals 150 is equal to the number of users.

The photographing device 110 is a photographing device installed in a photographing spot such as a theme park or a tourist spot. FIG. 2 is a diagram showing the imaging device 110. As shown in FIG. 2, FIG. 2(a) is an external perspective view of the photographing device 110, FIG. 2(b) is a hardware configuration diagram of the photographing device 110, and FIG. 2(c) is a photographing device 110. It is a functional block diagram of. As shown in the appearance of FIG. 2A, the imaging device 110 has a housing 110j, and a display device 110f and an imaging device 110h are installed at positions visible from the outside of the housing 110j.

As shown in FIG. 2B, the photographing apparatus 110 includes a CPU (Central Processing Unit) 110a, a RAM (Random Access Memory) 110b as a main memory, and a non-volatile memory for storing programs and data executed by the CPU 110a. a physical storage device 110c (for example, hard disk, flash memory, etc.); an instruction input unit 110d such as a keyboard or touch panel; a printer 110e for outputting data; A communication unit 110g for performing network communication with other computers such as the management server 120, a photographing device 110h (such as a CCD camera) for photographing images such as still images and moving images, and other terminals and the network 160. A short-range wireless communication unit 110i for direct wireless communication is provided, and they are connected to each other via a bus.

In this embodiment, the instruction input unit 110d and the display device 110f are implemented as touch panel displays. The communication method used by the short-range wireless communication unit 110i is not particularly limited as long as it is a format for wireless communication at a relatively short distance, and various known methods such as Bluetooth and wireless LAN can be used. Each component shown in FIG. 2B is housed in a housing 110j as shown in FIG. In this specification, the term "image" is interpreted as both a still image and a moving image.

FIG. 2C is a functional block diagram of the imaging device 110. As shown in FIG. In the hardware configuration shown in FIG. 2(b), the CPU 110a loads the program stored in the storage device 110c into the RAM 110b and executes it, so that the photographing device 110 implements each means shown in FIG. 2(c). can do. As shown in FIG. 2C, the photographing device 110 includes image display means 111, photographing means 112, content reproducing means 113, facial expression recognition photographing means 114, image recording means 115, photographing control means 116, and facial expression recognition means 117. have

The image display means 111 is a means for displaying an image included in the content reproduced by the content reproduction means 113, and is realized by the CPU 110a executing a program and outputting it to the display device 110f such as a liquid crystal display. The photographing means 112 is a means for photographing a user who is a subject (a person to be photographed) positioned in front of the photographing device 110. The CPU 110a executes a program to photograph a photographing device 110h having an image pickup device such as a CCD or a CMOS. It is realized by giving instructions. The content reproduction means 113 is means for reproducing content including images. The facial expression recognition photographing means 114 is means for photographing the facial expression of the user, who is the subject, in order to recognize the facial expression of the user, and is realized by the CPU 110a executing a program and giving an instruction to the photographing device 110h. It should be noted that, in order to implement the facial expression recognition photographing means 114, two photographing devices 110h are required in addition to those constituting the photographing means 112. FIG. The image recording means 115 is a means for recording the photographed image photographed by the photographing means 112 . The image recording unit 115 realizes a mode of recording a photographed image as data and a mode of recording a photographed image on a paper medium. In the case of recording as data, the captured image is recorded by storing it in the storage device 110c. In the case of recording a photographed image on a paper medium, the photographed image is printed out by outputting it to the printer 110e. The imaging control means 116 controls imaging timing by the imaging means 112 . The facial expression recognition means 117 analyzes the captured image captured by the facial expression recognition imaging means 114 and recognizes the user's facial expression.

The photographing device 110 loads the program stored in the storage device 110c into the RAM 110b, executes it by the CPU 110a, and issues instructions to other components to display the image display means 111, the photographing means 112, the content reproduction means 113, the facial expressions, and the like. It realizes each function of the photographing means 114 for recognition, the image recording means 115, the photographing control means 116, the facial expression recognition means 117, and the like. Further, the photographing device 110 loads programs stored in the storage device 110c into the RAM 110b and executes them by the CPU 110a, thereby realizing various data processing and user interfaces.

The management server 120 is a server computer that is connected to the photographing device 110 via the network 160 so as to be capable of data communication. FIG. 3 is a diagram showing details of the management server 120. As shown in FIG. 3A is a hardware configuration diagram of the management server 120, and FIG. 3B is a functional block diagram of the management server 120. As shown in FIG.

The management server 120 can be realized by a general-purpose server computer, and as shown in FIG. A large-capacity storage device 120c (for example, hard disk, flash memory, etc.) for storing programs and data executed by the CPU, and an input I/F (interface) 120d for receiving input from input devices such as a keyboard and mouse. , a data input/output I/F (interface) 120e for data communication with an external device (data storage medium, etc.), and a display output I/F (interface) 120f for sending information to a display device (liquid crystal display, etc.) and a communication unit 120g for performing network communication with other computers such as the photographing device 110, the content publishing server 130, the advertisement server 140, and the user terminal 150 via the network 160, and are connected to each other via a bus. ing.

FIG. 3B is a functional block diagram of the management server 120. As shown in FIG. In the hardware configuration shown in FIG. 3(a), the CPU 120a loads the program stored in the storage device 120c into the RAM 120b and executes it, so that the management server 120 functions as each means shown in FIG. 3(b). It is possible to As shown in FIG. 3B, the management server 120 has content storage means 121 , user state data storage means 122 , content supply means 123 , photographed image acquisition means 124 and user state analysis means 125 .

The content storage unit 121 is a storage unit that stores content. In this embodiment, content is content data including moving images. Audio may be added to the video. Note that the content only needs to include images, and may include multiple still images instead of moving images. When a plurality of still images are included, the still images are set to be switched at a predetermined timing. Moreover, there is also a type of content in which the timing of photographing by the photographing means 112 is recorded. The user state data storage means 122 is means for storing user state data obtained by the user state analysis means 125 .

The content supply means 123 is a means for supplying the content stored in the content storage means 121 to the photographing device 110 . The photographed image acquisition means 124 is means for acquiring the photographed image received from the photographing device 110 . The user state analysis unit 125 is a unit that analyzes the photographed image acquired from the photographing device 110 and determines what kind of user state is the state of the user.

The management server 120 loads the program stored in the storage device 120c into the RAM 120b, executes it by the CPU 120a, and issues commands to other components to store content storage means 121, user status data storage means 122, and content supply means. 123 , photographed image acquisition means 124 , and user state analysis means 125 . The storage device 120c also serves as a data storage area for the content storage means 121 and the user state data storage means 122. FIG. The management server 120 may be physically realized by one computer, or may be realized by a plurality of computers. Alternatively, it may be implemented as a cloud system that is distributed over the network 160 .

The content publishing server 130 is a server computer that publishes content including images. The content publishing server 130 has the same hardware configuration as the management server 120, and implements functions such as arithmetic processing means and communication means by executing programs stored in the storage device. When the content publishing server 130 receives the content together with the publication request from the management server 120, the content publishing server 130 stores the content in the storage device and sets the content to be accessible from the outside. When there is an acquisition request from an external computer via a network, it performs a process of transmitting the requested content.

Content publishing server 130 may be physically realized by one computer, or may be realized by a plurality of computers. Alternatively, it may be implemented as a cloud system that is distributed over the network 160 .

The advertisement server 140 is a server computer connected to the management server 120 via the network 160 so as to be capable of data communication. FIG. 4 is a diagram showing details of the advertisement server 140. As shown in FIG. 4A is a hardware configuration diagram of the advertisement server 140, and FIG. 4B is a functional block diagram of the advertisement server 140. FIG.

The advertisement server 140 can be realized by a general-purpose server computer, and as shown in FIG. A large-capacity storage device 140c (for example, hard disk, flash memory, etc.) for storing programs and data executed by the CPU, and an input I/F (interface) 140d for receiving input from input devices such as a keyboard and mouse. , a data input/output I/F (interface) 140e for data communication with an external device (data storage medium, etc.), and a display output I/F (interface) 140f for sending information to a display device (liquid crystal display, etc.) and a communication unit 140g for performing network communication with other computers such as the photographing device 110, the management server 120, the content publishing server 130, and the user terminal 150 via the network 160, and are connected to each other via a bus. ing.

FIG. 4B is a functional block diagram of the advertisement server 140. As shown in FIG. In the hardware configuration shown in FIG. 4(a), the CPU 140a loads the program stored in the storage device 140c into the RAM 140b and executes it, so that the advertising server 140 functions as each means shown in FIG. 4(b). It is possible to As shown in FIG. 4(b), the advertising server 140 has content storage means 141, advertising effect analysis data storage means 142, content transmission means 143, user state data acquisition means 144, and advertising effect analysis means 145.

The content storage unit 141 is a storage unit that stores content, like the content storage unit 121 of the management server 120 . The advertising effectiveness analysis data storage means 142 is means for storing the advertising effectiveness analysis data obtained by the advertising effectiveness analysis means 145 .

The content transmission means 143 is means for transmitting the contents stored in the content storage means 141 to the management server 120 . The user status data acquisition means 144 is means for acquiring user status data received from the management server 120 . The advertising effect analysis means 145 is a means for performing analysis based on the user state data acquired from the management server 120 and separately stored product sales data, and creating advertising effect analysis data indicating the relationship between the user state and product sales. is.

The advertisement server 140 loads the program stored in the storage device 140c into the RAM 140b, executes it by the CPU 140a, and issues commands to other components to perform content storage means 141, advertisement effect analysis data storage means 142, and content transmission. The functions of means 143, user state data acquisition means 144, and advertisement effect analysis means 145 are realized. The storage device 140 c also serves as a data storage area for the content storage means 141 and the advertising effect analysis data storage means 142 . Advertising server 140 may be physically realized by one computer, or may be realized by a plurality of computers. Alternatively, it may be implemented as a cloud system that is distributed over the network 160 .

FIG. 5 is a hardware configuration diagram of the user terminal 150. As shown in FIG. The user terminal 150 is implemented by a smart phone, which is a multifunctional portable terminal. As shown in FIG. 5, a CPU (Central Processing Unit) 150a, a RAM (Random Access Memory) 150b as a main memory, and a non-volatile storage device 150c (for example, hard disk, flash memory, etc.), a keyboard, an instruction input unit 150d such as a touch panel, a data input/output I/F (interface) 150e for data communication with an external device (data storage medium, etc.), a display device (liquid crystal display etc.) 150f, a communication unit 150g for performing network communication with other computers such as the management server 120 via the network 160, a photographing device 150h for photographing an image, and other terminals without the network 160. A short-range wireless communication unit 140i for direct wireless communication and an audio input/output unit 150j such as a microphone and a speaker are provided, and are connected to each other via a bus. The communication method used by the short-range wireless communication unit 150i is not particularly limited as long as it is a format for wireless communication at a relatively short distance, and various known methods such as Bluetooth and wireless LAN can be used.

As the user terminal 150, it is possible to use a device other than a smart phone, such as a notebook PC, a tablet, etc., as long as it has an arithmetic processing unit of a CPU, etc., and has an information processing function, a network communication function, and the like.

<2. Processing operation>
Next, the processing operation of the imaging system according to this embodiment shown in FIG. 1 will be described. FIG. 6 is a sequence diagram showing processing operations by the imaging system.

The advertisement server 140 stores content including advertisement information such as products and services. The mode of advertising information included in the content is not particularly limited. For example, the name of the product or service may be displayed as text information, or the image of the product itself may be included as an advertisement in some form that allows the viewer to recognize the product or service. Such content is produced at the request of an advertisement client or the like and registered in the advertisement server 140 .

The advertisement server 140 transmits content to the management server 120 at arbitrary timing (step S1). The management server 120 registers the received content. Specifically, the content received from the advertisement server 140 is stored in the storage device within the management server 120 .

Then, the management server 120 transmits the content to the imaging device 110 at arbitrary timing (step S2). Specifically, the content is supplied by the content supply means 123 transmitting it to the photographing device 110 . Then, the imaging device 110 registers the received content. Specifically, the content received from the management server 120 is stored in the storage device 110c within the imaging device 110 .

Next, the photographing device 110 reproduces the content and photographs the user (step S3). Specifically, the content reproduction means 113 of the photographing device 110 reproduces the stored content and displays it on the image display means 111 . This allows the user in front of the imaging device 110 to view the content. In parallel with the reproduction of the content by the content reproducing unit 113, the photographing unit 112 photographs the user. Preferably, the photographing means 112 starts photographing at the same time when the content reproducing means 113 starts reproducing the content. The photographing unit 112 may start photographing before the content reproducing unit 113 starts reproducing the content.

The photographing means 112 performs photographing at a timing corresponding to the content under the control of the photographing control means 116, and finishes photographing when the content reproduction means 113 finishes reproducing the content. Preferably, the photographing by the photographing means 112 is finished at the same time as the reproduction of the content by the content reproducing means 113 is finished. The photographing unit 112 may continue photographing even after the content reproduction unit 113 finishes reproducing the content. By continuing to shoot even after the playback of the content is finished, it is possible to shoot the expression including the afterglow of viewing the content. For example, the photographing unit 112 may finish photographing five seconds after content reproduction by the content reproducing unit 113 ends, or may continue photographing until an external operation is performed. The operation from the outside includes, for example, the user's operation to shift to the "photographed image selection screen (see FIG. 10D)". Details of the processing in step S3 will be described later.

After reproducing the content and photographing the user, the photographing device 110 provides information regarding the reproduced content to the user terminal 150 that requested acquisition of information regarding the content (step S4). The acquisition request for content-related information and the provision of content-related information are not particularly limited, and can be performed in various manners. For example, after reproducing the content, the imaging device 110 displays code information (barcode, two-dimensional code, etc.) recording information for accessing information about the content that has been displayed on the image display means 111 . Then, the code information displayed on the user terminal 150 is read using the code information reading function and communication function of the smartphone or the like. Further, the user terminal 150 accesses a predetermined site (server) using the recorded information and obtains information regarding the content. The information on the content includes information on the product advertised in the content, so that the user can know the details of the corresponding product in addition to the detailed information on the content itself.

After reproducing the content and capturing the image of the user, the image capturing device 110 records the captured image in parallel with the processing in step S4 (step S5). By the user's image capturing process in step S3, the image capturing apparatus 110 acquires the user's captured image at the content playback time. Therefore, the imaging device 110 records the captured image in a predetermined manner. Although there are no particular limitations on the manner in which the photographed image is recorded, for example, there is a manner in which the photographed image is printed and recorded on a paper medium, and a manner in which the photographed image is recorded as data in a predetermined format. The details of the processing in step S5 will be described later, similarly to step S3.

After the captured image is obtained, the imaging device 110 transmits the obtained captured image to the management server 120 (step S6). A captured image to be transmitted is associated with a content ID for specifying the content. In the management server 120, the photographed image obtaining means 124 obtains the photographed image. Note that the process of step S4 for providing the user terminal 150 with information about the content that has been reproduced may be performed after steps S5 and S6.

After acquiring the photographed image from the photographing device 110, the management server 120 analyzes the user state for the photographed image (step S7). Specifically, the user state analysis means 125 analyzes the captured image and determines what kind of state the user state, which is the state of the user, is. Any user state can be used as long as it can be obtained by analyzing an image. In this embodiment, the user's emotion is determined by analyzing the facial expression of the user appearing in the captured image. Emotional states to be determined can be classified into joy, anger, sadness, joy, and the like. Then, for each category of emotion, the degree is also determined in units of % from 0% to 100%. Such determination of emotion by image analysis can be realized by a known image analysis technique. In this embodiment, it is realized by image analysis using AI (artificial intelligence). As the facial expression analysis technology using AI, for example, “User Local Facial Expression Estimation AI” provided by User Local Co., Ltd. and emotion recognition AI “Affdex” provided by Affectiva can be used. As described above, the emotional state is obtained as user state data.

After obtaining the user state data, the management server 120 transmits the user state data to the advertisement server 140 (step S8). The advertisement server 140 stores the user state data acquired from the management server 120 in association with the content ID. In step S1, the content supplied from the advertisement server 140 is subjected to the processing shown in steps S2 to S8, and the user state data of the user who is the viewer of the content is accumulated in the advertisement server 140.

On the other hand, the management server 120 sets contents disclosure to the contents disclosure server 130 (step S9). Specifically, for the content managed by the management server 120 , information on the publication time is transmitted to the content publication server 130 . The content publishing server 130 registers the content received from the management server 120 and sets the timing of publishing the content. Specifically, the received content is stored in the storage device of the content publishing server 130, and the storage area is set to be accessible from the outside only at a predetermined publication time. By executing such processing, the content reproduced by the photographing device 110 can be viewed by accessing the content publishing server 130 at the time of publication. Even if the content publishing server 130 is accessed from the outside, the content cannot be viewed until the publication start time set as the publication time.

When the release time set in step S9 comes, the corresponding content can be viewed from the outside. In step S4, the user who has acquired the address of the content from the photographing device 110 as the information on the content can access the content viewed by the photographing device 110 by accessing the content publishing server 130 using the address. Yes (step S10). When there is an access to the content that is due to be published, the content publication server 130 transmits the content to the user terminal 150 or the like that is the source of the access (step S11).

The user terminal 150 can also access the advertisement server 140 (step S12). In step S4, the user who obtains the address (URL, etc.) of the advertisement information in the advertisement server 140 as the information on the content from the photographing device 110 accesses the advertisement server 140 using that address, thereby creating a content publishing server. It is possible to access a website that introduces products advertised in the content viewed at 130 . When the publicized advertisement information is accessed, the advertisement server 140 transmits the advertisement information to the user terminal 150 or the like that is the access source (step S13).

As described above, in step S8, the advertising server 140 receives user state data from the management server 120. FIG. Then, for each content supplied from the advertising server 140 , user state data of viewers of the content is accumulated in the advertising server 140 . The advertisement server 140 analyzes the accumulated user state data and product sales to analyze the advertisement effect (step S14). Specifically, based on the user state data associated with the content ID and the product ID associated with the content ID, the advertising effect analysis unit 145 generates sales data and user state data for each product identified by the product ID. associate. As a result, it is possible to obtain changes in user status and changes in product sales due to content as advertising effect analysis data. The obtained advertisement effect analysis data is stored in the advertisement effect analysis data storage means 142 .

<3. Details of the imaging device>
As described with the sequence diagram shown in FIG. 6, the imaging device 110 mainly performed the processing of steps S3 to S5. Next, the details of the processing operation of the imaging device 110 will be described. As described above, the photographing device 110 acquires content from the management server 120 through the processing in step S2 and stores it in the storage device 110c. Then, the content reproduction means 113 reproduces the content and displays it on the image display means 111 . At the same time, the photographing means 112 starts photographing and photographs the user in front of the photographing device 110 . There are various modes in which the imaging means 112 performs imaging.

FIG. 7 is a diagram showing a photographing mode of the photographing means according to this embodiment. FIG. 7(a) shows changes in the screen during content reproduction. FIG. 7(b) shows the passage of time with a rightward arrow, indicating that the more time goes to the right, the more time elapses. FIG. 7(c) shows the facial expression of the user during content reproduction. FIGS. 7(d) to 7(i) show shooting modes. The star marks in FIGS. 7(d) to 7(h) indicate the shooting timing. In this embodiment, there are first to fifth imaging modes. Among the first to fifth shooting modes, the first to third shooting modes are for acquiring still images as shot images, and the fourth and fifth shooting modes are for acquiring moving images as shot images. It is for obtaining.

As shown in FIG. 7D, in the first shooting mode, still images are shot at regular time intervals during content reproduction. In the case of the first shooting mode, the shooting unit 112 starts shooting at the same time when the content playback unit 113 starts playing back the content. The user is photographed at an arbitrary time interval, for example, at intervals of one second. For example, if the content is 60 seconds long, 60 pictures are taken.

As shown in FIG. 7(e), the second shooting mode is to shoot in a predetermined time period determined according to the content. In the case of the second shooting mode, as the content, data in which the shooting timing is added to the data of the main body of the content is used. After reading the content, the content reproducing means 113 passes the shooting timing recorded in the content to the shooting control means 116 . Then, the photographing control means 116 causes the photographing means 112 to photograph at the acquired photographing timing. The photographing timing to be added to the data of the content body is not particularly limited, and can be in various forms. For example, if the content is 60 seconds long, it can be set to 1 to 5 seconds, 15 to 30 seconds, etc. from the start. As for the shooting timing in the second shooting mode, the time when the content becomes exciting (highlight) may be set while actually viewing the content. By setting such exciting times as shooting timings, it becomes easier to shoot the user's rich facial expressions.

As shown in FIG. 7(f), the third shooting mode is to shoot according to the facial expression of the user during content reproduction. In the case of the third photographing mode, photographing means 114 for facial expression recognition is used in addition to photographing means 112 . The photographing means 114 for facial expression recognition starts photographing at the same time when the content reproduction means 113 starts reproducing the content. The photographing means 114 for facial expression recognition transfers the photographed image obtained by photographing to the photographing control means 116 . The photographing control means 116 analyzes the acquired photographed image and determines what kind of emotional state the subject is in. Specifically, similar to the management server 120 in step S7, determination can be made by a known image analysis technique. Then, the photographing control means 116 instructs the photographing means 112 to photograph when a preset condition is satisfied. For example, if the condition is set as "70% or more happy", the photographing control means 116 analyzes the photographed image acquired from the facial expression recognition photographing means 114, and as a result of the analysis, "70% or more happy" is determined. If determined, the photographing means 112 is instructed to photograph. If the photographing means 112 photographs at this moment, a photographed image with an expression of "70% or more happy" is obtained. By such processing, in the third shooting mode, shooting according to the user's facial expression can be performed. As can be seen by comparing FIG. 7(c) and FIG. 7(f), in the case of the third shooting mode, shooting is performed when the user's expression is rich.

As shown in FIG. 7(g), in the fourth shooting mode, moving images are shot continuously while content is being reproduced, and a part of the shots is extracted according to the user's facial expression. In the case of the fourth shooting mode, the shooting unit 112 starts shooting a moving image at the same time when the content playback unit 113 starts playing back the content. Then, video shooting is performed over the entire playback time of the content. Acquire a captured image that is a moving image. For example, if the content is 60 seconds, a 60-second captured image of the user is obtained. When the photographed image is obtained, the facial expression recognition means 117 analyzes it. Specifically, similarly to the case described in step S7 and the third imaging mode, the photographed image is analyzed to determine the emotional state of the user. Then, the facial expression recognition means 117 extracts an image within a range corresponding to a condition set in advance from the photographed image. For example, if the condition is set as "70% or more happy", the expression recognition means 117 analyzes the photographed image acquired from the photographing means 112, and as a result of the analysis, "70% or more happy" is determined. Extract only range images. As a result, only photographed images with expressions of "70% or more happy" are obtained. Through such processing, only the moving image corresponding to the user's facial expression can be acquired in the fourth shooting mode. For example, in the example of FIG. 7, only the range shown in FIG. 7(i) is extracted from the moving image captured over the entire area as shown in FIG. 7(g).

As shown in FIG. 7(h), the fifth shooting mode is to shoot a moving image in a predetermined time period determined according to the content. In the case of the fifth shooting mode as well, as in the case of the second shooting mode, data in which the shooting timing is added to the data of the main body of the content is used as the content. The difference from the second imaging mode is that the captured image is acquired as a moving image. After reading the content, the content reproducing means 113 passes the shooting timing recorded in the content to the shooting control means 116 . Then, the photographing control means 116 causes the photographing means 112 to photograph at the acquired photographing timing. The photographing timing to be added to the data of the content body is not particularly limited, and can be in various forms. For example, if the content is 60 seconds long, it can be set to 1 to 5 seconds, 15 to 30 seconds, etc. from the start. The shooting timing in the fifth shooting mode may be set at a time when the user actually gets excited while watching the content. By setting such exciting times as shooting timings, it becomes easier to shoot the user's rich facial expressions.

As described above, in the case of the photographing mode of the photographing device according to the present embodiment, it is possible to photograph the natural expression of the user when viewing the content. In other words, it is possible to shoot a facial expression that is not created for shooting.

<4. State at the time of shooting>
FIG. 8 is a diagram showing the relationship between the photographing device and the user at the time of photographing. In FIG. 8, U indicates the user, and H indicates the facial expression of the user U at that time. First, as shown in FIG. 8A, the user moves in front of a photographing device 110 installed in a theme park or the like. Then, the imaging device 110 starts playing back the content. At this time, as shown in FIG. 8B, the moving image of the content is displayed on the display device 110f. The reproduction of the content may be automatically started at a predetermined timing, or may be started by operating an operation button (not shown) displayed on the display device 110f.

During reproduction of the content, the photographing means 112 photographs the user in the first to fifth photographing modes, for example. FIG. 8(c) shows a state in which the user takes a picture when the contents are exciting. At this time, the facial expression H of the user U is in a smiling state. After finishing the reproduction of the content, the photographing device 110 combines the captured image and the background image prepared in advance to generate a composite image. Further, as shown in FIG. 8D, the imaging device 110 displays the synthesized image on the display device 110f. The facial expression H of the user U in the photographed image is the one when the user U is smiling, and a preferable composite image is obtained. If the user likes the displayed composite image, the user operates the display device 110f operation button to print out the composite image.

FIG. 9 is a diagram showing the state of the screen of the display device 110f immediately after the end of content reproduction. Among them, FIG. 9(a) is the same as the synthesized image displayed on the display device 110f shown in FIG. 8(d). FIG. 9B is a diagram showing detailed content information displayed on the display device 110f. After a certain period of time has passed since the synthesized image shown in FIG. 9A is displayed, the imaging device 110 switches the display to display detailed content information as shown in FIG. 9B. The display switching from the composite image to the content detailed information may be automatically performed by the imaging device 110, or may be performed by the user operating the operation button. Also, the display may be returned to the synthesized image from the content detailed information as shown in FIG. 9B.

As shown in FIG. 9B, as the content detailed information, the content title (name), content provider, content publication address (public site URL), publication start date, code information, and the like are displayed. . The code information records all or part of the detailed content information. In the example of FIG. 9B, a bar code is used as the code information. When the composite image is recorded on a paper medium and printed out, only the composite image as shown in FIG. A composite image may be printed, and the content detailed information as shown in FIG. 9B may be printed on the other side. When recording and outputting the synthesized image as data, the synthesized image and the content detailed information are transmitted as data to the user terminal 150 possessed by the user. When recording and outputting data, the imaging device 110 may transmit the data to the user terminal 150 via the network 160 using the communication unit 110g, or perform short-range wireless communication using the short-range wireless communication unit 110i to the user terminal. 150.

<5. How the screen changes>
FIG. 10 is a diagram showing how the screen of the display device 110f of the imaging device 110 changes. Here, the processing of the photographing device 110 will be described using the state of the screen of the display device 110f. When providing a photographed image providing service by the photographing device 110, the photographing device 110 displays an initial screen as shown in FIG. 10(a) on the display device 110f. The example of FIG. 10A shows a case where the service name is "natural photo service". When the user performs an operation to start the service on the imaging device 110, the imaging device 110 displays a content selection screen as shown in FIG. 10B on the display device 110f. FIG. 10B shows an example of a content selection screen in which titles corresponding to emotions such as "strange,""surprised," and "crying" are displayed so as to be selectable. The facial expression recognizing photographing means 114 photographs the facial expression of the user, and the facial expression recognizing means 117 analyzes the acquired photographed image, estimates user attributes such as age and gender by face recognition, and determines the user attributes. Matching recommended video titles and thumbnail images may be displayed.

When the content is selected by the user, the photographing device 110 reproduces the selected content and starts photographing as described above. During this time, the moving image of the reproduced content is displayed on the display device 110f as shown in FIG. 10(c). In the example of FIG. 10(c), the name of the food company providing the content is also displayed in the moving image of the content. As described above, while the content is being reproduced, the photographing means 112 photographs the user viewing the content.

When the reproduction of the selected content ends, the photographing device 110 displays a plurality of photographed images side by side on the display device 110f as shown in FIG. 10(d). The example in FIG. 10D shows a case where a still image (photograph) is displayed as the captured image. When the user performs an operation of selecting one of the captured images displayed on the display device 110f on the imaging device 110, the imaging device 110 recognizes the selected captured image. Then, the imaging device 110 displays the selected captured image on the display device 110f. The photographing device 110 also displays a plurality of background images that can be selected as synthesis targets on the same screen.

FIG. 10E is a diagram showing an example in which a plurality of selectable background images are displayed in the upper part of the screen, centering on the photographed image selected by the user. In the example shown in FIG. 10(e), the display device 110f further displays a "next" button and a "return" button. When the user touches the display device 110f, which also serves as a touch panel, to select a background image, the imaging device 110 generates a new composite image using the selected background image and displays it on the display device 110f. When another background image is selected, the photographing device 110 generates a new synthesized image synthesized using the selected other background image, and displays it on the display device 110f. Thereby, the user can select one to be synthesized from the background images displayed by switching. If you want to change the captured image, touch the "return" button to return to the screen shown in FIG. 10(d). When the user operates a separately provided button (not shown) or the like while the synthesized image is displayed as shown in FIG. Content detailed information is displayed as follows.

When the user touches the "Next" button after completing the selection of the captured image and the background image, the image capturing apparatus 110 displays the image recording mode selection screen on the display device 110f as shown in FIG. 10(f). indicate. In the example of FIG. 10(f), options of "print" for recording on a paper medium, "send by e-mail" for recording as data, and "both print and e-mail" are displayed. When the user performs an operation to select one of the options, the imaging device 110 performs image recording processing in a mode according to the content of the selection.

In the above embodiment, the user state analysis means 125 of the management server 120 analyzes the photographed image and determines the user's emotion as the user state. good. User states include body temperature, pulse rate, point of interest, etc., in addition to emotions. The point of interest is a point on the image on which the user is paying attention. As a technique for analyzing the point of interest, for example, “User Local Gaze Estimation AI” provided by User Local Co., Ltd. can be used. This technology uses deep learning to estimate where the person in the video is looking. VitalBit (registered trademark) provided by Asahi Kasei Electronics Corporation can be used as the body temperature/pulse detection technology. This technology uses a near-infrared camera to sense the pulse without contact.

As shown in the above embodiments, the content includes animation. This moving image usually follows one story. However, the content may have multiple stories that can be branched, and videos according to different stories may be played based on the user's selection. FIG. 11 is a diagram showing how an image of content having a plurality of stories that can be branched changes.

Also in this case, the imaging device 110 displays the moving image on the display device 110f according to the story of the content. In the example of FIG. 11, since there is one story halfway through, the determined moving images are displayed as shown in FIGS. 11(a) and 11(b). Then, when the content branch position is reached, the photographing device 110 displays a user selection screen (FIG. 11(c)). FIG. 11(j) is a diagram showing details of the user selection screen. The photographing device 110 selectably displays images corresponding to a plurality of stories as a user selection screen. In the example of FIG. 11(j), images corresponding to the two stories "club" and "cafe" are displayed on the user selection screen. When the user selects an image on the user selection screen, the imaging device 110 displays a moving image according to the story corresponding to the selected image.

When "club" is selected on the user selection screen shown in FIG. 11(j), the imaging device 110 displays moving images as shown in FIGS. 11(d) to 11(f). On the other hand, when "Cafe" is selected on the user selection screen shown in FIG. 11(j), the photographing device 110 displays moving images as shown in FIGS. 11(g) to (i). After the reproduction of the content is finished, the photographing device 110 may display a questionnaire screen (not shown) and ask the user to answer the user attributes such as age and sex. As in the case of the above embodiment, while the content is being reproduced, the image capturing device 110h captures images at predetermined timings to obtain captured images. After that, the imaging device 110 transmits the captured image to the management server 120 . At this time, user attributes answered according to the questionnaire screen may be transmitted together.

In the photographing system according to the present embodiment, a photographed image is obtained by the above-described processing, so that the user can obtain a photograph of himself with a natural and rich expression. The photographing system according to the present embodiment may include a billing mechanism (not shown) so that the user himself/herself charges for the photographed image. In this case, a cash slot, an electronic money reader, or the like can be used as the billing mechanism. On the other hand, in the photographing system according to the present embodiment, advertisements are included in the content, and the user who sees the image of the content can recognize the advertiser and the product. Therefore, by using the imaging system according to this embodiment, the advertiser can advertise the product to the user. Therefore, the photographing device 110 may generate billing information based on the user's usage, transmit the billing information to the advertising server 140 , and the advertiser may pay based on the received billing amount collected by the advertising server 140 .

Although the preferred embodiments of the present disclosure have been described above, the present disclosure is not limited to the above embodiments, and various modifications are possible. For example, although the imaging system according to the above embodiment is configured to include the management server 120, the content publishing server 130, and the advertising server 140, the content publishing server 130 and the advertising server 140 are not essential, and can be added as necessary. is possible.

DESCRIPTION OF SYMBOLS 100... Photographing system 110... Photographing apparatus 111... Image display means 112... Photographing means 113... Content reproducing means 114... Photographing means for facial expression recognition 115... Image recording means 116. Photographing control means 117 Facial expression recognition means 120 Management server 121 Content storage means 122 User state data storage means 123 Content supply means 124 Photographed image acquisition means 125 ... User state analysis means 130 ... Content publishing server 140 ... Advertising server 150 ... User terminal 160 ... Network

Claims (7)

image display means for displaying an image included in content;
a photographing means for photographing a user;
content reproduction means for reproducing the content;
a photographing control means for controlling timing of photographing by the photographing means in accordance with reproduction of the content by the content reproducing means;
an image recording means for recording a photographed image acquired by photographing by the photographing means;
A photographing device having a 2. The photographing apparatus according to claim 1, wherein said photographing means photographs at regular time intervals during reproduction of content by said content reproducing means. 2. The photographing apparatus according to claim 1, wherein said photographing means photographs in a predetermined time period determined according to said content. 2. The photographing method according to claim 1, further comprising facial expression recognition photographing means for recognizing the expression of the subject, wherein the photographing means photographs the subject according to the subject's facial expression photographed by the facial expression recognition photographing means. Device. further comprising facial expression recognition means for recognizing facial expressions of a person;
The shooting means shoots moving images continuously during playback of the content,
The facial expression recognition means determines a range to be extracted according to the facial expression of the subject in the captured moving image,
2. The photographing device according to claim 1, wherein said image recording means records said determined moving image. The imaging device according to any one of claims 1 to 5;
a management server connected to the imaging device;
The management server is
a content supplying means for supplying the content reproduced by the content reproducing means to the photographing device;
a photographed image obtaining means for obtaining the photographed image from the photographing device;
user state analysis means for analyzing the user state of the captured image;
A shooting system with further comprising an advertisement server connected to the photographing device;
The ad server
content transmission means for transmitting the content to the management server;
user status data acquisition means for acquiring user status data, which is data of the user status analyzed by the user status analysis means, from the management server;
The imaging system according to claim 6, comprising:

Images