JP2021163335A - Information processing program and information processing system - Google Patents

Abstract

To provide an information processing program and an information processing system which impart a motion to a portion of a subject in an image while maintaining characteristics of the subject possessed by a user in the real space.SOLUTION: The information processing program is executed in a computer of an information terminal device, and causes the computer to identify a part which is a portion of a first subject in an image obtained by photographing the real space, and to overlay part data based on the part on a first object area including the part.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing program and an information processing system.

Augmented reality (AR) is a technology that adds information in virtual space to information in real space and provides the user with a space in which reality is visually expanded. In recent years, the use of AR in smartphones has been advancing, and for example, games or contents for displaying images in real space taken by a camera of a smartphone with visual information in virtual space added to them are provided.

In AR, the user can feel the real space by the captured image, and can also feel the virtual space by the visual information of the added virtual space. For example, in AR, it is possible to display a character such as an animation or a game as visual information in a virtual space against a background of a real space taken by a camera. That is, if AR is used, a virtual character can appear in the real space through the display of the smartphone. However, in most of the contents that use such AR, the character disappears when the display of the smartphone is turned off. In order for the user to embrace the character and feel attachment, it is preferable that the character is materialized even in the real space.

For example, a figure is a materialized character in the real space. By applying AR to the image of the figure, it is possible to add an object to the image of the figure (see, for example, Patent Document 1). For example, if a gun is held in the figure's hand as a weapon, you can add an object in the video that emits a beam from the muzzle. Also, even if the smartphone display is turned off, the figure remains in the real space. Therefore, the user can always feel a sense of unity with the figure.

Japanese Unexamined Patent Publication No. 2019-185796

In Patent Document 1, a figure is made to take a specific pose in order to determine the character of the figure. That is, the pose of the figure for applying AR is predetermined for each character. The purpose of this is to keep the position where the objects prepared in advance for each character are superimposed in the image, but the figure is always fixed in a predetermined pose. Therefore, Patent Document 1 superimposes a character object determined in the image on an object (for example, a weapon) attached to the figure, and does not give movement to the figure itself in the image. There wasn't. In addition, since the characters of the photographed figure have the same pose, there is a problem that the user actually realizes that the figure does not move.

In order to give movement to the character, it is also possible to superimpose an animation display object on the figure in the video. However, each user's figure has its own characteristics, such as variations in the figure manufacturing process, color changes due to aging, and the presence or absence of scratches. If the user is an attached figure, the difference in its characteristics is even more worrisome. Therefore, by simply superimposing the object on the figure in the image, the user often feels the figure and the object as separate objects rather than the figure and the object as one. Therefore, even if AR is used for the figure, the user still rarely feels that the figure in the image is moving. Therefore, the use of AR for figures has a problem that it is difficult to obtain user satisfaction and the availability is low.

In view of the above problems, the present invention provides an information processing program and an information processing system that give movement to a part of a subject in an image while maintaining the characteristics of the subject owned by the user in the real space. Is one of the issues.

The information processing program according to the embodiment of the present invention is an information processing program executed by a computer of an information terminal device, and is a part of a first subject in an image captured in a real space by the computer. Is specified, and the part data based on the part is superimposed on the first object area including the part.

The part data may be transmitted from the server device to the information terminal device. Further, the part data may be generated from the part.

The part data may be adjusted by the first object information based on the first object area.

The computer may have the part selected by the user identified.

The first object area may be selected by the user.

The movement of a part of the user's body in the image may be determined, and the part data may be further based on the determination.

Video data based on the terminal information of the information terminal device may be superimposed on the second object area.

An object in the image may be specified, and image data based on the object may be superimposed on the second object area.

The authentication mark of the first subject may be detected, and the shielding data may be superimposed on the third object area including the authentication mark.

The first subject may be identified by a certification mark.

The first subject may be a character.

The second subject may be specified, and the interaction data indicating the interaction between the first subject and the second subject may be superimposed on the fourth object area.

The information processing system according to the embodiment of the present invention includes an imaging unit that captures a real space, a display unit that displays an image captured by the imaging unit, and a part that is a part of a first subject in the image. Part data based on the parts is superimposed on the second processing unit to be specified, the third processing unit that defines the first object area including the parts, and the first object area.

It is a block diagram which shows the structure of the information processing system which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the server apparatus and information terminal apparatus of the information processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing which changes the part of a subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the shooting before and after the part of a subject in an image changes by using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the image which generated the part data by using the information processing system which concerns on one Embodiment of this invention, and superposed the generated part data. It is a flowchart which shows the information processing which displays the dialogue on the subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the image which the dialogue is displayed on the subject in the image by using the information processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing which displays the dialogue on the subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the shooting before and after displaying the dialogue on the subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing which changes the part of a subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the shooting before and after the part of a subject in an image changes by using the information processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing which displays the interaction between a 1st subject and a 2nd subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the shooting before and after displaying the interaction between a 1st subject and a 2nd subject in an image using the information processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the information processing which changes the part of a subject in an image according to the movement of a part of a user's body by using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the shooting before and after the part of the subject in an image changes according to the movement of a part of a user's body by using the information processing system which concerns on one Embodiment of this invention. It is a figure explaining the changed image by using the information processing system which concerns on one Embodiment of this invention, and using the operation part or the image pickup part of an information terminal apparatus. It is a figure explaining the image which customized the subject or the background by using the information processing system which concerns on one Embodiment of this invention.

Hereinafter, each embodiment of the present invention will be described with reference to the drawings and the like. However, the present invention can be implemented in various aspects without departing from the gist of the technical idea, and is not construed as being limited to the description contents of the embodiments illustrated below.

In order to clarify the explanation, the drawings may schematically represent the width, thickness, shape, etc. of each part as compared with the actual mode, but this is just an example, and the illustrated shape itself is a book. It does not limit the interpretation of the invention. Further, in the drawings, elements having the same functions as those described with respect to the drawings already mentioned in the specification may be designated by the same reference numerals even if they are separate drawings, and duplicate explanations may be omitted. ..

In the present specification, the "video" includes not only a still image but also a moving image.

As used herein, the term "character" includes living things such as people, animals, or plants, or inanimate objects such as robots, created in anime, cartoons, novels, games, advertisements, and the like.

First, the configuration of the information processing system 10 according to the embodiment of the present invention will be described with reference to FIGS. 1 and 2.

[1. Configuration of information processing system 10]
FIG. 1 is a block diagram showing a configuration of an information processing system 10 according to an embodiment of the present invention. As shown in FIG. 1, the information processing system 10 includes a server device 100, a first information terminal device 200-1, a second information terminal device 200-2, ..., And an nth information terminal device 200-n. However, the configuration of the information processing system 10 is not limited to this. The information processing system 10 may include a plurality of server devices 100. When each of the first information terminal device 200-1, the second information terminal device 200-2, ..., And the nth information terminal device 200-n is not particularly distinguished, the information terminal device 200 is described below. I will explain.

The server device 100 and the information terminal device 200 are connected by a communication network 300. The communication network 300 is a system in which a plurality of computers are connected by wire or wirelessly so that they can communicate with each other. The communication network 300 is, for example, the Internet. The communication network 300 may be via a communication base station managed by a communication company. That is, the server device 100 and the information terminal device 200 may be connected by a communication network 300 via a communication base station.

By connecting to the server device 100, the information terminal device 200 can transmit and receive programs, information, or data necessary for information processing in the information terminal device 200.

FIG. 2 is a block diagram showing the configurations of the server device 100 and the information terminal device 200 of the information processing system 10 according to the embodiment of the present invention. As shown in FIG. 2A, the server device 100 includes a computer 110. Further, as shown in FIG. 2B, the information terminal device 200 includes a computer 210. The computer 110 of the server device 100 includes a server processing unit 111. On the other hand, the computer 210 of the information terminal device 200 has a first processing unit 211, a second processing unit 212, a third processing unit 213, a fourth processing unit 214, a fifth processing unit 215, a sixth processing unit 216, and a seventh processing. A unit 217, an eighth processing unit 218, and a ninth processing unit 219 are included.

The server device 100 is, for example, a workstation or a personal computer and includes a computer 110. The computer 110 includes a central processing unit (CPU: Central Processing Unit) or an image processing unit (GPU: Graphics Processing Unit) as the calculation means. By reading the information processing program, the CPU functions as the server processing unit 111 and can execute predetermined information processing. In addition, the GPU can execute arithmetic processing in the video display. The server processing unit 111 may execute information processing using one CPU or GPU, or may execute information processing using a plurality of CPUs or GPUs.

The server device 100 may further include a communication unit 120, a storage unit 130, and the like.

The communication unit 120 is a communication interface capable of transmitting and receiving programs, information, or data.

The storage unit 130 is a memory or storage capable of storing information or data. For example, the storage unit 130 can store video data such as part data, dialogue data, shielding data, and interaction data, which will be described later. The video data includes video information such as pixel size or color. The video data may include audio data. Further, the storage unit 130 may store character information regarding the character.

The server processing unit 111 can determine the information and acquire the data based on the determined information. For example, the server processing unit 111 can determine what kind of information the information received by the communication unit 120 is, and can acquire data based on the determined information from the storage unit 130.

Further, the server processing unit 111 can identify the subject by determining the video data and generate the subject information. Further, the server processing unit 111 can also compare the subject information and the character information to determine the character of the subject.

Further, the server processing unit 111 can generate intermediate part data based on the part data and the part information.

The information terminal device 200 is, for example, a mobile phone, a smartphone, a tablet, a portable game machine, or the like, and includes a computer 210. Further, the computer 210 includes a CPU or a GPU as a calculation means. By reading the information processing program, the CPU functions as the first processing unit 211 to the ninth processing unit 219, and can execute predetermined information processing. In addition, the GPU can execute arithmetic processing in the video display. Each of the first processing units 211 to the ninth processing unit 219 may execute information processing using one CPU or GPU, or may execute information processing using a plurality of CPUs or GPUs.

The information terminal device 200 may further include an image pickup unit 220, a display unit 230, an operation unit 240, a communication unit 250, a storage unit 260, and the like.

The imaging unit 220 can capture a landscape in a real space. The image pickup unit 220 is, for example, a camera or a video. The information terminal device 200 may include a plurality of imaging units 220. When the information terminal device 200 includes two imaging units 220, one imaging unit 220 is provided on the display surface side of the information terminal device 200, and the other imaging unit 220 is on the side opposite to the display surface of the information terminal device 200. It may be provided on the surface.

The display unit 230 can display the image captured by the imaging unit 220. The display unit 230 is, for example, a liquid crystal display device or a light emitting diode (LED) display device.

The operation unit 240 is a user interface that can be operated by the user. The operation unit 240 is, for example, a keyboard, a mouse, a touch panel, a microphone, or the like. The operation unit 240 of the information terminal device 200 preferably includes a touch panel. Since the touch panel is provided on the display unit 230, the user can operate the information terminal device 200 using the touch panel while checking the image displayed on the display unit 230.

The communication unit 250 is a communication interface capable of transmitting and receiving programs, information, or data.

The storage unit 260 is a memory or storage capable of storing information or data.

The first processing unit 211 can identify the subject from the image displayed on the display unit 230. The first processing unit 211 identifies the subject based on the classification pattern having a predetermined feature included in the video. The classification pattern is, for example, a humanoid pattern, an animal pattern, a robotic pattern, and the like, but is not limited thereto. When the subject is a character, the balance between the head and the non-head is often different from the human balance, so the subject is further classified based on a classification pattern with a different classification method such as a two-headed body pattern or a five-headed body pattern. It may be specified. Further, the first processing unit 211 can also detect an authentication mark including information about the subject and identify the subject from the authentication mark. Machine learning, deep learning, or the like can also be used to identify the subject. The identification of the subject is not limited to these, and any method can be applied. In addition, the first processing unit 211 can generate subject information based on the specified subject. Further, when the subject is specified from the certification mark, the first processing unit 211 can generate the certification mark information related to the certification mark.

The second processing unit 212 can extract and determine the parts included in the subject in the image displayed on the display unit 230. The second processing unit 212 can extract parts based on the features or positions of the specified subject, determine which part of the subject the extracted parts correspond to, and specify the extracted parts. The second processing unit can also determine the selected part and specify the selected part. In addition, the second processing unit 212 can acquire part data based on the specified part.

The third processing unit 213 can define an object area for superimposing video data on the video displayed on the display unit 230. The third processing unit 213 acquires the coordinate information of the video and defines the object area according to a predetermined condition. The object area is preferably, but not limited to, a rectangle. The object area may be polygonal, circular, or elliptical. In addition, the third processing unit 213 generates object area information based on the object area.

It is also possible to extract parts and define the object area at the same time. That is, the object area including the parts may be defined, the object area may be extracted, and the parts may be specified from the object area. In this case, the third processing unit 213 defines the object area, and the second processing unit 212 can specify the part from the defined object area.

The part can also be an object area.

The fourth processing unit 214 can adjust the size of the video data according to the size of the object area. Specifically, the fourth processing unit 214 compares the object area information with the video data information, and enlarges or reduces the video data according to the object area information. Further, the fourth processing unit 214 can detect the angle of the subject and adjust the angle of the video data. Further, the fourth processing unit 214 can adjust the color of the video data based on the color information included in the object area information. It is preferable that the color of the contour portion of the video data is adjusted by using the color near the contour of the object area. By matching the color near the contour of the video data with the color near the contour of the object area, it is possible to make it difficult to see the boundary when the video data is superimposed on the subject.

The fifth processing unit 215 can superimpose the video data on the object area of the video displayed on the display unit 230. Specifically, the fifth processing unit 215 superimposes the adjusted video data on the object area based on the object area information.

The sixth processing unit 216 can generate intermediate part data necessary for animation display based on the part of the subject in the image and the part information. The sixth processing unit 216 can generate not only one but a plurality of part data. The number of part data to be generated may be determined by the part or may be determined by the setting.

The seventh processing unit 217 can acquire terminal information from the storage unit 260 of the information terminal device 200. The terminal information includes, but is not limited to, information about the function of the information terminal device 200 or the data stored in the information terminal device 200, such as, for example, the date, time, place, weather, or the birthday of the user. In addition, the seventh processing unit 217 can determine what kind of information the acquired terminal information is. Specifically, if today's date and the user's birthday match, the seventh processing unit 217 determines that it is the birthday. If today's date is a holiday, a holiday, or an event day, it is determined to be a specific day. In determining whether it is the day of the event, it is possible to connect to the Internet and obtain information about the event. Further, if it is a time, it can be determined as morning, noon, or night based on the time. Further, the seventh processing unit 217 can generate environmental information based on the determination.

The eighth processing unit 218 can detect the movement of a part of the user's body and determine the movement. For example, the eighth processing unit 218 can analyze the video data and detect the motion. It is also possible to detect movement based on the signal on the touch panel. In addition, the eighth processing unit 218 can determine the user's instruction (the user's action with respect to the subject) from the detected movement. Further, the eighth processing unit 218 can generate user behavior information based on the determination.

The ninth processing unit 219 can determine the facial expression of the user photographed by the imaging unit 220. For example, the ninth processing unit 219 can determine whether or not the user's facial expression is a smile from the shape of the corner of the mouth. The facial expression of the user determined by the ninth processing unit 219 is not limited to a smile. The ninth processing unit 219 can also determine a crying face or an angry face. In addition, the ninth processing unit 219 can generate user emotion information based on the determination.

Hereinafter, specific information processing using the information processing system 10, the information processing program, or the information processing method according to the embodiment of the present invention will be described with reference to FIGS. 3 to 17. The information processing in the information processing system, information processing program, or information processing method according to the embodiment of the present invention is not limited to the following examples.

[2. Information processing that changes the parts of the subject (information processing A)]
With reference to FIGS. 3 and 4, while the image pickup unit 220 of the information terminal device 200 captures the subject 500 in the real space, the parts of the subject 400 in the image displayed on the display unit 230 of the information terminal device 200 are captured. Information processing to be changed will be described.

FIG. 3 is a flowchart showing information processing in which the parts of the subject 400 in the video are changed by using the information processing system 10 according to the embodiment of the present invention. Hereinafter, for convenience, the information processing shown in FIG. 3 will be described as information processing A.

The information processing A shown in FIG. 3 includes a step of specifying the subject 400 in the image (S110), a step of specifying the part 401 included in the subject 400 (S120), and a step of acquiring the part data 411 related to the part 401. (S130), a step of defining the first object area 301 in the video (S140), a step of adjusting the part data 411 (S150), and a step of superimposing the part data 411 on the first object area 301 (S160). ) And, including. Hereinafter, each step will be described with reference to FIG.

FIG. 4 is a diagram illustrating shooting before and after the parts of the subject 400 in the image change by using the information processing system 10 according to the embodiment of the present invention. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 4 (A) displays an image before the parts of the subject 400 are changed, and the display of the information terminal device 200 shown in FIG. 4 (B). The image after the parts of the subject 400 are changed is displayed in the unit 230.

As shown in FIGS. 4A and 4B, the user uses the imaging unit 220 of the information terminal device 200 to photograph the subject 500 in the real space. The captured image is displayed on the display unit 230 of the information terminal device 200. That is, the user can visually recognize the subject 500 in the real space while checking the subject 400 of the image displayed on the display unit 230. The imaging unit 220 is provided not only on the display unit 230 side but also on the side opposite to the display unit 230. Hereinafter, for convenience, the imaging unit 220 on the opposite side of the display unit will be referred to as the first imaging unit 220-1, and the imaging unit 220 on the display unit side will be referred to as the second imaging unit 220-2. The subject 500 is photographed by the first imaging unit 220-1.

The subject 500 is, but is not limited to, a three-dimensional object such as a figure, a doll, a stuffed animal, or a plastic model in which a character is imitated. The subject 500 may be a flat object such as a poster on which a character is drawn.

In step S110, the first processing unit 211 identifies the subject 400 from the video displayed on the display unit 230. The first processing unit 211 identifies the subject 400 based on the classification pattern. It is preferable that the character is specified for the subject 400, but the character may not be specified. The first processing unit 211 may be able to identify the subject 400 separated from the background in the image. In addition, the first processing unit 211 generates subject information based on the specified subject 400.

The server device 100 can also be used to identify the subject 400. The communication unit 250 of the information terminal device 200 transmits the captured video data to the server device 100. The communication unit 120 of the server device 100 receives the video data. The server processing unit 111 identifies the subject 400 by determining the video data and generates the subject information. At that time, the server processing unit 111 may specify the character of the subject 400. For example, the server processing unit 111 can identify the character of the subject 400 by comparing the subject information with the character information stored in the storage unit 130. When the character of the subject 400 can be identified, the subject information includes information about the character. The communication unit 120 transmits the subject information generated by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the subject information, the information terminal device 200 can acquire the subject information. The acquired subject information may be stored in the storage unit 260 of the information terminal device 200.

In step S120, the second processing unit 212 identifies the part 401 included in the subject 400. Specifically, the second processing unit 212 extracts the part 401, determines which part of the subject 400 the extracted part 401 corresponds to, and identifies the part 401. The part 401 shown in FIG. 4A is a mouth, but the part 401 is not limited to the mouth. Part 401 may be a part of the subject 400 (face, torso, arms, legs, eyes, nose, mouth, ears, hair, hands, feet, etc.), and is the clothes worn by the subject. It may be a tool (such as a weapon or a wing) that the subject has. In the subject 400, it is particularly preferable that the parts 401 such as eyes or mouth are determined. By changing the eyes of the subject 400, it is possible to display an animation in which the subject 400 is blinking. Further, by changing the mouth of the subject 400, it is possible to display an animation in which the subject 400 is talking.

The second processing unit 212 may determine a plurality of parts 401. By determining a plurality of parts, it is possible to display an animation with a plurality of parts.

The part 401 can also be selected by the user. When it is difficult for the second processing unit 212 to extract the part 401, or when there is a specific part that the user wants to be animated, the second processing unit 212 extracts the part 401 by selecting the part 401. It becomes possible to determine the part 401 without doing so. Therefore, the accuracy of the determination of the part 401 by the second processing unit 212 is improved.

In addition, the second processing unit generates part information based on the specified part 401.

In step S130, the information terminal device 200 acquires the part data 411 based on the part information. The communication unit 250 of the information terminal device 200 transmits the parts information to the server device 100. The communication unit 120 of the server device 100 receives the part information. The server processing unit 111 determines the part information, and acquires the part data 411 based on the determination from the storage unit 130. The communication unit 120 transmits the part data 411 acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the part data 411, the information terminal device 200 can acquire the part data 411. The acquired part data 411 may be stored in the storage unit 260 of the information terminal device 200.

The part data 411 may be still image data or moving image data. Even if the part data is still image data, the animation of the subject 400 can be displayed by alternately displaying the parts originally possessed by the subject 400 and the still image data.

In step S140, the third processing unit 213 defines the first object area 301 including the part 401 in the image displayed on the display unit 230. Based on the part information, the third processing unit 213 acquires the coordinate information of the video so as to surround the part 401, and defines the first object area 301. In addition, the third processing unit 213 generates the first object area information based on the first object area 301.

In step S150, the fourth processing unit 214 adjusts the size of the acquired part data 411 according to the size of the first object area 301. Further, the fourth processing unit 214 adjusts the angle of the part data 411 based on the subject information. Further, the fourth processing unit 214 adjusts the color of the part data 411 based on the color information included in the first object area information. By adjusting the size, angle, and color of the part data 411, it is possible to display a comfortable image even when the part data 411 is superimposed on the subject 400.

In step S160, the fifth processing unit 215 superimposes the adjusted part data 411 on the first object area 301 based on the first object area information. By superimposing the part data 411 on the subject 400, the subject 400 is displayed as an animation, and the subject 400 can be moved.

Although the information processing A has been described above, when the information processing A is executed, the display unit 230 of the information terminal device 200 displays an image in which a part of the subject 400 is changed. Since the image other than a part of the subject 400 is an image of the subject 500, the user can feel the subject 400 as if it were the subject 500 in the real space. Further, since the part data 411 is adjusted based on the subject 400, it is possible to display the part data 411 without discomfort even if the part data 411 is superimposed on the subject 400. Therefore, the user can feel as if the subject 500 in the real space, which is originally stationary, is moving by the animation display in which a part of the subject 400 is changed.

[3. Part data generation]
In step S130 described above, the part data 411 is acquired from the storage unit 130 of the server device 100, but the part data 411 can also be generated. Here, a method of generating the part data 411 will be described with reference to FIG.

FIG. 5 is a diagram illustrating an image in which part data is generated by using the information processing system 10 according to the embodiment of the present invention and the generated part data is superimposed. In FIG. 5, the determined part 401 will be described as the eye of the subject 400.

5 (A) to 5 (D) show the first part data 411 to the fourth part data 411-4, respectively. The sixth processing unit 216 generates the second part data 411-2 to the fourth part data 411-4 as intermediate part data based on the original first part data 411-1 of the subject 400 and the part information. .. When the specified part is an open eye, the second part data 411-2 to the fourth part data 411-4 in the middle of closing the eyes are generated based on the eye information included in the part information. For example, by performing a process of deleting a part of the pupil of the eye, it is possible to generate intermediate second part data 411-2 to fourth part data 411-4.

The server device 100 can also be used in generating the part data 411. The communication unit 250 of the information terminal device 200 transmits the original first part data 411-1 of the subject and the part information to the server device 100. The communication unit 120 of the server device 100 receives the first part data 411-1 and the part information. The server processing unit 111 generates intermediate second part data 411-2 to fourth part data 411-4 based on the received first part data 411-1 and part information. The communication unit 120 transmits the second part data 411-2 to the fourth part data 411-4 generated by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the second part data 411-2 to the fourth part data 411-4, the information terminal device 200 can acquire the generated part data 411.

Instead of step S130 shown in FIG. 3, the information processing related to the generation of the part data 411 described above is executed, and the information processing of steps S140 to S160 is executed using the generated part data 411. As a result, as shown in FIG. 5 (E), the display unit 230 of the information terminal device 200 displays an image in which a part (part) of the subject 400 is changed by using the generated part data 411. Since the information processing system 10 can generate the part data 411, the subject 400 can be moved even when the part data 411 based on the part information is not stored in the storage unit 130 of the server device 100. Can be done. Further, since the generated part data 411 is generated from the subject 400, it is possible to display the part data 411 without discomfort even if the part data 411 is superimposed on the subject 400.

[4. Information processing that displays the lines of the subject (information processing B)]
With reference to FIGS. 6 and 7, while the image pickup unit 220 of the information terminal device 200 captures the subject 500 in the real space, the dialogue is displayed on the subject 400 of the image displayed on the display unit 230 of the information terminal device 200. Information processing will be described.

FIG. 6 is a flowchart showing information processing for displaying dialogue on a subject 400 in an image using the information processing system 10 according to the embodiment of the present invention. Hereinafter, for convenience, the information processing shown in FIG. 6 will be described as information processing B.

The information processing B shown in FIG. 6 includes a step of acquiring terminal information (S210), a step of generating environmental information (S220), a step of acquiring dialogue data 412 based on the environmental information (S230), and in the video. Includes a step of defining the second object area 302 (S240), a step of adjusting the dialogue data 412 (S250), and a step of superimposing the dialogue data 412 on the second object region 302. Hereinafter, each step will be described with reference to FIG. 7.

FIG. 7 is a diagram illustrating an image in which dialogue is displayed on a subject 400 in an image by using the information processing system 10 according to the embodiment of the present invention.

In step S210, the seventh processing unit 217 acquires terminal information such as the time stored in the storage unit 260 of the information terminal device 200. Here, the seventh processing unit 217 acquires information on the time as terminal information.

In step S220, the seventh processing unit 217 determines the acquired terminal information and generates the environmental information necessary for acquiring the serif data 412. Here, the seventh processing unit 217 determines that the information regarding the time is the "morning" information, and generates the environmental information including the "morning" information.

The server device 100 can also be used in generating the environmental information. The communication unit 250 of the information terminal device 200 transmits the terminal information to the server device 100. The communication unit 120 of the server device 100 receives the terminal information. The server processing unit 111 determines the terminal information and generates the environment information. The communication unit 120 transmits the environment information generated by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the environmental information, the information terminal device 200 can acquire the environmental information. The acquired environmental information may be stored in the storage unit 260 of the information terminal device 200.

In step S230, the information terminal device 200 acquires the dialogue data 412 based on the terminal information. The communication unit 250 of the information terminal device 200 transmits environmental information to the server device 100. The communication unit 120 of the server device 100 receives the environmental information. The server processing unit 111 determines the environmental information and acquires the dialogue data 412 based on the environmental information from the storage unit 130. The communication unit 120 transmits the dialogue data acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the dialogue data 412, the information terminal device 200 can acquire the dialogue data 412. The acquired serif data 412 may be stored in the storage unit 260 of the information terminal device 200.

When the generation of the environment information is executed by the server device 100, the server device 100 may acquire the dialogue data 412 following the generation of the environment information. That is, the server device 100 may acquire the dialogue data 412 based on the environment information from the storage unit 130 and transmit only the dialogue data 412 to the information terminal device 200 without transmitting the environment information to the information terminal device 200. ..

The dialogue data 412 includes at least conversation data, and may include speech balloon data. The conversation data of the dialogue data 412 is data of a conversation in which the subject 400 speaks to the user. For example, the conversation data is "Congratulations" if the environmental information includes birthday information, "Good morning" if the environmental information contains "morning" information, or "Specific date" in the terminal information. If the information is included, it may be something like "Let's go play somewhere". Further, the dialogue data 412 may include voice data.

When the character of the subject 400 is specified, the server processing unit 111 may acquire the dialogue data 412 based on the character information and the environment information. In that case, since the dialogue data 412 that matches the way the character speaks can be acquired, the user can share the world view of the character.

Further, instead of the dialogue data 412, the video data of the character based on the environmental information or the advertisement data based on the environmental information may be acquired. These data may be acquired from the storage unit 130 of the server device 100, or may be acquired from another server device. These data can also be superimposed on the second object area 302.

In step S240, the third processing unit 213 defines the second object area 302 in the image displayed on the display unit 230. The third processing unit 213 defines the second object area 302 in the vicinity of the face portion of the subject 400 so as not to overlap the face portion as much as possible. Since the face portion is often located above the subject 400, the third processing unit 213 may define the second object area 302 at a position adjacent to the upper portion of the subject 400. In addition, the third processing unit 213 generates the second object area information based on the second object area 302.

In step S250, the fourth processing unit 214 adjusts the size of the acquired serif data 412 according to the size of the second object area 302. The fourth processing unit 214 compares the second object area information of the second object area 302 with the dialogue information of the dialogue data 412, and adjusts the size of the dialogue data 412 based on the second object region information. If the dialogue data 412 is too small, the displayed dialogue will be too small, and the user will not feel immersive. Therefore, a lower limit may be set for the size of the dialogue data 412.

In step S260, the fifth processing unit 215 superimposes the adjusted serif data 412 on the second object area 302 based on the second object area information. When the dialogue data 412 includes voice data, the dialogue data 412 may be displayed and the voice data may be reproduced. The presence of voice can enhance the user's immersive feeling.

Information processing B can be executed in combination with information processing A.

Although the information processing B has been described above, when the information processing B is executed, the dialogue of the subject 400 is displayed on the image of the display unit 230 of the information terminal device 200. Since the dialogue is based on the terminal information, the user can feel as if the character of the subject 500 in the real space speaks to the user. Therefore, the user can immerse himself in the world view of the character of the subject 500.

[5. Information processing that displays dialogue based on the background of the subject (information processing C)]
With reference to FIGS. 8 and 9, information is displayed on the subject 400 based on the background of the image displayed on the display unit 230 while the image pickup unit 220 of the information terminal device 200 captures the subject 500 in the real space. The processing will be described.

FIG. 8 is a flowchart showing information processing for displaying dialogue on a subject 400 in an image using the information processing system 10 according to the embodiment of the present invention. Hereinafter, for convenience, the information processing shown in FIG. 8 will be described as information processing C.

The information processing C shown in FIG. 8 includes a step of generating background information from the background of the subject 400 (S310), a step of acquiring dialogue data 413 based on the background information (S320), and a second object area 302 in the video. The step (S330), the step of adjusting the dialogue data 413 (S340), and the step of superimposing the dialogue data 413 on the second object area 302 are included. Hereinafter, each step will be described with reference to FIG.

FIG. 9 is a diagram illustrating shooting before and after displaying dialogue on a subject 400 in an image using the information processing system 10 according to the embodiment of the present invention. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 9A displays an image before displaying the dialogue on the subject 400, and the display of the information terminal device 200 shown in FIG. 9B is displayed. The image after displaying the dialogue on the subject 400 is displayed in the unit 230. As shown in FIGS. 9A and 9B, an object 510 is placed in addition to the subject 500 in the real space. The object 510 shown in FIGS. 9 (A) and 9 (B) is a fan mainly used in the summer season.

In step S310, the first processing unit 211 identifies an object 450 included in a portion (background) other than the subject 400 from the video displayed on the display unit 230. The first processing unit 211 identifies the object 450 based on a predetermined classification pattern. In addition, the first processing unit 211 generates object information based on the specified object 450.

The server device 100 can also be used to identify the object 450. The communication unit 250 of the information terminal device 200 transmits the captured video data to the server device 100. The communication unit 120 of the server device 100 receives the video data. The server processing unit 111 identifies the object 450 by determining the video data and generates the object information. The communication unit 120 transmits the object information generated by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the object information, the information terminal device 200 can acquire the object information. The acquired object information may be stored in the storage unit 260 of the information terminal device 200.

In step S320, the information terminal device 200 acquires the dialogue data 413 based on the object information. The communication unit 250 of the information terminal device 200 transmits the object information to the server device 100. The communication unit 120 of the server device 100 receives the object information. The server processing unit 111 determines the object information and acquires the dialogue data 413 based on the object information from the storage unit 130. The communication unit 120 transmits the dialogue data 413 acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the dialogue data 413, the information terminal device 200 can acquire the dialogue data 413. The acquired serif data 413 may be stored in the storage unit 260 of the information terminal device 200.

When the server device 100 executes the identification of the object 450, the server device 100 may acquire the dialogue data 413 following the generation of the object information. That is, the server device 100 may acquire the dialogue data 413 based on the object information from the storage unit 130 and transmit only the dialogue data 413 to the information terminal device 200 without transmitting the object information to the information terminal device 200. ..

The dialogue data 413 includes at least conversation data, and may include speech balloon data. The conversation data of the dialogue data 413 is the data of the conversation that the subject 400 talks to the user, but is based on the object information. For example, conversation data is "hot" if the object information includes summer information, "pretty" if the object information contains flower information, or food information is included in the object information. If so, it means "I'm hungry." Further, the dialogue data 413 may include voice data.

When the character of the subject 400 is specified, the server processing unit 111 may acquire the dialogue data 413 based on the character information and the object information. In that case, since the dialogue data 413 that matches the way the character speaks can be acquired, the user can share the world view of the character.

Since step S330, step S340, and step S350 are the same as step S240, step S250, and step S260 described above, respectively, description thereof will be omitted here.

Information processing C can be executed in combination with information processing A or information processing B.

Although the information processing C has been described above, when the information processing C is executed, the lines of the subject 400 according to the shooting environment can be displayed on the display unit 230 of the information terminal device 200. Therefore, the user can feel the world view of the character of the subject 500 fused to the environment of the real space, and can embrace the character of the subject 500.

[6. Information processing that changes the parts of the subject based on the certification mark (information processing D)]
With reference to FIGS. 10 and 11, the image pickup unit 220 of the information terminal device 200 captures the subject 500 to which the authentication mark in the real space is added, while changing the parts of the subject 400 of the image displayed on the display unit 230. The information processing flow to be performed will be described.

FIG. 10 is a flowchart showing information processing in which the parts of the subject 400 in the video are changed by using the information processing system 10 according to the embodiment of the present invention. Hereinafter, for convenience, the information processing shown in FIG. 10 will be described as information processing D.

In FIG. 10, the information processing D includes a step of specifying the subject 400 based on the authentication mark 403 in the video (S405), a step of specifying the part 401 included in the subject 400 (S410), and part data related to the part 401. The step of acquiring 411 (S415), the step of defining the first object area 301 in the image (S420), the step of adjusting the part data 411 (S425), and the part data 411 in the first object area 301. A step of superimposing (S430), a step of acquiring shielding data 414 (S435), a step of defining a third object area 303 in the image (S440), and a step of adjusting the shielding data 414 (S445). A step (S450) of superimposing the shielding data 414 on the third object area 303 is included. Hereinafter, each step will be described with reference to FIG.

FIG. 11 is a diagram illustrating shooting before and after the parts of the subject 400 in the image change by using the information processing system 10 according to the embodiment of the present invention. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 11A displays an image before the parts of the subject 400 are changed, and the display of the information terminal device 200 shown in FIG. 11B is displayed. The image after the parts of the subject 400 are changed is displayed in the unit 230.

The subject 500 shown in FIGS. 11A and 11B is provided with a certification mark 520 that can identify the subject 500 (including information about the subject 500). Therefore, by reading the certification mark 520 (specifically, the certification mark 403 in the video), the form and character of the subject 500 can be specified. The certification mark 520 shown in FIGS. 11A and 11B is a QR (Quick Response) code, but the certification mark 520 is not limited to the QR code (registered trademark). The certification mark 520 may be, for example, a barcode, as long as the subject 500 can be identified by analyzing the certification mark 520 (specifically, the certification mark 403 in the video). ..

In step S405, the first processing unit 211 detects the certification mark 403 from the video displayed on the display unit 230, and identifies the subject 400 from the certification mark 403. In addition, the first processing unit 211 generates subject information based on the specified subject 400. Further, the first processing unit 211 generates the certification mark information.

Since step S410, step S415, step S420, step S425, and step S430 are the same as the above-mentioned steps S120, S130, step S140, step S150, and step S160, respectively, description thereof will be omitted here.

In step S435, the information terminal device 200 acquires the shielding data 414 for hiding the certification mark 403. The communication unit 250 of the information terminal device 200 transmits the authentication mark information to the server device 100. The communication unit 120 of the server device 100 receives the authentication mark information. The server processing unit 111 determines the authentication mark information, and acquires the shielding data 414 based on the determination from the storage unit 130. The communication unit 120 transmits the shield data 414 acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the shield data 414, the information terminal device 200 can acquire the shield data 414. The acquired shielding data 414 may be stored in the storage unit 260 of the information terminal device 200.

The shielding data 414 may be data for displaying a rectangle of a single color, or may be data for displaying a rectangle of a plurality of colors. Further, the shielding data 414 may be data for displaying a rectangle with a pattern. Further, the shielding data 414 is not limited to displaying a rectangle. The shielding data 414 may display a circular shape, an elliptical shape, or the like. Further, when the character of the subject 400 is specified, the shielding data 414 may be an image such as a trademark or an emblem peculiar to the character.

In step S440, the third processing unit 213 defines the third object area 303 including the certification mark 403 in the image displayed on the display unit 230. Based on the certification mark information, the third processing unit 213 acquires the coordinate information of the video so as to surround the certification mark 403, and defines the third object area 303. In addition, the third processing unit 213 generates the third object area information based on the third object area 303.

In step S445, the fourth processing unit 214 adjusts the size of the acquired shielding data 414 according to the size of the third object area 303. Further, the fourth processing unit 214 adjusts the angle of the shielding data 414 based on the subject information. Further, the fourth processing unit 214 adjusts the color of the shielding data 414 based on the color information included in the third object area information. For example, the fourth processing unit 214 can adjust the shielding data 414 so as to match the color of the third object area 303 from which the authentication mark 403 has been removed.

In step S450, the fifth processing unit 215 superimposes the adjusted shielding data 414 on the third object area 303 based on the third object area information. Since the superimposed shielding data 414 is adjusted based on the color information included in the third object area information, the certification mark 403 can be hidden without discomfort. Therefore, even if the authentication mark 403 of the subject 400 is hidden, the impression given to the user of the subject 400 does not change.

Information processing D can be executed in combination with information processing A to information processing C.

Although the information processing D has been described above, the authentication mark 403 is hidden in the image of the display unit 230 of the information terminal device 200 by executing the information processing D. Therefore, the user can immerse himself in the world view of the character of the subject 500 without being disturbed by the authentication mark 403.

[7. Information processing that displays interactions between multiple subjects (information processing E)]
With reference to FIGS. 12 and 13, the image pickup unit 220 of the information terminal device 200 captures the first subject 500-1 and the second subject 500-2 in the real space while displaying them on the display unit 230 of the information terminal device 200. Information processing for displaying the interaction between the first subject 400-1 and the second subject 400-2 in the generated image will be described.

FIG. 12 is a flowchart showing information processing for displaying the interaction between the first subject 400-1 and the second subject 400-2 in an image using the information processing system 10 according to the embodiment of the present invention. .. Hereinafter, for convenience, the information processing shown in FIG. 12 will be described as information processing E.

The information processing E shown in FIG. 12 includes a step of specifying the first subject 400-1 in the image (S510), a step of specifying the second subject 400-2 in the image (S520), and interaction data 415. (S530), the step of defining the fourth object area 304 in the video (S540), the step of adjusting the interaction data 415 (S550), and the interaction data 415 superimposed on the fourth object area 304. Step (S560), and the like. Hereinafter, each step will be described with reference to FIG.

FIG. 13 is a diagram illustrating shooting before and after displaying the interaction between the first subject 400-1 and the second subject 400-2 in an image using the information processing system 10 according to the embodiment of the present invention. Is. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 13A displays an image before displaying the interaction between the first subject 400-1 and the second subject 400-2. The display unit 230 of the information terminal device 200 shown in 13 (A) displays an image after displaying the interaction between the first subject 400-1 and the second subject 400-2.

As shown in FIGS. 13A and 13B, the user uses the imaging unit 220 of the information terminal device 200 to photograph the first subject 500-1 and the second subject 500-2 in the real space. .. In shooting, a part of the first subject 500-1 and a part of the second subject 500-2 may overlap, but the part showing the characteristics of the first subject 500-1 and the second subject 500-2 It is preferable that it does not overlap with the characteristic portion. The characteristic portion is, for example, the face portion.

Since step S510 and step S520 are the same as step S110 described above, description thereof will be omitted here.

In step S530, the information terminal device 200 acquires the interaction data 415 based on the first subject information and the second subject information. The communication unit 250 of the information terminal device 200 transmits the first subject information and the second subject information to the server device 100. The communication unit 250 of the server device 100 receives the first subject information and the second subject information, and the server processing unit 111 determines the first subject information and the second subject information. Specifically, the server processing unit 111 determines the relationship between the first subject 500-1 related to the first subject information and the second subject 500-2 related to the second subject information. The relationship between the first subject 500-1 and the second subject 500-2 is, for example, whether it is a person and a person, a person and an animal, or a person and a robot.

Further, the server processing unit 111 may specify each character from the first subject information and the second subject information. By identifying each character of the first subject 500-1 related to the first subject information and the second subject 500-2 related to the second subject information, the server processing unit 111 determines the relationship between the characters. be able to. That is, the server processing unit 111 can make a determination based on the character, such as whether the character appears in the same animation. Therefore, the server processing unit 111 can also determine whether the characters are allies, enemies, or lovers, and the first subject 500-1 and the second subject 500-1 and the second subject can be determined. The relevance to 500-2 can be determined in more detail.

The server processing unit 111 acquires the interaction data 415 based on the determination of the relationship between the first subject 500-1 and the second subject 500-2 from the storage unit 130. The communication unit 120 transmits the interaction data 415 acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the interaction data 415, the information terminal device 200 can acquire the interaction data 415. The acquired interaction data 415 may be stored in the storage unit 260 of the information terminal device 200.

The interaction data 415 may be generated by the information terminal device 200.

In step S540, the third processing unit 213 defines the fourth object area 304 in the image displayed on the display unit 230. The fourth object area 304 is preferably defined between the first subject 400-1 and the second subject 400-2, but is not limited to this. The third processing unit 213 acquires the coordinate information of the video and defines the fourth object area 304. In addition, the third processing unit 213 generates the fourth object area information based on the fourth object area.

In step S550, the fourth processing unit 214 adjusts the size of the acquired interaction data 415 according to the size of the fourth object area 304. Further, the fourth processing unit 214 adjusts the color of the interaction data 415 based on the color information included in the fourth object area information.

In step S560, the fifth processing unit 215 superimposes the adjusted interaction data 415 on the fourth object area 304 based on the fourth object area information.

Information processing E can be executed in combination with information processing A to information processing D.

Although the information processing E has been described above, when the information processing E is executed, the image of the display unit 230 of the information terminal device 200 has an interaction between the first subject 400-1 and the second subject 400-2. Is displayed. Since different interactions can be obtained depending on the combination of the subjects 500, the user can enjoy the game characteristics of the combination of the subjects 500. Therefore, the utilization rate of AR for the subject 500, which is a character, is improved.

[8. Information processing that changes the parts of the subject by detecting movement (information processing F)]
With reference to FIGS. 14 and 15, while the image pickup unit 220 of the information terminal device 200 captures the subject 500 in the real space, the movement of a part of the user's body is also captured, and the image displayed on the display unit 230. The information processing flow for changing the parts of the subject 400 according to the movement will be described.

FIG. 14 is a flowchart showing information processing in which the information processing system 10 according to the embodiment of the present invention is used to change the parts of the subject 400 in the image according to the movement of a part of the user's body. Hereinafter, for convenience, the information processing shown in FIG. 14 will be described as information processing F.

The information processing F shown in FIG. 14 determines the step of identifying the subject 400 in the image (S610), the step of determining the parts 401 included in the subject 400 (S620), and the movement of a part of the user's body. Step (S630), the step of acquiring the part data 416 based on the part 401 (S640), the step of defining the first object area 301 (S650), the step of adjusting the part data 416 (S660), and the first step. One step (S670) of superimposing the part data 416 on the object area 301 is included. Hereinafter, each step will be described with reference to FIG.

FIG. 15 is a diagram illustrating shooting before and after the information processing system 10 according to the embodiment of the present invention is used, and the parts of the subject 400 in the image change according to the movement of a part of the user's body. be. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 15 (A) displays an image before the parts of the subject 400 are changed, and the display of the information terminal device 200 shown in FIG. 15 (B). The unit 230 displays an image after the parts of the subject 400 have changed according to the movement of a part of the user's body.

As shown in FIGS. 15A and 15B, the user brings the finger 530, which is a part of the user's body, close to the subject 500, and moves the finger 530 with respect to the subject 500. For example, as shown in FIGS. 15 (A) and 15 (B), the user can move the finger 530 so as to stroke the subject 500. Alternatively, the user can move the finger 530 so as to poke the subject 500. The movement of the finger 530 is not limited to these.

Further, the finger 460 in the image may be in the vicinity of the subject 400, and the finger 530 in the real space may or may not touch the subject 500. Further, the finger 530 does not have to be in the vicinity of the subject 500, and may be away from the subject 500. For example, the finger 530 may be between the subject 500 and the information terminal device 200.

Since step S610 and step S620 are the same as step S110 and step S120 described above, respectively, description thereof will be omitted here.

In step S630, the eighth processing unit 218 detects the movement of the finger 460 and determines the movement of the finger 460. The eighth processing unit 218 detects the finger 460 that repeatedly makes a constant movement in the vicinity of the subject 400. In addition, the eighth processing unit 218 determines the user's behavior with respect to the subject 400 from the repeated constant movement. For example, when the finger 460 repeatedly moves in the vicinity of the subject 400, the eighth processing unit 218 determines that the user is stroking the subject 400. Further, when the finger 460 repeatedly moves toward and away from the subject 400, the eighth processing unit determines that the user is poking the subject 400. The eighth processing unit 218 generates user behavior information based on the determination.

In step S640, the information terminal device 200 acquires the part data 416 based on the part information and the user behavior information. The communication unit 250 of the information terminal device 200 transmits the parts information and the user behavior information to the server device 100. The communication unit 120 of the server device 100 receives the parts information and the user behavior information. The server processing unit 111 determines the part information and the user behavior information, and acquires the part data 416 based on the determination from the storage unit 130. The communication unit 120 transmits the part data 416 acquired by the server processing unit 111 to the information terminal device 200. When the communication unit 250 receives the part data 416, the information terminal device 200 can acquire the part data 411. The acquired part data 416 may be stored in the storage unit 260 of the information terminal device 200.

Step S650, step S660, and step S670 are the same as steps S140, S150, and S160 described above, respectively, and thus the description thereof will be omitted here.

Information processing F can be executed in combination with information processing A to E.

Although the information processing F has been described above, when the information processing F is executed, the display unit 230 of the information terminal device 200 displays an image in which a part of the subject 400 is changed according to the intention of the user. .. Therefore, the sense of unity between the user and the character of the subject 500 is enhanced, and the satisfaction level of the user is improved.

[9. Use of operation unit or imaging unit]
The information processing system 10 can change the parts of the subject 400 in the image or display the dialogue of the subject 400 by using the operation unit 240 or the imaging unit 220 of the information terminal device 200.

FIG. 16 is a diagram for explaining a changed image by using the information processing system 10 according to the embodiment of the present invention and using the operation unit 240 or the image pickup unit 220 of the information terminal device 200. Specifically, the display unit 230 of the information terminal device 200 shown in FIG. 16 (A) displays an image after the parts of the subject have changed, and the display unit of the information terminal device 200 shown in FIG. 16 (B). On the 230, an image in which the dialogue is displayed on the subject 400 is displayed.

In FIG. 16A, the user directly touches the display unit 230 of the information terminal device 200 while photographing the subject 500 in the real space. The display unit 230 is provided with a touch panel as the operation unit 240, and the operation unit 240 can detect the contact of the finger 530 with the display unit. The eighth processing unit 218 detects the movement of the finger 530 via the operation unit 240 and determines the movement of the finger 460. Specifically, the eighth processing unit 218 determines whether or not the movement of the finger 530 has regularity. When the movement of the finger 530 has a predetermined regularity, the eighth processing unit 218 generates the user behavior information.

Since the subsequent information processing is the same as the above description, the description thereof is omitted here, but the information terminal device 200 can acquire the part data 417 based on the part information and the user behavior information. Further, the part data 417 is superimposed on the defined first object area 301. In FIG. 16A, since it is determined that the finger 530 is stroking the subject 400, the part data 417 of the facial expression that the subject 400 wants to tickle is superimposed.

In FIG. 16B, the user is photographing the user's face with the second imaging unit 220-2 while the first imaging unit 220-1 is photographing the subject 500 in the real space. The ninth processing unit 219 determines the user's facial expression from the image of the user's face taken by the imaging unit 220. Specifically, the ninth processing unit 219 determines whether or not the user's facial expression is a smile. In addition, the ninth processing unit 219 generates user emotion information based on the determination result.

The determination by the ninth processing unit 219 is not limited to the facial expression of the user. The ninth processing unit 219 may be any image taken by the second imaging unit 220-2, and may determine the movement of the user's body. For example, the ninth processing unit may determine a predetermined movement of the user's body and generate user emotion information.

Since the subsequent information processing is the same as the above description, detailed description thereof will be omitted, but the information terminal device 200 can acquire the dialogue data 418 based on the dialogue information and the user emotion information. Further, the dialogue data 418 is superimposed on the defined second object area 302. In FIG. 16B, since it was determined that the user's facial expression was not a smile, dialogue data 418 including conversation data “tired” asking why the subject 400 was not smiling is superimposed.

In FIG. 16A, dialogue data may be acquired based on dialogue information and user behavior information, and in FIG. 16B, parts data may be acquired based on parts information and user emotion information. good.

Further, in the information processing system 10, dialogue data 418 or part data may be acquired based on the object photographed by the second imaging unit 220-2. For example, the second imaging unit 220-2 may photograph food, determine the photographed food, and acquire dialogue data such as "want to eat" or "looks delicious".

Further, although not shown, the information processing system 10 can also use the microphone of the operation unit 240 of the information terminal device 200. That is, the information terminal device 200 may determine the voice of the user via the microphone and acquire the part data or the dialogue data of the subject 400. By using the microphone of the information terminal device 200 in the information processing system 10, the user can feel as if he / she is interacting with the subject 400. Therefore, the user can achieve deeper communication with the character of the subject 400 through the dialogue with the subject 400.

Machine learning or deep learning can be used in determining facial expressions or voices.

As described above, even if the operation unit 240 or the imaging unit 220 of the information terminal device 200 is used, it is possible to display an image in which a part of the subject 400 is changed according to the user's intention on the display unit 230 of the information terminal device 200. .. Therefore, the sense of unity between the user and the character of the subject 500 is enhanced, and the satisfaction level of the user is improved.

[10. Subject customization]
The information processing system 10 can customize the subject 400 or the background in the image by using the operation unit 240 of the information terminal device 200.

17 (A) and 17 (B) are diagrams for explaining an image in which the subject 400 or the background is customized by using the information processing system 10 according to the embodiment of the present invention, respectively.

The subject 400 in the image of the display unit 230 of the information terminal device 200 shown in FIG. 17A has a different color of the hair 419 from the subject 500 in the real space. That is, the user can customize the subject 400 in the image according to the user's preference by using the information processing system 10 while photographing the subject 500 in the real space.

An object 420 that does not exist in the real space is arranged in the background in the image of the display unit 230 of the information terminal device 200 shown in FIG. 17 (B). That is, the user can customize the background of the subject 400 in the image according to the user's preference by using the information processing system 10 while photographing the subject 500 in the real space. The object 420 may be selected from the objects prepared in advance, or may be a user-generated object. Further, the object 420 may be an object generated from the video data stored in the information terminal device 200.

The user can customize the subject 400 displayed on the display unit 230 of the information terminal device 200 according to the user's preference. Therefore, the user can feel attachment to the character of the subject 500 by embracing the character, and the satisfaction level of the user is improved.

Each of the above-described embodiments of the present invention can be appropriately combined and implemented as long as they do not contradict each other. Further, based on each embodiment, those skilled in the art who appropriately add, delete, or change the design, or add, omit, or change the conditions of the process are also gist of the present invention. As long as it is provided, it is included in the scope of the present invention.

Of course, other actions and effects that are different from the actions and effects brought about by the aspects of each of the above-described embodiments that are clear from the description of the present specification or that can be easily predicted by those skilled in the art are described in the present invention. It is understood that it is brought about by the invention.

10: Information processing system, 100: Server device, 110: Computer, 111: Server processing unit, 120: Communication unit, 130: Storage unit, 200: Information terminal device, 200-1: First information terminal device, 200-2 : 2nd information terminal device, 200-n: nth information terminal device, 210: Computer, 211: 1st processing unit, 212: 2nd processing unit, 213: 3rd processing unit, 214: 4th processing unit, 215 : 5th processing unit, 216: 6th processing unit, 217: 7th processing unit, 218: 8th processing unit, 219: 9th processing unit, 220: imaging unit, 220-1: 1st imaging unit, 220- 2: Second imaging unit, 230: Display unit, 240: Operation unit, 250: Communication unit, 260: Storage unit, 300: Communication network, 301: First object area, 302: Second object area, 303: Third Object area, 304: 4th object area, 400: Subject, 400-1: 1st subject, 400-2: 2nd subject, 401: Parts, 403: Certification mark, 411: Parts data, 411: 1: 1st Parts data, 411-2: 2nd part data, 411-4: 4th part data, 412: Dialogue data, 413: Dialogue data, 414: Shielding data, 415: Interaction data, 416: Parts data, 417: Parts data , 418: Dialogue data, 419: Hair, 420: Object, 450: Object, 460: Finger, 500: Subject, 500-1: First subject, 500-2: Second subject, 510: Object, 520: Certification mark , 530: Finger

Claims (15)

An information processing program executed by a computer of an information terminal device, which is executed by the computer.
Identify the parts that are part of the first subject in the image of the real space,
An information processing program that superimposes part data based on the part on the first object area including the part. The information processing program according to claim 1, wherein the part data is transmitted from the server device to the information terminal device. The information processing program according to claim 1, wherein the part data is generated from the parts. The information processing program according to any one of claims 1 to 3, wherein the part data is adjusted by the first object information based on the first object area. The information processing program according to any one of claims 1 to 4, wherein the computer identifies the part selected by the user. The information processing program according to any one of claims 1 to 4, wherein the first object area is selected by the user. The movement of a part of the user's body in the video is determined, and the movement is determined.
The information processing program according to any one of claims 1 to 4, wherein the part data is further based on the determination. The information processing program according to any one of claims 1 to 7, wherein video data based on the terminal information of the information terminal device is superimposed on the second object area. Let the object in the image be identified
The information processing program according to any one of claims 1 to 7, wherein video data based on the object is superimposed on the second object area. Detecting the certification mark of the first subject,
The information processing program according to any one of claims 1 to 9, wherein the shielding data is superimposed on the third object area including the certification mark. The information processing program according to claim 10, wherein the first subject is identified by the certification mark. The information processing program according to any one of claims 1 to 11, wherein the first subject is a character. Let me identify the second subject
The information processing program according to any one of claims 1 to 12, wherein the interaction data indicating the interaction between the first subject and the second subject is superimposed on the fourth object area. An imaging unit that captures real space,
A display unit that displays an image captured by the imaging unit, and a display unit.
A second processing unit that identifies a part that is a part of the first subject in the image, and
A third processing unit that defines the first object area including the parts, and
An information processing system that superimposes part data based on the parts on the first object area. The information processing system according to claim 14, wherein the first subject is a character.

Images