JP2013073324A - Image display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image display system for clarifying and synthesizing the photographed contour part of a browser with a background image, and for displaying a synthetic image in a real time.SOLUTION: An image processor 2 includes: scenario data storage means for storing scenario data determining the timing of the synthesis of one or more persons on a video with contents; content storage means for storing a background image to be used for synthesis as a content; face detection means 21 for outputting the position and rectangular size of a face detection frame as face detection frame data; tracking means 23 for associating the acquired face detection frame data with the face detection frame data of the other frame; scenario data association means 83 for associating a face object including the face detection frame data with the person included in the scenario data; and synthetic image generation means 84 for masking the background image by using a segment result image segmented into the person and any part other than the person by using a segment area, and for synthesizing the person in a frame with the background image.

Description

  The present invention relates to a technique for processing and displaying a captured video, and more particularly to a technique for displaying a processed video according to the state of a viewer who is shooting.

  Digital signage, which is an advertising medium for displaying advertisements using a display device such as a display or a projector, has begun to be installed in various places. By using digital signage, not only will it be possible to provide rich content using video and audio, but it will also be possible to efficiently deliver advertisements according to the location of digital signage. Expansion is expected.

  Recently, various improvements have been made on digital signage, and a technique for changing an image to be displayed in accordance with the movement of a viewer existing before digital signage has been proposed (see Patent Document 1).

Japanese Patent No. 4238371

  In the technique described in Patent Document 1, a synthesized image is generated based on human recognition information and motion information. However, since tracking processing is not performed, it is possible to synthesize a background image according to each person's motion. Can not. In particular, it is not possible to combine a person and a background without a sense of incongruity. In addition, as a technique for synthesizing a person and a background image, there is a chroma key composition technique. However, in the chroma key composition technique, there is a problem that the background must be clearly distinguished from a person by a different color.

  Therefore, an object of the present invention is to provide an image display system capable of synthesizing a photographed viewer and a background image without a sense of incongruity and displaying them in real time.

  In order to solve the above problems, in the first aspect of the present invention, there is provided a camera for photographing a person, an image processing device for synthesizing a photographed image sent from the camera, and a display for displaying the synthesized image synthesized. The image processing system includes a scenario data storage unit that stores scenario data that determines the timing of combining one or more persons on the video and the content, and a background image used for the combination. Detects a face image recorded in one frame of video sent from the camera and content storage means stored as content, and detects the position / rectangular size of the face detection frame for each detected face image Corresponding face detection frame data acquired from the face detection means output from the face detection means and the face detection frame data of other frames Racking means, scenario data associating means for associating a face object including face detection frame data detected by the face detecting means with a person included in the scenario data, and the scenario in each frame A segment area is set in a range including a face detection frame associated with a person included in the data, and a segment result image segmented into a portion other than a person and a person using the segment area is used. There is provided an image display system comprising: a synthesized image generating unit that masks a background image and generates a synthesized image obtained by synthesizing a person in the frame and a background image.

  According to the first aspect of the present invention, scenario data that determines the timing of synthesis of a person and content in a captured video is prepared, a face detection frame is detected from the frame of the captured video, and tracking of the face detection frame is performed. Assign the obtained face object to the person in the scenario data, set a segment area in the range that includes the face detection frame of the face object, and use this segment area to segment into parts other than the person and person Since the background image is masked using the result image and the person in the frame and the background image are combined, it is possible to combine the photographed viewer and the background image without discomfort and display them in real time. Become. In particular, it becomes possible to synthesize a background image without a sense of incongruity by clarifying the contour of the viewer who is taking a photograph.

  In the second aspect of the present invention, in the image display system according to the first aspect of the present invention, the content storage means further stores a superimposed image, which is an image to be superimposed on the person or background, as the content, and generates the composite image. The means assigns the face object to the person of the scenario data according to the association by the scenario data association means, and sets the position and size of the superimposed image according to the position and size of the face detection frame data of the face object. It is modified to generate a synthesized image obtained by synthesizing the superimposed image on the frame.

  According to the second aspect of the present invention, the position and size of the superimposed image is changed in accordance with the position and size of the face detection frame data of the face object, and the superimposed image is synthesized on the frame. It is possible to display a video in which a superimposed image is superimposed on a person shown in the video in real time. Here, the superimposed image is an image that is overlaid with a person part or a background image in a frame in a captured video, and as a method of superimposing, the person part or There is also a method of overlapping while reflecting the pixel value of the background image.

  According to a third aspect of the present invention, in the image display system according to the first or second aspect of the present invention, the composite image generating means detects the face area based on the position of the face detection frame. A person who is set at a predetermined position in the segment area by setting the frame and the center in the left-right direction to coincide, and setting the center of the face detection frame above the center of the segment area in the up-down direction Segmentation is performed using pixels designated in a region where the character is present and a region where no person is present.

  According to the third aspect of the present invention, based on the position of the face detection frame, the segment detection area is aligned with the center of the face detection frame in the horizontal direction, and the center of the face detection frame is higher than the center of the segment area in the vertical direction. It is set so that it is positioned above, and segmentation is performed using the specified pixels in the area where the person exists and the area where the person does not exist set in the segment area. Based on the characteristics of the pixel value specified for the region and the pixel value specified for the region where no person exists, it is possible to identify the pixel at the border other than the person and the person, and to clarify the outline of the person .

  According to the fourth aspect of the present invention, an image display system includes a camera that shoots a person, an image processing device that synthesizes a captured video sent from the camera, and a display that displays the synthesized video that has been synthesized. The image processing apparatus includes scenario data storage means for storing scenario data that determines the timing of composition of one or more persons on the video and the content, and a part surrounded by the background image and the periphery. Content storage means that stores open superimposed images as content, and a face image captured in one frame of video sent from the camera, and a position of a face detection frame for each detected face image A face detection unit that outputs a rectangular size as face detection frame data; and the face detection frame data acquired from the face detection unit is used as face detection frame data of another frame. An association tracking unit, a scenario data association unit for associating a face object including the face detection frame data detected by the face detection unit with a person included in the scenario data, and according to the association The face object is assigned to the person in the scenario data, and the position and size of the superimposed image are changed and combined in accordance with the position and size of the face detection frame data of the face object, and the face image corresponds to the superimposed image. There is provided an image display system comprising: composite image generation means for generating a composite image combined with the background image masked by a mask image.

  According to the fourth aspect of the present invention, scenario data that determines the timing of synthesis of a person and content in a captured video is prepared, a face detection frame is detected from the frame of the captured video, and tracking of the face detection frame is performed. Assign the obtained face object to the person in the scenario data, and change the position and size of the superimposed image with the part surrounded by the face according to the position and size of the face detection frame data of the face object And combining with the background image masked by the mask image corresponding to the superimposed image, it is possible to combine the photographed viewer and the background image without discomfort and display in real time Become.

  According to the present invention, it is possible to synthesize a photographed viewer and a background image without feeling uncomfortable and display them in real time.

The figure explaining the structure of the image display system 1 in this embodiment. 1 is a hardware block diagram of an image processing device 2 that constitutes an image display system 1. FIG. FIG. 3 is a functional block diagram realized by a computer program installed in the image processing apparatus 2. The flowchart explaining the process which the image processing apparatus 2 analyzes a flame | frame. The flowchart for demonstrating a tracking process. The flowchart for demonstrating face detection frame data matching processing. The figure explaining the state transition table in this embodiment. The figure for demonstrating a human body and a face detection result. The flowchart explaining the process in which the image processing apparatus 2 produces a synthesized image. The figure which shows an example of the scenario data of an XML format in case a target is one person. The figure which shows an example of the scenario data of the XML format in case a target is two persons. The figure which shows the mode of the composition of the content using a face detection frame. The figure which shows the display state of a synthesized image in case a target is one person. The figure which shows the display state of the synthesized image in case there are two targets. The figure which shows an example of the scenario data of the XML format in the case of a touch button correspondence. The figure which shows the display state of the synthesized image in the case of a touch button corresponding | compatible. The figure which shows the mode of the image process in an instruction | indication determination process. The functional block diagram implement | achieved by the computer program mounted in image processing apparatus 2 '. The flowchart for demonstrating a face detection process and a tracking process. The figure which shows an example of the scenario data of the XML format in the case of scene composition. It is a figure which shows the relationship between the face detection frame and segment area | region in a picked-up image. It is a figure which shows the shape of an initial area | region. It is a figure which shows the initial region set according to the example of scenario data of FIG. It is a figure which shows the mode of synthetic mask image preparation. It is a figure which shows the mode of background mask image preparation. It is a figure which shows a mode that a scene synthetic | combination is performed with respect to a picked-up image, and a synthesized image is produced.

<< 1. System configuration >>
DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the drawings. FIG. 1 is a diagram illustrating a configuration of an image display system 1 according to the present embodiment, FIG. 2 is a hardware block diagram of an image processing apparatus 2 that configures the image display system 1, and FIG. It is a functional block diagram implement | achieved by the performed computer program.

  As illustrated in FIG. 1, the image display system 1 includes a display 3 that is a display device such as a liquid crystal display. The display 3 may display not only the photographed image but also the advertisement by dividing the display area.

  In this case, the image display system 1 also functions as a digital signage by installing the display 3 in a street or a store. When the image display system 1 functions as digital signage, a server that controls advertisement video displayed on the display 3 is required.

  An angle is set on the display 3 so that the face of the person who is watching the video reproduced on the display 3 is photographed, and a video for photographing a person who is viewing the advertisement video reproduced on the display 3 A camera 4 is installed.

  The video captured by the video camera 4 is input to the image processing apparatus 2 using a USB port or the like. The image processing apparatus 2 analyzes the frame included in the video transmitted from the video camera 4 and displays the display 3. And the face of the person who browsed the video reproduced on the display 3 are detected, and a log relating to the viewer (for example, the viewing time of the display 3) is stored.

  In the apparatus constituting the image display system 1 illustrated in FIG. 1, commercially available apparatuses can be used for the display 3 and the video camera 4, but the image processing apparatus 2 has features that are not found in the prior art. The image processing apparatus 2 will be described in detail.

  The image processing apparatus 2 can be realized by using a general-purpose computer, and includes hardware similar to that of the general-purpose computer. In the example of FIG. 2, the image processing apparatus 2 includes, as the hardware, a CPU 2a (CPU: Central Processing Unit), a ROM 2b (ROM: Read-Only Memory) on which a BIOS is mounted, and a RAM 2c that is a main memory of the computer. (RAM: Random Access Memory), a large-capacity data storage device 2d (for example, a hard disk) as an external storage device, and an input / output interface 2e for data communication with an external device (video camera 4), for network communication Network interface 2f, a display output interface 2g for sending information to a display device (display 3), a character input device 2h (for example, a keyboard), and a pointing device 2i (for example, a mouse).

  A computer program for operating the CPU 2a is installed in the data storage device 2d of the image processing apparatus 2, and the image processing apparatus 2 is provided with the means shown in FIG. 3 by this computer program. The data storage device 2d is capable of storing various data necessary for the image display system, and stores scenario data that determines the timing of combining one or more persons on the video with the content. It also serves as a scenario data storage means and a content storage means for storing content used for composition.

  Here, the content stored in the content storage means will be described. The content is a material image when a composite image is obtained by combining with the frame of the captured video. There are two types of content: a background image as a background and a superimposed image superimposed on a person in a captured image or a background image. FIG. 12A shows an example of content as a superimposed image. The superimposed image is not particularly limited, and various forms can be used. In the example of FIG. 12A, a case where a wig is prepared as the superimposed image is shown. This superimposed image has rectangular information (position, width, height in the x and y directions), and alignment with the captured frame is possible using this rectangular information.

  As shown in FIG. 3, the input of the image processing device 2 is an image captured by the video camera 4, and the output of the image processing device 2 is a processed image obtained by processing the captured image.

  The image processing apparatus 2 includes a background removing unit 20 that removes a background image of a frame of a video shot by the video camera 4 and a background removing unit 20 as means for analyzing the frame of the video shot by the video camera 4. Face detection means 21 for detecting the face of a person from the frame from which the background has been removed, human body detection means 22 for detecting a human body from the frame from which the background image has been removed by the background removal means 20, and the face detected by the face detection means 21 For each face image newly detected by the face detection means 21, the tracking means 23 for associating the frame with the preceding and following frames, the moving picture analysis means 24 for detecting the designated face image from the frame using a moving picture analysis technique such as a particle filter. Create a face object and associate the previous face detection frame data obtained from the tracking means 23 with the current face detection frame data Referring to the result, the state of the face object is transitioned according to a predetermined state transition table, and the state transition management unit 25 that stores a log corresponding to the state transition of the face object and the face detection unit 21 detect the state transition. A composite image obtained by processing each frame of a video shot by the video camera 4 according to the scenario data, the scenario data associating unit 83 for associating the face object whose state has been changed by the management unit 25 with the prepared scenario data The composite image creating means 84 for creating the scenario data, the instruction determining means 85 for determining whether or not the viewer is instructing the touchless button defined at the predetermined position, and the scenario data according to the determination by the instruction determining means 85 Command switching means 86 for switching the commands in the middle. Personal attribute estimation means 26 for estimating the personal attributes (age and gender) of the person from the face image detected by the face detection means 21 in order to include in the log data the attributes (age and gender) of the person who viewed the play 3; Log file output means 27 for outputting the log stored in the state transition management means 25 in a file format, synthetic target definition means 80 for defining the target (person or place) to be processed in the scenario data, content (image, A synthesized content defining unit 81 for defining voice, CG, etc.) in the scenario data, and an animation scenario defining unit 82 for defining the processing content in the scenario data.

  The scenario data can be created in advance by another system and stored in the data storage device 2d as the scenario data storage means, but the synthesis target definition means 80, the synthesis content definition means 81, and the animation scenario definition means 82. Can also be created. The composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 are used to create scenario data that indicates how to process each frame of a video obtained by shooting. . The format of the scenario data is not particularly limited, but in the present embodiment, XML (eXtensible Markup Language) is adopted. In the present embodiment in which XML is used as the scenario data, the synthesis target definition unit 80, the synthesis content definition unit 81, and the animation scenario definition unit 82 can be realized by a text editor. Therefore, scenario data is created when the text editor is activated and the administrator inputs characters using the character input device.

  FIG. 10 is a diagram illustrating an example of scenario data in the XML format when the number of targets is one. From now on, the synthetic target definition unit 80, the synthetic content definition unit 81, and the animation scenario definition unit 82 will be described in detail with reference to the scenario data of FIG. The synthetic target definition means 80 defines a target by setting four attributes of target ID, type, absolute coordinate, and transition setting. In the example of FIG. 10, it corresponds to a range surrounded by two tags, <Simulation Targets> on the third line and </ Simulation Targets> on the ninth line. Regarding the type, two types of person and place can be set, but in the example of FIG. 10, only the person is set using the <Human> tag. The transition setting can be set only for a person, and sets whether or not to associate with a new person when the associated person disappears. In the example of FIG. 10, the tag of the 5th line sets three attributes of target ID, type, and transition setting, the target ID is “0”, the type is “human”, and transition setting (IsTransfer) Is “false (not set)”.

  The composite content definition unit 81 defines the content by setting three attributes of a content ID, a content path, and an overlap setting. In the example of FIG. 10, it corresponds to a range surrounded by two tags, <Simulation Contents> on the 10th line and </ Simulation Contents> on the 18th line. In the example of FIG. 10, the content ID (ContentsID) is defined for seven contents from “0” to “6”. As shown in FIG. 10, a content path (ContentsPath) and an overlap setting (OverlapOrder) are set in units of one line for each content.

  The animation scenario definition means 82 defines an animation scenario by setting seven attributes of command ID, command type, start key, end key, key type, target ID, and content ID. In the example of FIG. 10, it corresponds to a range surrounded by two tags of <Animation Commands> on the 19th line and </ Animation Commands> on the 34th line. In the example of FIG. 10, the command ID (CommandID) is defined for seven commands from “0” to “6”. As shown in FIG. 10, for each command, a command type, a start key, an end key, a key type, a target ID, and a content ID are set in units of two lines. The command type indicates what type of effect is generated based on what frame, and layer synthesis, α blend synthesis, audio reproduction start, and scene synthesis are prepared. Of these, layer composition, α blend composition, and scene composition indicate the type of image composition, layer composition overwrites content, and α blend composition is performed according to the set α ratio. The content and the frame are combined to be transmitted, and the scene combination is a method in which a human body part is cut out and combined with a background image. In the example of FIG. 10, layer composition (LayerMontage) is set as the command type (CommandType). The start key and end key set the start time and end time of each command. In the present embodiment, the scenario data time is managed as “0.0” at the start of the scenario and “1.0” at the end of the scenario. Therefore, the start key (StartKey) of the command that starts first is “0.0”, and the end key (EndKey) of the command that ends last is “1.0”. The key type is to set a target as a reference for the start key and the end key, and three types of own, base, and global are prepared. own is based on the browsing time of the face object corresponding to each target ID, base is based on the browsing time of the face object corresponding to target ID = 0, and global is based on the time when the first frame of the captured video is acquired. To do. In the example of FIG. 10, since “own” is set as the key type (KeyType), the time point when the face object appears in the frame (the time point when the face object is determined to be “viewing start”) is set to “0.0”. The start key and end key will be recognized.

  The <CycleInterval> tag of the scenario data sets the time from the start to the end of the scenario in seconds. In the example of FIG. 10, “10” is set in the <CycleInterval> tag on the first line. Therefore, it is shown that it is 10 seconds from the start to the end of the scenario. The scenario is managed in real time that is 10 times the value of the start key and end key. The <IsAutoLoop> tag of the scenario data sets whether to perform loop processing (repeated processing). In the example of FIG. 10, “true” is set in the <IsAutoLoop> tag on the second line. , Loop processing is performed. The <CycleInterval> tag and <IsAutoLoop> tag can also be set using a text editor. In this way, the scenario data created by the composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 is stored in the data storage device 2d as the scenario data storage means.

  When the image processing device 2 analyzes the frames of the video captured by the video camera 4 in time series, the data storage device 2d of the image processing device 2 can view the display as a log file that can be used for browsing measurement. A browsing time log file in which time is stored, a position log file in which the position of a person who has viewed the display is stored, and a person attribute log file in which the personal attributes (for example, age and gender) of the person who has viewed the display are stored Log file output that stores the total number of people in front of the display, the number of people who are not browsing the display, the number of people who have viewed the display, and outputs these log files Means 27 is provided in the image processing apparatus 2. In the present invention, it is not essential to create a log file, but the face object and the browsing start time in the process of creating the log file are used to generate a composite image.

  First, each means provided for analyzing and processing a frame of a video shot by the video camera 4 will be described while explaining a process in which the image processing apparatus 2 analyzes a frame of a video transmitted from the video camera 4. To do.

≪2. Processing action >>
FIG. 4 is a flowchart for explaining processing in which the image processing apparatus 2 analyzes a frame of a video transmitted from the video camera 4. Although details of each processing will be described later, when one frame of video is input to the image processing device 2, the image processing device 2 performs background removal processing S1 on the frame, and about the frame after the background removal processing S1. Then, face detection processing S2 and human body detection processing S3 are performed.

  The image processing apparatus 2 performs the face detection process S2 and the human body detection process S3 on the frame after the background removal process S1, and then uses the result of the face detection process S3 as a frame to be processed this time. A state transition in which a tracking process S4 is performed for associating the face detected from the N frame with the face detected from the previous frame N-1 frame, and the state of the face object is changed based on the result of the tracking process S4. The management process S5 is executed.

  First, the background removal process S1 will be described. The means responsible for the background removal processing S1 is the background removal means 20 of the image processing apparatus 2. The image processing apparatus 2 executes the background removal process S1 because the position and angle of the video camera 4 provided at the upper part of the display 3 is fixed as shown in FIG. This is because the photographed video includes a background that does not change, and by removing this background, the human body / face can be detected with high accuracy.

  As the background removal process executed by the background removal unit 20 of the image processing apparatus 2, existing technology can be used, and the video captured by the video camera 4 may change in the morning, noon, and night, for example. It is preferable to use a dynamic background update method that takes into account the temporal change of.

  Dynamic background update methods that take into account temporal changes in the background include, for example, “Shinji Morita, Kazumasa Yamazawa, Nobuhiko Terasawa, Naokazu Yokoya:“ Network-enabled remote monitoring system using omnidirectional image sensors ”, electronic The method described in the Journal of Information and Communication Engineers (D-II), Vol. J88-D-II, No. 5, pp. 864-875, (2005.5) can be used.

  Next, the face detection process S2 executed by the face detection unit 21 of the image processing apparatus 2 will be described. Various face detection methods including the face detection method described in Patent Document 1 have been disclosed as face detection methods performed in the face detection process S2, but in this embodiment, as weak classifiers, The face detection method by Adaboost algorithm using black and white Haar-Like feature is adopted. For the face detection method by Adaboost algorithm using black and white Haar-Like features as weak classifiers, see “Paul Viola and Michael J. Jones,“ Rapid Object Detection using a Boosted Cascade of Simple Features ”, IEEE CVPR, 2001. ", Rainer Lienhart and Jochen Maydt," An Extended Set of Haar-like Features for Rapid Object Detection ", IEEE ICIP 2002, Vol. 1, pp. 900-903, Sep. 2002.

  Face detection frame data is obtained for each face image included in the frame by executing the face detection method using the Adaboost algorithm using the black and white Haar-Like feature as a weak classifier. The position of the face detection frame used when detecting the image (for example, the coordinates of the upper left corner) and the rectangular size (width and height) are included.

  Next, the human body detection process S3 executed by the human body detection unit 22 of the image processing apparatus 2 will be described. As a method of detecting a human body, a method of detecting a human body using an infrared sensor and utilizing a human body temperature is well known. However, in this embodiment, the human body detection method performed in the face detection process S2 is weak. A human body detection method based on the Adaboost algorithm using HOG (Histogram of Oriented Gradients) features is adopted as a discriminator. For the human body detection method using the Adaboost algorithm using the HOG (Histogram of Oriented Gradients) feature as a weak classifier, see “N. Dalal and B. Triggs,” Histograms of Oriented Gradientstional Conference on Computer Vision, pp. 734-741. , 2003. "

  By executing the human body detection method using the Adaboost algorithm using the HOG feature as a weak classifier, human body detection frame data is obtained for each human body included in the frame, and when this human body detection frame data is detected, The position (for example, the coordinates of the upper left corner) and the rectangular size (width and height) of the human body detection frame used in the above are obtained.

  FIG. 8 is a diagram for explaining the human body and face detection results. The person photographed in the frame 7 of FIG. 8 includes a total of six persons 7a to 7f, and the human body detection means 22 of the image processing apparatus 2 detects each person 7a to 7f, and each person 7a to 7f is detected. Human body detection frame data 70a to 70f corresponding to 7f are output. Further, the face detection means 21 of the image processing apparatus 2 detects the faces of the persons 7a to 7c in which both eyes are photographed, and outputs face detection frame data 71a to 71c corresponding to each face.

  Next, the tracking process S4 executed by the tracking unit 23 of the image processing apparatus 2 will be described. In the tracking process S4, a process for associating the face detection frame data detected by the face detection unit 21 from the N-1 frame with the face detection frame data detected by the face detection unit 21 from the N frame by the tracking unit 23 of the image processing apparatus 2. Is executed.

  From here, the tracking process S4 performed by the tracking means 23 of the image processing apparatus 2 will be described in detail. FIG. 5 is a flowchart for explaining the tracking process S4 executed by the tracking unit 23 of the image processing apparatus 2.

  The tracking unit 23 of the image processing apparatus 2 first acquires the face detection frame data and the human body detection frame data obtained from the N frame from the face detection unit 21 and the human body detection unit 22, respectively, in order to perform the tracking process S4 for the N frame. (S10).

  In the next tracking process S4, the face detection frame data obtained from the N frame is used as the face detection frame data of the N-1 frame. Therefore, the tracking unit 23 of the image processing apparatus 2 obtains the frame from the N frame. The face detection frame data thus obtained is stored in the RAM 2c or the data storage device 2d.

  When the tracking unit 23 of the image processing apparatus 2 acquires the face detection frame data and the human body detection frame data of N frames, the tracking means 23 performs display browsing determination for each human body detection frame data of N frames (S11).

  As described above, since the human body detection frame data includes the position and the rectangular size of the human body detection frame, and the face detection frame data includes the position and the rectangular size of the face detection frame, the face detection frame is included. The human body detection frame data can be determined as the human body detection frame data of the person who is browsing the display 3, and the human body detection frame data which does not include the face detection frame is the human body detection frame data of the person who is not browsing the display 3. Can be determined as data.

  When the tracking means 23 of the image processing apparatus 2 makes a display browsing determination for each human body detection frame data of N frames in this way, it is displayed in front of the display 3 as a number of people log file when the N frames are captured. The total number of persons who are present, that is, the number of human body detection frame data detected by the human body detection means 22, and the number of persons who are not browsing the display 3, that is, human body detection frame data not including a face detection frame. Number of people browsing the display 3, that is, the number of human body detection frame data including the face detection frame is generated, and a data storage is performed by assigning N frame number and the like Store in device 2d.

  When the tracking unit 23 of the image processing apparatus 2 performs display browsing determination for each human body detection frame data of N frames, the face detection frame data detected by the face detection unit 21 from the N−1 frame and the face detection unit 21. Executes face detection frame data association processing S12 for associating the face detection frame data detected from the N frames.

  FIG. 6 is a flowchart for explaining the face detection frame data association processing S12. In this embodiment, in the flow illustrated in FIG. 6, the evaluation value obtained by using the evaluation function of Equation 1 described below. Is used to associate the face detection frame data.

Note that the evaluation function f1 () in Equation 1 is an evaluation function using the nearest neighbor method, and the evaluation value obtained by the evaluation function f1 () is an evaluation indicating the difference between the position / rectangular size of the face detection frame data. Value. Further, the evaluation value obtained by the evaluation function f2 () of Expression 1 is obtained from the face image included in the face detection frame specified by the face detection frame data to the evaluation value obtained from the evaluation function f1 (). The difference between the SURF feature amounts indicating the features of the image becomes an evaluation value added by weighting.

  In order to associate the face detection frame data detected from the N-1 frame with the face detection frame data detected from the N frame, the tracking unit 23 of the image processing apparatus 2 first counts the number of face detection frame data obtained from the N frame. Only the loop processing L1 is executed.

  In this loop process L1, the tracking means 23 of the image processing apparatus 2 first executes the loop process L2 by the number of face detection frame data detected from the N-1 frame, and in this loop process L2, the loop process L1 The evaluation value is calculated by substituting the position / rectangular size of the face detection frame data to be processed and the position / rectangular size of the face detection frame data to be processed by the loop processing L2 into the evaluation function f1 () of Equation 1. In step S120, an evaluation value indicating a position / size difference from the face detection frame data to be subjected to the loop processing L1 is calculated for each face detection frame data detected from the N-1 frame.

  The tracking unit 23 of the image processing apparatus 2 calculates an evaluation value indicating a position / size difference from the face detection frame data to be processed by the loop processing L1 for each face detection frame data detected from the N-1 frame. Then, the minimum value of the evaluation value is searched (S121), and after calculating the difference between the minimum value of the evaluation value and another evaluation value (S122), it is determined whether there is the difference value equal to or less than the threshold value (S123). ).

  Then, the tracking unit 23 of the image processing apparatus 2 includes the difference between the minimum value of the evaluation value indicating the position / size difference from the face detection frame data to be processed by the loop processing L1 and the other evaluation values. When there is a difference equal to or less than the threshold, the tracking unit 23 of the image processing apparatus 2 executes the loop process L3 for the number of face detection frame data whose evaluation value is within the threshold.

  In this loop process L3, the face image within the face detection frame specified by the face detection frame data to be processed by the loop process L1 and the N-1 frame face detection frame data to be processed by the loop process L3 are specified. The difference of the SURF feature quantity with the face image in the face detection frame to be obtained is obtained, the difference of the SURF feature quantity is substituted into the evaluation function f2 () of Formula 1, and the evaluation value obtained by adding the difference of the SURF feature quantity is It is calculated for each face detection frame data detected from the N-1 frame (S124).

  The evaluation value obtained using the evaluation function f1 () using only the nearest neighbor method is used to calculate the evaluation value obtained by adding the difference of the SURF feature values using the evaluation function f2 () expressed by the mathematical formula 1. If the difference value between the minimum value and other evaluation values is equal to or smaller than the threshold value, it is considered that the face detection frames having similar sizes are close to each other (for example, the person 7a, b in FIG. 8), and the nearest neighbor method This is because the N-1 frame face detection frame data associated with the N frame face detection frame data cannot be determined from the above evaluation values.

  By using the evaluation function f2 () expressed by Equation 1 and calculating an evaluation value obtained by adding the difference between the SURF feature amounts, an evaluation value in which the facial features are added is calculated. When face detection frames having similar sizes are close to each other, the N-1 frame face detection frame data having a similar face is associated with the N frame face detection frame data.

  Then, the tracking unit 23 of the image processing apparatus 2 uses the N-1 frame face detection frame data having the minimum evaluation value obtained from the evaluation function of Formula 1 as the N frame face to be subjected to the loop processing L1. A process of associating with the detection frame data is executed (S125). If the evaluation value using the evaluation function f2 () shown in Equation 1 is not calculated, the evaluation value used in this process is the value obtained from the evaluation function f1 () shown in Equation 1. Thus, when the evaluation value using the evaluation function f2 () shown in Equation 1 is calculated, the evaluation value used in this process is the value obtained from the evaluation function f2 () shown in Equation 1. Become.

  When the loop processing L1 ends and the tracking means 23 of the image processing apparatus 2 associates the N frame face detection frame data with the N-1 frame face detection frame data, the N-1 frame face detection frame data overlaps. Then, it is confirmed whether it is associated with the face detection frame data of N frames (S126).

  When the face detection frame data of the N-1 frame overlaps and is associated with the face detection frame data of the N frame, the tracking unit 23 of the image processing apparatus 2 overlaps the N-1 frame. By referring to the evaluation value of the face detection frame data of the frame and recursively executing the process of associating the smaller evaluation value with the face detection frame data of the N frame, finally the face detection frame data of the N frame N-1 frame face detection frame data to be associated with is determined (S127).

  From here, it returns to description of the flow illustrated in FIG. When the tracking process S4 is completed, the state transition management unit 25 of the image processing apparatus 2 obtains the state transition obtained in advance from the tracking process S4 and refers to the result of association between the previous and current face detection frame data, and is determined in advance. The state transition management process S5 is executed to change the state of the face object according to the table and store a log corresponding to the state transition of the face object. If there is a predetermined state transition in the state transition management process S5, A corresponding predetermined log file is stored in the data storage device 2d.

  In the state transition management unit 25 of the image processing apparatus 2, in order to manage the state transition of the face object, a state transition table in which a state of the face object and a rule for state transition are defined in advance is defined. The tracking unit 23 of the image processing device 2 refers to this state transition table and changes the state of the face object based on the result of the face detection frame data association processing S12.

  From here, an example of the state transition table is illustrated, and the state transition management process S5 executed by the state transition management unit 25 of the image processing apparatus 2 will be described while explaining the state transition table.

  FIG. 7 is a diagram illustrating the state transition table 6 in the present embodiment. The state transition table 6 illustrated in FIG. 7 defines the state of the face object and the transition from the state of the N-1 frame to the state of the N frame. The vertical axis of the state transition table 6 indicates the state of the N-1 frame. The horizontal axis indicates the state of N frames, and the condition for state transition is described at a location where the vertical axis and the horizontal axis intersect. In the state transition table, “-” indicates an illegal state transition.

  In the state transition table 6 illustrated in FIG. 7, None, candidate face, current face, standby face, noise face, and end face are defined as face object states, and the state transition defined in the state transition table is described. However, each state will be described.

  None, which is one of the face object states, means a state in which no face object exists, and there is no N-1 frame face detection frame data associated with N frame face detection frame data (condition 1 in FIG. 7). ), The state transition management unit 25 of the image processing apparatus 2 selects a face object having an attribute value such as an ID for identifying the face object, face detection data of the N frame, data related to the state assigned to the face object, and the like. A new face is generated and the state of the face object is set as a candidate Face.

  The candidate face that is one of the face object states means a state in which the newly detected face image may be noise, and the candidate face is provided as one of the face object states. This is because in the case of a complex background, erroneous detection of a face image is likely to occur even if background removal processing is performed, and the newly detected face image may be noise.

  The face object in the candidate face state is given, as data related to the candidate face state, a state ID indicating the candidate face state, the date and time when the state transition is made to the candidate face, and a counter.

  The states that can be changed from the candidate face are the candidate face, the current face, and the noise face, and a predetermined number of face detection frames corresponding to the face objects that are in the candidate face state are continuous within a predetermined setting time. If the tracking is successful (condition 2-2 in FIG. 7), the state of the face object changes from the candidate face to the current face.

  The reason why a counter is provided for the attribute of the face object in the candidate face state is to count the number of times that the face detection frame corresponding to the face object in the candidate face state can be tracked continuously within the set time. Then, the state transition management unit 25 of the image processing apparatus 2 determines that when the state of the face object including the N-1 frame face detection data associated with the N frame face detection frame data is the candidate Face, The face detection frame data attached to the face object is updated to N frame face detection frame data, and the counter of the face object is incremented.

  Then, when executing the state transition management process S5, the state transition management unit 25 of the image processing apparatus 2 refers to the date and time when the state transition is made to the candidate face for each face object that is the candidate face, and within the set time When the value of the counter has reached a predetermined setting value, the state of the face object is changed to the current Face. Further, the state transition management unit 25 of the image processing apparatus 2 causes the state of the face object that has not reached the set value at the time when the set time has elapsed to change to the noise face (FIG. 7). Condition 2-3), the face object for which the set time has not elapsed does not change state (condition 2-1 in FIG. 7).

  The noise face that is one of the states of the face object is a state in which the face image detected by the face detection unit 21 of the image processing apparatus 2 is determined to be noise, and the face object that has transitioned to the noise face is considered to have disappeared. It is made and is not used for the subsequent state transition management process S5.

  The current face, which is one of the face object states, is a state in which a person corresponding to the face object can determine that the display 3 is in the browsing state. It is time to browse.

  When the state transition management unit 25 of the image processing apparatus 2 changes the state of the face object from the candidate Face to the current Face, the face detection frame data of the face object is updated to N frame face detection frame data and the Face As the data related to the above, a face ID indicating the current face state and the date and time when the state is changed to the current face are assigned to the face object.

  In addition, when the state of the face object is changed to “Face” in order to store the person attributes (for example, age and gender) of the person browsing the display as a log, the state transition management unit 25 of the image processing apparatus 2 The person attribute estimating means 26 is operated to acquire a person attribute obtained from the face detection frame specified by the face detection frame data of the face object whose state is currently changed to Face, and the object ID and person attribute of the face object are described. The attribute log file thus stored is stored in the data storage device 2d.

  The person attribute estimation means 26 provided in the image processing apparatus 2 will not be described in detail, but automatic identification of a person's attribute (age / gender) from a person's face image is an automatic cigarette sale. For example, a technique disclosed in Japanese Patent Application Laid-Open No. 2007-080057 can be used.

  Further, the state transition management unit 25 of the image processing apparatus 2 stores a position log file for storing the position of the person who is browsing the display 3 in time series when the state of the face object is currently changed to Face. Newly generated in the device 2d. The position log file at the time of generation is given the object ID of the face object whose state has been changed to Face and the face detection frame data included in the face object whose state has been changed to Face.

  The states that can be changed from the current face state are the current face and the standby face. The state transition management unit 25 of the image processing apparatus 2 is configured so that the state of the face object including the N-1 frame face detection data associated with the N frame face detection frame data is currently Face (condition 3-1). The face detection frame data attached to the face object is updated to face detection frame data in N frames, and the face detection frame data is added to the position log file specified by the object ID of the face object.

  Further, when the state transition management unit 25 of the image processing apparatus 2 performs the state transition management process S5, N-1 frame face detection frame data that is not associated with the N frame face detection frame data is assigned. When the state of the face object is currently “Face”, the moving image analysis unit 24 is operated to perform processing for detecting a face image corresponding to the face detection frame data of the N−1 frame from the N frame by the moving image analysis method.

  In the present embodiment, the moving image analysis unit 24 of the image processing apparatus 2 first has N frames that are already associated with N-1 frame face detection frame data that has not been associated with N frame face detection frame data. The occlusion state is determined between the face detection frame data, and it is confirmed whether the target person's face is completely occluded.

The moving image analysis means 24 of the image processing apparatus 2 executes processing for determining the occlusion state of the face object according to Equation 2 for all face objects that exist at this time and are currently in the face, candidate face, and standby face states. To do.

  When the moving image analysis unit 24 of the image processing apparatus 2 executes the process of determining the occlusion state of the face object according to Equation 2, the process branches based on the determination result.

  When it is determined that there is a high possibility that the person to be tracked is an occlusion in a completely hidden state (when the criterion 1 of Expression 2 is satisfied), tracking by the particle filter is performed, and the position of the target face object・ Detect the rectangle size. The particle filter is described in many literatures such as “Takekazu Kato:“ Particle filter and its implementation ”, IPSJ Research Report, CVIM-157, pp.161-168 (2007).” .

  Also, if it is judged that the possibility is high due to occlusion in which the person being tracked is half-hidden (corresponding to criterion 2 in Formula 2), tracking is performed using the LK method (Lucus-Kanade algorithm) The position / rectangular size of the face object is detected. The LK method is described in “Lucas, BD and Kanade, T .:“ An Iterative Image Registration Technique with an Application to Stereo Vision ”, Proc. DARPA Image Understanding Workshop, pp. 121-130, 1981.” ing.

  When it is determined that there is a high possibility that the person to be tracked does not have occlusion (when the criterion 3 in Expression 3 is satisfied), the moving image analysis unit 24 of the image processing apparatus 2 performs tracking using the CamShift method. To detect the position / rectangular size of the target face object. The CamShift method is described in “G. R. Bradski:“ Computer vision face tracking for use in a perceptual user interface, ”Intel Technology Journal, Q2, 1998.”.

  When the target face image can be detected from N frames by any one of these methods, the state transition management unit 25 of the image processing apparatus 2 uses the face detection data of the face object that is currently in the face state as these methods. If the target face image cannot be tracked by any of these methods, the state of the face object that is currently in the face state is changed to the standby face (see FIG. 7 condition 3-2).

  A standby face, which is one of the states of a face object, is a state in which a face image corresponding to the face object cannot be detected even using the moving image analysis means 24 provided in the image processing apparatus 2.

  Further, when the state transition management unit 25 of the image processing apparatus 2 changes the state of the face object to the standby face, the face detection frame data of the face object is not updated, and the state of the standby face is used as data related to the standby face. Is given to the face object, the date and time when the face object has made a transition to the current Face, and the date and time when the face object has made a transition to the standby Face.

  The state in which state transition is possible from the standby face is the current face or the end face. The state transition management unit 25 of the image processing apparatus 2 searches for a face object including face detection frame data of N frames before a predetermined time elapses after the state transition to the standby face, and the state of the face object Is a standby face, the state of the face object is changed from the standby face to the current face (condition 4-1 in FIG. 7).

  When the state of the face object is changed from the standby face to the current face, the state transition management unit 25 of the image processing apparatus 2 indicates that the date and time when the face object has changed to the current face is the state of the standby face. Sometimes the date and time assigned to the face object is used.

  Further, when executing the process for managing the state transition of the face object, the tracking unit 23 of the image processing apparatus 2 changes the state of the face object that has passed a predetermined time from the state transition to the standby face to the end face. The state is changed (condition 4-3 in FIG. 7), and the state of the face object for which the set time has not elapsed is not changed (condition 4-2 in FIG. 7).

  The end face, which is one of the face object states, is a state corresponding to a person who can no longer be detected by the image processing apparatus 2, and the face object whose state is the end face is considered to have disappeared. It is not used in the transition management process S5.

  Note that the state transition management unit 25 of the image processing apparatus 2 indicates the object ID of the face object and the date and time when the face object is currently transitioned to Face before the face object is transitioned to end Face. A browsing time log file describing the browsing start time and the browsing end time, which is the date and time when the face object changes to the standby face, is generated and stored in the data storage device 2d.

  As described above in detail, the image processing apparatus 2 has five states of the face object generated for each face detected by the face detecting unit 21: None, candidate Face, current Face, standby Face, noise Face, and end Face. Can be stored as a log according to the state transition of the face object by changing the state of the face object according to the tracking result of the face corresponding to the face object. become.

  According to the above-described contents, while the face object state is currently Face, the face corresponding to the face object can be continuously detected. It becomes browsing time.

  Further, by defining the candidate Face as the state of the face object, even when a face is erroneously detected due to noise, the influence on the browsing time of the display 3 is eliminated. Also, by defining the standby face as the state of the face object, even if the same face is detected after losing sight of the face, it can be handled as the same face.

≪3. Synthesis processing using scenario data >>
<< 3.1. When there is one target >>
FIG. 9 is a flowchart illustrating a process in which the image processing apparatus 2 creates a composite image based on a video frame transmitted from the video camera 4. When the image processing device 2 is activated and scenario data to be used is designated, first, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Next, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one. In the example of FIG. 10, since only one target ID “0” is set in the scenario data, the scenario data association unit 83 is identified by the object ID associated with the target ID “0”. The face detection frame data is targeted.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and the <Animation Commands> of the scenario data is referred to at this time “0.0”. Then, as shown in FIG. 10, from the start key “0.0” to the end key “0.1”, the key type “own”, the command type “LayerMontage (layer composition)”, the target ID “0”, the content Since the ID is “0”, the composite image creating unit 84 creates a composite image by layer-combining the content ID “0” with the face detection frame data with the target ID “0”.

  Since the content path of content ID “0” (material for composition) can be identified by referring to <Simulation Contents> of the scenario data, it can be determined from the storage location in the data storage device 2d identified by the path. Get content. As described above, since the rectangular information is set in the content, the composite image creating unit 84 changes the rectangular information and the size of the content so that the rectangular information and the rectangular size of the face detection frame data match, Layer composition of the changed content is performed at a position where the position of the rectangle information after the change and the face detection frame data coincide. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, the <CycleInterval> tag in the scenario data is referenced with the elapsed time from the start of image creation according to the scenario data. If the elapsed time is less than the scenario time, it is determined that the scenario is in progress and the elapsed time If the time is longer than the scenario time, it is determined that the scenario is over. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. When the acquired elapsed time is smaller than “0.1”, as shown in the scenario data of FIG. 10, between the start key “0.0” and the end key “0.1”, as in the case described above. Therefore, since the key type is “own”, the command type is “LayerMontage (layer composition)”, the target ID is “0”, and the content ID is “0”, the composite image creating unit 84 performs the face detection frame with the target ID “0”. A composite image is created by layer combining the content ID “0” with the data. When the acquired elapsed time exceeds “0.1”, the key type “own”, the command type is used from the start key “0.1” to the end key “0.3” as shown in FIG. Since “LayerMontage”, target ID “0”, and content ID “1”, the composite image creation unit 84 combines the content ID “1” with the face detection frame data of the target ID “0”. By doing so, a composite image is created. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

  FIG. 13 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. First, the composite image balloon "Hello" has been synthesized is displayed on the frame (a). The frame at the scenario begins (StartKey = "0.0") until the elapsed time of 1 second (EndKey = "0.1"), all balloon "Hello" (ContentsID = "0") is a synthetic image synthesized is displayed It will be. At this time, by referring to <SimulationContents> in ContentsID = "0", and specifying a location "fukidashi001.jpg" in the content storage unit of the content, it acquires the content of the balloon "Hello".

  Next, when the elapsed time is 1 second (StartKey = "0. 1"), a synthesized image in which the balloon "isn't your hair nice today?" (ContentsID = "1") is displayed (I). For frames from an elapsed time of 1 second (StartKey = "0. 1") to an elapsed time of 3 seconds (EndKey = "0.3."), A composite image in which all the balloons are nice today's hair is synthesized? Will be displayed. At this time, by referring to <SimulationContents> with ContentsID = "1", the location "fukidashi002.jpg" in the content storage means of the content is specified, and the content of the balloon "isn't your hair today nice?"

  When the elapsed time is 3 seconds (StartKey = "0.3"), a composite image in which the balloon "Is it a little more casual" (ContentsID = "2") is combined is displayed (c). At this time, by referring to <SimulationContents> with ContentsID = "2", the location "fukidashi003.jpg" in the content storage unit of the content is specified, and the content of the balloon "Is it a little more casual" is acquired.

  For the frames from the elapsed time of 3 seconds (StartKey = "0.3") to the elapsed time of 6 seconds (EndKey = "0.6"), a composite image in which all the balloons "I want to make it a little more casual" is displayed. Become. When the elapsed time is 6 seconds (StartKey = "0.6"), a composite image in which the black wig (Contents ID = "5") is combined is displayed (D). At this time, by referring to <SimulationContents> with ContentsID = "5", the location "afro.jpg" in the content storage unit of the content is specified, and the content of the black wig is acquired. For frames from an elapsed time of 6 seconds (StartKey = "0.6") to an elapsed time of 7 seconds (EndKey = "0.7"), a composite image in which all black wigs are combined is displayed.

  When the elapsed time is 7 seconds (StartKey = "0.7"), a composite image in which the balloon "This is good" (ContentsID = "3") and the blond wig (ContentsID = "6") is displayed. (E). At this time, by referring to <SimulationContents> with ContentsID = "6", the location "blond.jpg" in the content storage means of the content is specified, and the content of the blond wig is acquired. For frames from an elapsed time of 7 seconds (StartKey = "0.7") to an elapsed time of 9 seconds (EndKey = "0.9"), all the balloons are “this is good” and a composite image of a blonde wig is displayed. Will be. When the elapsed time is 9 seconds (StartKey = "0.9"), a balloon "Nedou will" (ContentsID = "4") and a composite image in which a blonde wig is synthesized are displayed (F). At this time, by referring to <SimulationContents> with ContentsID = "4", the location "fukidashi005.jpg" in the content storage means of the content is specified, and the content of the balloon "Nendodo" is acquired. For frames from an elapsed time of 9 seconds (StartKey = "0.9") to an elapsed time of 10 seconds (EndKey = "1.0"), a composite image in which all black wigs are combined is displayed.

<< 3.2. When there are two targets >>
Next, a case where there are two targets will be described. FIG. 11 is a diagram illustrating an example of scenario data in XML format when there are two targets. A significant difference from the example of FIG. 10 is that two target IDs are set in a <Human> tag sandwiched between <Simulation Targets> tags. In the example of FIG. 11, target IDs “0” and “1” are set. In the <Animation Commands> tag, one target ID is set for each command ID. In the example of FIG. 11, the target ID “0” is set for the command IDs “0”, “2”, “3”, and “5”, and the target ID “1” is set for the command IDs “1”, “4”, and “6”. "Is set.

  When the number of targets is two, the process is executed according to the flowchart of FIG. 9 as in the case of one target. If there are two targets, the scenario data association unit 83 acquires a plurality of pieces of face object data generated by the state transition management unit 25 in S23. Therefore, in S24, the acquired face object data is converted into scenario data. Associate with. In the example of FIG. 11, two target IDs “0” and “1” are set in the scenario data. Therefore, the scenario data associating unit 83 associates the object IDs “0” and “1” with each other. The face detection frame data specified by the ID is targeted. As a result, even if the face detection frame of the target face object is moved so as to be replaced by a frame, the composite image creating unit 84 can synthesize content in accordance with each face detection frame.

  FIG. 14 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. First, is displayed combined image on the left side of the frame balloon "Hello" is synthesized, at the elapsed time of 1 second, the composite image in the right frame balloon "Hello" is synthesized is displayed (A). When the elapsed time reaches 3 seconds, a composite image in which a blond wig and a black wig are combined is displayed (not shown). When the elapsed time reaches 4 seconds, a composite image in which the balloon “your hair today”, the blonde wig and the black wig are combined is displayed (I). When the elapsed time reaches 6 seconds, a composite image in which a blonde wig and a black wig are combined is displayed (not shown). When the elapsed time is 7 seconds, a composite image in which the balloon “your hair today”, the blonde wig, and the black wig are combined is displayed (c). When the elapsed time is 9 seconds, a composite image in which a balloon wig and a black wig are combined is displayed ("D").

<< 3.3. Example of interactive display >>
Next, a case where interactive display is performed using a virtual touchless button will be described. In order to perform interactive display, scenario data corresponding to the interactive display is required. FIG. 15 is a diagram illustrating an example of scenario data in the XML format in the case of corresponding to a touchless button (one target). Here, the functions of the synthesis target definition unit 80, the synthesis content definition unit 81, and the animation scenario definition unit 82 will be described with reference to the scenario data of FIG. The synthetic target definition means 80 defines a target by setting four attributes of target ID, type, absolute coordinate, and transition setting. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Simulation Targets> on the first line and </ Simulation Targets> on the seventh line. Regarding the type, two types of person and place can be set, but in the example of FIG. 15, only the person is set using the <Human> tag. The transition setting can be set only for a person, and sets whether or not to associate with a new person when the associated person disappears. In the example of FIG. 15, the tag in the third row sets three attributes of target ID, type, and transition setting, the target ID is “0”, the type is “human”, and transition setting (IsTransfer) Is “false (not set)”.

  The composite content definition unit 81 defines the content by setting three attributes of a content ID, a content path, and an overlap setting. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Simulation Contents> on the eighth line and </ Simulation Contents> on the eleventh line. In the example of FIG. 15, two contents whose content ID (ContentsID) is “0” and “1” are defined. As shown in FIG. 15, a content path (ContentsPath) and an overlap setting (OverlapOrder) are set for each content in units of one line.

  The animation scenario definition means 82 defines an animation scenario by setting eight attributes of command ID, command type, selector ID, start key, end key, key type, target ID, and content ID. In the example of FIG. 15, it corresponds to a range surrounded by two tags, <Animation Commands> on the 12th line and </ Animation Commands> on the 17th line. In the example of FIG. 15, two commands with command IDs (CommandID) “0” and “1” are defined. As shown in FIG. 15, for each command, a command type, a selector ID (SelectorID), a start key, an end key, a key type, a target ID, and a content ID are set in units of two lines. The command type indicates what type of effect is generated based on what frame, and layer synthesis, α blend synthesis, audio reproduction start, and scene synthesis are prepared. Of these, layer composition, α blend composition, and scene composition indicate the type of image composition, layer composition overwrites content, and α blend composition is performed according to the set α ratio. The content and the frame are transmitted and combined, and the scene combination is a method in which a human body part is cut out and combined on the background. In the example of FIG. 15, layer composition (LayerMontage) is set as the command type (CommandType).

  The selector ID is a condition for specifying a command to be executed, and a command with a command ID corresponding to the selector ID is executed. A set of one or more commands specified by the selector constitutes one scenario. Therefore, there is a one-to-one correspondence between selectors and scenarios. In the example of FIG. 15, one scenario is composed of one command. When the selector ID = 0, the command with the command ID = 0 is executed, and when the selector ID = 1, the command ID = 1. Indicates that the command will be executed. The start key and end key are used to set the start time and end time of each command, as in the example of FIG. In the present embodiment, the scenario data time is managed as “0.0” at the start of the scenario and “1.0” at the end of the scenario. In the example of FIG. 15, unlike the example of FIG. 10, the start key (StartKey) of all commands is “0.0”, and the end key (EndKey) of the last command is “1.0”. The key type is to set a target as a reference for the start key and the end key, and three types of own, base, and global are prepared. own is based on the browsing time of the face object corresponding to each target ID, base is based on the browsing time of the face object corresponding to target ID = 0, and global is based on the time when the first frame of the captured video is acquired. To do. In the example of FIG. 15, since “own” is set as the key type (KeyType), the time point when the face object appears in the frame (the time point when the face object is determined to be “viewing start”) is set to “0.0”. The start key and end key will be recognized.

  In the example of FIG. 15, the <CycleInterval> tag and the <IsAutoLoop> tag shown in FIG. 10 are surrounded by two tags, <Human> on the second line and </ Human> on the fourth line. Are included in the target ID tag. Even when the <CycleInterval> tag and <IsAutoLoop> tag are included in the target ID tag, the <CycleInterval> tag of the scenario data is the time from the start to the end of the scenario in seconds, as in the example of FIG. The <IsAutoLoop> tag of the scenario data sets whether to perform loop processing (repetition processing). The image display system 1 allows a plurality of description formats for scenario data, and even when the description positions of the <CycleInterval> tag and <IsAutoLoop> tag are different as shown in FIGS. 10 and 15, The scenario data association unit 83 recognizes the <CycleInterval> tag and the <IsAutoLoop> tag, and executes processing according to the description content. In the example of FIG. 15, since CycleInterval = “10” is set in the target ID tag, it indicates that it is 10 seconds from the start to the end of the scenario. The scenario is managed in real time that is 10 times the value of the start key and end key. In the example of FIG. 15, since IsAutoLoop = “true” is set in the target ID tag, this indicates that loop processing is performed. In this way, the scenario data created by the composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 is stored in the data storage device 2d as the scenario data storage means.

  Furthermore, the animation scenario definition means 82 defines an animation selector by setting four attributes: selector ID, select type, select number, and trigger ID. In the example of FIG. 15, it corresponds to a range surrounded by two tags of <AnimationSelectors SelectorsNum = “2” InitialSerectorID = “0”> on the 18th line and </ AnimationSelectors> on the 21st line. In the example of FIG. 15, two selectors having selector IDs (Selector ID) “0” and “1” are defined. As shown in FIG. 15, a selector ID (SelectorID), a select type (SelectType), a select number (SelectNum), and a trigger ID (TriggerID) are set for each selector in units of one line. The selector ID is an ID that identifies the selector. The select type (SelectType) defines a method for specifying the selector ID, and there are four types: selectnum, incrementnum, decrementnum, and returnnum. selectnum selects the directly specified selector ID. The incrementnum selects a selector ID obtained by adding a select number (SelectNum) to the currently selected selector ID. Decrementnum selects the selector ID obtained by subtracting the select number (SelectNum) from the currently selected selector ID. returnnum selects the selector ID corresponding to the trigger ID. The select number (SelectNum) is a number used for incrementnum and decrementnum. The trigger ID is an ID that is generated in response to a predetermined event. In the example of FIG. 15, the trigger ID is generated when a touchless button is designated.

  Furthermore, the animation scenario definition means 82 sets an animation trigger by setting seven attributes of a touchless button ID, a button image path, a mask file path, a button position, a button size, a continuous time threshold value, and a mask ratio threshold value. Define In the example of FIG. 15, it corresponds to a range surrounded by two tags of <AnimationTriggers> on the 22nd line and </ AnimationTriggers> on the 31st line. In the example of FIG. 15, two buttons having a touchless button ID (TouchlessButton ID) “0” and “1” are defined. As shown in FIG. 15, touchless button ID (TouchlessButton ID), button image path (ButtonImagePath), mask file path (MaskFilePath), button position (PositionX, PositionY), button size (ButtonWidth) in units of 4 lines for each button. , ButtonHeight), continuous time threshold (Thtime), and mask ratio threshold (Thmask). The touchless button ID is an ID that identifies the touchless button. The touchless button ID has a one-to-one correspondence with the trigger ID. When the touchless button ID is determined, the trigger ID is also set to the same value. The button image path is a path for specifying a position where an image of a touchless button to be displayed on the display 3 is recorded. The mask file path is a path for specifying a position where a mask image for masking to round the corners of the button shape is recorded. The button position is an xy coordinate on the screen on which the touchless button is displayed. In the button instruction determination described later, when a button area is set, since it is necessary to invert the left and right of the image, the x coordinate is converted and used accordingly. The button size is the width and height of the touchless button. The continuous time threshold value is a threshold value of time for determining that the touchless button is continuously pressed. The mask ratio threshold is an area ratio threshold for determining that the touchless button is pressed.

  The case where the scenario data shown in FIG. 15 is designated will be described using the flowchart of FIG. When the image processing device 2 is activated and designated as scenario data using the scenario data shown in FIG. 15, first, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Simultaneously with the start of image creation, the image processing apparatus 2 displays the button image of the touchless button on the display 3. The viewer is photographed by the video camera 4, and the photographed image is always acquired. When the viewer tries to touch the button image, the image is photographed.

  When the creation of the image is started, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one. In the example of FIG. 15, since only one target ID “0” is set in the scenario data, the scenario data association unit 83 is identified by the object ID associated with the target ID “0”. The face detection frame data is targeted.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and the <Animation Commands> of the scenario data is referred to at this time “0.0”. As shown in FIG. 15, in <Animation Commands>, there are two commands with command IDs “0” and “1”, both of which are a start key “0.0” and an end key “1.0”. Up to now, the key type is “own”, the command type is “LayerMontage (layer composition)”, and the target ID is “0”, and only the content ID is different between “0” and “1”. As described above, the command ID is determined by the selector ID. However, since the initial selector ID (InitialSerectorID) is set to “0” as shown in the 18th line in FIG. The creation unit 84 creates a composite image by layer-combining the content with the content ID “0” with the face detection frame data with the target ID “0”.

  Since the content path of content ID “0” (material for composition) can be identified by referring to <Simulation Contents> of the scenario data, it can be determined from the storage location in the data storage device 2d identified by the path. Get content. As described above, since the rectangular information is set in the content, the composite image creating unit 84 changes the rectangular information and the size of the content so that the rectangular information and the rectangular size of the face detection frame data match, Layer composition of the changed content is performed at a position where the position of the rectangle information after the change and the face detection frame data coincide. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, refer to “CycleInterval” in the <Target ID> tag in the scenario data with the elapsed time from the start of image creation according to the scenario data, and if the elapsed time is less than the scenario time, If the elapsed time is equal to or longer than the scenario time, it is determined that the scenario is ended. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. At this time, if the selector ID is “0”, the content of the content ID “0” is selected. If the selector ID is “1”, the content of the content ID “1” is converted to the face detection frame of the target ID “0”. A composite image is created by layering the data. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

  On the other hand, in parallel with the processing of S22 to S25, the instruction determination unit 85 always determines whether or not the touch button has been instructed (S28). Specifically, the determination is performed by performing image processing on the frame of the captured video. FIG. 17 is a diagram showing an image obtained by the image processing of the instruction determination unit 85. In FIG. 17, a white portion indicates a pixel having a pixel value “1”, and a black portion indicates a pixel having a pixel value “0”. First, the instruction determination unit 85 acquires the background removed image processed by the background removal unit 20 in S1 of FIG. Then, each pixel value of the multi-value background removed image is binarized with a predetermined threshold value. FIG. 17A shows a binarized background removed image. Subsequently, the instruction determination unit 85 performs HSV conversion on the input frame image, binarizes it with a predetermined threshold value, and obtains a binarized HSV image. FIG. 17B shows a binarized HSV image. A color portion such as skin color is extracted by this HSV conversion. Then, an And calculation process is performed on the binarized background removed image and the binarized HSV image. And the person image which extracted the person is obtained. FIG. 17C shows a person image.

  Next, the instruction determination unit 85 acquires the face detection frame acquired by the face detection unit 21 in S2 of FIG. The instruction determination unit 85 generates a binary face mask image masking the acquired face detection frame. FIG. 17D shows a face mask image. Then, the instruction determination unit 85 performs an And calculation process on the face mask image and the person image. As a result, a feature image excluding the face portion from the person is obtained. FIG. 17E shows a feature image.

  Next, the instruction determination unit 85 determines the button position (PositionX, PositionY) and button size (ButtonWidth, ButtonHeight) of the touchless button in the range surrounded by two tags <AnimationTriggers> and </ AnimationTriggers> in the scenario data. ) Information. Then, from the button position and the button size, the button area and the total number of pixels included in the button are obtained. Further, a ratio Rmask between the number of pixels of the pixel value “1” (expressed in white in the drawing) in the button area of the feature image and the total number of pixels included in the button is calculated. FIG. 17F shows the relationship between the feature image and the button area. In FIG. 17F, a rectangular area surrounded by a white line is a button area. As illustrated in FIG. 15, when a plurality of touchless buttons are set, the instruction determination unit 85 performs a process of calculating the ratio Rmask for each button.

  The instruction determination unit 85 performs the processing as described above for each frame, and calculates the ratio Rmask of each touchless button. Then, the mask threshold Thmask set in the scenario data is compared with the ratio Rmask of each touchless button. Then, based on the comparison result, it is determined which of the following states 1) to 3).

(Judgment of the state by the ratio and duration of the feature on the touchless button)
1) When Rmask <Thmask Normal state 2) When Rmask> Thmask and Time <Thtime Hand is defeated 3) When Rmask> Thmask and Time> Thtime Instructed state

  In 2) and 3) above, Time is the time calculated by multiplying the number of frames by the reciprocal of the frame rate, and Thtime is a threshold of time set in the scenario data. In the case of 1), since the area of the hand on the touchless button is small, it is determined that the hand is not in a state of being touched by the touchless button but in a normal state. In the case of the above 2), since the area of the hand on the touchless button is large, the hand is in a state of being touched, but since the time is short, it is not possible to determine even the instruction for the touchless button. In the case of 3), since the area of the hand on the touchless button is large and the time is long, it is determined that the instruction is for the touchless button. In the scenario data shown in FIG. 15, in both of the two touchless buttons, Thtime is set to 1.0 second and Thmask is set to 0.1 (10%).

  That is, in S28, the instruction determination unit 85 performs processing for each frame to determine whether the hand area ratio for each touchless button is equal to or greater than the threshold value, and the hand area ratio is equal to or greater than the threshold value. When a certain touchless button continues for a predetermined number of frames or more, it is determined that the touchless button has been instructed.

  If the instruction determination unit 85 determines in S28 that the touchless button has been instructed, the command switching unit 86 performs command switching (S29). Specifically, the selector ID is changed. In the example of FIG. 15, “0” is changed to “1”, and “1” is changed to “0”. By changing the selector ID, the command ID is also changed, and the command is switched.

  FIG. 16 is a diagram showing a display state of a composite image obtained according to the scenario data of FIG. In FIGS. 16A and 16B, a rectangle with rounded corners indicates a touchless button. In the example of FIG. 16, touchless buttons labeled “Blonde” and “Black hair” are displayed. When the viewer is recognized, first, as shown in the scenario data 18th line in FIG. 15, since the InitialSelectorID that defines the initial selector ID is “0”, the selector ID is set to “0”. The command ID is set to “0”. Then, according to the <Command ID> tag on the 13th and 14th lines, the content ID “0” is set for the frame from the scenario start time (StartKey = “0.0”) to the elapsed time of 10 seconds (EndKey = “1.0”). Are layered. At this time, by referring to <SimulationContents> with ContentsID = "0", the location "afro.jpg" in the content storage unit of the content is specified, and the content of the black wig is acquired. And as shown to Fig.16 (a), a black wig is synthesize | combined with a viewer's image. When there is no command switching by the instruction of the touchless button, a composite image in which all black wigs are synthesized for the frames from the start of the scenario (StartKey = "0.0") to the elapsed time 10 seconds (EndKey = "1.0"). Will be displayed.

  When the viewer touches the “blonde” touchless button and the instruction determination unit 85 determines that the touchless button is instructed according to the condition of <AnimationTriggers>, the instruction determination unit 85 determines that the touchless button ID (TouchlessButton ID) is set to “1”, the command switching means 86 sets the trigger ID corresponding to the touchless button ID on a one-to-one basis to “1”, the selector ID is set to “1”, and the command ID is set to “1”. Switch to 1 ”. Then, according to the <Command ID> tag in the 13th and 14th lines, the content ID “1” is set for the frame from the scenario start time (StartKey = “0.0”) to the elapsed time of 10 seconds (EndKey = “1.0”). Are layered. At this time, by referring to <SimulationContents> with ContentsID = "1", the location "blond.jpg" in the content storage means of the content is specified, and the content of the blond wig is acquired. And as shown in FIG.16 (b), it replaces with a black wig and a blond wig is synthesize | combined with a viewer's image. After this, if there is no command switching by the instruction of the touchless button, all the frames from the start of the scenario (StartKey = "0.0") to the elapsed time 10 seconds (EndKey = "1.0") are combined with a blonde wig. The synthesized image thus displayed is displayed.

<< 3.4. Scene composition processing (1) >>
Next, the scene composition function that is a feature of the present invention will be described. As the scene composition function, there are an embodiment in which the contours other than the person and the person are clarified and the composition is synthesized, and an embodiment in which the contours other than the person and the person are clarified are not particularly clarified. First, as a first embodiment, a description will be given of an embodiment in which a person and a contour other than a person are clarified and combined. In order to perform scene synthesis, scenario data corresponding to the scene synthesis function is required. FIG. 20 is a diagram illustrating an example of XML-format scenario data in the case of a scene synthesis function (two targets). Here, functions of the synthesis target definition unit 80, the synthesis content definition unit 81, and the animation scenario definition unit 82 will be described with reference to the scenario data of FIG. The synthetic target definition means 80 defines a target by setting four attributes of target ID, type, absolute coordinate, and transition setting. In the example of FIG. 20, it corresponds to a range surrounded by two tags, <Simulation Targets> on the first line and </ Simulation Targets> on the seventh line. There are three types of targets that can be set: Human, Place, and Scene. For the ID type, three types of person, place, and scene corresponding to the target can be set. In the example of FIG. 20, two human tags that start with HumanID are used to set two people, a place tag that starts with Place ID is used to set a place, and a scene tag that starts with SceneID is used to set a scene. . The transition setting is for setting whether to associate with a new target when the associated target disappears. In the example of FIG. 20, three attributes of target ID (HumanID, Place ID, SceneID), type, and transition setting are set in the tags in the second to fourth lines, and the transition setting (IsTransfer) is “false ( Do not set) ”.

  The composite content definition unit 81 defines the content by setting three attributes of a content ID, a content path, and an overlap setting. In the example of FIG. 20, it corresponds to a range surrounded by two tags of <Simulation Contents> on the 8th line and </ Simulation Contents> on the 24th line. In the example of FIG. 20, the content ID (ContentsID) is defined for three contents “0”, “1”, and “2”. There are two types of content: a superimposed image superimposed on a person or background, and a background image. As for the superimposed image and the background image, as shown in the ninth and tenth lines in FIG. 20, the content path (ContentsPath) and the overlap setting (OverlapOrder) are set in units of one line for each content.

  The animation scenario definition means 82 defines an animation scenario by setting eight attributes of command ID, command type, selector ID, start key, end key, key type, target ID, and content ID. In the example of FIG. 20, it corresponds to a range surrounded by two tags, <Animation Commands> on the 25th line and </ Animation Commands> on the 28th line. In the example of FIG. 20, a command with a command ID (CommandID) “0” is defined. As shown in FIG. 20, a command type, a selector ID (SelectorID), a start key, an end key, a key type, a target ID, and a content ID are set in units of two lines for one command. The command type indicates what type of effect is generated based on what frame, and layer synthesis, α blend synthesis, audio reproduction start, and scene synthesis are prepared. Of these, layer composition, α blend composition, and scene composition indicate the type of image composition, layer composition overwrites content, and α blend composition is performed according to the set α ratio. The content and the frame are transmitted and combined, and the scene combination is a method in which a human body part is cut out and combined on the background. In the example of FIG. 20, scene composition (SceneMontage) is set as the command type (CommandType).

  The selector ID is a condition for specifying a command to be executed, and a command with a command ID corresponding to the selector ID is executed. A set of one or more commands specified by the selector constitutes one scenario. Therefore, there is a one-to-one correspondence between selectors and scenarios. In the example of FIG. 20, one scenario includes one command, and when the selector ID = 0, the command with the command ID = 0 is executed. The start key and end key are used to set the start time and end time of each command, as in the example of FIG. In the present embodiment, the scenario data time is managed as “0.0” at the start of the scenario and “1.0” at the end of the scenario. In the example of FIG. 20, as in the example of FIG. 15, the start key (StartKey) of all commands is “0.0”, and the end key (EndKey) of the last command is “1.0”. The key type is to set a target as a reference for the start key and the end key, and three types of own, base, and global are prepared. own is based on the browsing time of the face object corresponding to each target ID, base is based on the browsing time of the face object corresponding to target ID = 0, and global is based on the time when the first frame of the captured video is acquired. To do. In the example of FIG. 20, since “own” is set as the key type (KeyType), the time when the first face object appears in the frame (the time when the face object is determined to be “viewing start”) is “0”. .0 ", the start key and the end key are recognized.

  In the example of FIG. 20, the <CycleInterval> tag and the <IsAutoLoop> tag are included in each target ID tag represented by HumanID and PlaceID in the second to fourth lines, as in FIG. As described above, even when the <CycleInterval> tag and the <IsAutoLoop> tag are included in the target ID tag, the <CycleInterval> tag of the scenario data from the start to the end of the scenario is the same as in the example of FIG. The time is set in seconds, and the <IsAutoLoop> tag of the scenario data sets whether to perform loop processing (repeated processing). The image display system 1 allows a plurality of description formats for scenario data, and even when the description positions of the <CycleInterval> tag and <IsAutoLoop> tag are different as shown in FIGS. 10 and 15, The scenario data association unit 83 recognizes the <CycleInterval> tag and the <IsAutoLoop> tag, and executes processing according to the description content. In the example of FIG. 20, since CycleInterval = “10” is set in the target ID tag, it indicates that it is 10 seconds from the start to the end of the scenario. The scenario is managed in real time that is 10 times the value of the start key and end key. In the example of FIG. 20, since IsAutoLoop = “true” is set in the target ID tag, it indicates that loop processing is performed. In this way, the scenario data created by the composite target definition means 80, the composite content definition means 81, and the animation scenario definition means 82 is stored in the data storage device 2d as the scenario data storage means.

  The case where the scenario data shown in FIG. 20 is designated will be described using the flowchart of FIG. When the image processing apparatus 2 is activated and designated as scenario data using the scenario data shown in FIG. 20, first, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Next, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one. In the example of FIG. 20, since scenario data has four target IDs, HumanID = "0", HumanID = "1", PlaceID = "2", and SceneID = "3", the scenario data is supported. The attaching unit 83 targets the data specified by the object ID associated with the target ID “0”.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and the <Animation Commands> of the scenario data is referred to at this time “0.0”. As shown in FIG. 20, in <Animation Commands>, only a command having a command ID “0” exists, and the key type “own” is used from the start key “0.0” to the end key “1.0”. , Command type “SceneMontage (scene synthesis)”, target ID “3”, and content ID “2”. As described above, the command ID is determined by the selector ID. However, in FIG. 20, since there is only one command ID, the composite image creation unit 84 determines that the target ID is “3”. By synthesizing the content with the content ID “2”, a composite image is created.

  A content path of content ID “2” (composition material image) can be identified by referring to <Simulation Contents> of the scenario data. Specifically, the content ID “2” indicates “SceneContentsID = 2” on the 11th line, and consists of two contents “Human ContentsID = 0” and “Place ContentsID = 1” in the SceneContentsID tag. I understand. Furthermore, “Human ContentsID = 0” indicates “ContentsID = 0” on the 9th line, and “Place ContentsID = 1” indicates “ContentsID = 1” on the 10th line. Each content is acquired from the storage position in the data storage device 2d. Then, in the content (background image) specified by “Place Contents ID = 1”, the person portion cut out by the contour is masked. Specifically, a mask image is generated by masking the inside of the outline obtained according to the contents described in the SegmentationArea, SourceArea, and SinkArea from the 12th line to the 22nd line, and specified by “Place ContentsID = 1” Composite with the background image. On the other hand, since the content (superimposed image) specified by “Human ContentsID = 0” has already been masked, it is used as it is as a superimposed image.

  Subsequently, the composite image creating means 84 cuts out the inside of the contour line obtained according to the contents described in the Segmentation Area, Source Area, and Sink Area from the 12th line to the 22nd line from the photographed image, and masked background image And synthesize. Further, the portion corresponding to the face detection frame data of the photographed image is used to synthesize the superimposed image. Since the rectangular information is also set in this superimposed image as in the above-described content, the composite image creating unit 84 uses the rectangular information and the superimposed image so that the rectangular information matches the rectangular size of the face detection frame data. And the layered composition of the changed superimposed image is performed at a position where the position of the rectangular information after the change matches the position of the face detection frame data. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. For a portion where the background image and the superimposed image overlap, the superimposed image has priority according to the value of OverlapOrder in the ninth and tenth lines. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, referring to “CycleInterval” in the target ID tag (HumanID, PlaceID) in the scenario data with the elapsed time from the start of image creation according to the scenario data, if the elapsed time is less than the scenario time, If it is determined that the scenario is in progress and the elapsed time is equal to or longer than the scenario time, it is determined that the scenario is completed. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. However, since PlaceID = "2" is set in FIG. 20, HumanID can only be set to "0" or "1". That is, the scenario data in FIG. 20 can support only a maximum of two people. Of course, if the scenario data is rewritten, it is possible to handle a larger number of people. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. Then, according to <Animation Commands>, a composite image is created by scene-synthesizing the content specified by ContentsID = “2” on the target specified by TargetID = “3”. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

  Next, details of the cut-out process of the captured image in the 12th to 22nd lines of the scenario data in FIG. 20 will be described. The 12th to 22nd lines are composed of a SegmentationArea tag on the 12th and 13th lines, a SourceArea tag on the 14th to 18th lines, and a SinkArea tag on the 19th to 22nd lines. In the SegmentationArea tag, SourceArea tag, and SinkArea tag, a segment area, an initial area of the source data string, and an initial area of the sink data string are set, respectively.

  First, the segment area will be described. The segment area is an area necessary for initial area setting, and is a range including a face detection frame. As shown in the 12th and 13th lines in FIG. 20, the segment area is set by the face detection frame upper magnification (TopRate), the face detection frame lower magnification (BottomRate), and the aspect ratio (AspectWidth, AspectHeight).

  FIG. 21 is a diagram illustrating a relationship between a face detection frame and a segment area in a captured image. Of the two rectangles in FIG. 21, the inner rectangle indicates the face detection frame, and the outer rectangle indicates the segment region frame. The segment area shown in FIG. 21 is set according to the contents of the scenario data shown in FIG. 20 (TopRate = "1.2, BottomRate =" 2.0, AspectWidth = "5", AspectHeight = "3"). The upper magnification of the face detection frame is the ratio of the length from the center of the face detection frame to the upper side of the face detection frame and the length from the center of the face detection frame to the upper side of the segment area. It is the ratio of the length from the center to the lower side of the face detection frame and the length from the center of the face detection frame to the lower side of the segment area. A segment area is set in a range including the entire face detection frame according to the face detection frame upper magnification, the face detection frame lower magnification and the aspect ratio. As shown in FIGS. 20 and 21, the segment area is set by numerical designation based on the position / size of the face detection frame, and the numerical value can be changed as appropriate. It is desirable to set the direction centers so that the center of the face detection frame is positioned above the center of the segment area in the vertical direction. Such a numerical value may be automatically set by the composite content definition means 81.

  FIG. 22 is a diagram showing the shape of the initial region. In the present embodiment, the initial region can be set in three shapes: a circle, an ellipse, and a rectangle (including a square). Each initial area is expressed by a relative position, a relative width, and a relative height when the segment area is 0 to 1. Specifically, in the case of a circle, the center position (CenterX, CenterY) and radius (Radius), in the case of an ellipse, the center position (CenterX, CenterY), X-axis radius (XRadius) and Y-axis radius (YRadius), In the case of a rectangle, the upper left vertex position (X, Y), width (Width), and height (Height) are set. Of course, it is possible to set other than the three shapes of the present embodiment.

  FIG. 23 is a diagram showing an initial area set according to the example of scenario data of FIG. FIG. 23A is a diagram showing the initial area in the shooting frame, and FIG. 23B is a script of the SourceArea tag and the SinkArea tag part in the scenario data for setting the initial area of FIG. is there.

  In order to cut out the outline of a person by the process described later using the initial area, the initial area of the source needs to be set to a position where the person exists, and the initial area of the sink needs to be set to a position where no person exists. In the scenario data of FIG. 20, the initial area is designated by a numerical value. This numerical value is a numerical value obtained by calculation based on the relationship between the face detection frame and the segment area. In the SourceArea tag and the SinkArea tag, the number of pixels randomly selected in the initial region is specified as SamplingNum = “50”.

  When the initial area is set, the composite image creating means 84 segments the image in the segment area. This segmentation is a process for separating the target and the non-target, and is performed so that the source initial area is included in the target and the sink initial area is included in the non-target portion. Various known techniques can be used as the segmentation technique. In this embodiment, a known graph cut segmentation technique is used. Therefore, the composite image creating unit 84 selects 50 pixels at random in each of the initial area of the source and the initial area of the sink, and executes the graph cut segmentation technique using the value of each pixel, thereby obtaining an image in the segment area. Are segmented into targets and non-targets. The number of pixels selected at random in each of the Source initial area and the Sink initial area can be arbitrarily set by “SamplingNum” in the SourceArea tag. The segment result image obtained by segmentation is shown in FIG. In FIG. 24B, a white part is a part indicating a target (person), and a black part on both sides is a part indicating other than the target (person). The segment result image is created as a binary image.

  When the segment result image is obtained, the composite image creating unit 84 composites the content mask image and the segment result image to create a composite mask image. The content mask image is a mask image corresponding to the content image specified by Human Contents ID, and is an image in which portions other than the portion extracted by the content image are masked. This content mask image is a binary image created in advance for each content image, and is stored in the data storage device 2d. As shown in FIG. 24, the composite mask image of FIG. 24C is obtained by ORing the content mask image as shown in FIG. 24A and the unmasked portion of the segment result image as shown in FIG. By doing so, a binary image obtained by adding the unmasked portions of both is obtained. Therefore, when the content image is a helmet and clothes, the content mask image is an image in which the helmet, clothes, and person portions are not masked.

  Next, the composite image creating means 84 creates a background mask image by inverting the composite mask image. Specifically, the value of each pixel of the composite mask image that is a binary image is changed to the other value. As a result, a background mask image which is a binary image in which the masked pixels and the unmasked pixels are reversed is obtained. FIG. 25 shows an example of the composite mask image and the background mask image. FIG. 25A is a composite mask image, and FIG. 25B is a background mask image. In the background mask image, the helmet, clothes, and person portions are masked.

  Subsequently, the composite image creating means 84 creates a composite image by masking the background image using the background mask image, masking the superimposed image with the content mask image, and superimposing the obtained image on the captured image. As a result, a composite image as shown in FIG. FIG. 26 shows how a composite image is created. 26 (a) is a captured image, FIG. 26 (b) is a background image, FIG. 26 (c) is a background mask image, FIG. 26 (d) is a superimposed image, and FIG. 26 (e) is a superimposed mask. The image, FIG. 26 (f), is a composite image. In FIG. 26 (f), for the convenience of the drawing, the background is shown in a narrowed state, but in reality, the composition is performed with an aspect ratio that matches the display range of the display 3. In this way, it is possible to combine the content such as clothes and the background with the person browsing the display 3 and display it on the display 3.

<< 3.5. Scene composition processing (2) >>
In the above scene composition process, the segment area is segmented so that the boundary between the person and the person is clarified and the background image is synthesized. In the second embodiment, the process of synthesizing the person and the background image is performed without using the segment area and without clarifying the boundary between the person and the person. The second embodiment is also the same as the first embodiment in that processing according to the flowcharts shown in FIGS. 4 to 6 and 9 is executed by the apparatus configuration shown in FIGS. Although illustration is omitted, scenario data that determines the timing of combining one or more persons on the video with the content is stored in the data storage device 2d as scenario data storage means.

  The second embodiment will be described with reference to the flowchart of FIG. When the image processing device 2 is activated and scenario data is designated, the scenario data association unit 83 reads the designated scenario data from the data storage device 2d (S21). Then, the scenario data association unit 83 interprets the scenario data and starts creating an image according to the scenario data (S22).

  Next, the scenario data association unit 83 acquires the face object data generated by the state transition management unit 25 (S23). The face object data includes an object ID, face detection frame data (position, rectangular size), and browsing time.

  Subsequently, the scenario data association unit 83 performs a process of associating the face object data acquired from the state transition management unit 25 with the scenario data (S24). Specifically, the object ID of the face detection frame data included in the face object data is associated with the target ID. When a plurality of face detection frame data is acquired from the state transition management unit 25, the date and time when the state transition to the candidate face is the earliest date and time is set to “0”, and thereafter the date and time when the state transition to the candidate face is performed. In order from the earliest, “1”, “2” and “3” are set while increasing the number by one.

  Next, the composite image creating means 84 performs processing for creating a composite image for display in units of frames (S25). Specifically, first, the start time is set to “0.0”, and at this time “0.0”, the <Animation Commands> of the scenario data is referenced and the command with the specified command ID is executed. .

  By referring to the <Simulation Contents> of the scenario data with the content ID specified by the command ID tag, the content path can be specified. In the second embodiment, unlike the first embodiment, two contents of a superimposed image and a background image must be specified. In addition, as the superimposed image, an image that is open only near the face of a person and surrounds the entire periphery is prepared. For example, a face signboard and a wedding dress can be mentioned. The composite image creation unit 84 acquires each content from the storage position in the data storage device 2d specified by the path. Then, the content (background image) specified by the Place Contents ID is masked using the background mask image obtained by inverting the content mask image of the superimposed image. On the other hand, since the content (superimposed image) specified by “Human ContentsID” has already been masked, it is used as it is as a superimposed image.

  Subsequently, the composite image creating unit 84 synthesizes the superimposed image using the portion corresponding to the face detection frame data of the photographed image. Since the rectangular information is also set in this superimposed image as in the above-described content, the composite image creating unit 84 uses the rectangular information and the superimposed image so that the rectangular information matches the rectangular size of the face detection frame data. And the layered composition of the changed superimposed image is performed at a position where the position of the rectangular information after the change matches the position of the face detection frame data. Specifically, the size of the rectangle set in the content is changed to match the rectangle of the face detection frame data set in the face image as shown in FIG. The content is resized according to the change ratio, and the composition process is performed so that the two rectangles match. As a result, for example, when the content is a wig as shown in FIG. 12A, a composite image (FIG. 12C) in a state where the wig is covered with a human face is obtained. In the present embodiment, as the content of the superimposed image, an image is prepared in which only the face of a person such as a face signboard or a wedding dress is open and surrounding all of the surroundings. Compositing will be done in such a way that a face is put out in the hole of the signboard or a wedding dress is put on. Subsequently, the composite image creating unit 84 synthesizes the background image masked using the background mask image with the composite of the captured image and the superimposed image. For the portion where the background image and the superimposed image overlap, the superimposed image has priority according to the value of the OverlapOrder in the 8th and 9th rows. The composite image creating unit 84 displays the obtained composite image on the display 3. As a result, a composite image in which the frame of the captured video is processed is displayed on the display 3.

  When the creation of the composite image for one frame is completed, the scenario data association unit 83 determines whether or not the scenario is in progress (S26). Specifically, referring to “CycleInterval” in the target ID tag (HumanID, PlaceID) in the scenario data with the elapsed time from the start of image creation according to the scenario data, if the elapsed time is less than the scenario time, If it is determined that the scenario is in progress and the elapsed time is equal to or longer than the scenario time, it is determined that the scenario is completed. If it is determined that the scenario is in progress, the scenario data association unit 83 returns to S23 and acquires face object data.

  In S24, the scenario data association unit 83 performs processing for associating the next face object data acquired from the state transition management unit 25 with the scenario data. Also at this time, as in the first loop, the earliest date and time when the state transition to the candidate face is set to “0”, and thereafter “1” and “2” in order of the date and time when the state transition to the candidate face occurs. Set "3" while increasing the number by one. Then, according to the scenario data, the scenario data association unit 83 targets the face detection frame data specified by the object ID associated with the target ID “0”.

  Next, in S25, the composite image creating unit 84 performs a process of creating a composite image in units of frames. Specifically, the elapsed time is acquired, and the <Animation Commands> of the scenario data is referenced with the acquired elapsed time. Then, in accordance with <Animation Commands>, a composite image is created by synthesizing the content specified by ContentsID with the target specified by TargetID. In this way, the process of executing the scenario data is repeated according to the elapsed time until it is determined in S26 that the scenario is ended.

  If it is determined in S26 that the scenario has been completed, the scenario data association unit 83 determines whether or not to repeat the process (S27). Specifically, the <IsAutoLoop> tag in the scenario data is referenced, and if “true” is set, it is determined that the process is repeated. If it is determined that the process is to be repeated, the scenario data association unit 83 resets the elapsed time to “0”, starts measuring the elapsed time again, returns to S22, and follows the scenario data. Start creating an image. In this manner, the composite image obtained from each frame of the video is sequentially displayed on the display, so that it is displayed as a processed video.

<4. Configuration not using state transition management means>
In the image display system of the above embodiment, the state transition management unit 25 is used, and when it is determined that the detected face image is noise, the browsing state is not determined, but the state transition management unit 25 is used. It is also possible to determine that all detected face images are in the browsing state. Next, a configuration not using the state transition management unit 25 will be described.

  FIG. 18 is a functional block diagram realized by a computer program installed in the image processing apparatus 2 ′ when the state transition management unit 25 is not used. 18, components having the same functions as those in FIG. 3 are denoted by the same reference numerals, and detailed description thereof is omitted.

  An image processing apparatus 2 ′ illustrated in FIG. 18 includes a tracking unit 23 ′ instead of the tracking unit 23 illustrated in FIG. This tracking means 23 'also has a function corresponding to the moving picture analysis means 24 shown in FIG.

  The image processing apparatus 2 ′ illustrated in FIG. 18 performs the processes of S1 and S3 in the same manner as the image processing apparatus 2 among the processes of S1 to S5 illustrated in FIG. Further, the face detection process and the tracking process are executed in cooperation. As described above, the state transition management process in S5 is not performed.

  FIG. 19 is a flowchart showing face detection processing and tracking processing. First, after performing the background removal processing S1, in processing N frames, it is determined whether or not the number of face detection frames in N-1 frames is greater than 0 (S31). When the number of N-1 frame face detection frames is larger than 0, the tracking unit 23 'executes the tracking process (S32).

  The tracking means 23 'tracks each face detection frame in the N-1 frame and specifies a corresponding face detection frame in the N frame. As the tracking unit 23 ′, a known tracking method such as “particle filter”, “LK method”, or “CamShift method” executed by the moving image analysis unit 24 can be employed.

  When the tracking process of the face detection frame from the N-1 frame to the N frame is finished, the face detection unit 21 performs the face detection process in the N frame (S33). The face detection process in S33 is the same as the face detection process in S2 shown in FIG. If it is determined in S31 that the number of face detection frames in the N-1 frame is not greater than 0, the face detection unit 21 does not perform the tracking process from the N-1 frame to the N frame and the face detection unit 21 The face detection process is performed.

  Subsequently, it is determined whether or not the number of N frame face detection frames newly detected in the face detection process S33 is greater than 0 (S34). The newly detected N frame face detection frame is obtained by excluding the face detection frame tracked from the N-1 frame to the N frame from the face detection frames detected in the N frame.

  Next, the face detection means 21 assigns an object ID to each face detection frame data newly detected in the N frame, and sets a face object composed of the face detection frame data, the object ID, and the tracking time ( S35). The face object is specified by the object ID, and the face detection frames associated by tracking are specified by the same object ID. The initial value of the tracking time is set to zero.

  Subsequently, it is determined whether or not the number of face detection frames in the N frame is greater than 0 (S36). In S36, it is determined whether or not a face detection frame in which an object ID has already been issued exists in the N frame regardless of whether or not a new detection has been performed in the N frame.

  When the face detection frame exists, the tracking time is calculated for the face object of each face detection frame (S37). Specifically, the tracking time of each face object up to N frames is calculated by adding the time corresponding to one frame to the tracking time calculated up to the immediately preceding N-1 frame. When the tracking time is calculated, N is incremented (S38), and the process proceeds to the next N frame. If the result of determination in S36 is that there is no face detection frame, there is no target to be tracked in N frames, so tracking time is not calculated and N is incremented (S38). ), And shifts to processing for the next N frame.

  The face detection unit 21 and the tracking unit 23 ′ of the image processing apparatus 2 ′ repeatedly execute the process shown in FIG. 19 for each frame on which the background process has been performed by the background removal unit 20.

  The face object given in the process shown in FIG. 19 is associated with the scenario data by the scenario data association unit 83 in S24 shown in FIG. In the processing shown in FIG. 19, the object ID of the face object is set in increments of “0”, “1”, “2”, and “3” in the order in which the face detection frames are detected.

  The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above embodiments, and various modifications can be made. For example, in the first embodiment, the person portion cut out from the photographed image and the background image are combined and the superimposed image is superimposed on the person portion or the background image. Even when it is synthesized with only the image, it is possible to create a synthesized image having a production effect sufficiently in real time. When the superimposed images are superimposed, the effect of production is further enhanced.

  In the first embodiment, the graph cut segmentation is used as a specific method of segmenting the image in the segment area by the composite image creating unit 84. However, other known methods such as the Watershed algorithm and the average value algorithm are used. A technique may be used.

  INDUSTRIAL APPLICABILITY The present invention is applicable to industries that display images on a display using a computer and digital signage that displays advertisements as video.

DESCRIPTION OF SYMBOLS 1 Image display system 2, 2 'Image processing apparatus 20 Background removal means 21 Face detection means 22 Human body detection means 23, 23' Tracking means 24 Movie analysis means 25 State transition management means 26 Person attribute estimation means 27 Log file output means 3 Display 4 video camera 6 state transition table 80 composite target definition means 81 composite content definition means 82 animation scenario definition means 83 scenario data association means 84 composite image creation means 85 instruction determination means 86 command switching means

Claims (4)

An image display system comprising a camera for photographing a person, an image processing device for synthesizing a captured video sent from the camera, and a display for displaying the synthesized video that has been synthesized,
The image processing apparatus includes:
Scenario data storage means for storing scenario data defining the timing of composition of one or more persons on the video and the content;
Content storage means for storing a background image used for composition as content;
Face detection means for detecting a face image captured in one frame of a video transmitted from the camera and outputting the position / rectangular size of the face detection frame as face detection frame data for each detected face image; ,
Tracking means for associating the face detection frame data acquired from the face detection means with face detection frame data of another frame;
Scenario data associating means for associating a face object including face detection frame data detected by the face detecting means with a person included in the scenario data;
A segment result is set in a range including a face detection frame associated with a person included in the scenario data in each frame, and segmented into portions other than the person and the person using the segment area Using an image, masking the background image, and a combined image generating means for generating a combined image obtained by combining the person in the frame and the background image;
An image display system comprising: The content storage means further stores a superimposed image, which is an image to be superimposed on the person or background, as content,
The composite image generation unit assigns the face object to the person of the scenario data according to the association by the scenario data association unit, and matches the position and size of the face detection frame data of the face object to match the superimposed image. The image display system according to claim 1, wherein a combined image is generated by changing the position and size and combining the superimposed image on the frame. The composite image generating means matches the segment area with the face detection frame at the center in the left-right direction based on the position of the face detection frame, and the center of the face detection frame is the center of the segment area in the up-down direction. Set to be located above the center,
The segmentation is performed using a pixel set in a predetermined position in the segment area and designated in an area where a person exists and an area where no person exists. Item 3. The image display system according to Item 2. An image display system comprising a camera for photographing a person, an image processing device for synthesizing a captured video sent from the camera, and a display for displaying the synthesized video that has been synthesized,
The image processing apparatus includes:
Scenario data storage means for storing scenario data defining the timing of composition of one or more persons on the video and the content;
Content storage means for storing a background image and a superimposed image in which a part surrounded by the periphery is opened as content,
Face detection means for detecting a face image captured in one frame of a video transmitted from the camera and outputting the position / rectangular size of the face detection frame as face detection frame data for each detected face image; ,
Tracking means for associating the face detection frame data acquired from the face detection means with face detection frame data of another frame;
Scenario data associating means for associating a face object including face detection frame data detected by the face detecting means with a person included in the scenario data;
According to the association, the face object is assigned to the person of the scenario data, and the position and size of the superimposed image are changed and combined in accordance with the position and size of the face detection frame data of the face object, and the superposition is performed. A composite image generating means for generating a composite image combined with the background image masked by the mask image corresponding to the image;
An image display system comprising: