JP6934402B2 - Editing system - Google Patents

Description

The present invention relates to an editing system, for example, in a broadcasting station accumulating images, in order to produce a celebration program, a memorial program, or the like, the appearance scene of a specific performer is detected from the past images. , Regarding an editing system having a function of assisting editing for program production.

Traditionally, past video assets have been recorded on VTR tapes, and a huge number of VTR tapes have been stored in warehouses. A tape number was attached to each VTR tape, and along with the tape number, information on the program name, performers, and program content outline recorded on the tape was managed. Therefore, when a video of a specific performer is desired, the person in charge of the broadcasting station identifies the VTR tape on which the program in which the performer is appearing is recorded based on the management information of the VTR tape.

For example, in Patent Document 1, as the same program editing technique, when the original material information data which is the information extracted from the relationship between the original material for editing and the edited material is created and edited again, the edited material and the project A technique for editing using data and original material information data has been proposed.

Japanese Unexamined Patent Publication No. 2012-34218

By the way, conventionally, in order to identify which scene of a VTR tape is appearing, it is necessary to play the VTR tape on a VTR device and visually search for the appearance scene of the performer, and the appearance scene is found. In this case, the time code information of the appearance scene is written down and used for editing, and a new technology for countermeasures is required from the viewpoint of work efficiency and accuracy.

In recent years, there has been a change in the method of dubbing video assets from VTR tapes to magnetic media such as LTO tapes and optical media such as Blu-ray discs (registered trademarks) and storing them as video files in these media. However, in order to find out the appearance scene, it is still necessary to reproduce and visually inspect the video files in these media, and there is a similar problem.

In addition, if the performer's broadcast becomes impossible due to a problem caused by the performer immediately before the broadcast after the program editing is completed, the performer is mosaicked or the appearance scene is cut. Need to be re-edited to do. In order to find the appearance scene for re-editing, it was necessary for the editor to visually find the appearance scene by playing back the video after the editing was completed. In this respect as well, there was a similar problem.

The present invention has been made in view of such a situation, and an object of the present invention is to solve the above problems.

The present invention is an editing system provided with an editing device for editing a video file used for broadcasting, and acquires a face image of a performer included in the video file, and obtains the face image and the appearance video of each performer. The face image storage server that records in association with the time code information, the face image recorded in the face image storage server, and the face image to be searched included in the video file of the specific program are compared and specified. Other than the specific program in which the person with the face image to be searched appears in the specific program based on the appearance image detection unit that detects the appearance image in the program and the appearance image detected by the appearance image detection unit. When the distance of the feature amount between the face image of the appearance video and the face image of the person in the specific program is closer than the preset threshold , the search target is obtained by the similar face image search for determining the same person. A similar face that detects an appearance video other than the specific program in which a person with a face image is appearing , associates the time code information of the detected appearance video with the performer information to be searched, and notifies the editing device. An image detection device is provided, and the editing device edits a video file of the specific program using the time code information.
In addition, when editing the video file, the editing device captures the video file of the recording device when the video file of the performer is stored in the recording device to be processed by the similar face image detection device. By playing back, the video confirmation of the appearance video may be displayed.
In addition, the editing device may reproduce the appearance video detected by using the low-resolution video.
Further, the face image storage server can save the face image to be detected in association with the face type, and the similar face image detection device may perform a similar face image search according to the face type.

According to the present invention, it is possible to provide a technique for facilitating the detection of a face image of a person to be searched during a program (in video data) and effectively performing editing such as mosaic processing.

It is a block diagram which shows the schematic structure of the video editing system which concerns on embodiment. It is a block diagram which shows the schematic structure of the recording apparatus which concerns on embodiment. It is a block diagram which shows the schematic structure of the similar face image detection apparatus which concerns on embodiment. It is a block diagram which shows the schematic structure of the editing apparatus which concerns on embodiment. It is a functional block which shows the schematic structure of the automatic editing information creation apparatus which concerns on embodiment. It is a flowchart which shows an example of the editing process which concerns on embodiment. It is a flowchart which shows an example of the editing process which concerns on embodiment. It is a flowchart which shows an example of the editing process which concerns on embodiment. It is a flowchart which shows the operation example of the process target recognition part which concerns on embodiment. It is a figure which shows the example of the display method (display of a touch panel display) in an editing apparatus which concerns on embodiment. It is a figure which shows the example of the feature amount of the image which became the candidate of the search key image which concerns on embodiment. It is a flowchart which shows the procedure which carries out the similar person search (similar face image detection processing) which concerns on embodiment. It is a figure which shows the example of the search screen which can be used in the similar face image search system which concerns on embodiment. It is a figure which shows the procedure example which stores a face image in a face image storage server which concerns on embodiment. It is a figure which shows the procedure example which stores a face image in a face image storage server which concerns on embodiment. It is a figure which shows the example which detected the similar face image from the face image storage server with the face image of the target performer as the detection target which concerns on embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The outline of this embodiment is as follows.
(1) From the huge amount of past images stored in the broadcasting station, only the face images of all the performers are cut out as much as possible and saved in the face image storage server together with the time code information of the appearance scenes.
(2) The accumulated face image is compared with the face image of the target performer, and the appearance scene of the performer is detected.
(3) Using the face image of the appearance scene as a result of detection, the similar appearance scene is narrowed down by the similar face image detection process.
(4) By passing the time code information of the detected appearance scene to the editing machine, it is easy to produce a special program in which the performer is appearing.
(5) The appearance scene detected by using the low-resolution video is easily reproduced.
(6) If the video of a performer becomes unbroadcast immediately before broadcasting, the scene in which the performer is appearing is specified and edited (mosaic, cut, etc.).

FIG. 1 is a block diagram showing a schematic configuration of a video editing system 1 according to the present embodiment. The video editing system 1 includes a camera 10, a recording device 11, a recording device 12 (video server), an automatic editing information creation device 13, an editing device 14, a management terminal 17, a transmission server 18, and a system control unit. 15 is provided, and they are connected by a network 2 such as a LAN line or a predetermined communication line. The system control unit 15 controls the entire video editing system 1 in an integrated manner, and may be configured independently or is included in the same configuration as other devices (recording device 12, editing device 14, etc.). May be done.

The camera 10 performs digital conversion processing on an image captured by a CCD (Charge Coupled Device), CMOS (Complementary Metal Oxide Semiconductor) element, or the like, and obtains image data (for example, HD-SDI standard material video data) of the conversion result. The recording device 11 is used to output data to the recording device 12 via the network 2. The recording device 12 (video server) stores this. The automatic editing information creating device 13 is connected to the recording device 12 via the network 2, and the material video data is input to the automatic editing information creating device 13 and then stored in the recording device 12. However, the material video data may be directly input to and stored in the recording device 12 without going through the automatic editing information creating device 13.

FIG. 2 is a block diagram showing a schematic configuration of the recording device 12. The recording device 12 includes a recording function, a similar face image detection function (similar face image detection device 16), and a data storage function (121-127).

The data storage function of the recording device 12 will be described. The recording device 12 includes a material video data unit 121 for recording material video data, an edited video data unit 122 for recording edited video data, and an automatically edited video data unit 123 for recording automatically edited video data. The final edit information unit 124 that records the final edit information, the automatic edit information unit 125 that records the automatic edit information, the low resolution server 126 that records the low resolution file, and the face image included in the video are recorded and stored. It includes a face image storage server 127.

The reason for providing the low resolution server 126 is as follows. That is, in general, the video file in the media 5 (optical media 5a, magnetic media 5b, VTR tape 5c) needs to be stored with high image quality, so that the video file size becomes large. That is, it cannot be saved in the HDD storage that can always be accessed. If it is a low-resolution file, since the file size is small, it is possible to save the video file in the HDD storage that is always accessible and preview it. Therefore, when dubbing the media 5, a low-resolution video (low-resolution file) is simultaneously created and recorded on the low-resolution server 126.

Subsequently, the similar face image detection device 16 that realizes the recording function and the similar face image detection function of the recording device 12 will be described with reference to FIG.

FIG. 3 is a block diagram showing a schematic configuration of the similar face image detection device 16. The similar face image detection device 16 includes an image transmission / reception unit 210, an image recording unit 211, a reproduction control unit 212, a person area detection unit 213, a person feature amount extraction unit 214, a person feature amount recording unit 215, and attributes. It has an information recording unit 216, a request receiving unit 217, a similar person search unit 218, an appearance event search unit 219, a search result transmission unit 220, a keyword recording unit 110, and a keyword search unit 111.

The image transmission / reception unit 210 is a processing unit that inputs / outputs images from the outside of the device, receives input image data from the camera 10 or another device, and transmits output image data to another device (editing device 14 or the like). I do.

The image recording unit 211 writes the input image data to the recording medium and reads the output image data from the recording medium (in the case of the VTR tape 5c, it is connected to the media reproduction device 19). At the time of writing, in addition to the image data, an image ID (image identification information) which is information when reading the image data is also recorded. The playback control unit 212 controls video playback to the editing device 14.

The person area detection unit 213 detects the person using the image recognition technique for the input image data, determines the existence of the person in the image, and if the person exists, calculates the coordinates of the area. Further, the person area detection unit 213 identifies the area of the "face" of the person, extracts the face image including the area, and records it in the face image storage server 127.

The person feature amount extraction unit 214 calculates the feature amount for the area detected by the person area detection unit 213 by using the image recognition technique. The person feature amount calculated here is, for example, the shape and direction of the outline of the person, the color of the skin, the gait (how to handle the legs such as which leg is moved how and at what timing), or the person is specified. The shape and direction of the contour of the face, which is a typical part of the face, and the size, shape, and arrangement of the main components such as eyes, nose, and mouth can be mentioned. Or any number. The person feature amount extraction unit 214 can determine the face type (front, profile, diagonal face, back face, laughing face, angry face, etc.) as the type of feature amount, and the face image to be detected and such a face image to be detected. Features can be associated.

The person feature amount recording unit 215 writes and reads the feature amount calculated by the person feature amount extraction unit 214 to the recording medium. At this time, the person feature amount is associated with the face image extracted by the person area detection unit 213 when it is recorded in the face image storage server 127. The face image is associated with the person's name at a predetermined timing (input by the user or automatically assigned by a similar face image search).

The image data recording medium in the image recording unit 211 and the personal feature amount recording medium in the processing unit may be the same or separate.

The attribute information recording unit 216 writes and reads the attribute information related to the image data to the recording medium. The attribute information is, for example, an image shooting time, an image pickup device number, or the like.

The request receiving unit 217 receives a search request and a keyword assignment request from the editing device 14. The search request includes a similar face image search request and an appearance event search request.

The similar person search unit 218 performs a similar face image search when the request received by the request receiving unit 217 is a similar person search request.

The appearance event search unit 219 performs an appearance event search when the request received by the request receiving unit is an appearance event search request.

The search result transmission unit 220 transmits the similar person search result and the appearance event search result obtained from the similar person search unit 218 and the appearance event search unit 219 to the editing device 14.

The keyword recording unit 110 writes and reads the keyword to the recording medium based on the keyword addition request received by the request receiving unit 217.

The keyword search unit 111 performs a keyword search when a keyword is included in the search request data received by the request receiving unit 217.

Subsequently, the editing device 14 (editing machine) will be described with reference to FIG. FIG. 4 is a block diagram showing a schematic configuration of the editing device 14. The editing device 14 performs an editing process that actually performs a rendering process or the like on the material video data.

The editing device 14 includes an editing control unit (editing means) 141 including a processor that actually performs this editing work, and a display unit that displays material video data and a video based on the video data after being edited. It includes 142 (display), an operation panel 143 (operation means) for selecting each part of the image or sound, or inputting an instruction, and a similar face image detection operation unit 103. The display unit 142 and the operation panel 143 may be provided as an integrated touch panel display 144.

The editing control unit 141 reads the material video data and the above-mentioned automatic editing information from the recording device 12 (automatic editing information unit 125), and edits the material video data based on the automatic editing information. New video data (automatically edited video). Data) is created, and the automatically edited video data is stored in the recording device 12 (automatically edited video data unit 123).

However, in the editing device 14, the user confirms the image based on the automatically edited video data on the display unit 142, and then operates the operation panel 143 to process the automatically edited video data. Of these, an instruction for canceling the processing of the portion recognized as inappropriate can be issued to the editing control unit 141, and this processing can be canceled. In this case, the material video data can also be referred to.

Similarly, the editing control unit 141 can further perform additional processing on the automatically edited video data. At this time, the part to be newly processed is specified by the user. Also at this time, the user can perform this operation by operating the operation panel 143 after confirming the video based on the automatically edited video data on the display unit 142. By such an operation by the user, the final editing information in which the automatic editing information is rewritten is generated. This final editing information is reflected in the editing process for the material video data, and is used for updating the processing target information as described later.

Similarly, the editing control unit 141 directly reads the material video data from the recording device 12, causes the user to confirm the image based on the material video data on the display unit 142, and then operates the operation panel 143 to perform the automatic editing. It is also possible to specify the part to be processed and perform the rendering process without using the information. In this operation, the user can perform the rendering process on the material video data regardless of the automatic editing information.

The editing control unit 141 transfers the automatically edited video data edited based on the automatic editing information, the automatically edited video data by the user, or the edited video data edited by the user to the recording device 12 in this way. Can be recorded.

The similar face image detection operation unit 103 has functional configurations such as a search request transmission unit 221 and a search result reception unit 222, a search result display unit 223, a reproduction image display unit 224, a screen operation detection unit 225, and a keyword assignment request transmission unit 112. It has each processing unit of the plurality of search key selection units 113.

The search request transmission unit 221 transmits the search request to the recording device 12. In the case of a similar person search, the search request data includes a person's name, a search key image (particularly a face image), or a feature amount thereof as a search key for the similar person search. It is also possible to include narrowing parameters in the search request data.

The search result receiving unit 222 receives the search result from the recording device 12 (similar face image detecting device 16). The data received as the search result includes a set of images obtained by performing a similar person search or an appearance event search in the recording device 12 (similar face image detecting device 16). The individual images constituting the set are generated by performing image size reduction processing or the like from the images recorded in the recording device 12 (similar face image detecting device 16). Hereinafter, these individual images are also referred to as "search result images", and the data transmitted and received as search results are also referred to as "search result data".

The search result display unit 223 displays the screen of the search result received by the search result receiving unit 222. An example of the displayed screen will be described later.
The reproduced image display unit 224 displays a continuous moving image on the screen of the image data input from the recording device 12 (similar face image detecting device 16).
The screen operation detection unit 225 detects and acquires the operation content by the user.
The keyword assignment request transmission unit 112 transmits the keyword assignment request to the recording device 12 (similar face image detection device 16).
The multiple search key selection unit 113 performs a process of appropriately selecting a smaller number of search key images when a plurality of search key image candidates are selected.

FIG. 5 is a functional block of the automatic editing information creation device 13. The automatic editing information creation device 13 includes a processing target recognition unit 131 and an information storage unit 132. The information storage unit 132 includes a final editing information unit 124, an automatic editing information unit 125, and a processing target information unit 128. The final editing information unit 124 and the automatic editing information unit 125 may be the same as those provided in the recording device 12, or may be provided separately.

The automatic editing information creation device 13 reads the material video data, and the processing target recognition unit 131 recognizes a portion to be rendered. At this time, the processor in the processing target recognition unit 131 performs this recognition based on the processing target information stored in the information storage unit 132, and thus performs the processing target portion and the information about the processing (automatic). The editing information) is stored in the recording device 12.

The information about the part to be processed in the content of the automatic editing information is specifically, the video frame position (time code information) of the part to be processed, the coordinates on the video, or when the processing target is audio. Has the range of audio sample positions, the content of processing, and so on. The contents of the processing include mosaic processing, blur processing, video cut, brightness increase / decrease processing when the processing target is video, mute processing, volume adjustment and the like when the processing target is audio. In addition, the reason for processing (for example, broadcasting ban, specific company name, etc.) is also included in the processing target information.

As the processing target information, for example, a plurality of information can be set according to the video distribution destination (purpose) and the like. According to this, for example, the part that is not the target of processing for one delivery destination is the target of processing for other delivery destinations, and the content of the above processing is changed according to the delivery destination. Etc. can be operated. In such a case, the processing target information is selected by the user.

Further, as will be described later, when the material video data is finally edited, the portion to be processed and the content of the processing are corrected after being confirmed by the user. Such final editing information (final editing information) or information regarding the difference between the final editing information and the automatic editing information is also stored in the information storage unit 132.

An operation example with the above configuration will be described.
First, an example of editing processing will be described with reference to FIGS. 6 to 10, then a similar person search processing (particularly similar face detection processing) will be described with reference to FIGS. 11 to 13, and further with reference to FIGS. 14 to 16. A processing example in which the similar face detection processing is applied to the editing processing will be described.

FIG. 6 is an example of a flowchart showing a specific operation performed by the system control unit 15. Here, for the sake of simplicity, it is assumed that the process specified by the user using the editing device 14 is not performed. Further, in FIG. 1, the material video data is input (stored) to the recording device 12 only via the automatic editing information creation device 13.

First, the recording device 11 obtains the material video data (S1). The automatic editing information creation device 13 obtains this material video data and analyzes whether or not there is a portion to be processed in the image in the material video data (S2). Here, the processing target recognition unit 131 refers to the information in the information storage unit 132, recognizes whether there is a part to be processed in the image in the material video data, and if this part is recognized, the processing target recognition unit 131 is recognized. The processing for this portion is also determined based on the information in the information storage unit 132 (S3). As a result, automatic editing information is created. When the portion to be processed is not recognized (No in S4), the material video data is stored in the recording device 12 as it is (S5).

When the portion to be processed is recognized (Yes in S4), the system control unit 15 asks the user whether or not to store the material video data (S6). When not stored (No in S6), as described above, the editing device 14 is used to create automatically edited video data in which the material video data is edited based on the automatic editing information (S7). The automatically edited video data and the automatically edited information are stored in the recording device 12 (S8). In this case, the video data stored in the recording device 12 is only the automatically edited video data after editing, or when the material video data is stored in the recording device 12, the material video data is Replaced with automatically edited video data.

When storing the material video data (Yes in S6), the system control unit 15 stores the material video data and the automatic editing information in the recording device 12 (S9), and then whether or not to perform automatic editing for the user. Is confirmed (S10). When automatic editing is not performed (No in S10), the process ends. In this case, the recording device 12 stores the material video data before editing and the automatic editing information. Therefore, although the automatically edited video data does not exist at this point, the automatically edited video data can be easily created later by using the editing device 14.

When performing automatic editing (Yes in S10), the system control unit 15 causes the editing device 14 to create the automatically edited video data (S11), and stores the automatically edited video data in the recording device 12 (S12). In this case, the recording device 12 stores all of the original material video data, automatic editing information, and automatically edited video data. Therefore, for example, when a plurality of processing target information is set as described above, it becomes easy to perform processing using other processing target information on the same material video data later.

When the recording device 12 directly receives the material video data from the recording device 11 and stores it, the above steps S6 to S8 are unnecessary. However, the material video data may be deleted after the automatically edited video data is stored (S12).

In the flowchart of FIG. 6, the operation of the system control unit 15 after the input of the material video data is shown. On the other hand, there is a video distribution (output) request to the system control unit 15 in a state where the material video data is stored in the recording device 12 in advance, and the video data after the material video data is edited in response to the request. It may be output.

FIG. 7 is a flowchart showing an example of the operation of the system control unit 15 in such a case. Here, it is assumed that at least the material video data is stored in the recording device 12.

First, when there is a distribution request (S21), the system control unit 15 confirms whether or not the automatically edited video data is stored in the recording device 12 (S22). When the automatically edited video data is not stored (No in S22), it is confirmed whether or not the automatically edited information is stored (S23).

When the automatic editing information exists (Yes in S23), the system control unit 15 creates the automatically edited video data using the editing device 14 as described above, and stores the automatically edited video data in the recording device 12 (S24). When the automatic editing information does not exist (No in S23), the system control unit 15 creates the automatic editing information using the automatic editing information creation device 13 (S25), and similarly uses the editing device 14 to create the automatically edited video. Data is created and stored in the recording device 12 (S24). As a result, when the automatically edited video data is not stored (No in S22), the automatically edited video data is newly created and stored in the recording device 12.

When the automatically edited video data is stored (Yes in S22), or when a new automatically edited video data is created and stored as described above (S24), the system control unit 15 is automatically edited. An image based on the video data is displayed on the editing device 14 (display unit 142) (S26), and the user is inquired as to whether or not the content may be distributed (S27).

When the content may be distributed (Yes in S27), the automatically edited video data is set as the edited video data for which distribution is permitted (S28). On the other hand, when it is desired to change from this content (No in S27), the system control unit 15 further edits the automatically edited video data using the editing device 14 (S29), and the edited video data is displayed. It is set as the edited video data for which distribution is permitted, and stored in the recording device 12 (S30). At this time, as described above, the final editing information is also created and stored.

After that, the system control unit 15 distributes the edited video data stored in the recording device 12 as described above (S31).

Further, even if the automatic editing information is created, it takes time to apply the automatic editing information to create the automatically edited video data, and it also takes time to store various video data in the recording device 12. Therefore, it is preferable not to store or create video data that is clearly unnecessary for distribution. Further, in order to shorten the processing time, the user may check the video at the same time using another device.

FIG. 8 is an example of a flowchart showing the operation of the system control unit 15 in consideration of these points.

Here, when the recording device 11 obtains the material video data (S41), it is determined whether or not the material video data is stored in the recording device 12 as it is (S42). When it is recognized that the storage of the material video data is unnecessary (No in S42), the automatic editing process is performed as described above to create the automatically edited video data (S43), and the automatically edited video data is used. It is stored in the recording device 12 as video data for distribution (S44). In this case, the video data recorded in the recording device 12 is only the automatically edited video data.

When it is recognized that the material video data is to be stored (Yes in S42), the material video data is stored in the recording device 12 (S45). After that, an inquiry is made as to whether or not to analyze the material video data using another device (S46). When the material video data is analyzed using another device (Yes in S46), the user analyzes the material video data using the other device (S47), and then the subsequent processing using the editing device 14. Can be started. This analysis result can be used for the following determinations (S50, S56).

After that, an inquiry is made as to whether or not to perform automatic editing immediately (S48), and if it is not performed immediately (No in S48), automatic editing information is created by the automatic editing information creation device 13 (S49), and then the editing device 14 , Inquire whether the content of this automatic editing information is acceptable (S50).

When making this inquiry, the automatically edited video data is not actually created, but the user can check the edited contents based on this automatically edited information at a certain point in time as described above. It is possible to confirm this using a still image.

When it is desired to change this content (No in S50), the editing device 14 causes the user to make the correction (S51). After that, when there is no change in the content (Yes in S50), the edited video data in which the actual editing work for the material video data is performed is created based on the automatic editing information as it is (S52). This edited video data is stored in the recording device 12 as video data for distribution (S53). In this case, the edited video data is not created until the content is finally finalized.

When the automatic editing is performed immediately (Yes in S48), the automatic editing information and the automatically edited video data based on the automatic editing information are immediately created (S54), and the automatically edited video data is displayed on the display unit 142 (S55). In this case, the user can confirm in detail whether or not the edited content is appropriate at all points in the automatically edited video data (S56).

After that, when it is desired to correct the edited content (No in S56), the correction work and confirmation are performed in the same manner as described above (S57), and then new video data (No) based on the corrected edited information is performed again. Edited video data) is created (S58), and the edited video data is stored in the recording device 12 as video data for distribution (S59). The final editing information created at this time is also stored at the same time.

When it is confirmed that the editing based on the automatic editing information is appropriate (Yes in S56), the already created automatically edited video data is stored in the recording device 12 as the video data for distribution (S60). ..

In the above operation, the processing time is shortened by keeping the actual editing process on the material video data to the minimum necessary, and the user checks whether the editing process is appropriate or not. Can be done, and the correction is also made.

Next, the processing target information stored in the information storage unit 132 in order to recognize the processing target portion of the material video data will be described. The parts to be processed by such processing include the time display as described above, the registration number of the reflected automobile, the company name, the face of the reflected person, and the like. The time display and the registration number can be recognized by recognizing the numbers by pattern recognition, the company name can be recognized by recognizing the character pattern, and the face can also be recognized by the pattern recognition method.

As described above, in the above-mentioned editing device 14, the user can set the target portion of such processing by operating the operation panel 143 by himself / herself, and then the final editing information reflecting this operation is displayed. Created. In this case, the processing target recognition unit 131 may recognize the final editing information and update (or create) the processing target information. In this case, the processing target recognition unit 131 functions as a processing target information modification means for updating the processing target information to more preferable contents.

FIG. 9 is a diagram showing the flow of such an operation in the processing target recognition unit 131.
First, the processing target information in the initial state (initial setting) is created by the user (P1). Here, for example, only the minimum necessary processing target and relatively easy to recognize are selected as the target. Further, for example, the time display in the image as described above can be such an object. Using this processing target information, the video editing system 1 is repeatedly used as described above. At this time, as described above, in addition to the editing by the automatic editing information, or instead of this editing, the editing work is also performed by the user, and finally the final editing information applied to the material video data is created. This final editing information is also stored in the information storage unit 132.

Therefore, the processing target recognition unit 131 becomes the basis of the automatic editing information and the information storage unit 132 by comparing the automatic editing information which is the basis of the automatically edited video data with the final editing information generated after that. The processing target information stored in can be modified. For example, a certain character string in the image was not included in the processing target in the processing target information, so that it was not processed in the automatic editing information, but it was later specified by the user and processed in the final editing information. When it is targeted, the processing target information can be modified so that this character string is added as the processing target. On the contrary, since a certain character string in the image was included in the processing target in the processing target information, it was targeted for processing in the automatic editing information, but this designation was later canceled by the user and in the final editing information. If it is not the target of processing, the processing target information can be modified so that this character string is deleted from the processing target. The processing content (blurring processing, etc.) in the processing target information can also be modified in the same manner. Such work can be performed in the same manner even when a specific person is targeted for processing by using face recognition in an image, for example.

Further, the processing target recognition unit 131 uses the processing target information based on the statistical processing of the plurality of recorded final editing information without using a simple determination as to whether or not the processing target is selected in this way. It can also be modified. At this time, for example, it is possible to numerically evaluate each of the differences between the final editing information and the automatic editing information, calculate the total of the numerical values as a score, and modify the processing target information based on this numerical value. For example, it is possible to extract the final editing information with a large score (large difference), and newly incorporate the information that was treated as a common processing target and was not included in the processing target information. can.

Therefore, in the flow of FIG. 9, the video editing system 1 is used using the processing target information (P1) in the initial state, and at this time, the final editing information is created by the user's operation, and the information storage unit 132. It is stored in (P2). After that, as described above, the difference between the final editing information and the automatic editing information is quantified and evaluated (P3). Based on this numerical value, as a comprehensive analysis, it is determined whether it is preferable to rewrite the current processing target information or how to rewrite it (P4), and finally the processing target information is updated (P5). Here, as shown in FIG. 9, in the final determination (P4), not only the difference between the final editing information and the automatic editing information as described above, but also the tendency of the editing work by the user in the editing device 14 (for example,). Some users have a lot of editing work, and some other users have a little editing work), and conditions for images newly added due to circumstances newly generated after the initial setting (P1) can be taken into consideration.

Such modification work of the processing target information may be repeated every time the video editing system 1 is used to create the final editing information, or may be performed periodically. Further, when the above points are used, it may be performed according to the cumulative value of the points.

In this way, the work of modifying the processing target information based on a large amount of final editing information can also be performed by using a well-known machine learning method (deep learning) or the like. As described above, when a plurality of processing target information is set according to the video distribution destination and the like, these operations can also be performed for each processing target information.

Inquiries and inputs for setting the user himself / herself to input the evaluation of the automatically edited video data can be performed by using the display unit 142 and the operation panel 143 (touch panel display 144) in the editing device 14.

FIG. 10 is an example of such a display. Here, in the display K, the automatic editing information (each of the parts to be processed and the content of the processing in each) is explained and whether or not the application is possible is performed, and in the upper display L, the automatic editing information at this time is described. The rating is entered by the user. After that, by operating the lower display M, an editing process using the final editing information in which the automatic editing information is modified to reflect the operation of the display K is executed.

There are various types of material video data, and in some cases, processing is applied to uncommon special parts. In such a case, even if the difference between the automatic editing information and the final editing information becomes large, it is preferable that the final editing information in this case is not used for modifying the generally used processing target information. As shown in FIG. 10, when it is selected not to evaluate the automatic editing information in this case, the final editing information in such a special case is not used for modifying the processing target information.

As described above, various methods other than the above can be applied as a method of feeding back the newly created final editing information and updating the processing target information.

Further, for example, the portion that can be the target of the above processing is the face of the reflected person, and the processing target recognition unit 131 can recognize the face in the image. Here, for example, when there are a plurality of reflected persons and it is desired to apply the processing only to the face of a specific person, or conversely, when it is desired to apply the processing to the faces of all persons other than this specific person. be. In such a case, if the face of the person is set to the above-mentioned first level in the processing target information, only a warning is issued as in the case of the above-mentioned broadcast prohibited term, and unless this warning is canceled, automatic editing is performed. The configuration may be such that the finished video data is not created and the material video data is not distributed. After that, the user finally controls the operation panel 143 so that the processing is performed only on the face of a specific person among all the reflected faces, or conversely, the processing is not performed only on the face of a specific person. After creating the edited information and creating the edited video data according to the final edited information, it can be distributed.

In the above configuration, the recording device 12 (video server) is provided with a processing target recognition unit (processing target recognition means, processing target information modification means) 131, and an information storage unit (information storage means) 132. The creation device 13 was connected to an editing device 14 including an editing control unit (editing means) 141, a display unit (display means) 142, and an operation panel (operating means) 143, and the above operation was performed. However, processing target recognition means, processing target information modification means, information storage means, editing means, display means, etc. having the same functions as described above are provided in relation to the material video data, and the automatically edited video data, the automatic editing information, etc. As long as the final editing information and the like can be created, the specific configuration of the device is arbitrary. That is, how each of the above means is provided in each device used is arbitrary, and all of the above means may be provided in a single device.

Next, a similar person search process (particularly a similar face detection process) will be described with reference to FIGS. 11 to 18. This process is executed by the functions of the similar face image detection device 16 and the editing device 14 (particularly the similar face image detection operation unit 103), and the technique disclosed in Japanese Patent Application Laid-Open No. 2013-101431 is used for face image recognition processing. It is applied to. In the following, the main parts disclosed will be illustrated.

11 (a) to 11 (g) exemplify the feature amount of the image that is a candidate for the search key image according to the procedure for performing the similar person search in this embodiment. FIG. 12 illustrates a procedure for performing a similar person search (similar face detection process).

First, in the search process 6001 using the first key image, the first search is performed by the first search key image selected by the user. Here, an image having a feature amount close to the feature amount of the image selected as the first search key image (in this example, the feature amount of the person in the image) is transmitted through the similar person search unit 218 in the recording device 12. As a result of the search, for example, 10 images are searched.

In FIG. 11A, the feature amount of the first search key image is indicated by “◯”. Here, the feature amount of the image is expressed in two dimensions for the sake of clarity of explanation, but in reality, the feature amount of the image often has a very large number of dimensions such as several hundred dimensions. ..

Here, it is assumed that 3 out of 10 images that are the search results are the same target as the first search key image. In the process 6002 of selecting the same person from the search results, three target images are selected from the ten search result images. Specifically, for example, the user operates the operation panel 143 of the editing device 14 or a mouse (not shown) to select a target image. A threshold value is set for the feature amount of the image, and if the distance between the feature amount of the first search key image and the feature amount of the search result image is equal to or less than the threshold value, it is judged that they are the same target (same person), and the corresponding search result is applied. It may be a method of automatically selecting an image.

In FIG. 11B, in addition to the contents of FIG. 11A, the feature amount of the image selected by the process 6002 for selecting the same person from the search results is indicated by “Δ”. The image selected by such processing becomes a candidate for a new search key image.

Here, if the search result image is one of the continuous images forming the moving image, the image of the same person is often included before and after the search result image in the moving image. In the process 6003 of selecting the same person before and after the search result, the position, progress speed, etc. of the person are selected from a plurality of images included in a predetermined length of time before or after the search result image in the video from which the search result image is extracted. The image determined to be the same person as the person in the search result image (that is, the same person as the person in the search key image) is automatically selected based on. The user may be able to specify it.

In FIG. 11 (c), in addition to the contents of FIG. 11 (b), the feature amount of the image selected by the process 6003 for selecting the same person before and after the search result is indicated by “□”. The image selected by such processing becomes a candidate for a new search key image.

In the image processing 6004 to add a mask, an image with a mask covering the nose and mouth is generated by image processing for the image of the person who is a candidate for a new search key image in the previous processing, and a new search key is added. Performs processing to add to image candidates. On the contrary, when the image of the person before the image processing has a mask covering the nose and mouth, a process of removing the mask by the image processing may be performed. Further, a plurality of types of mask images may be prepared.

In the image processing 6005 to add sunglasses and glasses, an image to which sunglasses and glasses are added is generated by image processing for the image of the person who is a candidate for a new search key image in the previous processing, and a new search key image is generated. Performs the process of adding to the candidates of. On the contrary, when the image of the person before the image processing is wearing sunglasses or glasses, a process of removing the sunglasses or glasses may be performed by the image processing. Further, a plurality of types of images may be prepared for the images of sunglasses and eyeglasses.

In the image processing 6006 for changing the direction of a person, an image in which the direction of the person is changed by image processing is generated for the image of the person who is a candidate for a new search key image in the previous processing, and a new search key image is generated. Performs the process of adding to the candidates of. The orientation of the person is usually plural, but it may be a simple left-right reversal.

In addition to the contents of FIG. 11C, FIG. 11D is generated as a result of image processing 6004 for adding a mask, image processing 6005 for adding sunglasses and eyeglasses, and image processing 6006 for changing the orientation of a person. The feature quantities of a plurality of images are indicated by "x". The image generated by such processing is added as a candidate for a new search key image.

The image processing 6004 for adding a mask, the image processing 6005 for adding sunglasses and glasses, and the image processing 6006 for changing the direction of a person are the results of the first search key image and the process 6002 for selecting the same person from the search results. It may be applied to either the image or the image of the result of the process 6003 for selecting the same person before and after the search result. Further, any one image processing may be performed on the target image, any two image processings may be performed, or all three image processings may be performed. Further, image processing other than the above may be performed, such as image processing for changing the brightness of the target image.

Next, in the clustering process 6007, a plurality of images that have been candidates for the search key image by the previous processes 6001 to 6006 are clustered, and an image (or a feature amount thereof) representing each cluster is obtained. As the clustering method, a known technique such as the k-means method can be used. As an image representing each cluster, for example, an image closest to the average of the feature amounts of the images included in the cluster is used, and the feature amount of the image is used as a new search key. The average of the feature amounts of the images included in the cluster may be used as a new search key as it is.

FIG. 11 (e) illustrates how the new search key image candidates obtained by the previous processes 6001 to 6006 are divided into clusters by the clustering process 6007, and the feature amount of the image representing each cluster. It is done. In FIG. 11 (e), three clusters are surrounded by a frame, and the feature amounts P11, P12, and P13 of the image closest to the center of gravity of each cluster are selected as the feature amounts of the images representing each cluster. Has been done.

In the search process 6008 using the representative search key, a similar image search is performed using the feature amount of the image representing each cluster obtained by the clustering process 6007 as a new search key, and the result is output.

Here, in the example of FIG. 11 (e), since there are 29 images (images obtained by processing 6001 to 6006) related to the first search key image, conventionally, the feature amounts of these images are used. When the search using the new search key was repeated 29 times, in this embodiment, a similar face image search was performed using the feature quantities of the three images representing each cluster obtained by the clustering process 6007. , I try to search only 3 times while balancing the features. Here, the number of clusters is set to 3, but this can be changed by setting.

Next, the screen of the similar face image detection operation unit 103 of the editing device 14 will be described with reference to FIG. FIG. 13 illustrates a search screen that can be used in the similar face image search system of this example.

The search screen includes a reproduced image display area 3001, an image reproduction operation area 3003, a search key image designation area 3004, a search narrowing parameter designation area 3008, a search execution area 3017, and a search result display area 3020.

The reproduced image display area 3001 is an area for displaying an image recorded on the recording device 12 (similar face image detecting device 16 or) as a moving image. Further, the moving image 3002 in the reproduced image display area 3001 displays the image recorded in the recording device 12 as a moving image.

The image reproduction operation area 3003 is an area for reproducing an image recorded in the recording device 12. A unique reproduction type is assigned to each button constituting the area 3003. In this figure, an example is shown in which rewind, reverse playback, playback stop, forward playback, and fast forward playback types are assigned in order from the left. When the user appropriately presses each button with the mouse 282, the moving image 3002 is switched to the playback type assigned to the button.

The search key image designation area 3004 is an area for designating and displaying the search key image. This area 3004 has a search key image 3005, a video designation button 3006, and a file designation button 3007.

The search key image 3005 is an image used as the first search key image for a similar search. In the initial state, since the search key image is not specified, the image is not displayed. If it is not specified, it may be indicated that it is not specified, such as displaying a separately prepared image showing the unspecified state.

The video designation button 3006 is a button for designating the image displayed in the reproduced image display area 3001 when pressed as the search key image 3005.

The file designation button 3007 is a button for designating an image other than the image recorded in the recording device 12, such as an image taken by a digital still camera or an image captured by a scanner, as a search key image 3005. When this button 3007 is pressed, a dialog box for specifying files of those images is displayed, in which the user can specify desired images.

The search narrowing parameter specification area 3008 is an area for designating the type of the narrowing parameter and its value (range) at the time of searching. This area 3008 has image pickup device designation check boxes 3009, 3010, 3011, 3012, time code designation check boxes 3013, 3014, and time code designation fields 3015, 3016.

The image pickup device designation check boxes 3009, 3010, 3011, and 3012 are check boxes for designating the image pickup device (camera 10 and the like) to be searched at the time of search. When the check boxes 3009, 3010, 3011, and 3012 are pressed, check marks indicating that they have been selected are displayed. When this mark is pressed again, it disappears, and when it is pressed, it is repeatedly displayed and hidden.

The time code designation check boxes 3013 and 3014 are check boxes for designating the time range to be searched at the time of search. This check box is the same as other check boxes in terms of display mode. When the time code specification check box 3013 is selected, the start time is given to the time range. When the non-selected state is set, it means that the start time is not given to the time range, that is, the image up to the oldest time recorded in the recording device 12 is set as the search target range.

Similarly, when the time code specification check box 3014 is selected, the last time is given to the time range. When the non-selected state is set, it means that the last time is not given to the time range, that is, the search target range is up to the image of the latest time recorded in the recording device 12.

The time code designation fields 3015 and 3016 are input fields for designating the above-mentioned start time and end time values. In the initial state, since all time zones are searched, the time code designation check boxes 3013 and 3014 are all unselected states, and the time code designation fields 3015 and 3016 are left blank.

The search execution area 3017 is an area for instructing the search execution. In this area 3017, in addition to the similar person search button 3018 and the appearance event search button 3019, the similar person search button 3300 from the search results, the same scene check box 3201, the mask check box 3202, the sunglasses check box 3203, and the different angle check box 3204 Have.

The similar person search button 3018 is a button for instructing the execution of the similar person search (search process 6001 by the first key image) by the search key image 3005. When the parameter is specified in the search narrowing parameter specification area 3008, the execution of the similar person search is instructed according to the specified parameter.

The appearance event search button 3019 is a button for instructing the execution of the appearance event search. When a parameter is specified in the search narrowing parameter specification area 3008, the execution of the appearance event search is instructed according to the specified parameter.

The search result display area 3020 is an area for displaying search results. The search results are displayed by displaying a list of search result images. In the initial state, nothing is displayed in the search result display area 3020.

Here, the user presses the image designation button 3006, the imaging device designation check boxes 3009, 3010, 3012, and further presses the time code designation check boxes 3013, 3014, and the time code designation fields 3015, It is assumed that "15:30:20:17" and "12:30:20:17" are input to 3016, respectively.

As a result, as shown in FIG. 13, the image of the person "Mr. A" displayed in the moving image 3002 is designated as the search key image in the search key image 3005, and the "camera" is used as the image pickup device 201 to be searched. "1", "Camera 2", and "Camera 4" are specified, and "from 15:30:20:17 to 12:30:20:17" is specified as the time range to be searched.

After that, it is assumed that the user presses the similar person search button 3018. Then, in the search result display area 3020, the search result obtained by executing the similar person search using the search key image 3005 is displayed. FIG. 13 shows an example of the search screen in this state. The search result is displayed by displaying a list of search result images (search result images 3033-1141 in this example).

The search result images 3033-1141 are displayed, for example, from the top left to the right and then from the second left to the right in order of similarity to the search key image 3005. In this display example, the search result image 3031 has the highest degree of similarity to the search key image 3005, and the search result image 3141 has the lowest degree of similarity.

In the notation of the example shown in this figure, the circles and alphabets shown on the search result images 3033-1141 in the search result display area 3020 are simplified displays of a person's face and person's name, for example, a search. The result image 3031 shows that the person "Mr. A" appears. Of course, in this simplified display part, the actual image is displayed in the display in the actual system.

A cue playback button 3032, a search key image designation button 3033, and a search target check box 3301 are provided around the search result image 3031. The same applies to the other search result images 3041 to 141.

The cueing playback button 3032 is a button for instructing the start of continuous moving image playback starting with the search result image 3031. For example, when the cue playback button 3032 is pressed, the moving image 3002 is switched to the search result image 3031, and the user can watch the moving image starting with the search result image 3031.

The search key image designation button 3033 is a button for designating the search result image 3031 as a new search key image. For example, when the search key image designation button 3033 is pressed, the search result image 3031 is displayed on the search key image 3005. As a result, the search result image 3031 can be used for re-searching.

The search target check box 3301 is a check box for designating the search result image 3031 as a new search key image (or a candidate thereof) when the similar person search button 3300 from the search result is pressed. For example, all the images of "Mr. A" appearing in the search results (in this example, the search result images 2031 to 3061, 3081, 3091, 3121, 3141) are checked, and the similar person search button 3300 from the search results is pressed. By pressing, it is possible to search for "Mr. A" in various patterns.

The similar person search button 3300 from the search result is a button for instructing the execution of another similar person search (search process 6008 by the representative search key) based on the result of the similar person search by the search key image 3005. In the similar person search again, the image selected by the user (the search target check box is checked) from the display of the search result display area 3020 (the result of the search process 6001 by the first key image) is a new search key. Re-execute the similar person search as an image (or its candidate).

The same scene check box 3201 executes the process 6003 of selecting the same person before and after the search result for the image selected by the user from the display of the search result display area 3020, and the resulting image (target image). This is a check box that specifies that images before and after the same person as the person inside) should be added as new search key image candidates.

The mask check box 3202 executes image processing 6004 for adding a mask to an image selected by the user from the display of the search result display area 3020, and the resulting image (a person in the target image). This is a check box that specifies that an image with a mask added to the image or an image with the mask removed from the person concerned) is added as a candidate for a new search key image.

Further, the sunglasses check box 3203 executes image processing 6005 for adding sunglasses and eyeglasses to an image selected by the user from the display of the search result display area 3020, and the resulting image (in the target image). This is a check box for designating an image in which sunglasses or the like are added to the person in question or an image in which sunglasses or the like are removed from the person in question) as a candidate for a new search key image.

The different angle check box 3204 executes image processing 6006 for changing the direction of a person for an image selected by the user from the display of the search result display area 3020, and the resulting image (in the target image). This is a check box that specifies that an image with a changed person's orientation) should be added as a new search key image candidate.

When one or more of these check boxes 3201 to 204 are checked and the similar person search button 3300 from the search results is pressed, each of the displays selected by the user from the display of the search result display area 3020. For images, perform image processing corresponding to the checked check box, add a new search key image candidate to the resulting image, and then make it a new search key image candidate. On the other hand, the clustering process 6007 is executed to obtain a search key image representing each cluster, and a similar image search is executed using the feature amount of the image representing each cluster as a search key.

As described above, in the above example, the search key determining means for determining the feature amount of the image to be the search key based on the feature amounts of a plurality of images that are candidates for the search key, and the search key determining means by the search key determining means. In a configuration including a search means for searching an image having a feature amount similar to the feature amount of the image determined in, the search key determination means clusters the feature amounts of a plurality of images that are candidates for the search key. , The feature amount of the image representing the cluster is determined for each cluster as a search key, and the search means is configured to perform the search using each of the search keys for each cluster determined by the search key determining means.

In this example, the search key determination means is realized by the function of the multiple search key selection unit 113 of the similar face image detection operation unit 103 of the editing device 14, and the function of the similar person search unit 218 of the similar face image detection device 16 is used. Although the search means is realized, the search key determination means and the search means may be realized by other aspects.

Next, a processing example in which the above-mentioned similar person search processing (similar face detection processing) is applied to the editing processing will be described with reference to FIGS. 14 to 16.

As described above, in the conventional flow of searching for a performer's appearance scene (appearance video), the person in charge (editor, etc.) searches for the performer's information on the management terminal, and the performer finds out. A list of programs appearing and VTR tape numbers in which the programs are recorded is displayed. After that, the person in charge takes out the VTR tape having the output tape number from the shelf and puts it on a VTR player to play it. Then, the appearance scene was searched by visually observing the reproduced video, and the time code information of the appearance scene was recorded. In such a flow, improvement was required from the viewpoint of work efficiency and accuracy. Therefore, we will introduce the following flow technology.

FIG. 14 shows a procedure for accumulating a face image on the face image storage server 127 when the original image is recorded on the media 5 (optical media 5a, magnetic media 5b, and VTR tape 5c). The procedure for finding the media 5 is the same as before.

When the original video is recorded on the optical media 5a or the magnetic media 5b, the video file is taken out from the found media (optical media 5a or magnetic media 5b), the video file is played back by the similar face image detection device 16, and the video file is played back. Using the above-mentioned similar person search processing technique, only the image of the face portion is cut out from the reproduced video, and the cut out face image is stored in the face image storage server 127 together with the time code information.

The accumulated face images are not limited to one type (generally the front face), but multiple face types (front, profile, diagonal face, back face, laughing face, angry face, etc.) are used as detection target face images. It can be registered and saved, and the face image and its face type are associated and recorded. By preparing a plurality of face images to be detected, particularly a plurality of face images of different types, it is possible to more accurately detect a video in which a specific performer is appearing, and a specific face image can be detected. It is possible to detect the situation that is particularly desired among the images of the performers (such as wanting an image of a laughing face). Further, when the name of the performer can be specified at the time of accumulating the face image, that name may also be registered. Further, when a plurality of face images of the same performer are recorded on the face image storage server 127, a reference face image (reference face image) may be specified. The reference face image is not limited to one, but from the viewpoint of workability, it can be set to one for each face type or one at a predetermined appearance time (for example, 5 years).

When the original video is recorded on the VTR tape 5c, the found VTR tape 5c is reproduced by the VTR playback device (media playback device 19) and captured in the similar face image detection device 16. The similar face image detection device 16 cuts out only the image of the face portion from the captured reproduced image by using the technique of similar person search processing as in the case of the optical media 5a and the magnetic media 5b, and cuts out the face image. Is saved in the face image storage server 127 together with the time code information.

FIG. 15 shows a procedure for accumulating a face image on the face image storage server 127 when the original video is recorded on the low resolution server 126.

When the original video is recorded on the low resolution server 126, when the person in charge searches for the performer's information on the management terminal 17, the program in which the performer is appearing and the program in which the performer is recorded are recorded. The video file name in the resolution server 126 is output. The information is passed online as it is, that is, to the similar face image detecting device 16 via the network 2. As a result, the video file is taken out from the low resolution server 126, and the video file is played back by the similar face image detection device 16 to cut out only the image of the face part from the reproduced video, and the cut out face image is used as the face image together with the time code information. Save to the storage server 127.

FIG. 16 is a diagram in which a similar face is detected from the face image storage server 127 with the face image of the target performer as the detection target.

The editor loads the face image file (detection target face image) of the target performer into the similar face image detection device 16. The detection target face image may be an image extracted as a representative face image from the video file to be edited, a face image selected from the face images included in the face image storage server 127, or an image on the web. It may be an image captured from. The similar face image detection device 16 compares the face image to be detected with the face image in the face image storage server 127, and searches for the face image of the scene in which the performer of the same face appears and the time code information.

The time code information of the appearance scene detected here is passed to the editing device 14. The editor does not have to find out the scene in which the target performer is appearing, and can create a special program of the performer or apply a mosaic to the performer.

Further, when it is desired to preview the detected appearance scene without using the editing device 14, the appearance scene preview can be easily performed by playing back the video file in the low resolution server 126.

By performing such similar face detection processing, for example, when a scene in which a target performer is appearing is searched for from a huge past image of a broadcasting station, the similar face image detection device 16 automatically appears. It detects the scene. As a result, the editor does not need to watch the image of the media 5 (optical media 5a, magnetic media 5b, VTR tape 5c). In the meantime, the editor can do other work, which can greatly improve the work efficiency of the editor.

Further, since the number of editing devices 14 is limited, if the editing device 14 cannot be used, the appearance scene of the target performer is searched for in advance, and the appearance scene is preliminarily used by using the video file of the low resolution server 126. By previewing it in, it is possible to perform pre-work before editing.

In addition, if the performer's broadcast becomes impossible due to a problem caused by the performer immediately before the broadcast after completing the program editing, the above-mentioned technology can be used to easily change the appearance scene of the target performer. It is possible to search for and apply a mosaic to the performer, or to cut the appearance scene, which prevents complaints from sponsors and viewers.

In the above process, performers are detected from the past images of the broadcasting station. However, since the face of the performer changes several decades after the video is recorded, if the current face image of the target performer is targeted for detection, there is a high possibility that the detection accuracy will drop. In order to solve this problem, the face image of the result of detecting the current face image as the detection target (past face image with reduced detection accuracy) is replaced with a new face image to be detected, and the face image is newly re-registered. The detection accuracy can be improved by detecting the similar face image again. That is, it is expected that the detection accuracy will be improved by the two-step detection (re-registration of new information of the reference face (reference face image) → search for similar face image).

Then, by passing the time code information of the detected appearance video to the editing machine, the editor can apply a mosaic to the video in which the performer is appearing or cut the appearance video.

In addition, if the face image has a similar appearance time (shooting time), it is considered that the same feature amount appears. Therefore, it is possible to detect the face image whose appearance time is open while tracing the same feature amount. .. Similarly, if a profile image is desired, the profile can be re-registered as a face image to be detected and a similar face image can be detected, so that a narrower appearance scene can be detected.

Then, by passing the time code information of the detected appearance scene to the editing machine, the editor can create a special program that summarizes only the scene in which the performer is appearing.

In the current broadcasting station, the edited video (edited video) is recorded on the optical media 5a and broadcasted by the optical media 5a, or the optical media 5a is taken into the transmission server 18 and broadcast output. There are many. Therefore, the video file is taken out from the optical media 5a, the video file is reproduced in the similar face image detection device 16 (similar face detection device), only the face image is cut out from the reproduced video, and the cut out face image is combined with the time code for the face. By storing the image in the image storage server 127 and detecting similar faces by using the face image of the target performer as a detection target, the person in charge can search for the appearance scene without visually observing the image. Here, by preparing a plurality of face images to be detected, such as a front face, a profile face, and an oblique face, it is possible to more accurately detect the scene in which the performer to be searched for appears.

The present invention has been described above based on the embodiments. This embodiment is an example, and it will be understood by those skilled in the art that various modifications are possible for the combination of each of these components, and that such modifications are also within the scope of the present invention.

1 Video editing system 2 Network 5 Media 5a Optical media 5b Magnetic media 5c VTR tape 10 Camera 11 Recording device 12 Recording device 13 Automatic editing information creation device 14 Editing device 15 System control unit 16 Similar face image detection device 17 Management terminal 18 Sending server 19 Media playback device 103 Similar face image detection operation unit 110 Keyword recording unit 111 Keyword search unit 112 Keyword assignment request transmission unit 113 Multiple search key selection unit 121 Material video data unit 122 Edited video data unit 123 Automatically edited video data unit 124 Final editing information unit 125 Automatic editing information unit 126 Low resolution server 127 Face image storage server 128 Processing target information unit 131 Processing target recognition unit 132 Information storage unit 141 Editing control unit 142 Display unit 143 Operation panel 144 Touch panel display 210 Image transmission / reception unit 211 Image recording unit 212 Playback control unit 213 Person area detection unit 214 Person feature amount extraction unit 215 Person feature amount recording unit 216 Attribute information recording unit 217 Request reception unit 218 Similar person search unit 219 Appearance event search unit 220 Search result transmission unit 221 Search Request transmission unit 222 Search result reception unit 223 Search result display unit 224 Playback image display unit 225 Screen operation detection unit

Claims (4)

An editing system equipped with an editing device that edits video files used for broadcasting.
A face image storage server that acquires the face image of the performer included in the video file and records the face image in association with the time code information of the appearance video of each performer.
A face image detection unit that compares the face image recorded in the face image storage server with the face image to be searched included in the video file of the specific program, and detects the appearance video in the specific program.
Based on the appearance video detected by the appearance video detection unit, the face image of the appearance video other than the specific program in which the person of the face image to be searched in the specific program appears, and the face image of the person in the specific program. The specific program in which a person with a face image to be searched appears by a similar face image search that determines that the person is the same when the distance between the feature amount and the face image is closer than a preset threshold. It is provided with a similar face image detection device that detects appearance videos other than the above, associates the time code information of the detected appearance video with the performer information to be searched, and notifies the editing device.
The editing device is an editing system characterized by editing a video file of the specific program using the time code information. When editing the video file, the editing device reproduces the video file of the recording device if the video file of the performer is stored in the recording device to be processed by the similar face image detection device. The editing system according to claim 1, wherein the video confirmation of the appearance video is possible. The editing system according to claim 1 or 2, wherein the editing device reproduces an appearance video detected by using a low-resolution video. The face image storage server can save the face image to be detected in association with the face type.
The editing system according to any one of claims 1 to 3, wherein the similar face image detection device performs a similar face image search according to the face type.

Images