JP5994612B2 - Video editing apparatus, video editing method, and video editing program - Google Patents

Description

  The present disclosure relates to a video editing apparatus, a video editing method, and a video editing program for generating a summary video.

  In some cases, for example, a summary video may be created by cutting out and collecting a part in which a moving process of an object such as a person or a vehicle is captured from a video obtained by a surveillance camera or the like.

  As a technique for creating a summary video, for example, a technique for editing so that the motion of an object does not overlap in the space-time represented by the summary video has been proposed (see Patent Document 1).

Special table 2009-516257

  By the way, for example, by analyzing how each person moves immediately after entering the store based on the video taken near the entrance of the store, there is a case where an effective arrangement of equipment in the store is examined. is there. In such a case, for example, a summary video may be created from a video taken near the entrance of a store over a period from the start to the end of business hours.

  In this case, if a summary video is created by superimposing videos of each person passing through the camera's shooting range, a summary video can be generated that presents a large number of people moving together. Can do. However, in such a summary video, if images of a large number of people are displayed in an overlapping manner, attributes such as the gender and age group of each person, the detailed actions such as the gesture of each person, and the direction of the line of sight are grasped. It becomes difficult to present information for doing so.

  On the other hand, in the method of summarizing videos so that the motions of objects do not overlap, it is not possible to generate a summary video that lists the motions of a plurality of objects in the same time series.

  An object of the present disclosure is to provide a video editing apparatus, a video editing method, and a video editing program capable of generating a summary video that collectively reproduces attributes and movement paths of a plurality of objects.

  A video editing apparatus according to one aspect receives a plurality of videos representing a moving process of each object with a predetermined location as a background, and identifies an object region including the image of each object in each image included in each video And a portion where the object regions corresponding to the objects overlap each other in each image in the summary video generated by superimposing images having the same relative time from the beginning of each video. A detection unit for detecting, and a calculation unit for obtaining a reduction rate for reducing the overlapping part by reducing the overlapping part in the respective object regions with reference to a vertex away from the overlapping part; An adjustment unit that adjusts the position of the object region so that the gravity center position of each object region reduced by the reduction ratio obtained by the calculation unit is close to the gravity center position of each object region before the reduction. Obtain.

  Further, a video editing method according to another aspect receives a plurality of videos representing a movement process of each object with a predetermined background as a background, and an object region including an image of each object in each image included in each video In each image in the summary video generated by superimposing images with the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected. Then, in each of the object regions, a reduction rate for reducing the overlapped part is obtained by reducing the vertexes away from the overlapped part as a reference, and reduction is performed at the obtained reduction rate. The position of the object area is adjusted so that the center of gravity of each object area is close to the position of the center of gravity of each object area before the reduction.

  Further, a video editing program according to another aspect receives a plurality of videos representing a moving process of each object with a predetermined place as a background, and an object region including an image of each object in each image included in each video In each image in the summary video generated by superimposing images with the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected. Then, in each of the object regions, a reduction rate for reducing the overlapped part is obtained by reducing the vertexes away from the overlapped part as a reference, and reduction is performed at the obtained reduction rate. The computer is caused to execute a process of adjusting the position of the object region so that the center of gravity of each object region is close to the center of gravity of each object region before the reduction.

  According to the video editing apparatus, the video editing method, and the video editing program of the present disclosure, it is possible to generate a summary video that collectively reproduces the attributes and movement paths of a plurality of objects.

It is a figure which shows one Embodiment of a video editing apparatus. It is a figure which shows the example of the image | video input into a video editing apparatus. It is a figure which shows the example of a partial image | video. It is a figure which shows the example of an object area | region. It is a figure which shows the example of an object information holding part. It is a figure which shows the example of the relative position of two object area | regions. It is a figure which shows the example of reduction of an object area | region. It is a figure which shows the example which adjusts the position of the object area | region after reduction by the adjustment part. It is a figure which shows the example of the object area | region superimposed on each image of the summary image | video. It is a figure which shows the example of the flowchart of a video editing process. It is a figure which shows the example which adjusted the position of the object area | region after reduction. It is a figure which shows another embodiment of a video editing apparatus. It is a figure which shows the example of similarity conversion. It is a figure which shows the example of the solution of a multi-objective optimization problem. It is a figure which shows the hardware structural example of a video editing apparatus. It is a figure which shows another example of the flowchart of a video editing process. It is a figure which shows the example of the flowchart of the process which produces | generates a constraint condition. It is a figure which shows the example of the flowchart of the process which calculates | requires an evaluation formula. It is a figure which shows the example of the flowchart of the process which calculates the parameter of similarity transformation.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows an embodiment of a video editing apparatus. The video editing apparatus 10 shown in FIG. 1 receives a video representing the movement process of each object with a predetermined location as a background from the motion video extraction apparatus 3, generates a summary video by synthesizing each received video, The generated summary video is stored in the summary video storage device 4.

  The video storage device 1 shown in FIG. 1 stores video obtained by shooting a predetermined shooting range by a shooting device 2 such as a surveillance camera. In addition, the motion video extraction device 3 extracts each video from the video stored in the video storage device 1 during a period in which some kind of object motion is captured, and inputs the extracted video to the video editing device 10.

  FIG. 2 shows an example of a video input to the video editing device 10. FIG. 2A shows an example of the shooting range VW of the video stored in the video storage device 1 shown in FIG. FIG. 2B shows an example of the video V stored in the video storage device 1.

  In the photographing range VW shown in FIG. 2A, the symbol Dr indicates an automatic door, and the symbol P1 indicates an example of a person who has entered the room from the automatic door Dr. Further, in FIG. 2A, reference numeral Tr1 indicates the movement path of the person P1 described above.

  Also, the video V shown in FIG. 2B includes a video that captures a process in which a plurality of persons and objects including the person P1 described above move with the imaging range VW shown in FIG. 2A moving as a background. It is out.

  Here, for example, in order to examine how an object including a person who has entered through the automatic door Dr illustrated in FIG. 2 moves in the imaging range VW, an image representing the movement process of each object is extracted from the image V. Consider a case where a summary video is generated by synthesizing extracted videos.

  For example, the video representing the moving process of each object is detected from the video V at the timing when the event of opening the automatic door Dr is taken, and the video from the detected timing until a predetermined time τ elapses is extracted from the video V. You can get it. In the following description, “an event that the automatic door Dr opens” is referred to as a “door event”.

  For example, when a total of L objects enter the room through the automatic door Dr during the video V shooting period, the motion video extraction device 3 shown in FIG. By extracting, L videos are acquired.

  Hereinafter, in the video V shown in FIG. 2 (B), the part of the time τ from the time Tk when the kth door event was photographed as viewed from the top of the video V is the moving process of the kth detected object Pk. This is referred to as a partial video Vpk representing For example, when the person P1 shown in FIG. 1 enters the room from the automatic door Dr at the time T1 when the above-described door event is first photographed when viewed from the top of the video V, for example, between the time T1 and the time τ. The part is a partial video Vp1 representing the movement process of the person P1. In the following description, the partial video extracted from the video V is collectively referred to as a partial video Vp as the video representing the movement process of each object.

  Each of the partial videos Vp including the partial videos Vp1 and Vpk described above is an image representing a process in which an object entering from the automatic door Dr moves with the shooting range VW as a background on the basis of the timing at which the door event occurs. It is. That is, each partial video Vp is an example of a video that corresponds to each object that moves with a predetermined location as a background, and that represents the movement process of the object with a predetermined timing as a reference. The event detected when each partial video Vp is extracted from the video V is not limited to the door event described above, and may be an event that is included in the process of moving each object. An event passing through a predetermined location may be used as an opportunity for extracting the partial video Vp.

  FIG. 3 shows an example of the partial video Vp. Note that among the elements shown in FIG. 3, the same elements as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof is omitted.

  In FIG. 3, reference symbol G <b> 1 </ b> _m indicates an image with an image number m among images included in the partial video Vp <b> 1. Similarly, a symbol Gk_m indicates an image with an image number m among images included in the partial video Vpk. Like these images G1_m and Gk_m, the images indicated by the same image number in the partial video Vp are all images taken after the same time has elapsed from the beginning of each partial video Vp.

  The partial video Vpk shown in FIG. 3 shows a case where a person Pk is photographed as an object that has entered the room from the automatic door Dr at the time of the k-th door event.

  In FIG. 3, reference sign Vs indicates an example of a summary video obtained by superimposing the partial videos Vp as described later, and reference sign Gs_m represents an image among images included in the summary video Vs. The image of number m is shown. In FIG. 3, illustration of images other than the images G1_m, Gk_m, and Gs_m is omitted.

  The image Gs_m included in the summary video Vs shown in FIG. 3 is obtained by superimposing the portions of the persons P1 and Pk extracted from the images of the image number m included in the partial videos Vp1 and Vpk, respectively, on the image representing the background. It is an example. In this way, it is possible to generate an image in which the moving object parts included in each image in each video Vp are superimposed while maintaining the position in each image, and the summary video Vs can be generated from these images. . Each image included in the summary video Vs generated in this way includes a part of an object extracted from an image having the same image number in each partial video Vp, corresponding to the position in the extraction source image. . Therefore, according to such summary video Vs, for example, it is possible to collectively represent the movement of each object including the persons P1 and Pk on the basis of the opening timing of the automatic door Dr.

  That is, by performing the above-described editing processing including superposition on each partial video Vp photographed at different timings, a summary video Vs that can list how each object including the persons P1 and Pk moves is generated. Can do. It should be noted that the length of the summary video Vs generated by the above-described editing process including superposition is a time equivalent to the length of each partial video Vp, that is, time τ, as shown in FIG.

  As shown in FIG. 1, the video editing apparatus 10 of the present disclosure, in order to realize the generation of the summary video Vs by the editing process including superposition described above, includes a specifying unit 11, an object information holding unit 12, and a detection unit. 13, a calculation unit 14, a reduction unit 15, and an adjustment unit 16. The summary video storage device 4 may hold in advance n images representing the same number of backgrounds as the images included in each partial video Vp shown in FIG. In addition, the video editing apparatus 10 performs an editing process described with reference to FIGS. 7 and 8 on an image representing an object extracted from each of the images of the partial videos Vp shown in FIG. The summary video Vs may be generated by superimposing the images on the summary video storage device 4. In the following description, an image representing n backgrounds prepared in advance in the summary video storage device 4 for generating each image included in the summary video Vs may be referred to as each image of the summary video Vs.

  The identification unit 11 illustrated in FIG. 1 receives the partial video Vp described above as a video representing the movement process of each object with a predetermined location as a background from the motion video extraction device 3 described above, and is included in each partial video Vp. In each image, an object region including an image of each object is specified. Here, each partial video Vp includes n images taken during time τ, and each of these images is an image representing n backgrounds held in the summary video storage device 4. Each is associated. For example, as illustrated in FIG. 4, the specifying unit 11 specifies a rectangular region surrounding an object image included in the image as an object region in each of n images included in each partial video Vp.

  FIG. 4 shows an example of the object region. 4 that are the same as those shown in FIGS. 2 and 3 are given the same reference numerals, and descriptions thereof are omitted.

  4, G1_1, G1_2,..., G1_m,..., G1_n−1, G1_n indicate images of image numbers 1, 2,..., M, ..., n−1, n included in the video Vp1, respectively. Yes. In FIG. 4, illustration of images other than the image G1_m is omitted. Reference sign Rm1 is a rectangular area surrounding the image of the person P1 included in the image G1_m, and is an example of the object area specified by the specifying unit 11 shown in FIG.

  Also, coordinate axes x and y shown in FIG. 4 indicate a coordinate system having the origin at the upper left vertex of the above-described image G1_m. By using such coordinate systems x and y to indicate the position of the upper left vertex Q1_m of the object region R1m and the width w1m and height h1m of the object region R1m, the specifying unit 11 can The position and size of R1m may be specified. Similarly, the specifying unit 11 specifies a rectangular area surrounding the image of the person P1 included in each image as an object area in each image included in the video Vp1. In the example of FIG. 4, the illustration of the object region specified by the specifying unit 11 in each image other than the image G1_m is omitted.

  Further, the specifying unit 11 similarly specifies a rectangular region surrounding the image of the object included in each image as the object region in the image in each image included in each partial video Vp other than the above-described Vp1. . In the following description, the object area specified in each image included in each partial video Vp by the specifying unit 11 is simply referred to as an object area.

  Further, the specifying unit 11 detects object information including information indicating the position and size of the object region specified in each image included in each partial video Vp via the object information holding unit 12 as illustrated in FIG. 13 may be passed.

  FIG. 5 shows an example of the object information holding unit 12. The object information holding unit 12 illustrated in FIG. 5 stores object information including the position and size of the object region specified in the image for each image indicated by the same image number in each of the partial videos Vp corresponding to the objects. keeping.

  In the object information holding unit 12 shown in FIG. 5, the object ID (ID: IDentifier) identifies the object whose movement process is indicated by each partial video Vp extracted by the motion video extraction device 3 shown in FIG. And corresponds to information for identifying each partial video Vp.

  In FIG. 5, for example, coordinates (X11, Y11) corresponding to the combination of the image number 1 and the object ID “P1” indicating the person P1 are the objects specified in the image of the image number 1 of the partial video Vp1. The position of the top left vertex of the area is shown. The information (w11, h11) indicating the size corresponding to the combination described above indicates the width and height of the object region described above. Similarly, in FIG. 5, coordinates (Xk1, Yk1) corresponding to the combination of the image number 1 and the object ID “Pk” indicating the person Pk are the object regions in the image of the image number 1 of the partial video Vpk. The position of the top left vertex of the position is shown. The information (wk1, hk1) indicating the size corresponding to the combination described above indicates the width and height of the object region described above.

  In FIG. 5, for example, the coordinates (X1m, Y1m) corresponding to the combination of the image number m and the object ID “P1” described above are the images of the image number m of the partial video Vp1 shown in FIG. The position of the top left vertex of the object region R1m at G1_m is shown. The information (w1m, h1m) indicating the size corresponding to the combination described above indicates the width and height of the object region R1m described above. Similarly, in FIG. 5, the coordinates (Xkm, Ykm) corresponding to the combination of the image number m and the object ID “Pk” described above are the upper left of the object region in the image of the image number m of the partial video Vpk. The position of the vertex is shown. The information (wkm, hkm) indicating the size corresponding to the combination described above indicates the width and height of the object region described above.

  Further, the object information holding unit 12 holds the image data of the object region specified in each image of each partial video Vp in correspondence with the combination of the image number and the object ID. For example, as described later, You may use for the reduction process by the reduction part 15 shown in FIG. Note that the object information holding unit 12 may hold, as the image data of each object region, for example, information that represents a portion representing the background other than the image of the object included in each object region as a transparent image.

  The detection unit 13 shown in FIG. 1 superimposes the image indicated by the corresponding image number of the summary video Vs based on the object information held corresponding to each image number in the object information holding unit 12 described above. In addition, a plurality of object regions having overlapping portions are detected.

  For example, the detection unit 13 determines whether or not the object information indicated by the two pieces of object information selected from the plurality of pieces of object information held in correspondence with each image number in the object information holding unit 12 has overlapping portions. The determination may be made as follows.

  For example, consider a case where it is determined whether or not two object regions Ra and Rb shown in FIG. 6 have overlapping portions.

  FIG. 6 shows an example of the relative positions of the two object regions Ra and Rb. In FIG. 6, the symbol Qa indicates the upper left vertex indicating the position of the object region Ra, and the symbol Qa ′ indicates the diagonal vertex of the vertex Qa. The symbol Qb indicates the upper left vertex indicating the position of the object region Rb, and the symbol Qb ′ indicates the diagonal vertex of the vertex Qb. The coordinate axes x and y indicate, for example, a coordinate system having the origin at the upper left corner of the image.

  In the example of FIG. 6, the positions of the vertex Qa and the vertex Qa ′ of the object region Ra are coordinates (xa, ya) and coordinates (xa + wa, ya + ha), respectively. Similarly, the positions of the vertex Qb and the vertex Qb ′ of the object region Rb are coordinates (xb, yb) and coordinates (xb + wb, yb + hb), respectively.

Therefore, the condition having a portion where the two object regions Ra and Rb overlap as shown in FIG. 6 can be expressed as Equation (1).
When xa <xb, xb + wb> xa + wa
When xa> xb, xb + wb <xa + wa
And when ya <yb, yb + hb> ya + ha
When ya> yb, yb + hb <ya + ha (1)
That is, the detection unit 13 illustrated in FIG. 1 calculates the above equation (1) from the pair of object information selected from the plurality of object information held in the object information holding unit 12 corresponding to each image number. By finding a satisfying pair, a pair of object regions having overlapping portions can be detected.

  The calculation unit 14 receives two pieces of object information corresponding to the pair of object regions detected by the detection unit 13, and based on these pieces of object information, the two object regions are minimized so as to minimize the area of the overlapping portion. The reduction ratio when reducing one of the vertices as a reference point is obtained.

  Based on the two pieces of object information received from the detection unit 13, the calculation unit 14 illustrated in FIG. 1 identifies the vertex that is farthest from the overlapping portion among the vertices of each object region, and the identified vertex May be selected as a reference point for reduction. In addition, for example, as illustrated in FIG. 7, the calculation unit 14 reduces the reduction rate so that there is no overlap between the two object regions due to the reduction in the direction toward the reduction reference point specified for each of the two object regions. May be calculated.

  FIG. 7 shows an example of reduction of the object area. In FIG. 7, symbols Ra and Rb indicate object regions, respectively, and a portion shaded by the symbol Rd indicates a portion where the two object regions Ra and Rb overlap. Further, the symbol Ua indicates a reduction reference point for the object region Ra, and the symbol Ub indicates a reduction reference point for the object region Rb.

  In the example shown in FIG. 7, the reference point Ua for reducing the object region Ra is the upper left vertex of the rectangle indicating the object region Ra, and corresponds to the vertex Qa shown in FIG. On the other hand, the reference point Ub for reducing the object region Rb is the lower right vertex of the rectangle indicating the object region Rb, and corresponds to the vertex Qb 'shown in FIG. In FIG. 7, coordinate axes x and y indicate, for example, a coordinate system with the upper left corner of the image as the origin.

  In FIG. 7, the thick broken line area indicated by the symbol Ras is a reduced object area Ra obtained by reducing the object area Ra with the reduction ratio αa toward the reduction reference point Ua specified for the object area Ra. Is shown. A thick solid line area indicated by reference sign Rbs indicates a reduced object area obtained by reducing the object area Rb at a reduction ratio αb toward the reduction reference point Ub specified for the object area Rb. ing.

When the object regions Ras and Rbs after reduction shown in FIG. 7 have overlapping portions, the area of the overlapping portions can be calculated by using, for example, the equation (2) using the reduction rate αa and the reduction rate αb described above. It can be expressed as In the expression (2), a symbol Sab indicates an area of a portion where the object regions Ras and Rbs after the reduction overlap. In Expression (2), symbols Xa and Ya indicate the X and Y coordinates of the reduction reference point Ua specified for the object region Ra, respectively, and symbols Xb and Yb indicate the reduction specified for the object region Rb. The X and Y coordinates of the reference point Ub are shown respectively. In Expression (2), the symbol wa indicates the width of the object region Ra before reduction, and the symbol ha indicates the height of the object region Ra before reduction. Similarly, the symbol wb indicates the width of the object region Rb before reduction, and the symbol hb indicates the height of the object region Rb before reduction. Note that the condition where the reduced object regions Ras and Rbs have overlapping portions is a range in which the positions of the above-described reduction reference points Ua and Ub both have positive values of the two factors included in Expression (2). It is indicated by the coordinate value of.
Sab = ((αa · wa + αb · wb) − | Xa−Xb |) ((αa · ha + αb · hb) − | Ya−Yb |) (2)
That is, the calculation unit 14 illustrated in FIG. 1 sets the object region pair in which the overlapping portion is detected as the object regions Ra and Rb illustrated in FIG. 7, and the above-described formula ( The reduction ratios αa and αb that minimize the area Sab represented by 2) may be calculated.

When calculating the reduction ratios αa and αb from Expression (2), the calculation unit 14 may use a technique such as an iterative calculation using a gradient method, for example. Further, the calculation unit 14 analytically solves the expression (3) simplified by setting a common reduction ratio α for the two object regions Ra and Rb instead of the expression (2), for example, You may obtain | require the reduction rate (alpha) which sets the area Sab to the numerical value 0. FIG.
Sab = (α (wa + wb) − | Xa−Xb |) (α (ha + hb) − | Ya−Yb |) (3)
Here, when the overlap is eliminated by reducing the object areas Ra and Rb with reference to the reference points Ua and Ub shown in FIG. 7, for example, the centers of gravity Ca and Cb of the object areas Ra and Rb are used as the reference. The size of each object region after the reduction can be maintained as compared with the case where the reduction is squarely performed. That is, the calculation unit 14 selects the vertex farthest from the overlapping portion among the vertices of each object region as the reference point for reduction, thereby maintaining the size of each object region after reduction. A reduction ratio that can eliminate the overlap can be obtained.

  Based on the reduction ratio calculated corresponding to each object region obtained in this way, the reduction unit 15 shown in FIG. 1 is held in the object information holding unit 12 corresponding to each object region. By executing a reduction process on the image data, an image of the reduced object region is generated.

  By the way, the positions of the centroids Cas and Cbs of the object regions Ras and Rbs after the two object regions Ra and Rb are reduced using the most distant vertex as a reference point for reduction, as shown in FIG. It is different from the centers of gravity Ca and Cb of the object areas Ra and Rb.

  In the example of FIG. 7, the center of gravity Cas of the object region Ra after reduction is moved in a direction away from the other object region Rb as compared to the center of gravity Ca of the object region Ra before reduction. Similarly, the center of gravity Cbs of the object region Rbs after reduction has moved in a direction away from the other object region Ra as compared with the center of gravity Cb of the object region Rb before reduction.

  Therefore, the image of the reduced object regions Ras and Rbs obtained by the reduction unit 15 shown in FIG. 1 is superimposed on the corresponding image of the summary video Vs based on the positions of the above-described reduction reference points Ua and Ub. When combined, the object is displayed at a position shifted from the original position in the partial video Vp.

  The video editing apparatus 10 shown in FIG. 1 includes an adjustment unit 16 in order to reduce such a shift.

  As shown in FIG. 8, the adjusting unit 16 positions the positions of the image of the reduced object regions Ras and Rbs obtained by the reducing unit 15 on the summary video Vs, as shown in FIG. Adjustment is made in a range corresponding to Rb. When the adjustment unit 16 performs such adjustment, the moving distance of the center of gravity of the object region before and after the reduction can be minimized.

  FIG. 8 shows an example in which the position of the reduced object region is adjusted by the adjustment unit 16. Note that among the elements shown in FIG. 8, elements equivalent to those shown in FIG. 7 are denoted by the same reference numerals and description thereof is omitted.

  In FIG. 8A, the two object regions Ras and Rbs that have been reduced by the reduction unit 15 touch each other at the side parallel to the y-axis among the sides including the diagonal vertices of the respective reduction reference points Ua and Ub. In this case, the adjustment unit 16 adjusts the positions of the object regions Ras and Rbs.

  In FIG. 8A, the shaded area indicated by reference numeral Rac can move the reduced object area Ras while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. The movable range is shown. Similarly, the shaded area indicated by the symbol Rbc indicates a movable range in which the reduced object area Rbs can be moved while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. ing.

  The movable range Rac shown in FIG. 8A is a reference point for reduction among the two regions obtained by dividing the object region Ra before reduction using the straight line L1 including the side parallel to the y axis as described above. This corresponds to a part of the region closer to Ua. Similarly, the movable range Rbc is obtained by dividing the unreduced object region Rb with the straight line L1 including the side parallel to the y-axis described above as a boundary, on the side closer to the reduction reference point Ub. It corresponds to a part of the area.

  8B shows that two object regions Ras and Rbs that have been reduced by the reduction unit 15 are sides that are parallel to the x-axis among the sides that include the diagonal vertices of the respective reference points Ua and Ub for reduction. In this example, the adjustment unit 16 adjusts the positions of the object regions Ras and Rbs when they are in contact with each other.

  In FIG. 8B, the shaded area indicated by reference numeral Rac can move the reduced object area Ras while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. The movable range is shown. Similarly, the shaded area indicated by the symbol Rbc indicates a movable range in which the reduced object area Rbs can be moved while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. ing.

  The movable range Rac shown in FIG. 8B is a reference point for reduction among the two regions obtained by dividing the object region Ra before reduction using the straight line L2 including the side parallel to the x axis as described above. This corresponds to a part of the region closer to Ua. Similarly, the movable range Rbc is obtained by dividing the object region Rb before reduction with the straight line L2 including the side parallel to the x-axis described above as a boundary, on the side close to the reference point Ub for reduction. It corresponds to a part of the area.

  FIG. 8C shows that when the two object regions Ras and Rbs after reduction by the reduction unit 15 touch each other at the diagonal vertices of the respective reduction reference points Ua and Ub, the adjustment unit 16 In this example, the positions of the object regions Ras and Rbs are adjusted.

  In FIG. 8C, the shaded area indicated by reference numeral Rac can move the reduced object area Ras while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. The movable range is shown. Similarly, the shaded area indicated by the symbol Rbc indicates a movable range in which the reduced object area Rbs can be moved while maintaining the area of the portion where the reduced object areas Ras and Rbs overlap with zero. ing.

  One of the movable ranges Rac shown in FIG. 8C is obtained by dividing the object region Ra before reduction using a straight line L3 parallel to the y axis passing through the contact point between the object regions Ras and Rbs after reduction as a boundary. Of the two regions, this corresponds to a part of the region closer to the reduction reference point Ua. The other of the movable ranges Rac is a side closer to the reduction reference point Ua, out of two areas obtained by dividing the object area Ra before reduction with the straight line L4 passing through the contact point and parallel to the x-axis as a boundary. Corresponds to a part of the area. Similarly, one of the movable ranges Rbc corresponds to a part of a region closer to the reduction reference point Ub among two regions obtained by dividing the above-described straight line L3 as a boundary. The other of the movable ranges Rbc corresponds to a part of a region closer to the reduction reference point Ub among the two regions obtained by dividing the above-described straight line L4 as a boundary.

  The adjustment unit 16 illustrated in FIG. 1 selects and selects one of the movable ranges illustrated in FIGS. 8A to 8C based on the relative positions of the two object regions Ras and Rbs after the reduction. In the movable range, the position where the reduced object regions Ras and Rbs are superimposed on the image of the summary video Vs is adjusted.

  For example, the adjustment unit 16 moves the reduced object region Ras illustrated in FIG. 8A in the y-axis direction, and adjusts the y coordinate of the centroid Cas so as to coincide with the y coordinate of the centroid Ca. The reduced object region Ras may be superimposed on the corresponding image of the summary video Vs. Thus, the reduced object region Ras can be superimposed on the corresponding image of the summary video Vs so as to minimize the distance between the center of gravity Cas and the center of gravity Ca before reduction. Similarly, the adjustment unit 16 adjusts, for example, the position where the reduced object region Rbs is superimposed on the corresponding image of the summary video Vs so that the y coordinate of the centroid Cbs matches the y coordinate of the centroid Cb. The distance between the center of gravity Cbs and the center of gravity Cb can be minimized. Such adjustment can be performed while maintaining the area of the overlapping portion between the object region Ras and the object region Rbs at zero.

  That is, the adjustment unit 16 performs the above-described adjustment of the positions of the reduced object areas Ras and Rbs, so that the object positions in each image of the summary video Vs and the extraction source are not overlapped with each other. The deviation from the position of each partial video Vp in each image can be reduced. The adjustment unit 16 corresponds to the object region in which the summary video Vs has been superimposed on the corresponding image, and uses the information indicating the position after adjustment described above to correspond to the object region in the object information holding unit 12. Object information may be updated.

  According to the video editing apparatus 10 having the adjusting unit 16 described above, for example, as illustrated in FIG. 9, even when the movement paths of two individually photographed objects are close to each other, Each moving path can be faithfully reproduced without overlapping the objects.

  FIG. 9 shows an example of an object region superimposed on each image of the summary video Vs. In FIG. 9, the symbol t indicates a time axis representing the passage of time in the summary video Vs. On the time axis t, symbols Ti, Tj, and Tk indicate timings corresponding to the images of image number i, image number j, and image number k, respectively.

  In FIG. 9, thin broken line areas indicated by reference signs Rai, Raj, and Rak are object areas extracted from the images indicated by the image numbers i, j, and k included in the partial video Vp representing the movement process of the object A. Is shown. Similarly, the thin solid line areas indicated by reference characters Rbi, Rbj, and Rbk are object areas extracted from the images indicated by the image numbers i, j, and k included in the partial video Vp representing the movement process of the object B. Show.

  In the example of FIG. 9, a part in which the object region corresponding to the object A and the object region corresponding to the object B overlap with the summary video image corresponding to each timing included in the period from the timing i to the timing k described above. It shows the case of being superimposed with

  Further, in FIG. 9, the thick broken line areas indicated by reference numerals Rasi, Rasj, and Rask are obtained by reducing the above-described object areas Rai, Raj, and Rak with the vertex farthest from the object areas Rbi, Rbj, and Rbk as a reference. Each of the reduced object regions is shown. Similarly, thick solid line areas denoted by reference characters Rbsi, Rbsj, and Rbsk are obtained by reducing the above-described object areas Rbi, Rbj, and Rbk with reference to a vertex that is farthest from the object areas Rai, Raj, and Rak. Each object region is shown.

  In FIG. 9, the symbol Ga indicates the original trajectory of the object A indicated by the centers of gravity Cai, Caj, Cak of the object regions Rai, Raj, Rak before reduction. Similarly, the symbol Gb indicates the original trajectory of the object B indicated by the centroids Cbi, Cbj, Cbk of the object regions Rbi, Rbj, Rbk before reduction.

  As shown in FIG. 9, the trajectory Gas indicated by the centroids Casi, Casj, and Casks of the reduced object regions Rasi, Rasj, and Rask is deviated from the original trajectory Ga of the object A described above. Similarly, the trajectory Gbs indicated by the centroids Cbsi, Cbsj, and Cbsk of the reduced object regions Rbsi, Rbsj, and Rbsk is also deviated from the original trajectory Gb of the object B.

  On the other hand, the centroids Casi ′, Casj ′, and Cas ′ of the reduced object regions Rasi ′, Rasj ′, and Rask ′ after applying the position adjustment described above are changed to the centroids Casi, Casj, and Cask before the adjustment. Compared to the original center of gravity Cai, Caj, Cak, it is in a position. Similarly, the centroids Cbsi ′, Cbsj ′, and Cbsk ′ of the reduced object regions Rbsi ′, Rbsj ′, and Rbsk ′ after applying the position adjustment described above are compared with the centroids Casi, Casj, and Cask before adjustment. It is in a position close to the original center of gravity Cbi, Cbj, Cbk.

  Similarly, objects corresponding to the object A and the object B are displayed in the summary video images indicated by the image numbers between the image numbers i and j and the image numbers between the image numbers j and k. When the regions are superimposed, the above-described reduction processing and position adjustment processing can be applied. Thereby, in each image from the image number i to the image number k of the summary video, the reduced object regions corresponding to the object A and the object B can be overlapped while minimizing the deviation from the center of gravity position before the reduction. .

  Therefore, the locus of the center of gravity of the object area to which the reduction process and the position adjustment process are applied, including the centroids Casi ′, Casj ′, and Cask ′, is more effective than the locus Gas indicated by the centroids Casi, Casj, and Cassk before the adjustment. A locus close to the original locus Ga of A can be shown. Similarly, the locus of the center of gravity of the object region to which the reduction process and the position adjustment process are applied, including the centroids Cbsi ′, Cbsj ′, and Cbsk ′, than the locus Gbs indicated by the centroids Cbsi, Cbsj, and Cbsk before the adjustment. A locus closer to the original locus Gb of the object B can be shown. In FIG. 9, the trajectories indicated by the centroids Casi ′, Casj ′, and Cas ′ of the object regions Rasi ′, Rasj ′, and Rask ′ to which the reduction process and the position adjustment process are applied substantially coincide with the trajectory Ga described above. Therefore, the illustration is omitted. Similarly, the trajectories indicated by the centroids Cbsi ′, Cbsj ′, Cbsk ′ of the object regions Rbsi ′, Rbsj ′, Rbsk ′ to which the reduction process and the position adjustment process are applied substantially coincide with the trajectory Gb described above. The illustration in 9 is omitted.

  The same reduction processing and position adjustment processing are applied to each of two object regions that are overlapped with each image of the summary video Vs with overlapping portions, whereby objects photographed in each partial video Vp are mutually connected. The original position of each object can be preserved without overlapping.

  Here, when the object region extracted from the partial video corresponding to the movement process of each of the many objects is superimposed on each image of the summary video Vs, by eliminating the mutual overlap, for example, the facial expression and clothes of the person And information indicating detailed attributes such as gestures can be grasped. Further, by storing the position of the object in each image of the original partial video Vp, the movement path of each object can be faithfully reproduced by the trajectory of each object region in each image of the summary video Vs.

  That is, according to the video editing apparatus 10 of the present disclosure shown in FIG. 1, the detailed attributes of each object and the actual movement path are collectively reproduced from the partial images obtained by capturing the movement processes of the plurality of objects. The summary video Vs to be generated can be generated and displayed on the display device 5.

  According to the summary video Vs obtained in this way, the behaviors of a plurality of persons after the occurrence of a common event such as the door event described above can be presented so as to be listed in chronological order from the timing of the door event. Can do.

  A user who has received such a summary video Vs intuitively knows, for example, the direction of action and the direction in which a large number of persons point their eyes in common, as well as the behavioral trends and movement paths of persons of different age groups and genders. It becomes possible to grasp it. Information indicating the flow seen in the movement of a large number of people, as well as information indicating behavior trends of different people for each attribute and behaviors common to many people, for example, each corner of large retail stores and amusement parks This information is useful when evaluating the placement and display effects. Therefore, the video editing apparatus 10 disclosed herein is useful in the field of designing and refurbishing large-scale retail stores including department stores, amusement facilities including amusement parks and game centers. In addition, the summary video generated by the video editing apparatus 10 disclosed herein can be used for designing and remodeling offices of companies and the like and designing and renovating public facilities such as parks, stations, and underground passages.

  Further, in the video editing method disclosed in the present disclosure, the video editing apparatus 10 illustrated in FIG. 1 executes the video editing process according to the flowchart illustrated in FIG. 10 on the plurality of partial videos Vp received from the motion video extracting apparatus 3. May be realized.

  FIG. 10 shows an example of a flowchart of the video editing process. Steps 301 to 310 shown in FIG. 10 are executed by the specifying unit 11, the detecting unit 13, the calculating unit 14, the reducing unit 15, and the adjusting unit 16 included in the video editing apparatus 10 shown in FIG.

  First, the specifying unit 11 specifies a part representing an object moving with respect to the background as an object area in each image included in each partial video Vp, and sets object information indicating the specified object area as object information. It is held by the holding unit 12 (step 301). Further, the identification unit 11 corresponds to each object information, for example, as object image data, for example, information indicating a portion representing a background other than an object image included in each object area as a transparent image. You may hold | maintain to the holding | maintenance part 12. FIG.

  Next, the detection unit 13 selects, for example, each image of the summary video Vs as an image to be processed in order from the image with the image number 1, and stores it in the object information holding unit 12 corresponding to the image number indicating the image to be processed. The object area to be processed is sequentially selected from the held object information (step 302). For example, the detection unit 13 sequentially selects object regions to be processed according to the object ID from the object regions held corresponding to the image numbers selected by the object information holding unit 12 illustrated in FIG. Also good.

  Next, the detection unit 13 determines whether or not the object information corresponding to the object area to be processed and the object information corresponding to another object area satisfy the relationship expressed by the above formula (1). It is determined whether or not the object area overlaps with other object areas (step 303).

  When an object area having a portion overlapping with the object area to be processed is not found (No determination in step 303), the video editing apparatus 10 directly uses the image data of the object area to be processed as the summary video Vs. The image is superimposed on the corresponding image (step 304). For example, when the detection unit 13 illustrated in FIG. 1 does not detect an object region having a portion that overlaps the object region to be processed, the detection unit 13 notifies the adjustment unit 16 of the fact, thereby notifying the adjustment unit 16. The process of step 304 may be executed. When the adjustment unit 16 receives the detection result indicating that the overlapping portion is not detected, the adjustment unit 16 reads the image data of the object region to be processed from the object information holding unit 12 and stores the read image data in the summary video storage device 4. The images may be combined so as to be superimposed on the corresponding image.

  On the other hand, when an object area having a portion overlapping with the object area to be processed is detected (Yes in step 303), the video editing apparatus 10 executes the processes in steps 305 to 308.

  In the affirmative determination route of step 303, the detection unit 13 passes a pair of object information corresponding to the object region to be processed and the detected object region to the calculation unit 14 and causes the process of step 305 to be executed.

  Based on the object information received from the detection unit 13 and the above-described equation (2) or equation (3), the calculation unit 14 calculates a pair of the object region to be processed and the detected object region using these object regions. A reduction ratio for minimizing the area of the overlapping portion is calculated (step 305). Further, the calculation unit 14 passes the information indicating the reduction ratio and the reduction reference point calculated in the process of step 305 to the reduction unit 15.

  The reduction unit 15 receives the image of the object area represented by the image data held in correspondence with the two object areas described above in the object information holding unit 12 from the calculation unit 14 as illustrated in FIG. Using the reference point as a reference, a process of reducing at a specified reduction rate is executed (step 306).

  As described with reference to FIGS. 8 and 9, the adjustment unit 16 makes the movement of the center of gravity from the original object region to be minimum for the two reduced object regions thus obtained. Then, the position of the reduced object area is adjusted (step 307). For example, the adjustment unit 16 obtains a movement amount when adjustment is performed so that the center of gravity of each reduced object region approaches the original center of gravity within the movable range illustrated in FIG. 8, and the obtained movement amount is calculated for each object region. The position of the adjusted reference point may be specified by applying to the coordinates indicating the position of the reference point for reduction.

  Next, the adjusting unit 16 converts the image of each reduced object region received from the reducing unit 15 into an image to be processed of the summary video Vs based on the position of the adjusted reference point specified as described above. Overlay (step 308). For example, in the image corresponding to the above-described image number in the summary video storage device 4 illustrated in FIG. 1, the adjustment unit 16 reduces the image of the reduced object region to the position indicated by the coordinates specified in step 307. By superimposing, the summary video Vs may be superimposed on the processing target image. Further, the adjustment unit 16 may update the corresponding object information held in the object information holding unit 12 based on the position of the reduction reference point specified in step 307. For example, the adjusting unit 16 adjusts the coordinates of the upper left vertex after adjusting the position of the object region to be processed based on the coordinates of the reference point after applying the position adjustment described above and the applied reduction ratio, and the object region. The reduced width and height can be calculated. In this way, if the object information is rewritten using the calculated coordinates of the upper left vertex and the reduced width and height, the updated object information will be used when performing the subsequent processing while paying attention to the new object region. Can be used to determine whether there is overlap between object regions.

  As described above, after the processing of the affirmative determination route or the negative determination route in step 303 is completed, the video editing apparatus 10 has not yet processed the object information corresponding to the image number indicating the image to be processed. It is determined whether there is object information (step 309).

  When there is unprocessed object information (Yes in step 309 (YES)), the video editing apparatus 10 returns to the process of step 302 and selects one of the object areas indicated by the unprocessed object information as a processing target. The processing from step 303 onward is executed as the object region.

  After that, when the processing from step 302 to step 309 is completed for all the object information held in the object information holding unit 12 corresponding to the image number indicating the image to be processed, the video editing apparatus 10 performs step 309. The process proceeds to step 310 according to the negative determination route.

  In step 310, the video editing apparatus 10 determines whether or not the processing in steps 302 to 309 described above has been completed for the images indicated by all the image numbers of the summary video Vs. If there is an image that has not yet been generated among the n images included in the summary video Vs, the video editing apparatus 10 determines that the processing has not been completed for all the images (No in step 310). Judgment (NO)). In this case, the video editing apparatus 10 returns to the process of step 302, selects an image to be processed from ungenerated images, and holds it in the object information holding unit 12 corresponding to the image number indicating the selected image. With respect to the object information thus obtained, the processing from step 303 described above is executed.

  By executing the above-described processing for the object information held in the object information holding unit 12 corresponding to each of the image numbers 1 to n, each partial video Vp is associated with the image of the summary video Vs indicated by each image number. The process of superimposing the object region extracted from the image to be completed is completed. In this way, when the processing for the images indicated by all the image numbers is completed (Yes in step 310), the video editing apparatus 10 determines that the generation of the summary video Vs has been completed, The video editing process ends.

  According to the video editing method of the present disclosure described above, the overlap is eliminated when the object region extracted from the corresponding image of each partial video Vp is superimposed on the image of the summary video Vs indicated by each image number. Thus, the deviation from the original position can be minimized.

  As a result, the movement process of the plurality of objects is collectively displayed from the partial images obtained by capturing the movement processes of the plurality of objects and the detailed attributes represented by the entire image of each object. Thus, the summary video Vs to be reproduced can be generated.

  By the way, as described above, when the reduction is performed so that the overlapping part of the two object areas is completely eliminated, for example, when the ratio of the overlapping parts is large, the size of the object area after the reduction is the original. May be reduced to about half of the object area.

  Here, if the individual object regions superimposed on the images included in the summary video Vs are too small compared to the original object region, it becomes difficult to reproduce the details of the individual objects. It is desirable to set a predetermined lower limit for the reduction ratio applied to the above. Further, in order to give the user an impression that images of each object are continuous in time series in the summary video Vs, the size of the image of the object indicated by the same object ID in each image included in the summary video Vs. It is desirable to determine the reduction ratio of each object region so that the fluctuation does not vary greatly.

  Further, in the method of adjusting the positions of the reduced object regions within a range that does not overlap each other as shown in FIG. 8, when the ratio of overlapping parts is large, as shown in FIG. There may be a case where the deviation of the position of the center of gravity which cannot be ignored compared to the size of the area remains.

  FIG. 11 shows an example in which the position of the reduced object region is adjusted. Note that among the elements shown in FIG. 11, elements equivalent to those shown in FIG. 7 are denoted by the same reference numerals and description thereof is omitted.

  In FIG. 11, the object regions Ra and Rb are reduced with the vertices Ua and Ub as reference points until there is no overlapping portion, and the reduced object regions Ras and Rbs are moved within a range that does not overlap. This shows an example in which the reduced gravity centers Cas and Cbs are brought close to the position of.

  In this case, for example, if the reduction rate αa applied to the object region Ra is 2/3, the center of gravity Ca of the object region Ra shown by a thin broken line in FIG. 11 and the object after reduction and position adjustment shown by a thick broken line The distance from the center of gravity of the region Ras is equal to 1/4 of the width of the reduced object region Ras.

  The ratio of the portions where the object regions extracted from the two partial videos Vp overlap each other is, for example, that the movement path of the object photographed in each of the partial videos Vp passes the same time from the beginning of each partial video Vp. It becomes large when crossing later. Then, in the summary video Vs generated from the two partial videos Vp, a large shift in the center of gravity position as shown in FIG. 11 remains in the image after the elapse of time from the beginning of the summary video Vs. In addition, the movement path of each object cannot be faithfully reproduced.

  On the other hand, if the movable range is expanded in a direction that allows the overlapping part to remain, even when the ratio of the overlapping part between the two object regions is large, the shift of the center of gravity position as described above is reduced, It is possible to faithfully reproduce the movement path of each object.

  Here, the operation of reducing each of the two object regions Ra and Rb shown in FIG. 11 and moving them within the range of the original object regions Ra and Rb does not involve rotation with the reference points Ua and Ub as the origin. It can be understood as an operation for performing similarity transformation. That is, the functions of the reduction unit 15 and the adjustment unit 16 illustrated in FIG. 1 may be realized by, for example, similarity conversion processing using the reduction rate and the movement amounts of the reference points Ua and Ub as parameters.

  In addition, if the parameters of similarity transformation are determined so as to minimize the movement amount of the center of gravity before and after the similarity transformation rather than the area where the two object regions after the similarity transformation overlap, between the two object regions Even when the ratio of overlapping parts is large, the reproducibility of the movement route can be improved.

  In the following, by using FIG. 12 to FIG. 19 to perform overlapping that allows overlapping of object regions, even when the ratio of overlapping portions between two object regions is large, the movement path of each object is faithfully observed. A preferred embodiment for generating a summary video that can be reproduced will be described.

  FIG. 12 shows another embodiment of the video editing apparatus 10. 12 that are the same as those shown in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The video editing apparatus 10 illustrated in FIG. 12 includes a similarity conversion processing unit 17 instead of the reduction unit 15 and the adjustment unit 16 illustrated in FIG. Further, the calculation unit 14 illustrated in FIG. 11 includes a ratio calculation unit 141, a setting unit 142, a condition generation unit 143, and a determination unit 144.

  The calculation unit 14 applies to each object region, as described later, based on the object information held in the object information holding unit 12 for each of the two object regions in which overlapping portions are detected by the detection unit 13. The similarity transformation parameters including the reduction ratio and the movement amount of the reference point are calculated.

  The similarity conversion processing unit 17 receives from the calculation unit 14 parameters including a reduction ratio and a reference point movement amount applied to the two object regions in which the overlapping portions are detected by the detection unit 13, and based on the received parameters, Similarity transformation processing as shown in FIG. 13 is performed on the object region.

  FIG. 13 shows an example of similarity transformation. Note that among the elements shown in FIG. 13, elements equivalent to those shown in FIGS. 6 and 7 are denoted by the same reference numerals and description thereof is omitted.

  In FIG. 13, reference characters Rat and Rbt indicate object regions after conversion obtained by applying similarity conversion to the object regions Ra and Rb, respectively, and shaded regions indicated by reference characters Rd indicate converted object regions. The overlapping portions of the regions Rat and Rbt are shown. Reference symbols Uat and Ubt indicate converted reference points corresponding to the reference points Ua and Ub specified for the object regions Ra and Rb before conversion in the object regions Rat and Rbt after similarity conversion. Symbols αa and αb indicate the reduction ratio when the two object regions Ra and Rb are reduced as the reference points Ua and Ub with the vertices farthest from each other.

  In FIG. 13, symbols dxa and dya indicate the amount of change from the reference point Ua before conversion to the reference point Uat after conversion, and symbols dxb and dyb indicate reference points after conversion from the reference point Ub before conversion. The amount of change to Ubt is shown. In FIG. 13, the symbol Dx indicates the difference between the X coordinates of the converted reference points Ua and Ub, and the symbol Dy indicates the difference between the Y coordinates of the converted reference points Ua and Ub.

The area of the overlapping portion Rd of the converted object regions Rat and Rbt shown in FIG. 13 is calculated using, for example, the X-coordinate difference Dx and the Y-coordinate difference Dy, the reduction rate αa, and the reduction rate αb. It can be expressed as equation (4). In equation (4), the symbol Sd indicates the area of the overlapping portion Rd described above, the symbol wa indicates the width of the object region Ra, and the symbol ha indicates the height of the object region Ra. Show. Similarly, the symbol wb indicates the width of the object region Rb, and the symbol hb indicates the height of the object region Rb. Further, the X-coordinate difference Dx and the Y-coordinate difference Dy are expressed by equations (5) and (6), respectively, using the above-described change amounts dxa and dxb in the X-axis direction and change amounts dya and dyb in the Y-axis direction. Can be expressed as: In Equation (5), symbols Xa and Ya indicate the X and Y coordinates of the reference point Ua of the object region Ra, respectively, and symbols Xb and Yb indicate the X and Y coordinates of the reference point Ub of the object region Rb, respectively. Show.
Sd = ((αa · wa + αb · wb) −Dx) ((αa · ha + αb · hb) −Dy) (4)
Dx = | Xb−Xa | − | dxa | − | dxb | (5)
Dy = | Yb−Ya | − | dya | − | dyb | (6)
Further, the distance between the center of gravity Ca of the object area Ra before conversion and the center of gravity Cat of the object area Rat after conversion, that is, the distance Da of movement of the center of gravity before and after conversion is expressed by the reduction rate αa and the width wa of the object area Ra. Using the height ha, it can be expressed as equation (7). Similarly, the distance between the centroid Cb of the object region Rb before conversion and the centroid Cbt of the object region Rbt after conversion, that is, the moving distance Db of the centroid before and after conversion is expressed by the reduction rate αb and the width wb of the object region Rb. And the height hb can be expressed as in Expression (8).

上述した面積Ｓｄと、重心の移動距離Ｄａ，Ｄｂとをともに最小化するパラメータは、式(４)、式(７)および式(８)のそれぞれを目的関数とする多目的最適化問題を解くことによって求めることができる。なお、上述した多目的最適化問題を解く際に、例えば、物体領域Ｒａ，Ｒｂに適用する縮小率αａ，αｂについての上限および下限を示す制約条件を設定することで、各物体領域Ｒａ，Ｒｂが縮小されすぎないようにすることができる。

The parameters for minimizing both the area Sd and the movement distances Da and Db of the center of gravity are to solve the multi-objective optimization problem using the equations (4), (7), and (8) as objective functions. Can be obtained. When solving the above-mentioned multi-objective optimization problem, for example, by setting the constraint conditions indicating the upper and lower limits for the reduction ratios αa and αb applied to the object regions Ra and Rb, the object regions Ra and Rb It can be prevented from being reduced too much.

  Here, among the six variables used in the objective function expressed by the equations (4), (7), and (8), for example, the reduction rate αb that is a variable related to the object region Rb and the reference point Ub The movement amounts dxb and dyb may be fixed values to simplify the multi-objective optimization problem described above. By such simplification, the problem of optimizing three objective functions using six variables including variables relating to the object region Rb is changed to two objective functions using three variables relating to the object region Ra, ie, equations It can be solved as a problem of optimizing the objective function represented by (4) and (7).

  Using the reduction ratio αa, which is a variable related to the object region Ra, and the movement amounts dxa and dya of the reference point Ua, both the area Sd expressed by the equation (4) and the moving distance Da of the center of gravity expressed by the equation (7) are minimized. The problem to be solved can be solved using, for example, a gradient method. Further, a Pareto solution as shown in FIG. 14 can be obtained by solving the above-described problem of minimizing the area Sd and the movement distance Da of the center of gravity.

  FIG. 14 shows an example of a solution to the multi-objective optimization problem for the purpose of minimizing the area Sd and the movement distance Da of the center of gravity. The horizontal axis shown in FIG. 14 indicates the size of the area Sd, and the vertical axis indicates the size of the moving distance Da of the center of gravity.

  The code | symbol V shown in FIG. 14 has shown the example of the graph showing the set of the Pareto solution of the multi-objective optimization problem mentioned above. As can be seen from the graph V shown in FIG. 14, a solution that gives a small value as the area Sd tends to give a large value as the movement distance Da of the center of gravity. Conversely, a solution that gives a small value to the movement distance Da of the center of gravity is There is a tendency to give a large value to the area Sd.

Therefore, it is considered that an optimum solution is obtained from the graph V shown in FIG. 14 by the evaluation formula expressed by the formula (9) using the area Sd and the movement distance Da of the center of gravity. In equation (9), symbol C represents an arbitrary constant, and symbols ω1 and ω2 represent a first weight given to the area Sd and a second weight given to the center-of-gravity movement distance Da, respectively.
ω1 · Sd + ω2 · Da + C = 0 (9)
A symbol L1 illustrated in FIG. 14 is an example of a graph indicating an evaluation formula in which the second weight ω2 is set to a value larger than the first weight ω1 in the above-described formula (9). According to such an evaluation formula, the optimum when the reduction rate αa giving the contact Zc1 between the graph L1 and the graph V and the movement amounts dxa and dya of the reference point Ua are more important than the area Sd. The solution can be specified.

  Moreover, the code | symbol L2 shown in FIG. 14 is an example of the graph which shows the evaluation type | formula which set larger value than 1st weight omega1 in 2nd weight omega2 in the formula (9) mentioned above. According to such an evaluation formula, the optimum when the area Sd is more important than the movement distance Da of the center of gravity as the reduction rate αa that gives the contact Zc2 between the graph L2 and the graph V and the movement amounts dxa and dya of the reference point Ua. The solution can be specified.

  Similarly, the area Sd expressed by the equation (4) and the moving distance Db of the center of gravity expressed by the equation (8) using the reduction ratio αb and the movement amounts dxb and dyb of the reference point Ub, which are variables related to the object region Rb, To solve the problem of minimizing both.

  The calculation unit 14 illustrated in FIG. 12 includes a ratio calculation unit 141 and a setting unit 142 in order to generate the above-described evaluation formula.

  Based on the object information held in the object information holding unit 12 corresponding to the two object areas in which the overlapping parts are detected by the detecting unit 13, the ratio calculating unit 141 detects the overlapping parts in the two object areas. Calculate the share. For example, the ratio calculation unit 141 may obtain the ratio of the area of the overlapping portion to the area of each object region, and pass the obtained ratio value to the setting unit 142 as information indicating the above-described ratio. The ratio calculation unit 141 obtains a ratio between a value indicating the size of each object region, such as the width or height of each object region, and a value indicating the size of the overlapping portion, and calculates the value of the obtained ratio. You may pass to the setting part 142 as information which shows the ratio mentioned above.

  In addition, the setting unit 142 sets a value greater than the first weight ω1 to the second weight ω2 of Equation (9) when at least one of the ratio values obtained by the ratio calculation unit 141 is equal to or greater than a predetermined threshold. Generate the set evaluation formula. On the other hand, when both of the ratio values obtained by the ratio calculation unit 141 are smaller than the predetermined threshold, the setting unit 142 sets a value greater than the second weight ω2 to the first weight ω1 of Equation (9). Generated evaluation formula. For example, as shown in FIG. 8, the predetermined threshold described above is the amount of movement in the x-axis direction or the y-axis direction of the center of gravity position before and after the reduction when each object region is reduced so that there is no overlapping portion. It is desirable to set a value indicating a ratio such that is an allowable upper limit value.

  According to the ratio calculation unit 141 and the setting unit 142 described above, when the ratio of the overlapping portions in the two object regions is large, an evaluation formula for obtaining an optimal solution that emphasizes the moving distance of the center of gravity is generated. In addition, it is possible to generate an evaluation formula that obtains an optimal solution that places importance on the area of overlapping portions. Note that the values of the first weight ω1 and the second weight ω2 set in Expression (9) in each case are, for example, first values having various values when generating a summary video from a video shot in the past. It is desirable to determine in advance by an experiment using an evaluation formula in which the weight ω1 and the second weight ω2 are set.

  In addition, the calculation unit 14 illustrated in FIG. 12 includes a condition generation unit 143 in order to generate a constraint condition regarding the reduction ratio applied to each object region.

Based on the input minimum value αmin of the reduction rate and the reduction rate applied to the object region corresponding to the same object in the processing for the image indicated by the previous image number, the condition generation unit 143 A constraint condition is generated for each reduction ratio applied to the two object regions. The condition generation unit 143 may generate a constraint condition as shown in, for example, Expression (10) using the above-described minimum value αmin. In Expression (10), symbols βa and βb are processes for the image indicated by the previous image number, and are superimposed as object ID images corresponding to the two object regions Ra and Rb shown in FIG. The reduction ratio applied to the two combined object regions is shown. Further, the symbol κ represents a predetermined coefficient having a value of 0 or more and less than 1.
max (αmin, (1-κ) · βa) ≦ αa ≦ min (1, (1 + κ) · βa)
max (αmin, (1-κ) · βb) ≦ αb ≦ min (1, (1 + κ) · βb) (10)
As the value of the predetermined coefficient κ described above, for example, it is desirable to set a value indicating a permissible variation ratio with respect to the size of the object region corresponding to the same object ID included in the image indicated by the consecutive image numbers. . The minimum value αmin of the reduction ratio is preferably set in advance based on, for example, the ratio between the average size of the object region and the limit size at which the object attribute can be grasped.

  According to the constraint condition shown in Expression (10), the lower limit of the reduction ratio applied to each object region is set to the above-described minimum value αmin or more, and the reduction ratio applied to the corresponding object region of the previous image is reduced. The amount of change can be suppressed.

  The constraint conditions generated in this way by the condition generation unit 143 and the evaluation formulas generated by the ratio calculation unit 141 and the setting unit 142 described above are passed to the determination unit 144 illustrated in FIG.

  The determination unit 144 is a Pareto for the optimization problem that minimizes the above-described Expression (4) and Expression (7) or Expression (4) and Expression (8) within the range of the constraint conditions obtained by the condition generation section 143. Find each solution.

  The determining unit 144 is included in the above-described formula (4), formula (7), and formula (8) based on the object information held in the object information holding unit 12 corresponding to the two object areas Ra and Rb. The coordinates of the reference points Ua and Ub can be obtained. Further, as described with reference to FIG. 14, the determination unit 144 obtains contact points between a graph indicating a set of Pareto solutions for each of the two object regions and a graph indicating the above-described evaluation formula. The parameters of similarity transformation for the object area are specified. The determination unit 144 may pass, for example, the reduction ratio that gives the Pareto solution obtained as the contact point and the movement amount of the reference point to the similarity conversion processing unit 17 as parameters for similarity conversion.

  Further, when determining the similarity transformation parameters as described above, the determination unit 144 passes the reduction ratio of each object region to the object information holding unit 12 and holds it corresponding to the image number indicating the image being processed. It may be added to the object information corresponding to each object region. When such object information is added, the above-described condition generation unit 143 refers to the object information holding unit 12 in the process for the image indicated by the subsequent image number, and the previous information is displayed. The reduction ratio applied to the corresponding object region in the image can be acquired.

  According to the calculation unit 14 described above, when the ratio of the overlapping parts in the two object regions detected by the detection unit 13 is large, the movement amount of the center of gravity before and after the similarity transformation is more important than the area of the overlapping part. Through the optimization, the parameters of the similarity transformation can be determined.

  When the similarity conversion processing unit 17 executes similarity conversion using such parameters, even when a portion where two object regions overlap each other, such as when the movement paths of two objects intersect, The area of the overlapping portion can be minimized while suppressing the shift of the center of gravity position.

  That is, by superimposing the converted object regions obtained by the above-described similarity conversion on the corresponding images held in the summary video storage device 4, the deviation of the center of gravity position is suppressed in each image of the summary video Vs. However, the area of the overlapping portion can be minimized.

  That is, according to the video editing apparatus 10 of the present disclosure including the calculation unit 14 and the similarity conversion processing unit 17 as illustrated in FIG. 12, even when the movement paths of two objects intersect, While reproducing, it is possible to generate the summary video Vs in which the overlap of the images of each object is reduced.

  Furthermore, as shown in FIG. 12, by providing a condition generation unit 143 in the calculation unit 14, a desired constraint condition for the reduction ratio to be applied to each object region is set in the similarity transformation parameters obtained by the determination unit 144. can do.

  For example, the condition generation unit 143 sets a constraint condition including a condition indicating that the reduction ratio applied to each object region is equal to or greater than the minimum value αmin as a desired constraint condition, so that each image of the summary video Vs is set. The size of the image of each object to be superimposed can be maintained above a certain value. Thereby, based on the summary video Vs, the attribute of each object can be grasped through the whole movement process of each object.

  In addition, as shown in Expression (10), the condition generation unit 143 sets the constraint condition including the condition for suppressing the variation from the reduction ratio applied in the previous image, so that it is included in the summary video Vs. Variations in the size of the image of each object in each image can be suppressed. Accordingly, it is possible to provide a summary video Vs that gives the user an impression that images of each object are continuous in time series.

  The video editing apparatus 10 disclosed above can be realized by using a computer device such as a personal computer, for example.

  FIG. 15 shows an example of the hardware configuration of the video editing apparatus 10. 15 that are the same as those shown in FIG. 1 are denoted by the same reference numerals and description thereof is omitted.

  The computer device 20 includes a processor 21, a memory 22, a hard disk device 23, a general-purpose interface 24, a display control unit 25, and an optical drive device 26. The processor 21, the memory 22, the hard disk device 23, the general-purpose interface 24, the display control unit 25, and the optical drive device 26 illustrated in FIG. 15 are connected to one another via a bus. The video editing device 10 includes a processor 21, a memory 22, a hard disk device 23, and a general-purpose interface 24.

  The optical drive device 26 described above can be mounted with a removable disk 27 such as an optical disk, and reads and records information recorded on the mounted removable disk 27. In addition, the display control unit 25 is connected to the display device 5, and the processor 21 passes the summary video Vs generated by the video editing device 10 to the display device 5 via the display control unit 25, for example. The summary video Vs can be presented to the user.

  Prior to the above-described video editing process, the computer device 20 receives a plurality of partial videos Vp from the partial video extraction device 3 via the general-purpose interface 24, and a partial video storage unit provided in the hard disk device 23 with the received partial videos Vp. It may be stored in H.231. Further, the computer device 20 extracts, from the video stored in a video storage device (not shown), a portion where the process of moving each object with a predetermined shooting range as a background is extracted as a partial video corresponding to each object. The partial video may be stored in the partial video storage unit 231.

  In the hard disk device 23 shown in FIG. 15, the summary video holding unit 232 is an element corresponding to the summary video storage device 4 shown in FIG. 1, and each partial video described above prior to the start of the video editing process. As many background images as the number of images included in Vp are held.

  A part of the capacity of the hard disk device 23 can also be allocated to the object information holding unit 12. Note that the object information holding unit 12 provided in the hard disk device 23 preferably holds information indicating a reduction rate obtained by processing to be described later as part of object information corresponding to each object region. Moreover, the object information holding | maintenance part 12 may hold | maintain the image of the object area | region extracted from each image of each partial image | video Vp mentioned above as a part of each object information, for example.

  The memory 22 illustrated in FIG. 15 stores an application program for the processor 21 to execute the above-described video editing process together with the operating system of the computer device 20. Note that an application program for executing the above-described video editing process can be recorded and distributed on a removable disk 27 such as an optical disk, for example. Then, an application program for executing video editing processing may be stored in the memory 22 and the hard disk device 23 by loading the removable disk 27 in the optical drive device 26 and performing read processing. In addition, an application program for executing video editing processing can be read into the memory 22 and the hard disk device 23 via a communication device (not shown) connected to a network such as the Internet.

  FIG. 16 shows another example of a flowchart of the video editing process. Of the steps shown in FIG. 16, the same steps as those shown in FIG.

  The flowchart shown in FIG. 16 performs the processing of Step 311 to Step 315 instead of the processing of Step 305 to Step 308 shown in FIG. The function is realized. That is, each process including step 311 to step 315 illustrated in FIG. 16 is an example of a process included in the application program for the video editing process, and is an example of the video editing method and the video editing program according to the present disclosure. It is. Each process including the above-described steps 311 to 315 is executed by the processor 21.

  The processor 21 performs the processing of step 311 to step 315 for a pair of the object region to be processed and another object region having an overlapping portion in the affirmative determination route (YES) of step 303. Run. When two or more object regions satisfying the above-described expression (1) are detected for the object region to be processed during the process of step 303, the processor 21 determines that the object with the smallest overlapping area is the object. You may perform the process of step 311-step 315 sequentially from the pair of area | regions. In the following description, the object region having the smaller y coordinate of the upper left vertex in the coordinate system shown in FIG. 13 among the pair of object regions having overlapping portions is referred to as the object region Ra, and the other is the object. This is referred to as region Rb.

  In step 311, the processor 21 generates a constraint condition related to the reduction ratio applied to each object region by executing the processing of steps 321 to 327 shown in FIG. 17 for each of the two object regions Ra and Rb.

  FIG. 17 shows an example of a flowchart of processing for generating a constraint condition. The processing of step 321 to step 327 shown in FIG. 17 is an example of the processing of step 311 shown in FIG. 16, and the processing of each of these steps is executed by the processor 21 shown in FIG.

  First, the processor 21 refers to the object information holding unit 12 provided in the hard disk device 23 shown in FIG. 15 on the basis of the object IDs corresponding to the object regions Ra and Rb, so that the previous image can be handled. The reduction ratio applied to the object area to be acquired is acquired (step 321). For example, when the image with the image number j is an image to be processed, the processor 21 indicates two object regions Ra and Rb to be processed among the object information held corresponding to the image number j-1. Reference is made to object information corresponding to the object ID. Here, prior to the process for the image with the image number j, if the process for the image with the image number j-1 is completed, the object information corresponding to each object ID is handled as described later. Information indicating the reduction ratio applied to each object region is added. Therefore, the processor 21 refers to the object information held in the object information holding unit 12 corresponding to the image number j-1 and the above-described object ID, thereby corresponding to the object areas Ra and Rb to be processed. The reduction ratio applied in the processing of the previous image for one object region can be acquired. In the following description, the reduction ratios applied in the previous image processing for the two object areas corresponding to the object areas Ra and Rb to be processed are referred to as reduction ratios βa and βb, respectively. If the image with the image number 1 is the processing target, the processor 21 may omit the process of step 321 and set the numerical values “1” for the reduction ratios βa and βb.

  Next, the processor 21 calculates an upper limit value and a lower limit value indicating an allowable range for fluctuations in the reduction ratio, based on the reduction ratios βa and βb acquired in the process of step 321 (step 322). For example, the processor 21 multiplies the above-described reduction rates βa and βb by the coefficients (κ + 1) and (κ−1) expressed by using the above-described predetermined coefficient κ, so that the change in the reduction rate is changed. The upper limit value and the lower limit value of the allowable range may be obtained respectively.

  Next, the processor 21 determines whether or not each lower limit value of the allowable range obtained in the process of step 322 is equal to or larger than a preset minimum value αmin of the reduction rate (step 323).

  When at least one of the lower limit values of the allowable range described above is less than the minimum value αmin (No determination in step 323), the processor 21 replaces the lower limit value that is less than the minimum value αmin with the minimum value αmin (step 324). . When the processor 21 executes the process of step 324, the lower limit of the range of the reduction ratio applied to each object region Ra, Rb can be limited by a predetermined minimum value αmin.

  On the other hand, in the affirmative determination route (YES) of step 323 described above, the processor 21 omits the process of step 324. The processing of the affirmative determination route and the negative determination route in step 323 merges in step 325.

  In step 325, the processor 21 determines whether or not each upper limit value of the allowable range obtained in the processing in step 322 is equal to or less than a numerical value “1” that is a preset maximum value of the reduction ratio.

  When at least one of the upper limit values of the allowable range described above is equal to or greater than the maximum value “1” (No at Step 325), the processor 21 sets the upper limit value that is equal to or greater than the maximum value “1” to the maximum value “1”. (Step 326). By executing the processing of step 326 by the processor 21, the upper limit of the range of the reduction ratio applied to each object region Ra, Rb can be limited by a predetermined maximum value “1”.

  On the other hand, in the affirmative determination route (YES) of step 325 described above, the processor 21 omits the process of step 326. The processing of the affirmative determination route and the negative determination route in step 325 merges in step 327.

  In step 327, the processor 21 outputs the constraint condition indicating the upper limit and the lower limit of the reduction rate determined for each of the object regions Ra and Rb as described above, and ends the process of generating the constraint condition. Note that the processor 21 may execute the processes in steps 323 and 324 and the processes in steps 325 and 326 in the order opposite to the order shown in FIG.

  The processor 21 executes the processing from step 321 to step 327, thereby generating a constraint condition as shown in the above equation (10), and the reduction ratio value obtained by the parameter calculation processing shown in step 313 in FIG. Can be limited.

  In step 311 of FIG. 16, the processor 21 sets the allowable range indicated by the above-described minimum value αmin and the maximum value “1” for the reduction ratio, instead of the above-described processing in steps 321 to 327. Conditions may be set. Further, the processor 21 may execute the process of generating the above-described restriction condition for the reduction ratio after the process of step 312 described below or in parallel with the process of step 312.

  In step 312 shown in FIG. 16, the processor 21 minimizes the area Sd of the portion where the two object regions Ra and Rb overlap after similarity transformation and the center-of-gravity movement distances Da and Db for each object region. Find the evaluation formula to use. The processor 21 may execute the process of step 312 by executing the processes of step 331 to step 335 shown in FIG. 18, for example.

  FIG. 18 shows an example of a flowchart of processing for obtaining an evaluation formula. The processing of Step 331 to Step 335 shown in FIG. 18 is an example of the processing of Step 312 shown in FIG. 16, and the processing of each of these steps is executed by the processor 21 shown in FIG.

  First, the processor 21 calculates the ratio of the overlapping portions in the two object areas Ra and Rb to be processed in the object areas Ra and Rb (step 331). The processor 21 calculates the areas of the overlapping portions together with the areas of the object regions Ra and Rb based on the object information held in the object information holding unit 12 corresponding to the object regions Ra and Rb to be processed. The ratio described above may be calculated as the area ratio. Further, the processor 21 may calculate the above-described ratio based on a ratio between a value indicating the size such as the width or height of each of the object regions Ra and Rb and a value indicating the size of the overlapping portion.

  Next, the processor 21 compares the ratio calculated for each of the object regions Ra and Rb in step 331 with the above-described predetermined threshold value, and determines whether at least one ratio is equal to or greater than the threshold value (step 332). .

  When at least one of the ratios calculated for each of the object regions Ra and Rb is equal to or greater than the above-described threshold (affirmative determination in step 332), the processor 21 proceeds to the process of step 333, while on the other hand, in the case of a negative determination in step 332 Then, the process proceeds to step 334.

  In step 333, the processor 21 sets a larger value for the second weight ω2 applied to the movement distances Da and Db of the center of gravity than the first weight ω1 applied to the area Sd.

  On the other hand, in step 334, the processor 21 sets a value larger than the first weight ω1 applied to the above-described area Sd as the second weight ω2 applied to the movement distances Da and Db of the center of gravity.

  After completion of the processing in step 333 or 334 described above, the processor 21 proceeds to processing in step 335. In step 335, the processor 21 uses the first weight ω <b> 1 and the second weight ω <b> 2 whose values are set by any one of the processes to minimize the area Sd and the movement distances Da and Db of the center of gravity. An evaluation expression is generated and the generated evaluation expression is output.

  Next, the processor 21 performs optimal similarity transformation for each of the object regions Ra and Rb based on the evaluation formula generated in the process of step 312 described above and within the range of the constraint condition obtained in the process of step 311. Are calculated (step 313). For example, the processor 21 may execute the process of calculating the optimal similarity transformation parameter by executing the processes of Steps 341 to 344 shown in FIG. 19.

  FIG. 19 shows an example of a flowchart of processing for calculating parameters for similarity transformation. The process of Step 341 to Step 344 shown in FIG. 19 is an example of the process of Step 313 shown in FIG. 16, and the process of each of these steps is executed by the processor 21 shown in FIG.

  First, the processor 21 performs the Pareto solution under the constraint conditions for the reduction ratios αa and αb and the movement amounts dxa, dya, dab, and dyb of the reference points Ua and Ub generated in the process of step 311 described above. Is obtained (step 341). Here, the constraint conditions for the movement amounts dxa, dya, dxb, dyb of the reference points Ua, Ub may be set so as to indicate the ranges of the original object regions Ra, Rb, for example. For example, the processor 21 changes the reduction rates αa and αb of the object regions Ra and Rb and the movement amounts dxa, dya, dxb, and dyb of the reference points Ua and Ub within the range indicated by the above-described constraints. The area Sd and the movement distances Da and Db of the center of gravity are calculated. The processor 21, for example, uses the Pareto solution for the similarity transformation parameter for the object region Ra as a combination of the reduction rate αa and the movement amounts dxa and dya of the reference point Ua, which is the limit that the movement distance Da cannot be shortened unless the area Sd is increased. You can ask for. Similarly, the processor 21 can obtain a Pareto solution for the similarity transformation parameter for the object region Rb.

  Next, the processor 21 obtains a contact point between the graph V indicating the set of Pareto solutions as shown in FIG. 14 and the evaluation formula obtained in the processing of step 312 described above (step 342). The processor 21 plots the area Sd and the center-of-gravity movement distance Da given by each Pareto solution obtained, for example, corresponding to the object region Ra in the processing of step 341 described above in the coordinate space as shown in FIG. Thus, a graph V representing a set of Pareto solutions can be obtained. The processor 21 may obtain a contact point between the graph V indicating the Pareto solution for the object region Ra thus obtained and the straight line graph indicating the evaluation formula obtained in the process of step 312. Similarly, the processor 21 can obtain a contact point between the graph V indicating the Pareto solution for the object region Rb and the straight line graph indicating the evaluation formula obtained in the process of step 312.

  Next, the processor 21 uses the parameters indicated by the Pareto solution of the contact points obtained for the object regions Ra and Rb in the process of step 342 as parameters indicating the optimum similarity transformation applied to the two object regions Ra and Rb, respectively. Output (step 343). For example, the processor 21 reduces the reduction rate αa and the movement amount dxa that gives the area Sd and the movement distance Da of the center of gravity indicated by the contact point between the graph V indicating the set of Pareto solutions obtained for the object region Ra and the graph indicating the evaluation formula. , Dya are output as optimum parameters. Similarly, the processor 21 uses the reduction rate αb and the movement amounts dxb and dyb indicated by the contact points between the graph V indicating the set of Pareto solutions obtained for the object region Rb and the graph indicating the evaluation formula as optimum parameters. Output.

  In addition, the processor 21 adds the reduction ratios αa and αb included in the optimum parameters identified in the processing of step 342 described above to the object information corresponding to the object regions Ra and Rb (step 344). For example, the processor 21 passes the reduction ratios αa and αb specified as part of the optimum parameters to the object information holding unit 12 described above, and the image number indicating the image to be processed and the object ID indicating the object regions Ra and Rb. It is only necessary to add to the object information indicated by.

  As described above, every time the processor 21 specifies an optimum parameter corresponding to each object region extracted from the image indicated by the same image number of each partial video Vp as the processing target image of the summary video Vs. In addition, the reduction ratio included in each parameter is added to the corresponding object information. Thereby, in the subsequent processing, when executing the processing of step 311 in FIG. 16, the processor 21 converts each object from the object information held in the object information holding unit 12 to an image before the processing target image. It is possible to acquire the reduction ratio applied when the areas are overlapped.

  Thereafter, in step 314 shown in FIG. 16, the processor 21 executes similarity transformation to each of the object regions Ra and Rb by applying the optimum parameter obtained as described above. The processor 21 reduces the corresponding two object images by performing the above-described similarity transformation on the images of the object regions Ra and Rb held in the object information holding unit 12 in the processing of step 314. The position occupied by the images of these objects in the image is moved. Accordingly, it is possible to minimize the overlap of the images of the respective objects while preserving the positions in the images of the two objects corresponding to the object regions Ra and Rb.

  In this way, the processor 21 superimposes the images obtained as a result of the similarity transformation for the object regions Ra and Rb on the processing target images held in the summary video holding unit 232 shown in FIG. Step 315). The processor 21 may superimpose the image of the object region after the similarity transformation obtained in the process of step 314 on the image held in the summary video holding unit 232 corresponding to the image number indicating the image to be processed.

  In addition, the processor 21 reflects the similarity conversion result obtained for the object regions Ra and Rb as described above in the corresponding object information held in the object information holding unit 12, and is included in the image to be processed. May be used in the process of step 303.

  By executing the video editing program of the present disclosure described above by the processor 21, a summary video that reproduces a faithful movement route together with the attributes of these objects from the partial video Vp obtained by photographing the movement process of each object. Vs can be generated. In addition, in the summary video Vs obtained by the above-described processing, the change in the size of each object in each time-sequential image is suppressed, and the size of each object image is the same. It is maintained to the extent that the attributes of the object can be grasped. Also, as shown in FIG. 18, by controlling the weight of the evaluation formula according to the positional relationship of the object areas to be processed, even when the flow lines of a plurality of objects pass through the same part at the same timing The position of each object in the original partial video can be reflected in the position of the summary video Vs image.

  From the above detailed description, features and advantages of the embodiment will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Moreover, those who have ordinary knowledge in the technical field should be able to easily conceive various improvements and changes based on the above description. Therefore, there is no intention to limit the scope of the inventive embodiments to those described above, and appropriate modifications and equivalents included in the scope disclosed in the embodiments can be used.

Regarding the above description, the following items are further disclosed.
(Appendix 1)
Receiving a plurality of videos representing a movement process of each object with a predetermined background as a background, and in each image included in each video, a specifying unit for specifying an object region including the image of each object;
A detecting unit for detecting a portion in which each of the object regions corresponding to the object overlaps each other in each of the images in the summary video generated by superimposing images having the same relative time from the beginning of each video; ,
In each of the object regions, a calculation unit that obtains a reduction ratio for reducing the overlapping part by reducing the vertex from the overlapping part as a reference; and
An adjustment unit that adjusts the position of the object region so that the gravity center position of each object region reduced by the reduction ratio obtained by the calculation unit is close to the gravity center position of each object region before the reduction. A video editing apparatus characterized by comprising.
(Appendix 2)
In the video editing apparatus according to attachment 1,
The calculation unit includes:
When the ratio of the overlapping parts of the two object areas in each of the two object areas is smaller than a predetermined threshold, rather than shortening the moving distance of the center of gravity position before and after the reduction of each of the two object areas, Obtaining the reduction ratio and the amount of adjustment of the position of each object region in order to realize the reduction of the area of the overlapping portion with priority,
When at least one of the ratios is equal to or greater than the predetermined threshold, the reduction ratio and each object for realizing the reduction of the movement distance of the center of gravity position with priority over the reduction of the area of the overlapping portion A video editing apparatus characterized by obtaining an adjustment amount of a position of an area.
(Appendix 3)
In the video editing apparatus described in appendix 2,
The calculation unit includes:
The reduction ratio and each object region are minimized so as to minimize the sum of the product of the area of the overlapping portion and a predetermined first weight, and the product of the movement distance of the barycentric position and the predetermined second weight. A determination unit for determining an adjustment amount for the position of
When the ratio of the overlapping part of the two object areas in each of the two object areas is smaller than a predetermined threshold, the first weight is set to a value larger than the second weight, and at least the ratio A video editing apparatus comprising: a setting unit that sets a value greater than the first weight for the second weight when one of the predetermined threshold values is greater than or equal to the predetermined threshold.
(Appendix 4)
In the video editing apparatus according to any one of supplementary notes 1 to 3,
The calculation unit includes:
An image editing apparatus, wherein a reduction ratio to be applied to each object region is determined to be a value within a range indicated by a predetermined maximum value and a predetermined minimum value.
(Appendix 5)
In the video editing apparatus according to any one of appendix 1 to appendix 4,
The calculation unit
A video editing apparatus that obtains a reduction ratio to be applied to each object region so as to suppress variation in the size of an image of the same object included in the images that are continuous in time series in the summary video.
(Appendix 6)
Receiving a plurality of videos representing the movement process of each object with a predetermined background as a background, in each image included in each video, specify the object region including the image of each object,
In each image in the summary video generated by superimposing images having the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected,
In each object region, by reducing with respect to the vertex that is away from the overlapping portion as a reference, to obtain a reduction rate to reduce the overlapping portion,
Adjusting the position of the object region so that the gravity center position of each object region reduced by the reduction ratio obtained by the calculation unit is close to the gravity center position of each object region before the reduction. How to edit video.
(Appendix 7)
Receiving a plurality of videos representing the movement process of each object with a predetermined background as a background, in each image included in each video, specify the object region including the image of each object,
In each image in the summary video generated by superimposing images having the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected,
In each object region, by reducing with respect to the vertex that is away from the overlapping portion as a reference, to obtain a reduction rate to reduce the overlapping portion,
The computer adjusts the position of the object region so that the gravity center position of each object region reduced by the reduction ratio obtained by the calculation unit is close to the gravity center position of each object region before the reduction. A video editing program characterized by being executed.

DESCRIPTION OF SYMBOLS 1 ... Shooting device; 2 ... Video storage device; 3 ... Motion video extraction device; 4 ... Summary video storage device; 5 ... Display device; 10 ... Video editing device; 11 ... Specific part; Detection unit; 14 ... calculation unit; 15 ... reduction unit; 16 ... adjustment unit; 17 ... similarity conversion processing unit; 141 ... ratio calculation unit; 142 ... setting unit; 143 ... condition generation unit; 21; Processor; 22 ... Memory; 23 ... Hard disk device; 24 ... General-purpose interface; 25 ... Display control unit; 26 ... Optical drive device; 27 ... Removable disk

Claims (5)

Receiving a plurality of videos representing a movement process of each object with a predetermined background as a background, and in each image included in each video, a specifying unit for specifying an object region including the image of each object;
A detecting unit for detecting a portion in which each of the object regions corresponding to the object overlaps each other in each of the images in the summary video generated by superimposing images having the same relative time from the beginning of each video; ,
In each of the object regions, a calculation unit that obtains a reduction ratio for reducing the overlapping part by reducing the vertex from the overlapping part as a reference; and
An adjustment unit that adjusts the position of the object region so that the gravity center position of each object region reduced by the reduction ratio obtained by the calculation unit is close to the gravity center position of each object region before the reduction. A video editing apparatus characterized by comprising. The video editing apparatus according to claim 1,
The calculation unit includes:
When the ratio of the overlapping parts of the two object areas in each of the two object areas is smaller than a predetermined threshold, rather than shortening the moving distance of the center of gravity position before and after the reduction of each of the two object areas, Obtaining the reduction ratio and the amount of adjustment of the position of each object region in order to realize the reduction of the area of the overlapping portion with priority,
When at least one of the ratios is equal to or greater than the predetermined threshold, the reduction ratio and each object for realizing the reduction of the movement distance of the center of gravity position with priority over the reduction of the area of the overlapping portion A video editing apparatus characterized by obtaining an adjustment amount of a position of an area. The video editing apparatus according to claim 2, wherein
The calculation unit includes:
The reduction ratio and each object region are minimized so as to minimize the sum of the product of the area of the overlapping portion and a predetermined first weight, and the product of the movement distance of the barycentric position and the predetermined second weight. A determination unit for determining an adjustment amount for the position of
Wherein when the ratio of overlapping portions of the two object region occupied in each of the two object region is smaller than the predetermined threshold, a value greater than the second weight to the first weight, the ratio A video editing apparatus, comprising: a setting unit configured to set a value greater than the first weight for the second weight when at least one is equal to or greater than the predetermined threshold. Receiving a plurality of videos representing the movement process of each object with a predetermined background as a background, in each image included in each video, specify the object region including the image of each object,
In each image in the summary video generated by superimposing images having the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected,
In each object region, by reducing with respect to the vertex that is away from the overlapping portion as a reference, to obtain a reduction rate to reduce the overlapping portion,
Adjusting the position of the object region so that the gravity center position of each object region reduced at the obtained reduction ratio is close to the gravity center position of each object region before the reduction. . Receiving a plurality of videos representing the movement process of each object with a predetermined background as a background, in each image included in each video, specify the object region including the image of each object,
In each image in the summary video generated by superimposing images having the same relative time from the beginning of each video, a portion where the object regions corresponding to the objects overlap each other is detected,
In each object region, by reducing with respect to the vertex that is away from the overlapping portion as a reference, to obtain a reduction rate to reduce the overlapping portion,
Causing the computer to execute a process of adjusting the position of the object region so that the gravity center position of each object region reduced at the obtained reduction ratio is close to the gravity center position of each object region before the reduction. Feature video editing program.

Images