JP2019096130A - Morphing image generation device, and morphing image generation method - Google Patents

Abstract

To improve quality of a morphing image.SOLUTION: A morphing image generation device 1 has: an image acquisition unit 131 that acquires a first captured image and a second capture image; a propagation control unit 132 that makes a plurality of processing layers included in a machine learning model capable of outputting a category of a subject included in an input image on the basis of the input image propagate each of the first captured image and the second captured image; an extraction unit 133 that extracts one or more first captured image outputs and one or more second captured image outputs commonly activating in either processing layer of a rear-stage processing layer and a pre-stage processing layer selected from the plurality of processing layers; a characteristic point detection unit 137 that detects one or more first captured image characteristic points on the basis of the one or more first captured image outputs, and one or more second captured image characteristic points on the basis of the one or more second captured image outputs; and an intermediate image generation unit 139 that generates one or more intermediate images representing a process in which a subject changes step by step on the basis of the one or more first captured image characteristic points and the one or more second captured image characteristic points.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a morphing image generation apparatus and a morphing image generation method for generating a morphing image using a machine learning model.

  There is known an apparatus for generating a morphing image representing a process of changing an object stepwise based on an image in which an object before and after the change is captured. Patent Document 1 discloses a technique of generating a morphing image by generating an intermediate image of images of objects before and after changing.

Unexamined-Japanese-Patent No. 2001-076177

  In the method of generating a morphing image, a vertex in the image before the subject changes and a vertex of the image after the subject changes are extracted as feature points, and an intermediate value between the extracted feature points is calculated. Generate an image. However, when there is no common clear feature point between the images before and after the change, it is difficult to extract feature points associated in the images before and after the subject changes. As a result, there has been a problem that it is not possible to generate a smoothly changing high-quality morphing image.

  Therefore, the present invention has been made in view of these points, and it is an object of the present invention to provide a morphing image generation apparatus and a morphing image generation method that can improve the quality of morphing images.

  A morphing image generation device according to a first aspect of the present invention includes a first image which is an image before at least a part of a subject changes and a second image which is an image after at least a part of the subject changes. And a plurality of processes included in a machine learning model capable of outputting the type of the subject included in the image based on the input image to each of the first image and the second image. The activation control is common to both the propagation control unit for propagating the layer, the post-processing layer selected from the plurality of processing layers, and the pre-processing layer that is the processing layer immediately before the post-processing layer One or more output from the post-processing layer and the pre-processing layer based on the second image and the one or more first image outputs output from the post-processing layer and the pre-processing layer based on the first image Second picture An extraction unit for extracting an output; detecting one or more first image feature points based on the one or more first image outputs; and one or more second images based on the one or more second image outputs One or more step-by-step processes of changing the subject based on a feature point detection unit for detecting feature points, the one or more first image feature points and the one or more second image feature points And an intermediate image generation unit that generates an intermediate image.

  A plurality of second-stage first image outputs output from the second-stage processing layer by causing the first image to propagate in order of the first-stage processing layer and the second-stage processing layer which are a part of the plurality of processing layers; One or more post-stage first image outputs activated in common from the plurality of post-stage second image outputs outputted from the post-stage processing layer by the second image being propagated in order of the pre-stage processing layer and the post-stage processing layer Output from the pre-processing layer that has caused the activation of the one or more subsequent-stage first image outputs and the one or more subsequent-stage second image outputs; Among the plurality of first-stage first image outputs and the plurality of second-stage second image outputs output from the first-stage processing layer, one or more first-stage first image outputs and one or more second-stage second images activated in common Pre-stage extraction unit that extracts the output , It may have.

  The pre-stage extraction unit is configured to output the one or more pre-stage first image outputs and the one or more pre-stage first image outputs based on an activated size of the plurality of pre-stage first image outputs and the plurality of pre-stage second image outputs. The pre-stage second image output may be extracted.

The machine learning model may include a convolutional neural network, and the post-processing layer may be any of an output layer, a total connection layer, a normalization layer, a pooling layer, and a convolution layer.
The pre-processing layer may be any of a total bonding layer, a normalization layer, a pooling layer, a convolution layer, and an input layer.

  In the case where the last tail layer which is the last treatment layer among the plurality of treatment layers is selected as the second treatment layer, the extraction unit is configured to activate the one or more first common layers in the last tail layer. When there is no image output and the one or more second image outputs, the one or more first image outputs and the one or more second image outputs that are commonly activated in the processing layer before the last tail layer are extracted You may

  The morphing image generation device is configured to generate a part of the first image features based on the mutual correspondence relation from the one or more first image feature points specified by the feature point detection unit and the one or more second image feature points. The intermediate image generation unit further includes a selection unit that selects a point and a part of second image feature points, and the intermediate image generation unit is configured to select the part of the first image feature points and the part of the second image feature points. The image processing method may generate one or more intermediate images stepwise representing the process of changing the subject.

  The image acquisition unit acquires a plurality of second images in which a subject of the same type as the changed subject and having a different shape is captured, and the intermediate image generation unit is configured to receive the one or more first image feature points One second image may be selected from the plurality of second images based on the one or more second image feature points based on each of the plurality of second images.

The intermediate image generation unit may select one second image from the plurality of second images in which the number of second image feature points corresponding to the first image feature points is equal to or more than a predetermined reference value.
The morphing image generation device further includes an instruction receiving unit receiving an instruction to select a processing layer to be used as the post-processing layer among the plurality of processing layers, and the extraction unit receives the instruction received by the instruction receiving unit. The processing layer indicated by the instruction may be used as the subsequent processing layer.

  The extraction unit selects one of the plurality of processing layers as the post-processing layer and extracts the one or more first image outputs and the one or more second image outputs, and then the pre-processing layer. The selected processing layer may be selected as the post-processing layer to extract another one or more first image outputs and one or more second image outputs.

  A morphing image generation method according to a second aspect of the present invention includes a first image which is an image before at least a part of a subject changes and a second image which is an image after at least a part of the subject changes. And a plurality of processing layers included in a machine learning model capable of outputting, for each of the first image and the second image, the type of the subject included in the image based on the input image. The first image, which is commonly activated in both the processing step of propagating, the post-processing layer selected from the plurality of processing layers, and the pre-processing layer that is the processing layer immediately before the post-processing layer And one or more second image outputs from the subsequent processing layer and the preceding processing layer based on the one or more first image outputs outputted from the subsequent processing layer and the preceding processing layer and the second image. Extracting one or more first image feature points based on the one or more first image outputs, and one or more second image feature points based on the one or more second image outputs. Based on the one or more first image feature points and the one or more second image feature points, to generate one or more intermediate images representing the process of changing the object in stages And step.

  The step of extracting includes a plurality of subsequent first image outputs output from the subsequent processing layer by propagating the first image in order of the preceding processing layer and the subsequent processing layer that are a part of the plurality of processing layers; And one or more post-stage first image outputs activated in common from a plurality of post-stage second image outputs outputted from the post-stage processing layer by propagating the second image in order of the pre-stage processing layer and the post-stage processing layer And a pre-stage extraction step of extracting one or more second-stage second image outputs, and an output from the first-stage processing layer that has caused the activation of the one or more second-stage first image outputs and the one or more second-stage second image outputs. Among the plurality of pre-stage first image outputs and the plurality of pre-stage second image outputs output from the pre-stage processing layer, one or more pre-stage first image outputs and one or more pre-stage second images activated in common. Extract image output And subsequent extraction step of, may have.

  In the morphing image generation method, after performing the pre-stage extraction step, the one or more pre-stage first image outputs and the one or more pre-stage second image outputs, the plurality of post-stage first image outputs and the plurality of post-stages The second stage extraction step may be executed as a second image output.

  The morphing image generation method may execute the post-stage extraction step and the pre-stage extraction step on each of the plurality of processing layers.

  According to the present invention, it is possible to improve the quality of the morphing image.

It is a figure for demonstrating the outline | summary of the process which produces | generates a morphing image. It is a figure which shows an example of a structure of a machine learning model. It is a figure showing composition of a morphing picture generation device. It is a figure for demonstrating the extraction process which an extraction part performs. It is a figure for demonstrating the extraction process which an extraction part performs. It is a figure for demonstrating the extraction process which an extraction part performs. It is a figure for demonstrating the extraction process which an extraction part performs. It is a figure for demonstrating the extraction process which an extraction part performs. It is a figure for demonstrating the extraction process which an extraction part performs. It is a flowchart which shows the flow of the process which a morphing image generation apparatus performs. It is a flowchart which shows the flow of the process which an extraction part performs.

[Overview of Morphing Image Generation Device 1]
FIG. 1 is a diagram for describing an outline of processing for generating a morphing image. The morphing image generation device 1 is, for example, a PC (Personal Computer). The morphing image generation device 1 is a device that generates a morphing image using a machine learning model M based on a plurality of images in which subjects before and after change are shown. The morphing image A shown in FIG. 1 is an image that gradually shows the process of changing from the face of the person appearing in the first image A1 to the car appearing in the second image A3. In the example shown in FIG. 1, the human eye corresponds to the headlight of a car, and the human mouth corresponds to the radiator grille located between the left and right headlights of the car.

  The morphing image generation device 1 acquires a first image A1 which is an image before at least a part of the subject changes, and a second image A3 which is an image after at least a part of the subject changes. One (1)). The first image A1 shown in FIG. 1 is an image in which a human face is a subject. The second image A3 shown in FIG. 1 is an image in which a car is a subject.

  The morphing image generation device 1 inputs each of the acquired first image A1 and second image A3 to the machine learning model M, and propagates a plurality of processing layers included in the machine learning model M ((2) in FIG. 1). ). The machine learning model M is a model learned to output the type of the subject included in the image based on the input image.

  FIG. 2 is a diagram showing an example of the configuration of the machine learning model M. As shown in FIG. The machine learning model M includes a convolutional neural network (CNN). In this case, the machine learning model M includes the input layer M1, the first convolutional layer M2, the second convolutional layer M3, the first pooling layer M4, the normalization layer M5, the third convolutional layer M6, the second It has a pooling layer M7, a first total bonding layer M8, a second total bonding layer M9, and an output layer M10. In the present specification, of the two adjacent processing layers, the processing layer on the upstream side when the first image A1 and the second image A3 propagate is referred to as the pre-processing layer, and the processing layer on the downstream side is the post-processing layer. It is called.

  The processing layer that can be the post-processing layer is a first convolutional layer M2, a second convolutional layer M3, a first pooling layer M4, a normalization layer M5, a third convolutional layer M6, a second pooling layer M7, a second One of the total bonding layer M8, the second total bonding layer M9, and the output layer M10. Further, the processing layer which can be the pre-processing layer is an input layer M1, a first convolutional layer M2, a second convolutional layer M3, a first pooling layer M4, a normalization layer M5, a third convolutional layer M6, a second Pooling layer M7, the first total bonding layer M8, and the second total bonding layer M9. The morphing image generation device 1 inputs the acquired image to the machine learning model M, and propagates each processing layer from the input layer M1 to the output layer M10 in a forward direction, that is, infers the object shown in the image. Output the type.

  Returning to FIG. 1, the morphing image generation device 1 uses the calculation result in each processing layer at which the machine learning model M has output the type of the subject, that is, the first feature value with a high degree of abstraction by deep learning. A feature point common to the image A1 and the second image A3 is detected ((3) in FIG. 1). Here, the morphing image generation device 1 performs the detection of the common feature points in the order opposite to the order of propagation. By doing this, the morphing image generation device 1 can detect a feature point based on a feature amount having a high degree of abstraction.

  The morphing image generation device 1 detects the common feature points to detect the eyes and the face of the person's face in the first image A1, the car's headlights and the radiator grille in the second image A3, respectively. Detect that there is a correspondence. The correspondence relationship is a relationship in which the pixel included in the first image indicated by the first image feature point matches the pixel included in the second image indicated by the second image feature point.

Then, based on the feature points of the first image A1 and the second image A3 in the detected correspondence relationship, the morphing image generation device 1 generates an intermediate image A2 that represents the process of changing the object in stages ((1) FIG. 1 (4)). By doing this, the morphing image generation device 1 can improve the quality of the morphing image.
Hereinafter, details of the morphing image generation device 1 will be described.

[Configuration of Morphing Image Generating Device 1]
FIG. 3 is a diagram showing the configuration of the morphing image generation device 1. The morphing image generation device 1 includes an operation unit 11, a storage unit 12, and a control unit 13.

The operation unit 11 is an input device that receives a user's operation.
The storage unit 12 is, for example, a storage medium such as a read only memory (ROM), a random access memory (RAM), and a hard disk. The storage unit 12 stores various programs that the control unit 13 executes. The storage unit 12 stores the first image and the second image.

  The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 controls a function related to the morphing image generation device 1 by executing a program stored in the storage unit 12. The control unit 13 functions as an image acquisition unit 131, a propagation control unit 132, an extraction unit 133, an instruction reception unit 136, a feature point detection unit 137, a selection unit 138, and an intermediate image generation unit 139 by executing a program. .

  The image acquisition unit 131 acquires the first image and the second image stored in the storage unit 12. The image acquisition unit 131 inputs the acquired first image and second image to the propagation control unit 132.

  The propagation control unit 132 propagates a plurality of processing layers included in the machine learning model M to each of the first image and the second image. In the example illustrated in FIG. 2, the propagation control unit 132 causes each processing layer from the input layer M1 to the output layer M10 included in the machine learning model M to propagate in order to each of the first image and the second image.

  The extraction unit 133 is activated on the basis of the first image, which is activated in common in both the post-processing layer selected from the plurality of processing layers and the pre-processing layer that is the processing layer immediately before the post-processing layer. The one or more second image outputs output from the post-processing layer and the pre-processing layer are extracted based on the one or more first image outputs output from the processing layer and the pre-processing layer and the second image. Although the details of the extraction process performed by the extraction unit 133 will be described later, the extraction unit 133 is configured to output a second image output and a second image output that are part of the first image output activated commonly in the second processing layer. A post-stage extraction unit 134 that extracts a post-stage second image output that is a part, and a part of pre-stage first image output and a second image output that are part of a first image output that is activated commonly in the pre-stage processing layer And a pre-stage extraction unit 135 for extracting the pre-stage second image output.

  The first image output and the second image output extracted by the extraction unit 133 are information indicating an activated unit among the plurality of units included in the processing layer. A unit is one or more pixels included in an image. The definition of activation may be, for example, when the product of the output value of the unit or the output value of the unit and the weight of the combination of the unit exceeds a predetermined threshold, or a predetermined number or a predetermined number in the descending order of output. It may be included in the ratio. In addition, in the processing layers other than the total bonding layer, for example, the channels may be included in a predetermined number or a predetermined ratio in descending order of the output. The channel is an output that is convoluted for each filter.

  It is preferable that the extraction unit 133 select the last tail layer, which is the last treatment layer, among the plurality of treatment layers as the post-stage treatment layer. However, there may be cases where there is no first image output and second image output that are commonly activated in the last layer. Therefore, when the last tail layer is selected as the post-stage treatment layer among the plurality of treatment layers, the extraction unit 133 outputs one or more first image outputs and one or more second images that are commonly activated in the last tail layer. When there is no image output, one or more first image outputs and one or more second image outputs which are commonly activated in the processing layer before the last layer may be extracted.

  For example, when the extraction unit 133 selects the output layer M10, which is the last layer, as the post-processing layer, one or more first image outputs and one or more second image outputs that are commonly activated in the output layer M10 I do not have In this case, the extraction unit 133 repeatedly searches for one or more first image outputs and one or more second image outputs that are commonly activated with respect to each processing layer before the output layer M10. For example, when there is one or more first image outputs and one or more second image outputs that are commonly activated in the second total bonding layer M9, which is the processing layer immediately before the output layer M10, the extraction unit 133 , The second total bonding layer M9 is selected as the post-processing layer. Then, the extraction unit 133 extracts one or more first image outputs and one or more second image outputs that are commonly activated in the second entire combined layer M9 selected as the post-stage processing layer. By doing this, the extraction unit 133 can associate subjects shown in each of the first image and the second image even when there are few matching areas.

  The extraction unit 133 may select the processing layer specified by the user as the post-processing layer. Specifically, the instruction receiving unit 136 receives, via the operation unit 11, an instruction to select a processing layer to be used as a post-processing layer among a plurality of processing layers. Then, the extraction unit 133 uses the processing layer indicated by the instruction received by the instruction receiving unit 136 as a post-processing layer. In the example illustrated in FIG. 2, when the user selects the second total bonding layer M9 in the example illustrated in FIG. 2, the extraction unit 133 uses the second total bonding layer M9 indicated by the instruction received by the instruction receiving unit 136 as the post-processing layer. use. The extraction unit 133 inputs the extracted first image output and second image output to the feature point detection unit 137.

  The feature point detection unit 137 detects one or more first image feature points based on one or more first image outputs, and detects one or more second image feature points based on one or more second image outputs. Do. Specifically, the feature point detection unit 137 first searches for a corresponding feature point based on one or more first image outputs and one or more second image outputs. Then, the feature point detection unit 137 determines one or more first image feature points based on one or more first image outputs in a corresponding relationship, and one or more second image feature points based on one or more second image outputs. To detect The feature point detection unit 137 inputs the detected first image feature point and second image feature point to the selection unit 138.

  The selection unit 138 uses the one or more first image feature points specified by the feature point detection unit 137 and the one or more second image feature points to select one or more of the first image feature points and a part of the first image feature points. The second image feature point of is selected. Specifically, the selection unit 138 removes the erroneously detected correspondence, and selects one or more first image feature points and one or more second image feature points based on the correspondence relationship after the removal. The erroneous detection of the correspondence is a state in which the first image feature point and the second image feature point in the correspondence relationship are mutually wrinkled, for example, when the movement path of the feature point intersects in the process of generating the intermediate image is there. The selection unit 138 removes the correspondence by narrowing down, for example, based on the RANSAC (Random Sampling Consensus) method or the Least Median of Square (LMedS) method. In addition, the selection unit 138 may select the correspondence selected by the user via the instruction reception unit 136.

  Also, for example, when the second image is a CG (Computer Graphics) image including a coordinate system in a three-dimensional space, the second image of four or more of the second image feature points detected by the feature point detection unit 137 It is assumed that the feature points are found to be on the same straight line in real space. In this case, the selection unit 138 first calculates the complex ratio at each of the first image feature point and the second image feature point based on the correspondence relationship, and the first image feature point and the second image feature whose values are significantly different. Determine if there is a point. Then, the selecting unit 138 determines that the first image feature point and the second image feature point that are determined to have significantly different values as the misdetected feature point, and the first image feature point based on another correspondence relationship and The second image feature point is selected.

  The intermediate image generation unit 139 generates one or more intermediate images stepwise representing the process of changing the subject based on the one or more first image feature points and the one or more second image feature points. Specifically, the intermediate image generation unit 139 generates an object based on a part of first image feature points and a part of second image feature points based on the correspondence after the selection unit 138 removes the erroneous detection. Generates one or more intermediate images that gradually represent the process of change.

  For example, the intermediate image generation unit 139 is based on the coordinates in the pixel of the first image indicated by the first image feature point and the coordinates in the pixel of the second image indicated by the second image feature point corresponding to the first image feature point. Calculate the change step. Then, the intermediate image generation unit 139 generates one or more intermediate images based on the calculated change step. A known method can be used to calculate the change step.

  The intermediate image generation unit 139 may generate an interpolation feature point that interpolates between the corresponding first image feature point and the second image feature point when the predetermined condition is satisfied. Specifically, when a predetermined condition is satisfied, the intermediate image generation unit 139 generates an interpolation feature point by interpolating between the corresponding first image feature point and the second image feature point. An intermediate image may be generated based on the interpolated feature points. The predetermined condition is, for example, when the types of subjects of the first image and the second image are different, or when the number of intermediate images to be generated is large.

  By the way, when the types of subjects in the first image and the second image are different, there is a possibility that the corresponding feature points can not be detected depending on the shape of the subject selected as the second image. Therefore, the intermediate image generation unit 139 may select an image suitable for morphing from the plurality of second images. Specifically, first, the image acquisition unit 131 acquires a plurality of second images in which subjects of the same type as the subject after the change and of different shapes are captured. In this case, the intermediate image generation unit 139 generates one or more second images based on the one or more first image feature points and the plurality of second images based on the one or more first image outputs detected by the feature point detection unit 137. One second image is selected from the plurality of second images based on the image feature point.

  For example, the intermediate image generation unit 139 selects one second image from a plurality of second images in which the number of second image feature points corresponding to the first image feature points is equal to or more than a predetermined reference value. The predetermined reference value is, for example, a variation value that changes depending on whether the types of subjects of the first image and the second image are the same. When the types of subjects of the first image and the second image are the same, the intermediate image generation unit 139 sets the reference value lower than in the case where the types of subjects of the first image and the second image are different. On the other hand, when the types of subjects of the first image and the second image are different, the intermediate image generation unit 139 increases the reference value compared to the case where the types of subjects of the first image and the second image are the same. By doing this, the intermediate image generation unit 139 can improve the quality of the morphing image.

  Further, for example, the intermediate image generation unit 139 selects, from among the plurality of second images, the second image having the largest number of second image feature points corresponding to the first image feature points based on the first image. Specifically, it is assumed that the user designates a specific area (an area of the eye or mouth of the first image A1 in the example shown in FIG. 1) from the first image. In this case, the intermediate image generation unit 139 corresponds to the first image feature point included in the specific area in the first image specified by the user via the instruction reception unit 136 among the plurality of second images. The second image having the largest number of second image feature points is selected. Then, the intermediate image generation unit 139 generates one or more intermediate images based on the first image and the selected second image. By doing this, the intermediate image generation unit 139 can generate a morphing image intended by the user. The intermediate image generation unit 139 causes the storage unit 12 to store the generated intermediate image.

[Extraction processing]
Subsequently, the extraction process performed by the extraction unit 133 will be described. As described above, the extraction unit 133 includes the post-stage extraction unit 134 and the pre-stage extraction unit 135. The second-stage extraction unit 134 generates a plurality of second-stage first image outputs output from the second-stage processing layer by propagating the first image in order of the first-stage processing layer and the second-stage processing layer that are part of the plurality of processing layers. The image is propagated in the order of the pre-stage processing layer and the post-stage processing layer From the plurality of post-stage second image outputs outputted from the post-stage processing layer, one or more post-stage first image outputs commonly activated and one or more post-stages Extract the second image output.

  The pre-stage extraction unit 135 includes a plurality of pre-stage first image outputs output from a pre-stage processing layer that has caused one or more post-stage first image outputs and one or more post-stage second image outputs to be activated. Among the plurality of pre-stage second image outputs output from the image processing apparatus, one or more pre-stage first image outputs and one or more pre-stage second image outputs that are activated in common are extracted.

  4 to 9 are diagrams for explaining the extraction process performed by the extraction unit 133. FIG. FIGS. 4 to 9 show the state of propagation from the pre-processing layer to the post-processing layer. In FIG. 4 to FIG. 9, the connecting line connecting the units shown by the solid line shows that there is an output from the connecting unit, and the connecting line shown by the broken line shows that there is no output from the connecting unit. Also, the thickness of the line indicating the connecting line indicates the size of the output from the unit to be connected.

  In the case of FIG. 4, the post-stage treatment layer is the last layer (eg, output layer or total bonding layer) or the treatment layer (full bonding layer or pooling layer) prior to the last layer selected by the extraction unit 133. The pre-processing layer is a processing layer (for example, all bonding layer or pooling layer) immediately before the post-processing layer. In FIG. 4, it is assumed that the post-processing layer is the output layer M20 and the pre-processing layer is the total bonding layer M19.

  FIG. 4A shows a state before extraction, and FIG. 4B shows a state after extraction. In the first image, in the output layer M20, the units U5 and U8 are activated, and in the total bonding layer M19, the units U2, U5, U6, U7 and U8 are activated. In the second image, in the output layer M20, the units U3 and U5 are activated, and in the total bonding layer M19, the units U2, U4, U5, and U8 are activated.

  In this case, the post-stage extraction unit 134 outputs units U5 and U8 that are first-stage image outputs output from the output layer M20, which is the post-stage processing layer, and units that are second-stage image outputs that are output from the output layer M20. Compare U3 and U5. Then, the post-stage extraction unit 134 extracts the unit U5 of the post-stage first image output and the unit U5 of the post-stage second image output, which are commonly activated.

  Subsequently, the pre-stage extraction unit 135 sets units U2, U5, which are pre-stage first image outputs that are output from all the combined layers M19 that are pre-stage processing layers that are responsible for activating unit U5 of the post-stage first image output. U6 and units U2, U5, and U8, which are the pre-stage second image outputs output from all the combined layers M19 that cause activation of the unit U5 of the post-stage second image output, are compared. Then, the pre-stage extraction unit 135 extracts the units U2 and U5 of the pre-stage first image output and the units U2 and U5 of the pre-stage second image output, which are activated in common.

  When the extraction unit 133 extracts the output from the output layer M20 to the total coupling layer M19, the extraction unit 133 extracts an output for the next processing layer. Specifically, the extraction unit 133 performs the process of extracting the first image output and the second image output that are commonly activated for each processing layer in the reverse order of the order in which the propagation control unit 132 propagates the process. Repeat with. More specifically, the extraction unit 133 selects one of the plurality of processing layers as a post-processing layer and extracts one or more first image outputs and one or more second image outputs, and then performs pre-processing. The processing layer selected as the layer is selected as the post-processing layer to extract another one or more first image outputs and one or more second image outputs. By doing this, the extraction unit 133 can enhance the accuracy of comparison with the first image and the second image.

  FIG. 5 shows a state in which the pre-processing layer propagates to the post-processing layer based on the first image. FIG. 6 shows a state in which the pre-processing layer propagates to the post-processing layer based on the second image. In the case of FIG. 5 and FIG. 6, the post-processing layer is the total bonding layer M18, and the pre-processing layer is a processing layer (for example, a pooling layer or a convolution layer) other than the total bonding layer. In FIGS. 5 and 6, the pre-processing layer is described as the pooling layer M17. In FIGS. 5 and 6, the pre-processing layer has three channels. The upper first channel includes units U11, U12, U13, U14, and U15. The middle second channel includes units U21, U22, U23, U24, and U25. The lower third channel includes units U31, U32, U33, U34, and U35.

  In the first image, in the pooling layer M17, the unit U13 included in the first channel and the units U21 and U24 included in the second channel are activated. In the second image, in the entire bonding layer M18, units U22, U24, U25 included in the second channel and units U32, U33 included in the third channel are activated.

  The pre-stage extraction unit 135 outputs the unit U5 of the pre-stage first image output and the post-stage second image output that are output from the pooling layer M17 that is the pre-stage processing layer that is the factor that activates the unit U5 of the post-stage first image output. The pre-stage second image output outputted from the pooling layer M17 which has caused the activation is compared. The pre-stage extraction unit 135 checks the presence or absence of an activated unit, and unit U13 included in the first channel of the pre-stage first image output activated and U21 and U24 included in the second channel, and the pre-stage second image Note the units U22, U24, U25 included in the second channel of output and U32, U33 included in the third channel.

  Since the pre-stage extraction unit 135 uses the second channel as a channel in which a unit activated in both the pre-stage first image output and the pre-stage second image output is present, the first stage image output first The units U21 and U24 included in the two channels and the units U22, U24 and U25 included in the second channel of the pre-stage second image output are extracted.

  In the case of FIG. 7, the post-processing layer is a pooling layer M16, and the pre-processing layer is a processing layer other than the pooling layer (for example, a convolution layer or a normalization layer). In FIG. 7, the pre-processing layer is described as the convolution layer M15. Further, in FIG. 7, the pre-stage processing layer is described as having one channel. In the first image, the pooling layer M16 has the unit U5 activated, and the convolution layer M15 has the units U3 and U5 activated. In the second image, the pooling layer M16 has the unit U3 activated, and the convolution layer M15 has the units U3 and U4 activated.

  Here, in the pooling layer that compresses the image, the extraction unit 133 extracts the output based on the degree of activation for each channel among the plurality of units coupled from the immediately previous processing layer to the pooling layer. Do. Specifically, the pre-stage extraction unit 135 selects one or more pre-stage first image outputs and one or more pre-stage first image outputs based on the activated size of the plurality of pre-stage first image outputs and the plurality of pre-stage second image outputs. To extract the previous second image output. For example, among the plurality of first-stage first image outputs and the plurality of second-stage second image outputs, the first-stage extraction unit 135 selects one or more first-stage first image outputs and one or more second-stage second images that are most greatly activated for each channel. Extract image output.

  In this case, the post-stage extraction unit 134 selects the pooling layer M16 selected as the pre-stage processing layer in the previous extraction process, and outputs the unit U5 of the post-stage first image output output from the pooling layer M16 and the pooling layer M16. The unit U3 of the output second stage second image output is extracted. The pre-stage extraction unit 135 is, among the units U3 and U5 of the post-stage first image output and the units U3 and U4 of the post-stage second image output, the unit U5 of the pre-stage first image output activated most for each channel, And extract the unit U4 of the previous second image output. By doing this, the pre-stage extraction unit 135 can specify an area to be a feature in the image.

  In the case of FIG. 8, the post-processing layer is a convolution layer M14, and the pre-processing layer is another processing layer (for example, a normalization layer or a pooling layer) including the convolution layer. In FIG. 8, the pre-processing layer is described as the normalization layer M13. Further, in FIG. 8, the pre-processing layer is described as having one channel. In the first image, the convolution layer M14 has the unit U5 activated, and the normalization layer M13 has the units U3, U5 and U6 activated. In the second image, the convolution layer M14 has the unit U3 activated, and the normalization layer M13 has the units U3, U4 and U5 activated.

  In this case, the post-stage extraction unit 134 selects the convolution layer M14 selected as the pre-stage processing layer in the previous extraction process, and outputs the unit U5 of the post-stage first image output output from the convolution layer M14 and the convolution layer M14. The unit U3 of the output second stage second image output is extracted.

  Subsequently, the pre-stage extraction unit 135 outputs the pre-stage first image output and the post-stage second image output from the normalization layer M13, which is the pre-stage processing layer that is the cause of activating the unit U5 of the post-stage first image output. The pre-stage second image output outputted from the normalization layer M13, which is the pre-processing layer that has caused the unit U4 to be activated, is compared. Here, when the post-stage processing layer is a convolution layer, the pre-stage extraction unit 135 outputs the first stage image output among the plurality of units of the pre-stage processing layer coupled to the unit extracted by the post-stage extraction unit 134 from the post-stage processing layer. A unit whose position is relatively the same in the pre-stage second image output and whose channel is common is extracted. In this case, the pre-stage extraction unit 135 sets the units U5 and U6 of the pre-stage first image output as units whose positions are relatively the same between the pre-stage first image output and the pre-stage second image output and have a common channel. And extract the units U3 and U4 of the pre-stage second image output.

  In the case of FIG. 9, the post-processing layer is a normalization layer M12, and the pre-processing layer is a processing layer other than the normalization layer (for example, a convolution layer or a pooling layer). In FIG. 9, the pre-processing layer is described as the pooling layer M11. Further, in FIG. 9, the pre-processing layer is described as having one channel. In the first image, the normalization layer M12 has the unit U5 activated. In the second image, the normalization layer M12 has the unit U3 activated.

  Here, in the normalized layer that performs preprocessing on the image, the extraction unit 133 is the center of the plurality of units included in the pre-processing layer coupled to the unit activated in the post-processing layer. Extract a unit In this case, the post-stage extraction unit 134 outputs the unit U5 of the post-stage first image output outputted from the normalization layer M12 selected as the post-stage processing layer, and the unit U3 of the post-stage second image output outputted from the normalization layer M12. Extract

  Then, the pre-stage extraction unit 135 extracts the central unit U5 among the units of the pooling layer M11 coupled to the unit U5 of the post-stage first image output output from the normalization layer M12. Similarly, the pre-stage extraction unit 135 extracts the central unit U3 among the units of the pooling layer M11 coupled to the unit U3 of the post-stage second image output output from the normalization layer M12.

  It is preferable that the extraction unit 133 repeatedly perform the above-described extraction processing up to the input layer. However, the extraction unit 133 may end the processing on the way (for example, the pooling layer or the normalization layer) without performing the extraction processing up to the first processing layer. As described above, the extraction unit 133 can extract an output with a high degree of abstraction by performing the extraction process in the reverse order to the order in which the propagation control unit 132 propagates.

[Process of morphing image generation device 1]
Subsequently, the flow of processing performed by the morphing image generation device 1 will be described. FIG. 10 is a flowchart showing the flow of processing performed by the morphing image generation device 1. The flowchart starts when the morphing image generation device 1 stores the first image and the second image in the storage unit 12 and receives an operation to execute processing for generating a morphing image.

  The image acquisition unit 131 acquires the first image and the second image stored in the storage unit 12 (S1). The image acquisition unit 131 inputs the acquired first image and second image to the propagation control unit 132. The propagation control unit 132 includes a plurality of processing layers from the input layer M1 to the output layer M10 included in the machine learning model M in each of the first image and the second image input from the image acquisition unit 131 as the input layer M1. To propagate sequentially (S2).

  The extraction unit 133 performs a process of extracting one or more first image outputs and one or more second image outputs that are commonly activated in both the post-stage processing layer and the pre-stage processing layer (S3). FIG. 11 is a flowchart showing the flow of processing performed by the extraction unit 133. The extraction unit 133 determines whether the instruction receiving unit 136 receives an instruction to select a processing layer to be used as a post-processing layer among the plurality of processing layers via the operation unit 11 (S31).

  When the extraction unit 133 determines that the instruction receiving unit 136 receives an instruction, the extraction unit 133 selects the processing layer indicated by the instruction received by the instruction receiving unit 136 as a post-processing layer (S32). For example, when the extraction unit 133 determines that the instruction receiving unit 136 receives an instruction indicating the first entire combined layer M8, the extraction unit 133 performs post-processing on the first all combined layer M8 indicated by the instruction received by the instruction receiving unit 136. Use as a layer. On the other hand, when the extraction unit 133 determines that the instruction reception unit 136 does not receive the instruction, the extraction unit 133 outputs one or more first image outputs and one or more that are commonly activated in the last tail layer (for example, the output layer M10). It is determined whether there is a second image output (S33).

  If the extraction unit 133 determines that there is one or more first image outputs and one or more second image outputs that are commonly activated in the output layer M10, the output layer M10, which is the last layer, is processed as a post-processing layer It uses as (S34). On the other hand, when the extraction unit 133 determines that there is no one or more first image outputs commonly activated in the output layer M10 and one or more second image outputs, the extraction unit 133 applies to each processing layer before the output layer M10. And one or more first image outputs and one or more second image outputs that are commonly activated are repeatedly searched. Then, the extraction unit 133 uses a processing layer having one or more first image outputs and one or more second image outputs that are activated in common (for example, the second entire combined layer M9) as a post-processing layer. (S35). The extracting unit 133 is activated in common in both the selected post-processing layer and the pre-processing layer, and one or more of the first or more first layers output from the post-processing layer and the pre-processing layer based on the first image. Based on the image output and the second image, one or more second image outputs output from the post-processing layer and the pre-processing layer are extracted.

  Specifically, first, the post-stage extraction unit 134 is common to the plurality of post-stage first image outputs output from the selected post-stage processing layer and the plurality of post-stage second image outputs output from the selected post-stage processing layer The one or more subsequent stage first image outputs and the one or more subsequent stage second image outputs that have been activated are extracted (S36). Then, the pre-stage extraction unit 135 outputs the plurality of pre-stages output from the pre-stage processing layer that has caused the activation of the one or more subsequent stage first image outputs and the one or more subsequent stage second image outputs extracted by the subsequent stage extraction unit 134. Extract one or more preceding stage first image outputs and one or more preceding stage second image outputs that are activated in common among the plurality of preceding stage second image outputs outputted from the first image output and the preceding process layer ( S37).

  Subsequently, the extraction unit 133 determines whether there is another processing layer before the pre-processing layer (S38). If the extraction unit 133 determines that there is another processing layer (for example, the third convolution layer M6) before the pre-processing layer (for example, the second pooling layer M7), the second pooling layer M7 is The processing layer is selected (S39), and the process returns to S36. On the other hand, when the extraction unit 133 determines that there is no other processing layer before the pre-processing layer (for example, the input layer M1), the extracting unit 133 features one or more extracted first image outputs and one or more second image outputs. The point is input to the point detection unit 137, and the extraction process ends.

  Referring back to FIG. 10, the feature point detection unit 137 searches for a corresponding feature point based on one or more first image outputs and one or more second image outputs, and outputs one or more first image outputs in a corresponding relationship. And one or more second image feature points based on one or more second image outputs (S4). Subsequently, the selection unit 138 determines whether or not there are inappropriate feature points in the first image feature point and the second image feature point detected by the feature point detection unit 137 (S5). The selection unit 138 narrows down, for example, based on the RANSAC method.

  If the selection unit 138 determines that there is an inappropriate feature point in the first image feature point and the second image feature point, the selection unit 138 is an inappropriate feature point, that is, the first image feature point in the erroneously detected correspondence relationship and The second image feature points are removed (S6), and a part of first image feature points and a part of second image feature points are selected based on the correspondence relationship after removal. If the intermediate image generation unit 139 determines that the first image feature point and the second image feature point do not have inappropriate feature points, or one or more erroneously detected correspondences are removed. One or more intermediate images are generated based on the first image feature and the one or more second image features (S7).

  For example, the intermediate image generation unit 139 is based on the coordinates in the pixel of the first image indicated by the first image feature point and the coordinates in the pixel of the second image indicated by the second image feature point corresponding to the first image feature point. Calculate the change step. Then, the intermediate image generation unit 139 generates one or more intermediate images based on the calculated change step. The intermediate image generation unit 139 causes the storage unit 12 to store the generated intermediate image.

[Effect in the embodiment]
As described above, the morphing image generation device 1 propagates a plurality of processing layers included in the machine learning model M to each of the acquired first and second images. The morphing image generation device 1 has one or more first image outputs and one or more second images activated in common in both the post-processing layer and the pre-processing layer in the order opposite to the propagating order. An output is extracted for each processing layer, and a first image feature point and a second image feature point in a corresponding relationship are respectively detected. Then, the morphing image generation device 1 generates one or more intermediate images based on the one or more first image feature points from which the erroneously detected correspondence is removed and the one or more second image feature points.

  By doing this, the morphing image generation device 1 is based on the first image and the second image by using the machine learning model M including the convolutional neural network to obtain a feature with a high degree of abstraction by deep learning. An intermediate image can be generated. That is, the morphing image generation device 1 does not associate the specific area in the first image and the specific area in the second image with each other, and the first image feature point and the second image feature point in the correspondence relationship are respectively associated By detecting, an intermediate image based on the first image and the second image can be generated. As a result, the morphing image generation device 1 can improve the quality of the morphing image.

  The morphing image generation device 1 generates an intermediate image based on the preceding and following images without continuity, for example, when the images are not continuous and appear unnaturally in a portion cut by a short time video etc. By doing this, it is possible to make the image natural and continuous. In addition, the morphing image generation device 1 can automate the “split” process in animation production, for example, by generating one or more intermediate images based on two original pictures.

  As mentioned above, although the present invention was explained using an embodiment, the technical scope of the present invention is not limited to the range given in the above-mentioned embodiment, and various modification and change are possible within the range of the gist. is there. For example, a specific embodiment of device distribution and integration is not limited to the above embodiment, and all or a part thereof may be functionally or physically distributed and integrated in any unit. Can. In addition, new embodiments produced by any combination of a plurality of embodiments are also included in the embodiments of the present invention. The effects of the new embodiment generated by the combination combine the effects of the original embodiment.

1 morphing image generation device 11 operation unit 12 storage unit 13 control unit 131 image acquisition unit 132 propagation control unit 133 extraction unit 134 post-stage extraction unit 135 pre-stage extraction unit 136 instruction reception unit 137 feature point detection unit 138 selection unit 139 intermediate image generation unit

Claims (15)

An image acquisition unit that acquires a first image which is an image before at least a part of the subject changes and a second image which is an image after at least a part of the subject changes;
A propagation control unit configured to propagate, to each of the first image and the second image, a plurality of processing layers included in a machine learning model capable of outputting the type of the subject included in the image based on the input image;
The post-process according to the first image, which is activated in common in both the post-treatment layer selected from the plurality of treatment layers and the pre-treatment layer that is the treatment layer immediately before the post-treatment layer Extracting one or more first image outputs output from the layer and the pre-processing layer and one or more second image outputs output from the post-processing layer and the pre-processing layer based on the second image Department,
A feature point detection unit that detects one or more first image feature points based on the one or more first image outputs, and detects one or more second image feature points based on the one or more second image outputs When,
An intermediate image generation unit that generates one or more intermediate images representing a process of changing the subject stepwise based on the one or more first image feature points and the one or more second image feature points;
Morphing image generating apparatus having: The extraction unit
As the first image is propagated in the order of the pre-processing layer and the post-processing layer that are a part of the plurality of processing layers, the plurality of post-processing first image outputs and the second image output from the post-processing layer From the plurality of second-stage second image outputs outputted from the second-stage processing layer by propagating in order of the first-stage processing layer and the second-stage processing layer, one or more second-stage first image outputs and one or more second-stage latter output commonly activated. 2) A post-stage extraction unit that extracts an image output;
A plurality of pre-stage first image outputs output from the pre-stage processing layer that has caused the activation of the one or more post-stage first image outputs and the one or more post-stage second image outputs, and an output from the pre-stage processing layer A pre-stage extraction unit for extracting one or more pre-stage first image outputs and one or more pre-stage second image outputs that are commonly activated among the plurality of pre-stage second image outputs that have been activated;
Have
The morphing image generation device according to claim 1. The pre-stage extraction unit is configured to output the one or more pre-stage first image outputs and the one or more pre-stage first image outputs based on an activated size of the plurality of pre-stage first image outputs and the plurality of pre-stage second image outputs. Extract the pre-stage second image output,
The morphing image generation apparatus according to claim 2. The machine learning model includes a convolutional neural network
The post-processing layer is any one of an output layer, a total coupling layer, a normalization layer, a pooling layer, and a convolution layer,
A morphing image generation apparatus according to claim 2 or 3. The pre-processing layer is any of the total connection layer, the normalization layer, the pooling layer, the convolution layer, and the input layer,
The morphing image generation device according to claim 4. In the case where the last tail layer which is the last treatment layer among the plurality of treatment layers is selected as the second treatment layer, the extraction unit is configured to activate the one or more first common layers in the last tail layer. When there is no image output and the one or more second image outputs, the one or more first image outputs and the one or more second image outputs that are commonly activated in the processing layer before the last tail layer are extracted Do,
The morphing image generation device according to claim 1. From the one or more first image feature points specified by the feature point detection unit and the one or more second image feature points, a part of the first image feature points and a part of the second image feature points based on the mutual correspondence relationship The image processing apparatus further comprises a selection unit for selecting an image feature point,
The intermediate image generation unit generates one or more intermediate images stepwise representing the process of changing the subject based on the part of the first image feature points and the part of the second image feature points. Do,
The morphing image generation device according to any one of claims 1 to 6. The image acquisition unit acquires a plurality of second images in which subjects of the same type as the subject after change and of different shapes are captured,
The intermediate image generation unit may select one of the plurality of second images based on the one or more first image feature points and the one or more second image feature points based on each of the plurality of second images. 2Select an image,
The morphing image generation device according to any one of claims 1 to 7. The intermediate image generation unit selects one second image from the plurality of second images in which the number of second image feature points corresponding to the first image feature point is a predetermined reference value or more.
The morphing image generation device according to claim 8. The apparatus further includes an instruction accepting unit that accepts an instruction to select a processing layer to be used as the post-processing layer among the plurality of processing layers,
The extraction unit uses the processing layer indicated by the instruction received by the instruction receiving unit as the post-processing layer.
The morphing image generation device according to any one of claims 1 to 9. The extraction unit selects one of the plurality of processing layers as the post-processing layer and extracts the one or more first image outputs and the one or more second image outputs, and then the pre-processing layer. Selecting the processing layer selected as the post-processing layer to extract another one or more first image outputs and the one or more second image outputs,
The morphing image generation device according to any one of claims 1 to 10. Acquiring a first image, which is an image before at least a part of the subject changes, and a second image, which is an image after at least a part of the subject changes.
Propagating, to each of the first image and the second image, a plurality of processing layers included in a machine learning model capable of outputting the type of the subject included in the image based on the input image;
The post-process according to the first image, which is activated in common in both the post-treatment layer selected from the plurality of treatment layers and the pre-treatment layer that is the treatment layer immediately before the post-treatment layer Extracting one or more first image outputs output from the layer and the pre-processing layer and one or more second image outputs output from the post-processing layer and the pre-processing layer based on the second image When,
Detecting one or more first image feature points based on the one or more first image outputs, and detecting one or more second image feature points based on the one or more second image outputs;
Generating, based on the one or more first image feature points and the one or more second image feature points, one or more intermediate images representing a process of changing the subject in stages;
A morphing image generation method comprising: The extracting step is
As the first image is propagated in the order of the pre-processing layer and the post-processing layer that are a part of the plurality of processing layers, the plurality of post-processing first image outputs and the second image output from the post-processing layer From the plurality of second-stage second image outputs outputted from the second-stage processing layer by propagating in order of the first-stage processing layer and the second-stage processing layer, one or more second-stage first image outputs and one or more second-stage latter output commonly activated. A pre-extraction step of extracting two image outputs;
A plurality of pre-stage first image outputs output from the pre-stage processing layer that has caused the activation of the one or more post-stage first image outputs and the one or more post-stage second image outputs, and an output from the pre-stage processing layer A subsequent extraction step of extracting one or more preceding first image outputs and one or more preceding second image outputs that are activated in common among the plurality of preceding second image outputs;
The morphing image generation method according to claim 12, comprising: After performing the pre-stage extraction step, the one or more pre-stage first image outputs and the one or more pre-stage second image outputs are used as the plurality of second-stage first image outputs and the plurality of second-stage second image outputs. Execute the post extraction step,
A method of generating a morphing image according to claim 13. Performing the post-stage extraction step and the pre-stage extraction step on each of the plurality of processing layers;
A morphing image generation method according to claim 13 or 14.

Images