JP5876121B2 - Image processing apparatus, method, and program - Google Patents

Description

  Embodiments described herein relate generally to an image processing apparatus, a method, and a program.

  Various techniques for displaying a virtual image indicating a state of wearing a clothing to be tried on have been disclosed. For example, a technique for displaying a composite image showing a state where a human body has tried on a plurality of clothes is disclosed. Also disclosed is a technique for detecting an object in an acquired image by performing template matching using a reference image prepared in advance and superimposing a virtual image on the object.

JP 2010-152557 A

  However, the technique using template matching can improve accuracy, but the processing load is high. For this reason, it has been difficult to provide a composite image with low load while improving accuracy.

  The problem to be solved by the present invention is to provide an image processing apparatus, method, and program capable of creating a composite image with high accuracy and low load.

The image processing apparatus of the embodiment includes a first calculator, a second calculator, and a determination unit. The first calculation unit, based on the clothing image superimposition target object Utsushitai image, calculates a first position of the garment image in the object image. The second calculation unit calculates a second position of the clothing image in the subject image that is different from the first position, based on the subject image and the clothing image. The determining unit, when determining the superimposed position using the second position, further uses the first position calculated for the first subject image acquired before the subject image, The superposition position is determined .

1 is a block diagram of an image processing system according to a first embodiment. The schematic diagram of the external appearance of an image processing system. The figure of the data structure of the 1st table. The figure which shows an example of posture information and a clothing image. The figure which shows the calculation method of the direction of a to-be-photographed object. The figure which shows three types of clothing images from which posture information differs. Explanatory drawing of the size calculation of a feature area. The figure which shows an example of an outline extraction. The schematic diagram which shows calculation of a 2nd position. The flowchart which shows the procedure of an image process. The flowchart which shows the procedure of an adjustment process. The flowchart which shows the procedure of a 1st position calculation process. The flowchart which shows the procedure of a 2nd position calculation process. Explanatory drawing which shows the production | generation of a synthesized image. Explanatory drawing which shows the production | generation of a synthesized image. 1 is a schematic diagram showing an image processing system. The block diagram which shows the hardware structural example of an image processing apparatus.

  Hereinafter, an embodiment of an image processing apparatus, an image processing method, and an image processing program will be described in detail with reference to the accompanying drawings.

(Embodiment 1)
FIG. 1 is a block diagram showing a functional configuration of an image processing system 10 according to the present embodiment. The image processing system 10 includes an image processing device 12, an imaging unit 14, an input unit 16, a storage unit 18, and a presentation unit 20.

  In the present embodiment, the image processing system 10 will be described with respect to a configuration in which the image processing device 12, the imaging unit 14, the input unit 16, the storage unit 18, and the presentation unit 20 are provided separately. . However, the image processing system 10 may be configured such that the image processing device 12 and at least one of the imaging unit 14, the input unit 16, the storage unit 18, and the presentation unit 20 are integrally configured. .

  The imaging unit 14 images a subject and obtains a subject image of the subject. The imaging unit 14 images a subject every predetermined time. The imaging unit 14 sequentially outputs the subject images acquired by imaging to the image processing device 12. The imaging unit 14 continuously captures the subject and outputs it to the image processing device 12, whereby the image processing device 12 obtains a moving image including a plurality of subject images having different imaging times.

  The subject is a subject to try on clothes. The subject only needs to be an object to try on clothes, and may be a living organism or a non-living organism. Examples of the living organism include, but are not limited to, a person, and may be a pet such as a dog or a cat. The non-living body includes, but is not limited to, a mannequin that imitates the shape of a human body or a pet, clothes, and other objects. The subject may be a living creature or a non-living creature wearing clothes.

  Also, clothes are items that can be worn by a subject. Examples of the clothes include a jacket, a skirt, trousers, shoes, a hat, and the like.

  In the present embodiment, the imaging unit 14 includes a first imaging unit 14A and a second imaging unit 14B.

  The first imaging unit 14A acquires a depth image of the subject by imaging. The second imaging unit 14B acquires a color image of the subject by imaging.

  The color image is a bitmap image. The color image of the subject is an image in which pixel values indicating the color and luminance of the subject are defined for each pixel. The second imaging unit 14B is a known imaging device that can acquire a color image.

  The depth image may be referred to as a distance image. The depth image of the subject is an image that defines the distance from the first imaging unit 14A that captured the subject for each pixel constituting the subject image. In the present embodiment, the depth image of the subject may be created from the color image of the subject by a known method such as stereo matching, or the first imaging unit 14A is used under the same shooting conditions as the color image of the subject. May be obtained by imaging. A known imaging device capable of acquiring a depth image is used for the first imaging unit 14A.

  In the present embodiment, the first imaging unit 14A and the second imaging unit 14B image a subject at the same timing. It is assumed that the first imaging unit 14A and the second imaging unit 14B are controlled to sequentially perform imaging in synchronism with the same timing by a control unit (not shown) or the like. Then, the imaging unit 14 sequentially outputs a subject image including a depth image of the subject and a color image of the subject obtained by imaging to the image processing device 12.

  The presentation unit 20 is a device that presents various images. In the present embodiment, the presentation unit 20 presents a composite image (described later) generated by the image processing device 12. Note that the presentation indicates display, printing, transmission, and the like.

  The presentation unit 20 is, for example, a display device such as a liquid crystal display device, a printing device that prints an image, a known communication device that transmits information to an external device by wired communication, wireless communication, or the like. When the presentation unit 20 is a display device, the presentation unit 20 displays a composite image. When the presentation unit 20 is a communication device, the presentation unit 20 transmits the composite image to the external device. When the presentation unit 20 is a printing device, the presentation unit 20 prints a composite image.

  In the present embodiment, the term “presentation” indicates display, and the presenting unit 20 will be described as a display device.

  The input unit 16 is a means for the user to perform various operation inputs. Examples of the input unit 16 include a combination of one or more of a voice recognition device such as a mouse, a button, a remote control, a keyboard, and a microphone, and an image recognition device. Further, when the above-described image recognition device is employed as the input unit 16, it may be a device that accepts a user's gesture gesture or the like facing the input unit 16 as various instructions of the user. In this case, the image recognition apparatus stores in advance instruction information corresponding to each movement such as gestures and hand gestures, and can read the instruction information corresponding to the recognized gestures and hand gestures to accept an operation instruction by the user. That's fine.

  The input unit 16 may be a communication device that receives a signal indicating a user operation instruction from an external device such as a mobile terminal that transmits various types of information. In this case, the input unit 16 may accept a signal indicating an operation instruction received from the external device as an operation instruction by the user.

  In addition, the input part 16 and the presentation part 20 may be comprised integrally. Specifically, the input unit 16 and the presentation unit 20 may be configured as a UI (User Interface) unit having both an input function and a display function. The UI unit includes an LCD (Liquid Crystal Display) with a touch panel.

  FIG. 2 is a schematic diagram illustrating the appearance of the image processing system 10.

  As shown in FIG. 2, the presentation unit 20 in the image processing system 10 is incorporated on one surface of a rectangular housing 51, for example. Then, in the image processing system 10, a composite image W indicating a state in which the subject P has tried on various clothes is displayed on the presentation unit 20. The subject P such as a person visually recognizes the composite image W presented on the presentation unit 20 from a position facing the presentation unit 20, for example.

  The casing 51 supports the input unit 16 and the imaging unit 14 (the first imaging unit 14A and the second imaging unit 14B). In the example illustrated in FIG. 2, the input unit 16 and the second imaging unit 14 </ b> B are provided at both ends in the horizontal direction of the presentation unit 20 in the housing 51. In addition, the first imaging unit 14 </ b> A is provided in the housing 51 above the presentation unit 20. The installation position of the input unit 16 is not limited to this position. The first imaging unit 14A and the second imaging unit 14B may be provided at positions where the subject P can be imaged, and are not limited to these positions.

  Various information is input via the input unit 16 when the user instructs the input unit 16 to operate. In addition, as described above, the first imaging unit 14A and the second imaging unit 14B sequentially capture the subject P while synchronizing the same timing. Then, the first imaging unit 14A and the second imaging unit 14B sequentially output subject images obtained by imaging to the image processing device 12.

  Returning to FIG. 1, the storage unit 18 stores in advance a clothing image to be superimposed on the subject image. In the present embodiment, a case will be described in which the storage unit 18 stores a first table in which clothing numbers, posture information, clothing images, and attribute information are associated with each other.

  FIG. 3 is a diagram illustrating an example of the data structure of the first table.

  The clothes number is information for uniquely identifying clothes. Examples of the clothing number include, but are not limited to, a product number and a clothing name. For example, a known JAN code is used as the product number. As the name, for example, the name of clothes is used.

  The clothing number is information that uniquely identifies the clothing. For this reason, a plurality of clothing images having different posture information may be associated with one clothing number.

  The posture information is information indicating the posture of the subject that is the subject of the clothing at the time of acquiring the clothing image. The posture information indicates the orientation and movement of the subject relative to the imaging device 14.

  The orientation of the subject indicates the orientation of the subject wearing the clothing of the clothing image with respect to the imaging device 14 when the clothing image is acquired. For example, as the direction of the subject, the front direction in which the face and the body face the front with respect to the imaging device 14, the side direction in which the face and the body face the side with respect to the imaging device 14, other than the front direction and the side direction There are other directions.

  The movement of the subject is indicated by the skeleton information indicating the position of the skeleton of the subject wearing the clothes in the clothing image. The skeleton information is information that defines, for each pixel, information indicating the position of the skeleton of the subject wearing the clothing of the clothing image corresponding to the pixel position of each pixel constituting the clothing image.

  In the present embodiment, the posture information includes the orientation of the subject and the skeleton information.

  A clothing image is obtained by imaging a subject wearing clothing. A clothing image is an image in which pixel values indicating the color, brightness, etc. of clothing are defined for each pixel. For example, an image obtained by capturing an image of clothes with the second imaging unit 14B is used as the clothes image.

  The attribute information is information indicating the attribute of the corresponding clothing image. The attribute information includes, for example, the size of clothes, the color of clothes, the material of clothes, the price of clothes, and the like.

  Specifically, for example, assume that the image processing system 10 acquires a clothing image photographed from the front direction and a clothing image photographed from the side surface for one type of clothing. In this case, the storage unit 18 associates the clothes number indicating the clothes with the posture information including the “front direction” which is the direction of the subject and the skeleton information of the subject facing the front direction, and the posture information. The clothing image photographed from the front direction is stored in advance. Similarly, the storage unit 18 is associated with the clothing number indicating the clothing, the posture information including the “side direction” that is the direction of the subject and the skeleton information of the subject facing the side direction, and the posture information. The clothing image photographed from the side acquisition direction is stored in advance.

  Note that the first table may be a table in which clothing numbers, posture information, and clothing images are associated with each other, and may have a form that does not store attribute information.

  FIG. 4 is a schematic diagram illustrating an example of posture information and a clothing image.

  As shown in FIG. 4A, for example, a subject 60 such as a mannequin wearing a garment 61 is prepared. The subject 60 is imaged by the imaging unit 14 to obtain a subject image including a color image and a depth image. Then, by extracting a clothing region from the color image obtained by imaging, for example, a clothing image 61A shown in FIG. 4B is obtained. Further, a pixel position corresponding to the joint position of the human body is extracted from the depth image of the subject 60 by a known method, and the clothing of the clothing image corresponding to the pixel position of each pixel constituting each of the clothing images is worn. Skeleton information 61B indicating the position of the skeleton of the subject is created (FIG. 4C).

  In this way, the image processing apparatus 12 prepares in advance a first table including clothes images and posture information of clothes to be superimposed, and stores them in the storage unit 18 in advance.

  Returning to FIG. 1, the image processing apparatus 12 is a computer including a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The image processing device 12 is electrically connected to the imaging unit 14, the input unit 16, the storage unit 18, and the presentation unit 20.

  The image processing apparatus 12 includes a first acquisition unit 22, a reception unit 24, a second acquisition unit 26, a determination unit 28, an adjustment unit 30, an extraction unit 32, a first calculation unit 34, a second calculation unit 36, a determination unit 38, A generation unit 40 and a presentation control unit 42 are provided.

  The first acquisition unit 22 sequentially acquires subject images from the imaging unit 14. The first acquisition unit 22 includes a third acquisition unit 22A and a fourth acquisition unit 22B. The third acquisition unit 22A sequentially acquires depth images of the subject. The fourth acquisition unit 22B sequentially acquires color images.

  The subject image is an image showing the subject. The third acquisition unit 22A acquires a depth image of the subject by extracting a person region from the depth image acquired from the first imaging unit 14A. Similarly, the fourth acquisition unit 22B acquires a color image of the subject by extracting a person region from the color image acquired from the second imaging unit 14B.

  For example, the first acquisition unit 22 extracts a person region by setting a threshold for the distance in the depth direction among the three-dimensional positions of each pixel constituting the depth image. For example, in the camera coordinate system of the first image capturing unit 14A, the position of the first image capturing unit 14A is the origin, and the positive Z-axis direction is the optical axis of the camera extending from the origin of the first image capturing unit 14A toward the subject. In this case, out of the pixels constituting the depth image, pixels whose position coordinates in the depth direction (Z-axis direction) are equal to or greater than a predetermined threshold (for example, a value indicating 2 m) are excluded. As a result, the first acquisition unit 22 obtains a depth image composed of pixels of the person area existing within a range of 2 m from the first imaging unit 14A, that is, a depth image of the subject.

  The receiving unit 24 receives various information from the input unit 16.

  The second acquisition unit 26 acquires a clothing image to be superimposed on the subject image acquired by the first acquisition unit 22 from the storage unit 18.

  In the present embodiment, the second acquisition unit 26 receives clothes candidate information for specifying clothes to be superimposed from the input unit 16. The clothing candidate information includes information for specifying at least one of the attribute information such as the type, size, color, material, price, etc. of the clothing that the user desires to try on, or at least one of the attribute information. .

  For example, the user inputs “type”, “color”, “size”, and the like of clothes as clothes candidate information according to an operation instruction of the input unit 16.

  The second acquisition unit 26 analyzes the clothing candidate information received from the input unit 16, and stores a list of clothing images corresponding to the attribute information specified by the clothing candidate information or included in the clothing candidate information from the storage unit 18. Search for. And the presentation control part 42 performs control which shows the list of the clothing image searched in the 1st acquisition part 26 to the presentation part 20 for every corresponding clothing number. That is, when a plurality of types of clothing images having different posture information are stored in association with the storage unit 18 for one clothing number, the presentation control unit 42 displays representative posture information (for example, front). Control is performed to present one clothing image corresponding to 1 to the presentation unit 20.

  When a list of clothing images is presented to the presentation unit 20, a clothing image of a clothing to be superimposed is selected from the list of clothing images presented to the presentation unit 20 according to an operation instruction of the input unit 16 by the user. . Then, the clothing number of the selected clothing image is output from the input unit 16 to the image processing device 12.

  When the reception unit 24 receives the selected clothing number, the second acquisition unit 26 acquires the clothing image specified by the clothing number received by the reception unit 24 from the storage unit 18.

  The extraction unit 32 generates skeleton information from the subject image acquired by the first acquisition unit 22. Specifically, the extraction unit 32 generates skeleton information indicating the skeleton position of the human body for each pixel constituting the depth image of the subject acquired by the third acquisition unit 22A. The extraction unit 32 generates skeleton information by applying a human body shape to the depth image.

  Then, the extraction unit 32 obtains a fourth coordinate system of the pixel positions of each pixel of the skeleton information calculated from the depth image of the subject acquired by the third acquisition unit 22A (that is, the coordinate system of the first imaging unit 14A). The coordinate system of each pixel position of the color image of the subject acquired by the unit 22B (that is, the coordinate system of the second imaging unit 14B) is converted. That is, the extraction unit 32 captures the coordinate system of the pixel position of each pixel of the skeleton information calculated from the depth image of the subject imaged by the first imaging unit 14A by the second imaging unit 14B at the same timing as the depth image. To the coordinate system of the color image of the subject. This coordinate conversion is performed by performing known calibration.

  The adjustment unit 30 enlarges or reduces at least one of the clothing image and the subject image so that at least a part of the contour of the clothing image to be superimposed matches at least a part of the contour of the subject image. Further, the adjustment unit 30 extracts a feature region used in a first calculation unit 34 described later from the enlarged or reduced clothing image and the subject image.

  Specifically, the adjustment unit 30 first calculates the orientation of the subject from the skeleton information of the subject created by the extraction unit 32.

  The adjustment unit 30 calculates the orientation of the subject from the position of each joint indicated by the skeleton information as the posture information calculated by the extraction unit 32 from the depth image of the subject.

  FIG. 5 is an explanatory diagram illustrating a method for calculating the orientation of the subject by the adjustment unit 30.

Indicated skeletal information of the object calculated by the extraction unit 32, the coordinates of the pixel position corresponding to the left shoulder of the subject, and Psl in the coordinate system of the second image pickup unit 14B (in FIG. 5, reference pixel position 72 ₄₎ . Further, the coordinates of the pixel position corresponding to the right shoulder of the subject indicated by the skeletal information as the posture information calculated by the extraction unit 32 is set to Psr in the coordinate system of the second imaging unit 14B (in FIG. 5, the pixel location reference 72 _3).

  The adjustment unit 30 calculates the orientation of the subject with respect to the imaging unit 14 (second imaging unit 14B) from these coordinate information by the following equation (1).

Orientation of subject = arctan (Psl.z-Psr.z / Psl.x-Psr.x)
.... Formula (1)

  In formula (1), Psl. z indicates the z coordinate value of the pixel position corresponding to the left shoulder of the subject. Psr. z indicates the z coordinate value of the pixel position corresponding to the right shoulder of the subject. In the formula (1), Psl. x represents an x coordinate value of a pixel position corresponding to the left shoulder of the subject. Psr. x indicates an x coordinate value of a pixel position corresponding to the right shoulder of the subject.

  Note that the adjustment unit 30 may obtain the posture information of the clothing image in the same manner.

  Next, the adjustment unit 30 acquires, from the storage unit 18, a clothing image corresponding to the orientation of the subject (subject orientation information) indicated by the subject image among the clothing images corresponding to the clothing number selected by the user. .

  FIG. 6 is a schematic diagram illustrating a case where three types of clothing images having different posture information are stored in the storage unit 18 in association with the same clothing number for the same clothing.

For example, in the storage unit 18, “front” that is the front direction with respect to the second imaging unit 14 </ b> B provided in the casing 51 as a clothing image corresponding to the clothing number selected by the user, 20 from the front to the right. The clothing image 61A (61A ₀ , 61A ₂₀ , 61A ₄₀ ) corresponding to each posture information of the orientation “+20 degrees” rotated by 40 degrees and the orientation “+40 degrees” rotated 40 degrees to the right from the front is stored in advance. To do.

When the orientation of the subject indicated by the subject image is “front”, the adjustment unit 30 acquires these clothing images 61A (61A) corresponding to the clothing number selected by the user and acquired by the second acquisition unit 26. ₀ , 61 A ₂₀ , 61 A ₄₀ ), the clothing image 61 A ₀ corresponding to the direction “front” of the subject is read from the storage unit 18.

  Next, the adjustment unit 30 uses the coordinate system of the depth image of the subject acquired by the third acquisition unit 22A (that is, the coordinate system of the first imaging unit 14A) as the color image of the subject acquired by the fourth acquisition unit 22B. The coordinate system (that is, the coordinate system of the second imaging unit 14B) is converted. Then, the adjustment unit 30 causes the pixels at the respective pixel positions constituting the depth image of the subject after coordinate conversion to be positions corresponding to the pixel positions of the respective pixels constituting the color image of the subject acquired at the same timing. The resolution of the depth image of the subject is adjusted to the same resolution as that of the color image.

  For example, it is assumed that the resolution of the depth image of the subject obtained by the first imaging unit 14A is 640 × 480 pixels, and the resolution of the color image of the subject obtained by the second imaging unit 14B is 1080 × 1920 pixels. In this case, when each pixel constituting the depth image is projected onto the color image as a point of 1 pixel × 1 pixel, a gap is generated between the pixels constituting the depth image. For this reason, the adjustment unit 30 applies a filter using a known calculation such as a Gaussian filter or a morphological calculation as necessary, and there is a gap between each pixel constituting the depth image projected onto the color image. Adjust so that it does not occur.

  Next, the adjustment unit 30 calculates the size of the feature region in the clothing image corresponding to the orientation of the subject.

  The feature region is a region where the shape of the subject trying on clothes can be estimated. The feature region includes, but is not limited to, a shoulder region corresponding to the shoulder of the human body, a waist region corresponding to the waist, a foot region corresponding to the length of the foot, and the like. In the present embodiment, a case where a shoulder region corresponding to the shoulder of a human body is used as the feature region is described, but the feature region is not limited to the shoulder region.

  For example, the adjustment unit 30 calculates the shoulder width in the clothing image as the size of the feature region in the clothing image corresponding to the orientation of the subject.

  FIG. 7 is an explanatory diagram for calculating the size of the feature region of the clothing image by the adjustment unit 30.

  The adjustment unit 30 calculates the Y coordinate of the pixel at the pixel position corresponding to the left shoulder from the skeleton information included in the posture information corresponding to the clothing image in the clothing image corresponding to the orientation of the subject. And the Y coordinate of the pixel at the pixel position corresponding to the right shoulder. Then, an average Y coordinate of these Y coordinates is obtained. Next, the adjustment unit 30 performs a search from the X coordinate of the pixel position corresponding to the left shoulder to the region corresponding to the outside of the clothing at the obtained position (height) of the Y coordinate, and the left shoulder of the clothing. The X coordinate indicating the position of the side boundary line is obtained. Further, the adjustment unit 30 performs a search from the X coordinate of the pixel position corresponding to the right shoulder to the area corresponding to the outside of the clothing at the obtained position (height) of the Y coordinate, and the right shoulder of the clothing. The X coordinate indicating the position of the side boundary line is obtained.

  By obtaining the difference between the two X coordinates thus obtained, the shoulder width (number of pixels) on the clothing image can be obtained (see the shoulder width Sc in the clothing image 61A in FIG. 7A).

  Instead of calculating the Y coordinate of a certain shoulder joint, a search is performed for a plurality of horizontal lines with the width in the vertical direction of the Y coordinate centered on the Y coordinate of the shoulder joint, and horizontal You may obtain | require a shoulder width by calculating | requiring the average X coordinate of both sides about a direction.

  Next, the adjustment unit 30 calculates the shoulder width on the image of the subject using the depth image of the subject and the skeleton information of the subject adjusted to the same resolution as the color image.

  As shown in FIG. 7B, the adjustment unit 30 calculates the average Y coordinate of the Y coordinate of the pixel position corresponding to the left shoulder and the Y coordinate of the pixel position corresponding to the right shoulder in the depth image of the subject. Ask. Next, a search is performed from the X coordinate of the pixel position corresponding to the left shoulder toward the region corresponding to the outside of the subject, and the X coordinate that is one boundary line of the subject region is obtained.

  Further, the adjustment unit 30 searches from the X coordinate of the pixel position corresponding to the right shoulder in the depth image of the subject toward the region corresponding to the outside of the subject, and is the other boundary line X of the subject region. Find the coordinates.

  The adjustment unit 30 obtains the shoulder width (number of pixels) on the image of the subject by obtaining the difference between the two X coordinates obtained in this way (the shoulder width of the depth image 71A of the subject in FIG. 7B). See Sh).

  The adjustment unit 30 does not calculate the shoulder width for one Y coordinate of the shoulder joint, but searches for a plurality of horizontal lines by providing a width in the vertical direction of the Y coordinate around the Y coordinate of the shoulder joint. I do. And the adjustment part 30 may obtain | require a shoulder width by calculating | requiring the average X coordinate of both sides about a horizontal direction.

  Next, the adjustment unit 30 determines the clothing image scaling value (enlargement / reduction ratio) using the size of the feature region calculated above, that is, the shoulder width Sc of the clothing image and the shoulder width Sh of the subject image. To do.

  Specifically, the adjustment unit 30 calculates a division value (Sh / Sc) obtained by dividing the shoulder width Sh of the subject image by the shoulder width Sc of the clothing image as a scaling value. The scaling value may be calculated from different expressions using values such as the actual size of the clothes and the number of pixels of the width and height of the clothes image area.

  Then, the adjustment unit 30 enlarges or reduces the clothes image (color image) to be superimposed at the enlargement / reduction ratio specified by the scaling value. Similarly, the adjustment unit 30 enlarges or reduces the skeleton information included in the posture information of the clothing image to be superimposed at the enlargement / reduction ratio specified by the same scaling value (Sh / Sc).

  Next, the adjustment unit 30 extracts a feature region used in the first calculation unit 34 described later from the enlarged or reduced clothing image and the subject image.

  The feature area indicates an area in which the shape of the subject can be estimated in each of the clothing image and the subject image. The characteristic area is an area indicating, for example, a shoulder or a waist of a human body. In the present embodiment, a description will be given of a case where the adjustment unit 30 extracts an area (shoulder area) corresponding to the shoulder of the human body in each contour of a clothing image and a subject image as a feature area.

  First, the adjustment unit 30 extracts a contour from the depth image of the subject adjusted to coordinate conversion and resolution. The adjustment unit 30 also extracts the outline of the clothing image that has been enlarged or reduced so as to have the same scaling value as the subject image. A known method is used to extract the contour. And the adjustment part 30 extracts the outline of the area | region (shoulder area | region) equivalent to the shoulder of a human body from these outlines as a characteristic area.

  Here, it is preferable that the adjustment unit 30 extracts a contour according to the shape of the clothes image.

  FIG. 8 is a schematic diagram illustrating an example of contour extraction.

  As shown in FIG. 8A, it is assumed that the clothing image 61A has a shape having a vertically long opening on the front side of the human body. In this case, the contour is also shown at the center of the human body as shown in FIG. If template matching (details will be described later) is performed by the first calculation unit 34 to be described later using such an outline, the matching accuracy of the region corresponding to the central portion of the human body may be lowered.

  For this reason, when the clothing image 61A has a shape having a longitudinally long opening on the front side of the human body, the adjustment unit 30 determines the region corresponding to the central portion of the human body from the contour shown in FIG. By removing the contour, it is preferable to extract the contour 61C of the portion along the contour of the human body as the contour of the clothing image (FIG. 8D).

  In the image processing device 12, when a clothing image is acquired, a depth image of a subject such as a mannequin wearing the clothing of the clothing image is stored in advance in the storage unit 18 in association with the clothing image. deep. The adjustment unit 30 deletes a part of the inner region continuous with the contour in the depth image from the depth image by an image filtering process such as a known morphological calculation. Then, a depth image 71C (see FIG. 8C) after this deletion is prepared. And the adjustment part 30 deletes the area | region 61C along the external shape of a human body by deleting the area | region which overlaps with the depth image 71C (refer FIG.8 (C)) from the outline 61B shown in FIG.8 (B). It is extracted as the outline of the clothes image (FIG. 8D).

  Then, the adjustment unit 30 extracts a shoulder region corresponding to the shoulder of the human body as a feature region from the contours of the clothing image and the subject image (depth image).

  It may be difficult for the clothes specified by the clothes image to extract a shape (shoulder outline or the like) along the outer shape of the human body from the clothes image, such as a tank top or a bare top. In this case, a depth image of a subject such as a mannequin on which clothes are tried on may be stored in the storage unit 18 in advance, and the contour of the shoulder region may be calculated from the shoulder of the subject.

  Returning to FIG. 1, the description will be continued.

  The first calculation unit 34 adjusts the position of the clothing image in the subject image so that the position of the feature region in the clothing image to be superimposed matches the position of the feature region in the subject image acquired by the first acquisition unit 22. One position is calculated.

  Note that the first calculation unit 34 calculates the first position when the determination unit 28 described later determines that the subject image acquired by the first acquisition unit 22 satisfies a predetermined first condition. . Details of the determination process and the determination conditions of the determination unit 28 will be described later.

  The first calculation unit 34 searches for the subject image (depth image) by performing known template matching on the feature region of the subject image using the feature region of the clothing image as a template. Then, the first calculation unit 34 calculates, as the first position, a position on the subject image (depth image) that matches the specific area (shoulder area in the present embodiment) in the clothing image.

  The first position is indicated by position coordinates on the subject image. Specifically, the first position is the center position of the feature region when the feature region of the subject image matches the feature region of the clothing image on the subject image. In the present embodiment, the first calculation unit 34 calculates the center position of the feature area (shoulder area) of the subject image on the subject image as the first position.

  The second calculation unit 36 selects the second clothing image in the subject image so that the position of the feature point predetermined according to the feature region in the clothing image matches the position of the feature point in the subject image. Calculate the position.

  As described above, the first calculator 34 calculates the first position by performing template matching on the feature region. On the other hand, the second calculation unit 36 calculates the second position from the position of the feature point. Therefore, the first calculation unit 34 can calculate the first position with higher accuracy than the second calculation unit 36. On the other hand, the second calculation unit 36 can calculate the second position with a lower load than the first calculation unit 34, although the calculation accuracy of the second position is lower than that of the first calculation unit 34.

  The feature point is a position where the body shape of the subject trying on clothes can be estimated. The feature point is determined in advance according to the feature region. For example, the feature point is determined at a position corresponding to the center of the feature region used by the first calculation unit 34. For this reason, the feature points are set in advance according to the region used by the first calculation unit 34 as the feature region. The feature points are indicated by position coordinates on the image.

  In the present embodiment, the first calculation unit 34 uses a shoulder region as a feature region. Therefore, the second calculation unit 36 will describe a case where a position corresponding to the center of both shoulders of the human body is determined as the feature point as the center position of the shoulder region.

  FIG. 9 is a schematic diagram illustrating an example of calculation of the second position by the second calculation unit 36.

The second calculating section 36, for example, from the backbone information 81A of the clothing image 61A shown in FIG. 9 (A), determining the center positions to _{Q 1} between the shoulders. The second calculating section 36, the skeleton information 81B of the subject image 71A shown in FIG. 9 (B), obtaining a center position _{Q 2} between the shoulders. Then, the second calculating unit 36, the clothing image 61A, the center position to Q ₁ between the shoulders, the center position Q ₂ between the shoulders of the subject image 71A, so that the match, a subject image of the garment image 61A The second position at 71A is calculated. In this embodiment, the second calculating section 36, the center position Q ₂ between the shoulders of the subject image 71A, so that the calculating as a second position.

  Returning to FIG. 1, the determination unit 28 determines whether or not the subject image acquired by the first acquisition unit 22 satisfies a predetermined first condition.

  The first condition is a condition for determining whether or not the first calculation unit 34 executes the first position calculation process. That is, the first calculation unit 34 calculates the first position only when the subject image acquired by the first acquisition unit 22 is determined to satisfy the first condition by the determination unit 28. On the other hand, in the second calculation unit 36, when the subject image acquired by the first acquisition unit 22 is determined to satisfy the first condition by the determination unit 28, and when it is determined that the first condition is not satisfied In any case, the calculation of the second position is executed.

  The following conditions are mentioned as 1st conditions. Note that one of the following conditions may be defined as the first condition, and any of a plurality of conditions described below may be defined as the first condition.

  The first condition is when a subject existing in an area imaged by the imaging unit 14 is replaced with a different subject.

  In this case, the determination unit 28 includes a person as a subject within a predetermined distance from the presentation unit 20 based on the coordinate value of the joint position of the subject in the depth image as the subject image acquired by the first acquisition unit 22. It is determined whether or not. The subject image acquired at a certain time includes a person as a subject, the subject image acquired at a later time does not have a person as a subject, and the subject image acquired at a later time When it is determined that a person exists as a subject, the determination unit 28 determines that the person existing in the area imaged by the imaging unit 14 has been replaced with a different person. As a result, the determination unit 28 determines that the acquired subject image satisfies the first condition predetermined by the determination unit 28.

  For example, it is preferable to calculate the position (the first position or the second position) with higher accuracy because the body shape and the like are different when the subject to be tried on is switched in front of the presentation unit 20. For this reason, when the person who exists in the area imaged by the imaging unit 14 is changed to a different person is set as the determination condition of the determination unit 28, the accuracy of the superimposition position described later can be improved.

  If the first position is calculated from the subject image captured while the person located in front of the presentation unit 20 is moving, the calculation accuracy may be reduced. Therefore, the determination unit 28 satisfies the first condition with respect to the subject image acquired after detecting the stillness of the person after a predetermined time after the person existing in the area imaged by the imaging unit 14 is changed to a different person. It is preferable to discriminate from the subject image.

  A known image processing technique may be used for the movement of these persons and the detection of the stillness of the person.

  Another first condition is when a clothing number different from the clothing image included in the composite image being presented is designated as the clothing number to be tried on by the user's operation instruction on the input unit 16. It is done.

  In this case, the determination unit 28 determines whether the subject image acquired by the first acquisition unit 22 is a subject image acquired immediately after a new clothing number is instructed by an operation instruction of the input unit 16 by the user. Is determined. As a result, the determination unit 28 determines that the acquired subject image satisfies the first condition predetermined by the determination unit 28.

  Note that if the first position is calculated from the subject image that is captured when the subject that is located in front of the presentation unit 20 and is trying to operate the input unit 16 is moving, the calculation accuracy may be reduced. There is sex. For this reason, the determination unit 28 determines that the subject image acquired after detecting the stillness of the person after a predetermined time after determining that the operation instruction of the input unit 16 by the user has been made is the subject image satisfying the first condition. It is preferable.

  In addition, another first condition is that the acquired subject image is a subject image acquired after a predetermined number of subject images have been acquired since the previous determination that the acquired subject image is the first position calculation target subject image. There are some times.

  In this case, the determination unit 28 is the subject image acquired after the subject image acquired by the first acquisition unit 22 has acquired a predetermined number of images from the acquisition of the subject image previously determined as the first position calculation target. It is determined whether or not there is. Accordingly, the determination unit 28 determines that the acquired subject image satisfies the first condition by the determination unit 28.

  In addition, as this number of sheets, for example, 15 sheets (15 frames in the case of a moving image) can be cited, but the number is not limited thereto. In addition, this number may be set to a larger value as the processing load of the first calculation unit 34 becomes higher, or may be set to a larger value as the subject movement amount increases. Further, the interval between the acquired subject images that are determined to satisfy the first condition may be determined by combining these setting conditions.

  In addition, the determination unit 28 determines whether the subject image acquired by the first acquisition unit 22 is a subject image acquired after a predetermined time has elapsed since the acquisition of the subject image determined as the first position calculation target last time. It may be determined whether or not. Accordingly, the determination unit 28 determines that the acquired subject image satisfies the first condition by the determination unit 28.

  Also in this case, the determination unit 28 may determine an elapsed time for determining that the first condition is satisfied according to the amount of movement of the subject and the processing load of the first calculation unit 34.

  Another first condition is when the posture information of the clothing image instructed by the operation instruction of the input unit 16 by the user matches the posture information of the subject.

  In this case, the determination unit 28 is instructed by the skeleton information created by the extraction unit 32 from the subject image acquired by the first acquisition unit 22 and the operation instruction of the user input unit 16 stored in the storage unit 18. It is determined whether or not the skeleton information included in the posture information of the applied clothing image matches. If they match, the determination unit 28 determines that the acquired subject image satisfies the first condition by the determination unit 28.

  If the orientation of the subject and the orientation of the superimposition object stored in the storage unit 18, that is, the orientation of the clothing image of the clothing to be tried on, do not match, it is sufficient even if template matching is performed by the first calculation unit 34. Accurate matching may be difficult.

  Therefore, when the posture information of the clothing image instructed by the operation instruction of the input unit 16 by the user matches the posture information of the subject, the determination unit 28 determines that the acquired subject image has been acquired in advance by the determination unit 28. It is preferable to determine that the defined first condition is satisfied.

  Another example of the first condition is that the amount of movement of the subject included in the subject image is equal to or less than a predetermined value.

  In this case, the determination unit 28 determines the position of the person as the subject in the subject image from the coordinate value of the joint position of the subject in the depth image as the subject image acquired by the first acquisition unit 22. Then, by comparing the position of the person as the subject in the depth image as the subject image acquired last time and the position of the person as the subject in the depth image as the subject image acquired this time, Is calculated. When it is determined that the movement amount of the person is equal to or less than a predetermined value, the determination unit 28 determines that the acquired subject image satisfies the first condition predetermined by the determination unit 28.

  As another first condition, there is a case where a person as a subject included in the acquired subject image is in a state where his arm is lowered.

  In this case, the determination unit 28 determines that the position corresponding to the arm portion of the subject is below the shoulder portion of the subject based on the coordinate value of the joint position of the subject in the depth image of the subject acquired by the first acquisition unit 22 ( It is determined whether or not it is positioned in a direction extending to the foot side). Then, when the person as a subject included in the acquired subject image (the depth image of the subject) is in a state where the arm is lowered, the determination unit 28 satisfies the first condition that is determined in advance. Is determined.

  When the person as the subject has a posture with his arms raised, there is a high possibility that the posture information of the clothing image to be superimposed is different from the posture information of the subject. If template matching is performed by the first calculation unit 34 using a subject image of a subject in such a posture, the accuracy of template matching may be reduced. Therefore, when the determination unit 28 determines that the person as the subject included in the subject image is in a state where the arm is lowered, it is preferable that the determination unit 28 determines that the acquired subject image satisfies the predetermined first condition. .

  Next, the determination unit 38 will be described.

  When the determination unit determines that the subject image acquired by the first acquisition unit 22 satisfies the first condition, the determination unit determines the first position calculated by the first calculation unit as the subject image. Is determined as the superimposed position of the clothing image.

  In addition, when the determination unit 28 determines that the subject image acquired by the first acquisition unit 22 does not satisfy the first condition, the determination unit 38 is a subject image acquired before the subject image. A superposition position is determined from the difference between the first position calculated for one subject image and the second position calculated by the second calculation unit 36 from the first subject image.

  Specifically, the determination unit 38 is acquired by the first acquisition unit 22 this time when the subject image acquired by the first acquisition unit 22 is determined not to satisfy the first condition by the determination unit 28. A position obtained by shifting the second position calculated by the second calculation unit 36 based on the subject image according to the difference is determined as a superposition position.

  That is, the difference is determined when the subject image acquired by the first acquisition unit 22 is a subject image satisfying the first condition before the subject image acquired by the first acquisition unit 22 this time. This is the difference between the first position calculated by the first calculation unit 34 from the first subject image) and the second position calculated by the second calculation unit 36 from the subject image (first subject image).

  The generation unit 40 generates a composite image in which the clothing image is superimposed on the superimposed position determined by the determination unit 38 in the subject image acquired this time.

  Specifically, the generation unit 40 is instructed by a user's operation instruction of the input unit 16 at the superimposed position on the color image of the subject as the subject image acquired by the first acquisition unit 22 and adjusted by the adjustment unit 30. The clothing image (color image) to be superimposed is superimposed. Thereby, the generation unit 40 generates a composite image.

  Specifically, the generation unit 40 refers to the color value (Cr, Cg, Cb) and the alpha value (a) defined for each pixel for the clothing image adjusted by the adjustment unit 30. The alpha value a is a value between 0 and 1. For the subject color image, the color values (Ir, Ig, Ib) for each pixel are referred to. Then, for each pixel at the same pixel position, a composite image is generated by determining a pixel value (color value and alpha value) for each pixel using the following formula (2).

  When the clothing image occupies only a partial area of the color image of the subject, the alpha value is calculated as “0” (a = 0) in the area outside the occupied area of the clothing image.

  Ox = (1-a) × Ix + a × Cx (2)

  In formula (2), x represents r, g, or b.

  The presentation control unit 42 performs control for presenting the composite image to the presentation unit 20.

  Next, image processing executed by the image processing apparatus 12 according to the present embodiment will be described.

  FIG. 10 is a flowchart illustrating a procedure of image processing executed by the image processing apparatus 12 according to the present embodiment.

  Each time the image processing apparatus 12 receives one subject image (one depth image of the subject and one color image of the subject image taken at the same timing) from the imaging unit 14, the processing in steps S <b> 100 to S <b> 130 is performed. Execute. Note that when the image processing device 12 receives a moving image including a plurality of frames from the imaging unit 14, the image processing device 12 executes the processes of steps S100 to S130 for each frame.

  First, the first acquisition unit 22 acquires a subject image (step S100).

  Next, the extraction part 32 performs an extraction process (step S102). Specifically, in step S102, the extraction unit 32 generates skeleton information from the depth image of the subject acquired in step S100. In the present embodiment, the extraction unit 32 converts the coordinate system of the skeleton information of the subject (that is, the coordinate system of the first imaging unit 14A) into the coordinate system of the second imaging unit 14B.

  Next, the determination unit 28 determines whether or not the subject image acquired in step S100 satisfies the first condition (step S104).

  If an affirmative determination is made in step S104 (step S104: Yes), the process proceeds to step S106.

  In step S106, the adjustment unit 30 executes an adjustment process (step S106). In step S106, the adjustment unit 30 enlarges or reduces at least one of the clothing image and the subject image so that at least a part of the contour of the clothing image to be superimposed matches at least a part of the contour of the subject image. . Further, the adjustment unit 30 extracts a feature region used by the first calculation unit 34 from the enlarged or reduced clothing image and the enlarged or reduced subject image.

  Note that the clothing image to be superimposed is acquired during the adjustment process by the adjustment unit 30. Details of the adjustment process in step S106 will be described later.

  Next, the 1st calculation part 34 performs a 1st position calculation process (step S108). In step S108, the first calculation unit 34 makes the position of the feature region in the clothing image to be superimposed acquired by the adjustment unit 30 coincide with the position of the feature region in the subject image acquired in step S100. The first position of the clothing image in the subject image is calculated. Details of the procedure of the first position calculation process in step S108 will be described later.

  Next, the first calculation unit 34 stores the calculated first position in the storage unit 18 in association with information that can identify the subject image acquired in step S100 (step 110). As the identifiable information, for example, the acquisition date of the subject image is used.

  Next, the 2nd calculation part 36 performs a 2nd position calculation process (step S112). In step S112, the second calculation unit 36 causes the clothing image in the subject image so that the position of the feature point in the clothing image to be superimposed acquired by the adjustment unit 30 matches the position of the feature point in the subject image. The second position of is calculated. Details of the second position calculation process will be described later.

  Next, the second calculation unit 36 stores the calculated second position in the storage unit 18 in association with information that can identify the subject image acquired in step S100 (step S114). For this identifiable information, the same information as in step S110 may be used.

  Next, the determination unit 38 reads the first position calculated in step S108 and the second position calculated in step S112 from the storage unit 18, and calculates a difference between them (step S116). Accordingly, the determination unit 38 stores the calculated difference in the storage unit 18 in association with information that can identify the subject image used in Steps S110 and S114 (Step S118).

  When the difference is already stored in the storage unit 18, the determination unit 38 may overwrite the stored difference on the difference newly calculated in step S <b> 116. In this way, only the latest difference may be stored in the storage unit 18 as the difference calculated in the determination unit 38 and step S116.

  Next, the determination part 38 determines a superimposition position (step S120). In step S120, the determination unit 38 determines the first position calculated in step S108 as the superimposed position of the clothing image to be superimposed acquired by the adjustment process in step S106 in the subject image acquired in step S100. . And it progresses to step S126 mentioned later.

  In the image processing apparatus 12, when the subject image acquired in step S <b> 100 is determined to satisfy the first condition by the processing in steps S <b> 104 to S <b> 120, the first calculation unit 34 calculates the first image. One position is determined as a superimposed position of the clothing image in the subject image.

  On the other hand, if a negative determination is made in step S104 (step S104: No), the process proceeds to step S122. In step S122, similarly to step S112, the second calculation unit 36 executes a second position calculation process (step S122) (details will be described later).

  Next, the determination part 38 determines a superimposition position (step S124).

  Specifically, the determination unit 38 includes a first position calculated from the subject image (first subject image) acquired before the subject image acquired this time by the first calculation unit 34, and the first position. The difference between the subject image (first subject image) used for the calculation and the second position calculated by the second calculation unit 36 is read from the storage unit 18. The determination unit 38 reads the latest difference (that is, the previously calculated difference) among the differences stored in the storage unit 18. Then, the determination unit 38 determines a position where the second position calculated from the subject image acquired this time in step S122 is shifted according to the read difference as the superposition position.

  Note that the direction in which the second position calculated from the subject image acquired this time in step S122 is shifted is the second position calculated by the second calculation unit 36 last time, and is calculated by the first calculation unit 34 last time. The direction is parallel to the direction vector having the first position as the end point. Then, the process proceeds to step S126.

  Next, the generation unit 40 generates a composite image (step S126). In step S126, the generation unit 40 acquires the clothing image to be superimposed on the superimposed position on the color image of the subject acquired in step S100 and acquired by the adjustment process in step S106 and enlarged or reduced according to the subject image. Are combined to generate a composite image.

  Next, the presentation control unit 42 performs control to present the composite image generated in step S128 to the presentation unit 20 (step S128).

  Next, the image processing apparatus 12 determines whether or not to end the image processing (step S130). The image processing apparatus 12 performs, for example, by determining whether or not a signal indicating an end instruction is received by operating an end instruction button (not shown) of the image processing apparatus 12 according to a user operation instruction.

  If a negative determination is made in step S130 (step S130: No), the process returns to step S100. If an affirmative determination is made in step S130 (step S130: Yes), this routine is terminated.

  Next, the adjustment process in step S106 will be described.

  FIG. 11 is a flowchart illustrating a procedure of adjustment processing executed by the adjustment unit 30.

  First, the adjustment unit 30 calculates the orientation of the subject from the skeleton information of the subject created by the extraction unit 32 in step S102 (see FIG. 10) (step S200).

  Next, the adjustment unit 30 stores a clothing image having posture information corresponding to the orientation of the subject (subject orientation information) calculated in step S200 among the clothing images corresponding to the clothing number selected by the user. Read from the first table of the unit 18 (step S202).

  Next, the adjustment unit 30 adjusts the depth image of the subject acquired by the third acquisition unit 22A (step S204). Specifically, the adjustment unit 30 converts the coordinate system of the pixel positions of each pixel of the depth image of the subject (the coordinate system of the first imaging unit 14A) into the coordinate system of the second imaging unit 14B. Then, the adjustment unit 30 causes the pixels at the respective pixel positions constituting the depth image of the subject after coordinate conversion to be positions corresponding to the pixel positions of the respective pixels constituting the color image of the subject acquired at the same timing. The resolution of the depth image of the subject is adjusted to the same resolution as the color image of the subject.

  Next, the adjustment unit 30 calculates the size of the feature area in the clothing image read in step S202 and the size of the feature area in the subject image (step S206). As described above, in the present embodiment, the shoulder region is used as the feature region. Therefore, in the present embodiment, the adjustment unit 30 calculates each of the shoulder width in the clothing image read in step S202 and the shoulder width in the subject image acquired in step S100 (see FIG. 10).

  Next, the adjustment unit 30 determines the scaling value (enlargement / reduction ratio) of the clothing image from the size of the feature region calculated in step S206, that is, the shoulder width of the clothing image and the shoulder width of the subject image. (Step S208).

  Next, the adjustment unit 30 scales (enlarges or reduces) the clothing image read in step S202 with the enlargement / reduction ratio specified by the scaling value determined in step S208 (step S210). Similarly, the adjustment unit 30 enlarges or reduces the skeleton information included in the posture information of the clothing image to be superimposed at the enlargement / reduction ratio specified by the same scaling value.

  Next, the adjustment unit 30 extracts a feature region from the clothing image enlarged or reduced in step S210 and the subject image acquired in step S100 (see FIG. 10).

  First, the adjustment unit 30 extracts the contours of the clothing image that has been enlarged or reduced in step S210 and the subject image acquired in step S100 (see FIG. 10) (step S212). Next, the adjustment unit 30 extracts a shoulder region in each of the outline of the clothing image and the outline of the subject image (step S214). Then, this routine ends.

  In step S208, the case has been described in which the clothing image scaling value (enlargement / reduction ratio) is determined and the clothing image is scaled to the enlargement / reduction ratio specified by the scaling value. However, the enlargement / reduction ratio of at least one of the clothing image and the subject image may be obtained so that at least a part of the contour of the clothing image matches the contour of the subject image. Then, the subject image may be scaled by the reciprocal of the determined enlargement / reduction ratio of the clothing image.

  By executing the processing of step S200 to step S214, the adjustment unit 30 causes the clothing image and the subject to match at least a part of the contour of the clothing image to be superimposed and at least a part of the contour of the subject image. Enlarge or reduce at least one of the images. In addition, the adjustment unit 30 extracts a shoulder region as a feature region from the enlarged or reduced clothing image and the enlarged or reduced subject image.

  Next, the procedure of the first position calculation process executed by the first calculation unit 34 described in step S108 of FIG. 10 will be described.

  FIG. 12 is a flowchart illustrating a procedure of the first position calculation process executed by the first calculation unit 34.

  The first calculation unit 34 performs known template matching using the shoulder region of the subject image and the shoulder region of the clothing image as the feature regions extracted by the adjustment unit 30 (step S400). . In step S400, the first calculation unit 34 searches for the depth image of the subject adjusted by the adjustment unit 30 by template matching, and determines a position on the depth image that matches the specific region (shoulder region) in the clothing image. And calculated as the first position.

  Then, the first calculation unit 34 outputs the calculated first position to the determination unit 28 (step S402). Then, this routine ends.

  Next, the procedure of the second position calculation process executed by the second calculation unit 36 described in step S112 and step S122 in FIG. 10 will be described.

  FIG. 13 is a flowchart showing the procedure of the second position calculation process executed by the second calculation unit 36.

  First, the 2nd calculation part 36 calculates the center position between the both shoulders of a clothing image as a feature point of a clothing image (step S300).

  Next, the second calculation unit 36 calculates the center position between the shoulders of the subject image (step S302). Specifically, the second calculation unit 36 calculates the center position between both shoulders of the subject from the skeleton information of the subject created by the extraction unit 32 by the processing of step 106 (see FIG. 10).

  Next, the second calculation unit 36 calculates the second position so that the center position calculated in step S300 matches the center position calculated in step S302 (step S304). In the present embodiment, the second calculation unit 36 calculates the center position between both shoulders in the subject image calculated in step S302 as the second position.

  Then, this routine ends.

  FIG. 14 is an explanatory diagram schematically illustrating generation of a composite image in the image processing apparatus 12 according to the present embodiment.

  For example, it is assumed that the clothing image to be superimposed is a clothing image 61A (see FIG. 14A). Further, it is assumed that the depth image of the subject is a depth image 71C (see FIG. 14D).

  In this case, the outline 61B of the clothing image 61A is extracted by the adjustment process by the adjustment unit 30 (see FIG. 14B). Further, a shoulder region 61D is extracted as a feature region by the adjustment processing by the adjustment unit 30 (see FIG. 14C).

  Similarly, the contour 71B is extracted from the depth image 71C of the subject by the adjustment processing by the adjustment unit 30 (see FIG. 14E). Further, a shoulder region 71D is extracted as a feature region by the adjustment processing by the adjustment unit 30 (see FIG. 14F).

  When the determination unit 28 determines that the subject image satisfies the first condition, the first calculation unit 34 performs template matching using the shoulder region 61D of the clothing image 61A and the shoulder region 71D of the subject depth image 71C. By performing (FIG. 14G), the first position (not shown in FIG. 14) is calculated. When the determination unit 28 determines that the subject image satisfies the first condition, the determination unit 38 determines the first position as the superimposed position.

  Then, the generating unit 40 acquires the clothing image to be superimposed on the superimposed position on the color image of the subject acquired by the first acquiring unit 22 and acquired by the adjustment process in step S106 and enlarged or reduced according to the subject image. Are superimposed on each other to generate a composite image W (calculation in FIG. 14H).

  As described above, in the image processing apparatus 12 according to the present embodiment, when the subject image acquired by the first acquisition unit 22 is determined to satisfy the first condition by the determination unit 28, the first calculation unit 34 is used. The first position calculated in the above is determined as the superimposed position of the clothing image in the subject image. Then, the generating unit 40 acquires the clothing image to be superimposed on the superimposed position on the color image of the subject acquired by the first acquiring unit 22 and acquired by the adjustment process in step S106 and enlarged or reduced according to the subject image. Are combined to generate a composite image.

  On the other hand, in the image processing apparatus 12, when the subject image acquired by the first acquisition unit 22 is determined not to satisfy the first condition by the determination unit 28, the subject image is acquired before the subject image. The superposition position is determined from the difference between the first position calculated for the first subject image and the second position calculated by the second calculation unit 36 from the first subject image.

  The first condition is a condition for determining whether or not the first calculation unit 34 executes the first position calculation process. Then, the generation unit 40 superimposes the clothing image to be superimposed acquired by the adjustment process in step S106 and enlarged or reduced according to the subject image on the superposition position on the subject image acquired by the first acquisition unit 22. By doing so, a composite image is generated.

  As described above, according to the image processing device 12 of the present embodiment, the first calculation unit 34 calculates the first position only for the subject images determined to satisfy the first condition among the acquired subject images. Execute. Then, the determination unit 38 determines the first position as the overlapping position. On the other hand, for a subject image determined not to satisfy the first condition, the first calculation unit 34 does not perform the first position calculation process, and the first calculation unit 34 calculates the first position and the first position. The superimposed position is determined from the difference between the subject image (first subject image) used for calculating the position and the second position calculated by the second calculating unit 36.

  Therefore, the image processing apparatus 12 according to the present embodiment can provide a composite image with a low load while improving accuracy.

  In the image processing apparatus 12 according to the present embodiment, the adjustment unit 30 performs the adjustment process when it is determined by the determination unit 28 that the subject image acquired by the first acquisition unit 22 satisfies the first condition. Explained when to do. However, the adjustment unit 30 may execute the adjustment process described above regardless of whether the subject image acquired by the first acquisition unit 22 satisfies the first condition. However, the adjustment process performed by the adjustment unit 30 has a lower processing load than the first calculation unit 34 but a higher processing load than the second calculation unit 36. For this reason, the adjustment processing by the adjustment unit 30 is executed when the subject image acquired by the first acquisition unit 22 is determined to satisfy the first condition by the determination unit 28 as described in the above embodiment. It is preferable to do.

  In the present embodiment, the adjustment unit 30 may perform a weighting process on the extracted feature area according to the posture information of the subject. In this case, the first calculation unit 34 performs template matching using the weighted feature region.

  Specifically, when the adjustment unit 30 extracts a feature region from the contour extracted from the depth image of the subject, the luminance value of the region located closer to the imaging unit 14 when the subject image is captured is larger. The contour of the shoulder region is extracted so that the contour becomes high.

  For example, when the subject is facing the direction tilted to the left with respect to the imaging unit 14, the left shoulder of the subject is imaged in a state closer to the imaging unit 14 than the right shoulder. For this reason, when the posture information of the subject image indicates that the posture is tilted to the left from the posture facing the front with respect to the imaging unit 14, the adjustment unit 30 extracts the contour extracted from the depth image of the subject. The weighting process is performed so that the brightness value of the contour of the region corresponding to the left shoulder is higher than the contour of the region corresponding to the right shoulder.

  The adjustment unit 30 performs the weighting process by multiplying the pixel value of the pixel at the target pixel position to be subjected to the weighting process by a predetermined constant. Note that the adjustment unit 30 may obtain a pixel at a pixel position closer to the imaging unit 14 from the depth image, and perform the weighting process using a higher constant for a pixel at a pixel position closer to the imaging unit 14.

  Similarly, when the adjustment unit 30 extracts a feature region from the contour extracted from the clothing image, the adjustment unit 30 is closer to the imaging unit 14 at the time of imaging in the clothing image to be superimposed. The contour of the shoulder region is extracted so that the contour has a high luminance value.

  By performing such weighting, in the template matching process by the first calculation unit 34, the first position can be calculated so that the accuracy of the region closer to the imaging unit 14 side becomes higher. For this reason, further improvement in accuracy can be achieved.

  FIG. 15 is an explanatory diagram schematically illustrating generation of a composite image when the adjustment unit 30 performs weighting processing.

For example, it is assumed that the clothing image to be superimposed is a clothing image 61A ₂ (see FIG. 15A). Further, it is assumed that the depth image of the subject is the depth image 71C ₂ (see FIG. 15D). As described above, it is assumed that the subject is imaged with a posture inclined such that the left shoulder is closer to the imaging unit 14 than the right shoulder with respect to the imaging unit 14 (not illustrated in FIG. 15). In this case, as described above, the image processing apparatus 12 reads the clothing image corresponding to the tilted posture from the storage unit 18.

In this case, the adjustment process by adjusting unit 30, the outline 61B ₂ of the clothing image 61A ₂ are extracted (see FIG. 15 (B)). Moreover, the adjustment process by the adjusting unit 30, as a characteristic region, the shoulder region 61D ₂ are extracted (see FIG. 15 (C)). In this case, the adjusting section 30, after performing weighted processing area SA corresponding to the left shoulder, it extracts the shoulder region 61D _2. Therefore, the shoulder region 61D _2, as shown in FIG. 15 (C), a contour which has been treated weighted so that the brightness value of the region SA corresponding to the human body left shoulder is higher.

Similarly, the adjustment process by adjusting unit 30, the outline 71B ₂ from the depth image 71C ₂ of the object is extracted (see FIG. 15 (E)). Moreover, the adjustment process by the adjusting unit 30, as a characteristic region, the shoulder region 71D ₂ are extracted (see FIG. 15 (C)). In this case, the adjusting section 30, after performing weighted processing region SB corresponding to the left shoulder, it extracts the shoulder region 71D _2. Therefore, the shoulder region 71D _2, as shown in FIG. 15 (F), a corrected contoured so that the luminance value of the region SB corresponding to the human body left shoulder is higher.

If the subject image by the discrimination unit 28 is determined as the first condition is satisfied, the first calculating section 34, a shoulder region 61D ₂ of the garment image 61A _2, the shoulder region 71D ₂ of the depth image 71C ₂ of the object, the The first position (not shown in FIG. 15) is calculated by performing the used template matching (FIG. 15G).

In this case, the shoulder region 61D ₂ of the garment image 61A _2, the shoulder region 71D ₂ of the depth image 71C ₂ of the object, is higher luminance value of the region corresponding to the left shoulder is more area close to the imaging unit 14 side Thus, the contour is weighted. For this reason, the first calculation unit 34 can perform template matching so that the search accuracy of the region corresponding to the left shoulder is increased.

  When the determination unit 28 determines that the subject image satisfies the first condition, the determination unit 38 determines the first position as the superimposed position.

  Then, the generating unit 40 acquires the clothing image to be superimposed on the superimposed position on the color image of the subject acquired by the first acquiring unit 22 and acquired by the adjustment process in step S106 and enlarged or reduced according to the subject image. Are superimposed on each other to generate a composite image WA (see FIG. 15H).

  As described above, the adjustment unit 30 performs the weighting process, thereby further improving accuracy.

  In the present embodiment, the case where the shoulder region is used as the feature region has been described. This is because, in the present embodiment, as an example of a clothing image to be superimposed, a clothing image of clothing to be tried on the upper body side of the human body is given as an example.

  However, as described above, the feature region is not limited to the shoulder region as long as the shape of the subject trying on clothes can be estimated.

  The feature area may be set according to the posture of the subject of the subject image acquired by the image processing device 12.

  For example, when a subject image of a subject in a posture facing right side with respect to the imaging unit 14 is acquired, it is difficult to extract the contours of both shoulder portions of the subject, and it may be difficult to use the shoulder region as a feature region. .

  In view of this, the image processing apparatus 12 stores in advance in the storage unit 18 the orientation information (including the orientation of the subject) of the subject image, the feature region corresponding to the orientation information, and the feature point corresponding to the feature region. May be. In this case, an area in which the shape of the subject can be estimated from the subject having the posture of the corresponding posture information may be determined in advance as the feature region corresponding to the posture information. For example, a waist region or a neck region is determined in advance as a feature region in the posture information indicating that it is lateral to the imaging unit 14. Further, as described above, the feature points may be determined in advance according to the corresponding feature region.

  Then, the adjustment unit 30 calculates the orientation of the subject from the subject image acquired by the first acquisition unit 22, and then reads the feature area and feature point corresponding to the orientation information of the orientation of the subject from the storage unit 18. Then, the adjustment unit 30 performs the adjustment process using the read feature area. Further, the adjustment unit 30 outputs the read feature area to the first calculation unit 34 and outputs the read feature points to the second calculation unit 36. The first calculation unit 34 and the second calculation unit 36 may perform the calculation process using the feature regions and feature points received from the adjustment unit 30.

  In the present embodiment, the depth image of the subject acquired by the first imaging unit 14A is converted into the coordinate system of the second imaging unit 14B, and then superimposed on the color image of the subject acquired by the second imaging unit 14B. A case has been described in which a composite image in which a clothing image is superimposed on a position is generated.

  However, the image processing device 12 performs image processing using the clothes depth image captured by the first imaging unit 14A and the clothing color image captured by the second imaging unit 14B as the clothing image. Also good. In this case, after aligning the depth image of the clothes and the depth image of the subject on the coordinate system of the first image pickup unit 14A, the superimposed position of the clothes image is second from the coordinate system of the first image pickup unit 14A. Conversion to the coordinate system of the imaging unit 14B is performed.

  In this process, even if the position of the first imaging unit 14A when capturing clothes is different from the position of the first imaging unit 14A when capturing a subject, coordinate conversion can be performed by a known method. is there.

(Embodiment 2)
FIG. 16 is a schematic diagram showing the image processing system 10A.

  In the image processing system 10 </ b> A, a storage device 72 and a processing device 11 are connected via a communication line 74.

  The storage device 72 is a device including the storage unit 18 in the first embodiment, and is a known personal computer or the like. The processing device 11 is a device including the image processing device 12, the imaging unit 14, the input unit 16, and the presentation unit 20 in the first embodiment. In addition, the same code | symbol is provided to the same function part as Embodiment 1, and detailed description is abbreviate | omitted. The communication line 74 is a communication line such as the Internet, and includes a wired communication line and a wireless communication line.

  As shown in FIG. 16, the storage unit 18 is provided in a storage device 72 connected to the processing device 11 (image processing device 12) via a communication line 74, so that a plurality of processing devices (image processing) are provided. The same storage unit 18 can be accessed from the apparatus 12), and centralized management of data stored in the storage unit 18 becomes possible.

  In addition, the processing apparatus 11 can be installed in arbitrary places. For example, you may install the processing apparatus 11 in the place where a user visually recognizes a synthesized image, for example, a shop. Moreover, you may mount each function of the processing apparatus 11 in a well-known portable terminal.

(Embodiment 2)
Next, the hardware configuration of the image processing apparatus 12 according to the first embodiment and the second embodiment will be described. FIG. 17 is a block diagram illustrating a hardware configuration example of the image processing apparatus 12 according to the present embodiment.

  The image processing apparatus 12 according to the first and second embodiments includes a presentation unit 80, a communication I / F unit 82, an imaging unit 84, an input unit 94, a CPU (Central Processing Unit) 86, and a ROM (Read Only Memory) 88. A RAM (Random Access Memory) 90, an HDD 92, and the like are connected to each other by a bus 96, and have a hardware configuration using a normal computer.

  The CPU 86 is an arithmetic device that controls the overall processing of the image processing apparatus 12. The RAM 90 stores data necessary for various processes by the CPU 86. The ROM 88 stores a program for realizing various processes by the CPU 86 and the like. The HDD 92 stores data stored in the storage unit 18 described above. The communication I / F unit 82 is an interface for connecting to an external device or an external terminal via a communication line or the like and transmitting / receiving data to / from the connected external device or external terminal. The presentation unit 80 corresponds to the presentation unit 20 described above. The imaging unit 84 corresponds to the imaging unit 14 described above. The input unit 94 corresponds to the input unit 16 described above.

  A program for executing the various processes executed by the image processing apparatus 12 according to the first and second embodiments is provided by being incorporated in advance in the ROM 88 or the like.

  The program executed in the first and second embodiments is a file in a format that can be installed in these devices or in an executable format, and is a CD-ROM, flexible disk (FD), CD-R, DVD. (Digital Versatile Disk) or the like may be provided by being recorded on a computer-readable recording medium.

  Further, the program executed in the first embodiment and the second embodiment may be configured to be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. The program for executing each of the above processes in the image processing apparatus 12 according to the first and second embodiments may be configured to be provided or distributed via a network such as the Internet.

  In the program for executing the various processes executed in the first embodiment and the second embodiment, the above-described units are generated on the main storage device.

  Various information stored in the HDD 92, that is, various information stored in the storage unit 18, may be stored in an external device (for example, a server). In this case, the external device and the CPU 86 may be connected via a network or the like.

  In addition, although several embodiment of this invention was described above, these embodiment is shown as an example and is not intending limiting the range of invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

12 image processing device 14 imaging unit 16 input unit 18 storage unit 20 presentation unit 22 first acquisition unit 26 second acquisition unit 28 determination unit 30 adjustment unit 32 extraction unit 34 first calculation unit 36 second calculation unit 38 determination unit 40 generation Part

Claims (6)

Based on the superimposition target clothing image and the Utsushitai image, a first calculation unit for calculating a first position of the garment image in the object image,
A second calculation unit that calculates a second position of the clothing image in the subject image, which is different from the first position, based on the subject image and the clothing image;
When determining the superimposition position using the second position, the superimposition position of the clothing image is further determined using the first position calculated for the first subject image acquired before the subject image. a decision unit for determining,
The image processing apparatus provided with. A generator for generating a composite image in which the clothing image is superimposed at the superposition position in the subject image;
The image processing apparatus according to claim 1. The determination unit
Using said first position or said second position, determine the superimposing position of the garment image in the object image, based on a predetermined first condition, the superposition of the garment image using the first position Switching between determining a position or determining the superimposed position of the clothing image using the second position and the first position calculated for the first subject image;
The image processing apparatus according to claim 1 . The first calculation unit includes:
Calculating the first position so that the position of the feature region in which the shape of the subject in the subject image can be estimated matches the position of the feature region in the clothing image;
The second calculator is
Calculating the second position so that a position of a feature point predetermined according to the feature region in the subject image matches a position of the feature point in the clothing image;
The image processing apparatus according to claim 1. Based on the superimposition target clothing image and the Utsushitai image, it calculates a first position of the garment image in the object image,
Based on the subject image and the clothing image, the second position of the clothing image in the subject image, which is different from the first position, is calculated,
When determining the superimposition position using the second position, the superimposition position of the clothing image is further determined using the first position calculated for the first subject image acquired before the subject image. decide,
Images processing method. The computer,
Means for, based on the superimposition target clothing image and the Utsushitai image, calculates a first position of the garment image in the object image,
Means for calculating the second position of the clothing image in the subject image, which is different from the first position, based on the subject image and the clothing image;
When determining the superimposition position using the second position, the superimposition position of the clothing image is further determined using the first position calculated for the first subject image acquired before the subject image. Means to determine ,
The image processing program causing to function.

Images