JP2020178191A - Image processing device, photographing device, and program - Google Patents

Abstract

To provide a photographing device and the like capable of improving a commodity value.SOLUTION: The photographing device comprises a photographing unit for photographing a user, an image processing device, and the like. The image processing device extracts a person image 200 from an image photographed by the photographing unit and synthesizes content 300 to the person image 200 on the basis of a feature amount obtained from the person image 200, thereby generating a synthetic image 400. At this time, the image processing device selects the content 300 to be synthesized and sets the color and layout of the content 300, on the basis of the feature amount.SELECTED DRAWING: Figure 7

Description

The present invention relates to an image processing device, a photography device and a program.

There are various photography devices that photograph a user and print the photographed image. In such a photographic apparatus, a service that extracts a user's person image from the photographed image by taking a picture of the user with a background plate of a predetermined color as a background and chromakey-synthesizes the person image into a background image (content image). (For example, Patent Documents 1 and 2).

JP-A-2017-34354 JP-A-2016-146623

However, in a conventional photography apparatus, a user selects a content image prepared in advance, and a person image is often combined with the content image.

In this case, the content image provided to the user as a composite image is prepared in advance and selected by the user, and tends to be a uniform image that does not exceed a certain range regardless of the user. From this point of view, it is considered that there is room for further enhancing the attractiveness of the provided content image and further improving the commercial value.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing device or the like capable of further improving the commercial value of a content image.

The first invention for solving the above-mentioned problems is based on an image of a subject extracted from an image taken by the user as a subject or an image input by the user, and based on a feature amount obtained from the image of the subject. It is an image processing device characterized in that a content image is selected from a plurality of content image candidates.
It is also desirable to set the display form of the content image based on the feature amount.

In the second invention, the image of the subject is extracted from the image taken by the user as the subject or the image input by the user, and the display form of the content image is set based on the feature amount obtained from the image of the subject. It is an image processing apparatus characterized by this.

In the first and second inventions, the content image can be made variable according to the characteristics of the subject extracted from the image taken by the user or the image input by the user, and the content image can be made different depending on the subject. Therefore, a highly rare content image can be provided to each user, and the commercial value of the content image can be enhanced. The "subject" refers to all objects shown in the image.

It is desirable that the image processing device synthesize the content image and the image of the subject.
In the present invention, the content image is provided as a composite image synthesized with the image of the subject, and at this time, the content image can be changed according to the characteristics of the subject.

It is desirable to use AI at least when selecting the content image from a plurality of content image candidates based on the feature amount or when obtaining the feature amount.
This makes it possible to select a content image and set a display form that is close to human judgment.

It is also desirable to select a plurality of the content images from a plurality of content image candidates based on the feature amount using AI, and accept the user's selection of the content image from among them.
In the present invention, the user can select a favorite content image from a plurality of recommended content images selected by AI.

A third invention includes a photographing unit for photographing a user and an image processing device, and the image processing device extracts a person image of the user from an image photographed by the user in the photographing unit, and the person image. It is a photography apparatus characterized in that a content image is selected from a plurality of content image candidates based on the feature amount obtained from.
A fourth invention includes a photographing unit for photographing a user and an image processing device, and the image processing device extracts a person image of the user from an image photographed by the user in the photographing unit, and the person image. It is a photographic apparatus characterized by setting a display form of a content image based on a feature amount obtained from.
In the third and fourth inventions, it is possible to provide the user with a content image according to the characteristics of the user's person image in the captured image.

It is desirable that the image processing apparatus create a composite image by chroma keying the content image and the person image.
The image processing device creates a composite image by, for example, chroma key composition, and can be suitably applied as a service in a photography device provided with a background plate of a predetermined color or the like.

According to a fifth aspect of the present invention, a computer extracts a subject image from an image taken by the user as a subject or an image input by the user, and a plurality of content image candidates are obtained based on the feature amount obtained from the subject image. It is a program for functioning as an image processing device for selecting a content image from among.
A sixth aspect of the present invention is a display mode of a content image based on a feature amount obtained by extracting an image of a subject from an image taken by the user as a subject or an image input by the user and obtaining the feature amount from the image of the subject. It is a program for functioning as an image processing device for setting.

INDUSTRIAL APPLICABILITY According to the present invention, it is possible to provide an image processing device or the like capable of further improving the commercial value of a content image.

The figure which shows the appearance of the photography apparatus 1. The figure which shows the system configuration of the photography apparatus 1. The figure which shows the hardware composition of the image processing apparatus 10. The flowchart which shows the process of the photography apparatus 1. An example of a photographed image 100 and a person image 200. An example of content 300. The figure explaining the setting of the color and arrangement of the content 300. An example of a content selection screen. An example of an input image 100a and a subject image 200a.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

(1. Photographing device 1)
FIG. 1 is a diagram showing the appearance of the photographic apparatus 1 according to the embodiment of the present invention, and FIG. 2 is a diagram showing a system configuration of the photographic apparatus 1.

The photography device 1 has a box-like appearance in which a user enters and takes a picture, and has an image processing device 10, a display unit 11, a billing unit 12, a photographing unit 13, lighting 14, a printer 15, and a printed matter outlet 17. , A background plate 18 and the like. Other than the background plate 18, it is mainly provided on the front portion (corresponding to the left side of FIG. 1) of the photography apparatus 1.

The image processing device 10 is provided inside the photography device 1, and chroma key-combines a user's person image and a content image (hereinafter, simply referred to as content) according to a procedure described later to create a composite image. Further, the image processing device 10 controls the entire processing of the photography device 1 as a control device.

FIG. 3 is a diagram showing a hardware configuration of the image processing device 10. As shown in FIG. 3, the image processing device 10 can be realized by, for example, a computer configured by connecting a control unit 101, a storage unit 102, an input unit 103, a communication control unit 104, and the like by a bus or the like. However, the present invention is not limited to this, and various configurations can be taken as appropriate.

The control unit 101 is composed of a CPU, ROM, RAM, and the like. The CPU calls and executes a program related to processing of the image processing device 10 stored in a storage medium such as the storage unit 102 and ROM to a work area on RAM. ROM is a non-volatile memory that permanently holds programs such as boot programs and BIOS, and data. The RAM is a volatile memory, and includes a work area used by the control unit 101 to perform various processes while temporarily holding programs and data loaded from the storage unit 102, ROM, and the like.

The storage unit 102 is a hard disk drive, a solid state drive, a flash memory, or the like, and stores a program executed by the image processing device 10 at the time of processing described later, data necessary for program execution, an OS, and the like. These programs and data are read out and executed by the control unit 101 as needed. Further, in the present embodiment, the image data of the content described later is stored in the storage unit 102 in advance.

The input unit 103 inputs various settings to the image processing device 10.
The communication control unit 104 is a communication interface that mediates communication via a network or the like, and communicates with a server or the like (not shown).

Returning to the description of FIGS. 1 and 2. The display unit 11 is realized by a display with a touch panel or the like that also serves as an input unit of the photography device 1. The display unit 11 displays various screens related to the user's operation and selection, and accepts the user's instruction input and the like on the screen.

The billing unit 12 includes a coin / banknote slot, a change return slot, a receipt outlet, a receipt printer, etc. (not shown), and manages billing and issuance of receipts from users.

The photographing unit 13 is provided with a camera or the like inside the photographing window. For example, a high-resolution digital camera is used as the camera, but the camera is not limited to this.

The illumination 14 is an illumination for photographing the user. The illumination 14 is, for example, a surface light source that irradiates white light, and is provided above the photographing unit 13, but is not limited to this.

The printer 15 prints an image and is provided inside the photography apparatus 1. The printed matter on which the image is printed is discharged to the printed matter outlet 17. The method of the printer 15 and the number of installed printers are not particularly limited.

The background plate 18 is provided at the rear portion of the photography device 1 (corresponding to the right side of FIG. 1) and serves as a background when the user takes a picture. The background plate 18 is composed of a panel of a predetermined color or the like.

(2. Processing performed by the photography apparatus 1)
FIG. 4 is a flowchart showing the processing of the photography apparatus 1. Each step of FIG. 4 is performed by the image processing device 10 performing processing in each part of the photography device 1.

In the present embodiment, at the start of shooting, the photography device 1 displays a screen (not shown) for designating the content selection method on the display unit 11, and the user automatically selects the content on the screen. Specify whether to select by the user. The photography device 1 accepts the designation of the content selection method (S1).

When the content is automatically selected (S2; automatic selection), the photography device 1 displays a billing screen (not shown) for prompting the user to charge on the display unit 11, and the user charges on the charging unit 12. ..

When the photography device 1 receives a charge from the user (S4), the display unit 11 displays a guidance screen (not shown) to provide operation guidance related to photography, and the photography unit 13 takes a picture of the user (S5). ).

The photography device 1 extracts a person image of the user as a subject from the photographed image taken by the user (S6). 100 in FIGS. 5A and 5B is an example of a photographed image, and 200 is an example of a person image extracted from the photographed image. The person image 200 can be extracted by extracting a region excluding the color region a of the background plate 18 from the captured image 100.

Further, the photography device 1 acquires a feature amount from the person image 200 (S7). This feature amount is used for automatic selection of contents and setting of display form, which will be described later. For example, the person's clothes color, clothes shape, hair color, hairstyle, eye color, and facial skin portion in the person image 200. It is a value about the color, the facial expression, the direction of the person, the pose, the number of people, the position of the person image 200 in the photographed image 100, and the like. These features can be obtained by a known calculation method.

After obtaining the feature amount from the person image 200 in this way, the photography apparatus 1 synthesizes the content into the person image 200 based on the feature amount obtained from the person image 200. In the present embodiment, first, the content is automatically selected based on the feature amount (S8).

FIG. 6 is an example of the content 300. In the present embodiment, as shown in FIG. 6, a car image is used as the content 300, and a plurality of types (vehicle types) of the content 300 are prepared as selection candidates (content image candidates). These plurality of contents 300 are stored in advance in the storage unit 102 of the image processing device 10.

As a method of selecting the content 300, for example, attributes such as a person's race, age, gender, personality, and person composition are specified from one or a plurality of feature quantities using a known image recognition technology using AI (Artificial Intelligence). Then, the content 300 according to the specific result is selected. Various known techniques can be applied to AI (artificial intelligence), and for example, an expert system using a feature amount obtained by machine learning and a discriminant based on the feature amount can be applied. In addition, methods using deep learning and neural networks can also be applied.

For example, when the person of the person image 200 is identified as a young person, the content 300 of the off-road type or sports type vehicle type is selected, and when the person image 200 is identified as having a family, the content of the family type vehicle type is selected. Content 300 is selected.

In addition, in S8, for example, it is possible to use a value for the color of clothes as a feature amount and select the content 300 according to the color of the clothes from a plurality of contents 300 having different colors and designs. For example, individual content 300 having a design matching the hue (for example, content 300 of a vehicle model that matches the hue) is prepared for each hue, and the content 300 automatically selected in S8 is extracted as a feature amount. It is assumed that it matches the hue information of the color, and not only the color but also the design can be changed.

After the content 300 is automatically selected as described above, the photography device 1 sets the display form of the content 300 based on the feature amount obtained from the person image 200. Here, the color and arrangement of the content 300 are set as the display form of the content 300 (S9, S10), and the content 300 is combined with the person image 200 (S11).

In S9, for example, the color of clothes is used as the feature amount of the person image 200, and the color of the content 300 is changed to the color of the clothes.

The color of the clothes is obtained as a representative color for the area b of the clothes (jacket portion) of the person image 200 extracted, for example, as shown in b of FIG. 5A in S7. The representative color is, for example, the average or median of RGB values among the pixels in the above region b. As shown in FIG. 7, in S9, the color of the body portion of the car is changed to its representative color.

The feature amount used for setting the color of the content 300 is not limited to the color of clothes, but may be the color of hair, the color of eyes, or the color of the skin portion of the face. Also in this case, the representative color of each region is obtained, and the color of the content 300 is changed to the representative color. Further, the portion of the content 300 whose color is changed may be a part of the body portion of the car.

Further, in S10, for example, the orientation of the person is used as the feature amount of the person image 200, and the arrangement of the content 300 is adjusted to the orientation.

For example, when the person image 200 is oriented to the left as shown in FIG. 5A, the content 300 is arranged on the left side and the person image 200 is arranged on the right side in S10 as shown in FIG. 7, and these are arranged in S11. A composite image 400 in which the images are combined is created.

In the present embodiment, the orientation of the content 300 is also changed according to the arrangement of the content 300, and in the example of FIG. 7, the content 300 is changed to the right according to the arrangement of the content 300 on the left side.

The feature amount used for setting the arrangement (and orientation) of the content 300 is not limited to the orientation of the person, but may be the pose of the person or the position of the person image 200 in the captured image 100. For example, when the position of the person image 200 in the captured image 100 is to the left, the content 300 is arranged on the right side and the person image 200 is arranged on the left side.

The photography apparatus 1 prints the composite image 400 with the printer 15 (S12), and discharges the composite image 400 as a printed matter to the printed matter outlet 17. The composite image 400 can be transmitted to a server (not shown) so that it can be downloaded from the user's terminal, or can be directly transmitted from the photography device 1 to the user's terminal.

The above is the case where the content 300 is automatically selected, but when the content 300 is selected by the user himself (S2; the user himself selects the content 300), the photography device 1 first displays the content selection screen as shown in FIG. To display.

On the content selection screen of FIG. 8, a plurality of selectable content 300 candidates (content image candidates) are displayed. The user selects the content 300 to be used for compositing from the plurality of contents 300, and presses the confirmation button 111.

The photography device 1 accepts the user's selection of the content 300 (S3), and when the confirmation button 111 is pressed, the charging screen (not shown) is displayed on the display unit 11 to encourage the user to charge.

Hereinafter, the photographic apparatus 1 photographs the user by the processing of S4 to S7, obtains the feature amount from the person image 200, and sets the display form of the content 300 selected in S3 by the processing of S9 to S12. Then, it is combined with the person image 200, and the combined image 400 is printed. Here, unlike S8 described above, the content 300 is not automatically selected based on the feature amount, and the feature amount is used only for setting the display form of the content 300. On the contrary, the feature amount may be used only for the automatic selection of the content 300 as in S8, and the display form of the content 300 may be set by the user's operation input.

In some cases, the user is photographed as a moving image before the photographing (main shooting) in S5, and this is displayed as a preview screen on the display unit 11. In this case, the person image 200 may be extracted from the image of each frame of the moving image, and the composite image 400 obtained by synthesizing the content 300 with the person image 200 may be displayed on the display unit 11 as a preview screen.

In the above composition, after obtaining the feature amount from the person image 200, the same processing as in S8 to S11 is performed when the content 300 is automatically selected, and S9 to S11 when the content 300 is selected by the user. It can be performed by the same process as. For example, when the user changes the orientation, the arrangement of the content 300 on the preview screen also changes accordingly.

Further, before printing the composite image 400 in S12, a confirmation screen (not shown) for confirming the composite image 400 is displayed on the display unit 11, and the user decides whether to print the composite image 400 on the screen. It may be possible to select. The user confirms the composite image 400 on the screen, and if that is all right, selects print and proceeds to S12.

If the user is dissatisfied with the composite image 400, he / she can select to redo the composite without printing the composite image 400, or to print the image by performing graffiti editing on the background area of the portrait image 200. It can be carried out.

In the former case, for example, the person image 200 is maintained as it is, a new content 300 is selected on the content selection screen of FIG. 8, the display form of the content 300 is set, and the composite image 400 is created (S9 to S11) again. To do.

It is also possible to take a picture again without maintaining the person image 200. For example, when the content 300 is automatically selected, the composite image 400 is created again by the same procedure from the user's picture (S5). When the user has selected the content 300, the content 300 can be selected on the content selection screen of FIG. 8, and then the composite image 400 can be created again by the same procedure from the user's shooting (S5).

In the present embodiment described above, the content 300 provided as the composite image 400 can be made variable according to the characteristics of the person image 200 extracted from the captured image 100, and the content 300 can be made different depending on the user's person image 200. Therefore, the highly rare content 300 can be provided to each user, and the commercial value of the content 300 can be increased.

That is, in the present embodiment, the content 300 automatically selected by the photography device 1 is variable depending on the characteristics of the person image 200, and the color and arrangement of the content 300 are variable depending on the characteristics of the person image 200 as the display form of the content 300. Therefore, the content 300 having more commercial value can be obtained.

Further, in the present embodiment, the content 300 corresponding to the characteristics of the user's person image 200 in the photographed image 100 photographed by the photographing device 1 can be provided to the user, and the composite image 400 is created by chroma key composition. It can be suitably applied as a service of a photography apparatus 1 provided with a background plate 18 or the like having a predetermined color.

Further, by using AI for selecting the content 300, it is possible to select the content 300 that is close to human judgment.

However, the present invention is not limited to the above embodiment. For example, in the present embodiment, the content 300 is an image of a car, but the content 300 is not limited to this, and may be, for example, a person or an animal, a building or a natural landscape, or a background originally designed. The part for setting the color of the content 300 in S9 can also be freely determined according to the individual content 300.

Further, the feature amount used for selecting the content 300 and setting the display form of the content 300 is not limited to the above, and various types can be considered and are not particularly limited.

Further, in the present embodiment, the color of the content 300 is changed to the color of the clothes of the person image 200 in S9. For example, a pattern extracted from the clothing area b of the person image 200 is used as a representative pattern, and this is one of the contents 300. You may change the color so that it is pasted on a part or all.

Further, in the present embodiment, the color and arrangement (and orientation) of the content 300 are set as the display form of the content 300, but the display form to be set may be related to how the content 300 is displayed in the composite image 400. For example, the number and size of the contents 300 can be changed.

In addition, in S8, a plurality of contents 300 are selected by AI, the plurality of contents 300 are displayed as recommended contents 300 on the content selection screen similar to FIG. 8, and the contents 300 used for compositing are selected by the user. It may be possible to select. The photographing apparatus 1 accepts this selection and performs the processing after S9 using the selected content image. As a result, the user can select the favorite content 300 from the plurality of content 300 selected by AI.

Further, in the present embodiment, the content 300 is provided as a composite image 400 synthesized with the person image 200, but the data of the content 300 is transmitted to the user terminal as it is without synthesizing the content 300 with the person image 200. It is also possible to provide or print and provide only the content 300. Also in this case, the display form of the automatically selected content 300 and the content 300 can be made variable according to the feature amount in the same manner as described above. Further, the user may be able to select whether to provide the content 300 to the user without synthesizing with the person image 200 or to provide the content 300 as a composite image 400 by combining with the person image 200. In this case, for example, in the former case, the content 300 such as S8 is automatically selected, and in the latter case, the content 300 such as S8 is not automatically selected, and the content 300 selected by the user is referred to as the person image 200. It is also possible to synthesize.

Further, in the present embodiment, the user's person image 200 is extracted as the subject image from the captured image 100 captured by the photographing unit 13, but the subject image may be extracted from the image input by the user. .. For example, the user connects his / her terminal to the image processing device 10 in a communicable manner, transmits the image stored in the terminal to the image processing device 10, and inputs the image.

In this case, the photographing unit 13 is not essential, and the image processing device 10 of the present embodiment can be applied to other than the photography device 1. When an image is input by the user, the image of the subject extracted from the image is not limited to the image of a person. For example, it may be an image of an animal, an image of an artificial object such as a car or a house, or the like.

For example, FIG. 9 is an example of an input image 100a input by a user, and the image processing device 10 extracts an image 200a of a main subject (fox in the example of the figure) located at the center of the input image 100a, and the subject is the subject. Known image recognition using AI to determine what the subject is (for example, a person or an animal, an artificial object such as a car or a house, etc.) based on one or more values obtained from the image 200a of the above. Identify by technology.

Then, a feature amount corresponding to the specific result of the subject by AI is obtained, and the content 300 is selected by the same process as S8 described above based on the feature amount, or the display form of the content 300 is performed by the same process as S9 and 10. To set. In the former case, for example, the attribute of the subject is specified by AI using the feature amount, and the content 300 corresponding to the attribute is selected. For example, when the subject is an animal, the posture and color are obtained as feature quantities, and the type of animal (such as "fox") is specified as the attribute of the subject. When the subject is an artificial object, for example, the size and color are obtained as the feature amount, and the type of the artificial object (for example, "car" or "house") is specified as the attribute of the subject. The features and attributes when the subject is a person are the same as described above.

In the above example, AI is used not only when selecting the content 300 but also when obtaining the feature amount, and it is also possible to obtain the feature amount according to the specific result (what the subject is) of the subject by the AI. , Content selection close to human judgment and setting of the display form of the content 300 become possible.

Although preferred embodiments of the present invention have been described above with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person skilled in the art can come up with various modifications or modifications within the scope of the technical idea disclosed in the present application, and these also naturally belong to the technical scope of the present invention. Understood.

1: Photographing device 10: Image processing device 11: Display unit 12: Billing unit 13: Photographing unit 14: Lighting 15: Printer 17: Printed matter outlet 18: Background plate 100: Photographed image 100a: Input image 200: Person image 200a; Subject image 300: Content image 400: Composite image

Claims (11)

An image of a subject is extracted from an image taken by the user as a subject or an image input by the user, and a content image is selected from a plurality of content image candidates based on a feature amount obtained from the image of the subject. An image processing device characterized by. An image characterized in that an image of a subject is extracted from an image taken by the user as a subject or an image input by the user, and a display form of a content image is set based on a feature amount obtained from the image of the subject. Processing equipment. The image processing apparatus according to claim 1, wherein the display form of the content image is set based on the feature amount. The image processing apparatus according to any one of claims 1 to 3, wherein a composite image is created by synthesizing the content image and the image of the subject. Claims 1 to 4, wherein the AI is used at least when selecting the content image from a plurality of content image candidates based on the feature amount or when obtaining the feature amount. The image processing apparatus according to any one. Any of claims 1 to 5, wherein a plurality of the content images are selected from a plurality of content image candidates based on the feature amount using AI, and the user selects the content image from the plurality of content image candidates. The image processing device described in the above. The shooting unit that shoots the user and
Image processing equipment and
Have,
The image processing device is
Photographing characterized in that a user's person image is extracted from an image taken by the user in the shooting unit, and a content image is selected from a plurality of content image candidates based on a feature amount obtained from the person image. apparatus. The shooting unit that shoots the user and
Image processing equipment and
Have,
The image processing device is
A photography apparatus characterized in that a person image of a user is extracted from an image taken by the user in the photographing unit, and a display form of a content image is set based on a feature amount obtained from the person image. The photography apparatus according to claim 7, wherein the image processing apparatus creates a composite image by chromakey-synthesizing the content image and the person image. Computer,
An image in which an image of a subject is extracted from an image taken by the user as a subject or an image input by the user, and a content image is selected from a plurality of content image candidates based on a feature amount obtained from the image of the subject. A program for functioning as a processing device. Computer,
The image of the subject is extracted from the image taken by the user as the subject or the image input by the user, and is made to function as an image processing device that sets the display form of the content image based on the feature amount obtained from the image of the subject. Program for.