JP2020098409A - Image processing apparatus, image processing method, and image processing program - Google Patents

Abstract

To provide an image processing apparatus, an image processing method, and an image processing program capable of easily replacing an article in an image.SOLUTION: A server device 20 includes: a virtual object acquisition unit 232 that acquires a virtual object of an article based on article information included in user information; an image acquisition unit 231 that acquires an original image captured by an imaging device; a target object detection unit 233 that detects a target object from the original image based on the article; and an image processing unit 234 that generates a processed image obtained by replacing the target object in the original image with the virtual object.SELECTED DRAWING: Figure 2

Description

The present invention relates to an image processing device, an image processing method, and an image processing program.

Conventionally, there is an AR (augmented reality) technique for superimposing virtual information by processing an image or video in a physical space with a computer in order to present information intuitively to a user. For example, in recent years, there is an application that virtually installs a product such as furniture in an image captured by a user. A user who uses such an application can easily know the atmosphere and size feeling when a product before purchase is installed in an actual room.

However, in the above-mentioned application, when it is desired to arrange an AR image of new furniture in a place where existing furniture is installed, it takes time and effort to move the existing furniture to another place. On the other hand, Patent Document 1 discloses a method of using a DR (impairment reality) technique for erasing "things" from an original image in a physical space. Specifically, in Patent Document 1, furniture is removed from the original image, and a new furniture image is arranged on the 3D model based on the removed image. Thereby, the 3D model in which the existing furniture is virtually replaced with the new furniture is generated.

JP, 2005-125774, A

However, in the technique disclosed in Patent Document 1, the degree of the impaired reality of the original image in the physical space is adjusted based on the user's setting. Therefore, when it is desired to remove only the desired furniture from the original image, the user needs to set such a degree of impairment reality. For this reason, the setting by the user is required for each original image, and the user's labor is increased.

An object of the present invention is to provide an image processing device, an image processing method, and an image processing program that can easily replace an article in an image.

An image processing apparatus according to one aspect of the present invention acquires a virtual object acquisition unit that acquires a virtual object of an article and an original image imaged by an imaging apparatus, based on the information of the article included in the user information. An image acquisition unit, a target object detection unit that detects a target object from the original image based on the article, an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object, It is characterized by including.

An image processing apparatus according to another aspect of the present invention includes an image acquisition unit that acquires an original image captured by an imaging apparatus used by a user, and a target object that detects a target object arranged in a predetermined region from the original image. It is characterized by comprising a detection unit, a virtual object acquisition unit that acquires a virtual object of an article, and an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object.

According to the image processing device of one aspect of the present invention, it is possible to automatically replace an appropriate real object in an original image with a virtual object of an article desired by a user. As a result, the user can easily replace the article in the image without the trouble of moving the article in the physical space or setting the degree of the impaired reality.
According to the image processing device of one aspect of the present invention, it is possible to automatically replace an article arranged in a predetermined area in an original image with a virtual object of an arbitrary article. As a result, the user can easily replace the item in the image, as described above.

The block diagram which shows the schematic structure of the image processing system of 1st Embodiment. The block diagram which shows the schematic structure of the server apparatus of 1st Embodiment. The figure which shows an example of the cart information page of product selection content. The figure which shows an example of the product information page of product selection content. 3 is a flowchart showing an example of an image processing method of the server device according to the first embodiment. The figure which shows typically the flow of the process of the image processed in 1st Embodiment. The flowchart which shows an example of the image processing process of the server apparatus of 1st Embodiment. The figure which shows an example of the physical space and processed image of 1st Embodiment. The flowchart which shows an example of the image processing method of the server apparatus of 2nd Embodiment.

[First Embodiment]
A first embodiment of the present invention will be described with reference to the drawings.
[Image processing system 1]
FIG. 1 is a block diagram showing a schematic configuration of an image processing system 1 of this embodiment.
As shown in FIG. 1, the image processing system 1 of the present embodiment includes a user terminal 10 and a server device 20 that functions as an image processing device of the present invention. The user terminal 10 and the server device 20 are communicably connected via a network (for example, the Internet).

The image processing system 1 according to the present embodiment is, for example, a part of an electronic commerce system, and the server device 20 has registered therein information on products to be sold at online shops. In this image processing system 1, the server device 20 performs image processing on DR (impaired reality) and AR (augmented reality) on an image (original image) obtained by capturing the real space by the user terminal 10 to obtain a real space. A processed image is generated by virtually replacing the product in the item with a desired product. By looking at the processed image displayed on the user terminal 10, the user can intuitively know the mood and size feeling when the article in the physical space is replaced with the desired article.
Hereinafter, a specific configuration of the image processing system 1 will be described in detail.

[User terminal]
The user terminal 10 is, for example, a computer such as a smartphone or a tablet terminal, and has the basic hardware of this type of computer.
Specifically, as shown in FIG. 1, the user terminal 10 includes a terminal communication unit 11, an input operation unit 12, a terminal storage unit 13, a terminal control unit 14, a display 15, and a camera 16.

The terminal communication unit 11 is connected to a network via, for example, a LAN and communicates with other devices on the network.
The input operation unit 12 outputs an operation signal by a user operation to the terminal control unit 14. As the input operation unit 12, for example, a touch panel provided integrally with the display 15 can be exemplified.

The terminal storage unit 13 is composed of a data recording device such as a memory or a hard disk. The terminal storage unit 13 stores various programs for controlling the user terminal 10. Further, the terminal storage unit 13 temporarily stores an original image to be processed by the server device 20 described later.

The terminal control unit 14 includes an arithmetic circuit such as a CPU (Central Processing Unit) and a storage circuit such as a RAM (Random Access Memory), and controls each unit of the user terminal 10. The terminal control unit 14 develops a program (software) such as a predetermined application from the programs stored in the terminal storage unit 13 or the like in the RAM, and performs various processes in cooperation with the program loaded in the RAM. Execute. Accordingly, the terminal control unit 14 can communicate with the server device 20 via the network, and can use various services provided by the server device 20 and browse various data, for example.
In addition, the terminal control unit 14 transmits an image (original image) captured by, for example, the camera 16 to the server device 20 by the user operating the input operation unit 12.
Further, the terminal control unit 14 causes the display 15 to display (output) an image (processed image) subjected to image processing by the server device 20.

The camera 16 corresponds to the image pickup apparatus of the present invention, and is composed of a lens, an image sensor and the like. The image captured by the camera 16 is temporarily stored in the terminal storage unit 13 and transmitted to the server device 20 by the terminal control unit 14.

[Configuration of server device 20]
FIG. 2 is a block diagram showing a schematic configuration of the server device 20.
As shown in FIG. 2, the server device 20 is composed of a computer, and includes a communication unit 21, a storage unit 22, and a control unit 23.
The communication unit 21 is connected to the network, is connected to the network via, for example, a LAN, and communicates with other devices on the network.

The storage unit 22 is configured by a data recording device such as a memory or a hard disk.
The storage unit 22 stores various programs and various data for performing image processing described later. The storage unit 22 also includes a user database 221, a product database 222, a virtual object database 223, and an image processing database 224.

The product database 222 stores product information about products sold in online shops. The product information includes, for example, a product ID, a product name, category information and the like.
The user database 221 stores user information about users who use the image processing system 1. User information is managed for each user, and this user information includes, for example, a user ID, a user name, a user attribute, history information, recommendation information, and the like. The history information includes browsing history, bookmark information, cart information, purchase history, and the like.
The virtual object database 223 stores virtual objects associated with each product (for example, product ID) registered in the product database 222. The virtual object is, for example, 3D image data that represents the structure of a product in three-dimensional coordinates.
The image processing database 224 stores image processing information for performing image processing. The image processing information stores, for example, learning data for performing image recognition.

In addition, in the present embodiment, the storage unit 22 of the server device 20 exemplifies a configuration including the product database 222, the user database 221, the virtual object database 223, and the image processing database 224. It may be stored in the device.

The control unit 23 is configured by an arithmetic circuit such as a CPU and a storage circuit such as a RAM. The control unit 23 expands a program (software) stored in the storage unit 22 or the like into the RAM and cooperates with the program expanded into the RAM. , Executes various processes. Then, the control unit 23 reads the image processing program stored in the storage unit 22 and executes the image processing program, so that as shown in FIG. 2, the image acquisition unit 231, the virtual object acquisition unit 232, the target object detection unit 233, and the image. It functions as the processing unit 234 and the like. Each function will be described in detail later.

[Image processing method]
Next, each processing executed in the image processing system 1 will be described.
(Pre-processing)
In the present embodiment, the server device 20 performs a process of storing product information (for example, product ID) in the user database 221 before performing image processing.
Specifically, the server device 20 transmits the product selection content to the user terminal 10 based on the product selection content request received from the user terminal 10. As a result, the product selection content is displayed on the display 15 of the user terminal 10.
The product selection content provided by the server device 20 to the user terminal 10 is, for example, part of a service that provides online shopping, and may be provided through a general-purpose application such as a browser. It may be provided via an application for providing a service.

In the user terminal 10, the user performs an operation of selecting a product provided in the online shop while viewing the product selection content page displayed on the display 15. Here, the operation of “selecting a product” is, for example, an operation of browsing detailed information of a product, registering a product in a bookmark, or adding a product to a cart.

The server device 20 uses the product information (for example, product ID) of the product selected by the user as history information (for example, browsing history, bookmark information or cart information) of the user database 221 based on the operation information received from the user terminal 10. Remember.
Hereinafter, the products stored in the history information of the user database 221 are collectively referred to as user-selected products.

After at least one user-selected product is stored in the user database 221, the user taps the “place” button from any page of product selection content (see, eg, FIG. 3 or 4). The server device 20 executes the image processing program based on the operation information received from the user terminal 10, and performs the image processing described below.

(Image processing)
FIG. 5 is a flowchart showing image processing in the server device 20.
First, the virtual object acquisition unit 232 refers to the user-selected product stored in the user database 221, and acquires the virtual object associated with the product ID of the user-selected product from the virtual object database 223 (step S1; Virtual object acquisition step).

Here, the virtual object acquisition unit 232 may select, based on the operation information received from the user terminal 10, which storage area of the browsing history, the bookmark information, or the cart information in the user database 221 is referred to. ..
For example, when the operation information including that the “place” button in the page indicating the cart information is tapped is received, the virtual object acquisition unit 232 refers to the cart information and corresponds to the user-selected product in the cart information. You may acquire the virtual object. In this case, a plurality of virtual objects may be acquired.
In addition, when the operation information including that the “place” button in the page showing the product information of the specific product is tapped is received, the virtual object acquisition unit 232 refers to the browsing history, and the virtual object acquiring unit 232 refers to the browsing history to find the most A virtual object corresponding to the newly stored user-selected product may be acquired.
Hereinafter, for simplification of description, a case where one virtual object is acquired will be described.

The image acquisition unit 231 also transmits a camera activation request to the user terminal 10. The user terminal 10 that has received the request starts the operation of the camera 16. Then, when the user points the camera 16 at a desired space (for example, a part of the user's room), the camera 16 captures an original image with the real space as a subject. Then, the terminal control unit 14 transmits the original image captured by the camera 16 to the server device 20. Thereby, the image acquisition unit 231 acquires the original image transmitted from the user terminal 10 (step S2; image acquisition step).

Next, the target object detection unit 233 detects a real object included in the acquired original image (step S3). Here, the real object is image information of an article (for example, furniture) existing in the real space and can be detected by an image recognition technique. When there are a plurality of articles in the physical space, the target object detection unit 233 may detect the plurality of physical objects included in the original image.
For example, an example of the original image is shown in the upper part of FIG. In this example, the table and the chair are respectively detected as real objects.

Next, the target object detection unit 233 calculates the characteristic amount of the virtual object acquired in step S1 and the characteristic amount of the real object detected in step S3. Then, the similarity of the feature amount between the virtual object and the real object is calculated (step S4).
The feature amount of the virtual object is not calculated by the target object detection unit 233 and may be included in the information of the virtual object acquired in step S1. Further, the characteristic amount of the virtual object corresponds to the characteristic amount of the user-selected product corresponding to the virtual object.

Next, the target object detection unit 233 determines whether or not there is a real object whose feature amount similarity with the virtual object is equal to or more than a threshold value (step S5). In the case of No in step S5, the threshold value is decreased by a predetermined value set in advance (step S6), and the process returns to step S5.

On the other hand, in the case of Yes in step S5, the target object detection unit 233 specifies, as the target object, a physical object whose feature amount similarity with the virtual object is equal to or more than a threshold value (step S7). The above steps S3 to S7 correspond to the target object detecting step of the present invention.
According to the target object detection step (steps S3 to S7) of the present embodiment, among the physical objects existing in the original image, the physical object having the highest degree of similarity to the product selected by the user is detected as the target object.
For example, in the example of the original image shown in the upper part of FIG. 6, when the virtual object corresponds to the product of the rectangular table, the “real object of the round table” is specified as the target object instead of the “real object of the chair”. To be done.
Further, if the virtual object corresponds to the product of the rectangular table and only the “real object of the chair” is detected from the original image, the threshold value is repeatedly decreased to finally The "real chair object" is identified as the target object.

Next, the image processing unit 234 processes the original image to generate a processed image (step S8; image processing step).
FIG. 7 is a flowchart showing the image processing process of step S8.
As illustrated in FIG. 7, the image processing unit 234 first calculates various conditions regarding the target object (step S81). Here, the conditions include, for example, the distance from the user terminal 10 to the target object, the arrangement direction of the target object, the background information of the target object, the ground plane information of the target object, and the like.

The image processing unit 234 may calculate the placement direction of the target object based on the category of the product selected by the user. Specifically, the category of the user-selected product is referenced from the product database 222, and the front information associated with the category is acquired from the image processing database 224. Then, the arrangement direction of the target object may be calculated by detecting the front surface of the target object based on the acquired front surface information. For example, the front information corresponding to the category of the table is, for example, information indicating the central portion between any two adjacent leg portions of the four leg portions. Further, the front information corresponding to the chair category is information indicating a portion located on the opposite side in the horizontal direction with respect to the backrest portion.

Next, the image processing unit 234 determines the arrangement direction of the virtual object based on the arrangement direction of the target object or the real object arranged around the target object (step S82).
For example, when the front information can be acquired based on the target object, the rotation position of the virtual object about the vertical center axis is set such that the virtual object is arranged in the same direction as the target object.
Further, when the front information based on the target object cannot be acquired, and when the front information can be acquired based on the real object arranged around the target object, the rotation position of the virtual object is determined according to the front information. Set.
For example, in the example of the original image shown in the upper part of FIG. 6, the front information is not acquired based on the target object (round table), and based on the front information of the real objects (chairs) arranged around it, The orientation of the virtual object (square table) is set.

Next, the image processing unit 234 generates a mask image based on the background information of the target object, and superimposes the mask image on the display area (target area) of the target object in the original image (step S83). As a result, the actual impairment process of removing the target object from the original image is performed, and an impairment image is generated. A publicly known impairment reality (DR) technique can be used for the process of step S83.
An example of a depleted image is shown in the middle part of FIG. In this example, the image of the table detected as the target object is removed from the original image.

Next, the image processing unit 234 superimposes the virtual object on the impaired image (step S84).
Here, the arrangement direction of the virtual object in the impairment image is the arrangement direction set in step S82. Further, the vertical position of the virtual object in the impairment image is set based on the ground plane on which the real object was grounded. Also, the horizontal center position of the virtual object in the impairment image is set based on the horizontal center position of the physical object in the original image.
A publicly known augmented reality (AR) technique can be used for the process of step S84.

As a result, a processed image in which the target object in the original image is replaced with the virtual object is generated. For example, an example of a processed image is shown in the lower part of FIG. In this example, a rectangular table, which is a virtual object, is displayed in the room instead of the circular table that exists in the physical space.

When a plurality of target objects are specified, the processing of steps S81 to S84 may be performed for each target object. In this case, each of the plurality of target objects in the original image is replaced with the virtual object.
Alternatively, when a plurality of target objects are specified, some target objects may be selected. This selection process may be a process based on a user operation, or may be a process of the image processing unit 234 that selects a target object located in a predetermined area such as the center of the screen.

Returning to FIG. 5, the image processing unit 234 transmits the generated processed image to the user terminal 10 and causes the display 15 of the user terminal 10 to display the processed image (step S9).
For example, FIG. 8 shows an example of the physical space RS and the processed image PI displayed on the display 15 of the user terminal 10 when the physical space RS is imaged. As in the example shown in FIG. 8, the user can view the processed image in which the article in the real space is virtually replaced by the product selected by the user by imaging the real space. As a result, the user can intuitively know the atmosphere and the size feeling when the article in the physical space is replaced with the user-selected product.

When the camera 16 is capturing an image in the moving image mode, the image acquisition unit 231 acquires an original image as a frame image at a predetermined cycle. That is, after step S9, when the predetermined stop condition is not satisfied (step S10; No), the image acquisition unit 231 acquires the next original image (step S11), and returns to step S8. Here, the target object detection unit 233 performs a process of tracking the target object specified in the first flow, and the target object tracked by the target object detection unit 233 is used after the second flow.
When a predetermined stop condition is satisfied (step S10; Yes), such as the user performing a stop operation, the flow ends.

[Operation and effect of the first embodiment]
The server device 20 of the present embodiment functions as the image acquisition unit 231, the virtual object acquisition unit 232, the target object detection unit 233, and the image processing unit 234 by the control unit 23 reading and executing the image processing program. Then, the virtual object acquisition unit 232 acquires the virtual object of the user-selected product based on the information of the user-selected product, and the image acquisition unit 231 acquires the original image captured by the camera 16. The target object detection unit 233 detects the target object from the original image based on the information on the product selected by the user. The image processing unit 234 generates a processed image in which the target object in the original image is replaced with a virtual object.
In the present embodiment as described above, the server device 20 can automatically replace an appropriate real object in the original image with the virtual object of the user-selected product by acquiring the original image. As a result, the user can easily replace the article in the original image without the trouble of moving the article in the physical space or setting the degree of the impaired reality.

In the present embodiment, the target object detection unit 233 detects, as a target object, a real object that has a similarity to the user-selected product that is equal to or greater than a threshold value among the real objects that are present in the original image.
With such a configuration, even when the original image includes a plurality of types of physical objects, the physical object relating to the product selected by the user can be easily detected as the target object.

In the present embodiment, the target object detection unit 233 reduces the threshold value when the similarity between the real object and the virtual object is smaller than the threshold value.
With such a configuration, even if the real object in the original image is significantly different from the user-selected product, it can be detected as the target object.

In the present embodiment, the image processing unit 234 sets the arrangement direction of the virtual object in the processed image based on the arrangement direction of the target object or the real object arranged around the target object.
With such a configuration, the virtual object can be arranged with a natural appearance without requiring the user's trouble.

[Second Embodiment]
Next, a second embodiment will be described.
In the above-described first embodiment, the target object detection unit 233 detects the target object relating to the user-selected product from the original image. That is, in the first embodiment, the user needs to select the product in advance. On the other hand, in the second embodiment, the target object detection unit 233 detects a real object arranged in a predetermined area of the original image as a target object, and the user does not need to select a product in advance. It is different from the first embodiment in points.

The image processing method in the server device 20 of the second embodiment will be described below. The server device 20 executes the image processing program based on the operation information received from the user terminal 10, and performs the image processing described below.

FIG. 9 is a flowchart showing image processing in the server device 20 of the second embodiment. In FIG. 9, the same processes as those in the first embodiment are designated by the same reference numerals.
First, the image acquisition unit 231 transmits a camera activation request to the user terminal 10. The user terminal 10 that has received the request transmits the original image captured by the camera 16 to the server device 20 as in the first embodiment. Thereby, the image acquisition unit 231 acquires the original image transmitted from the user terminal 10 (step S2).

Next, the target object detection unit 233 detects, as a target object, a real object arranged in a predetermined area in the acquired original image (step S21). Here, the predetermined area can be arbitrarily set in the original image, and may be, for example, a central area in the original image or an area in which the camera 16 is in focus in the original image.

Next, the virtual object acquisition unit 232 acquires the virtual object of the article related to the target object (step S22).
For example, the virtual object acquisition unit 232 analyzes the target object and estimates which product among the products registered in the product database 222 the target object corresponds to. Then, based on the product (estimated product) estimated to correspond to the target object, a virtual object of the product belonging to the same category as the estimated product may be acquired.
Alternatively, the virtual object acquisition unit 232 may calculate the characteristic amount of the target object and acquire the virtual object whose similarity between the target objects is equal to or more than a threshold value.
Note that for simplification of description, it is assumed that any one virtual object is acquired here, and the subsequent processing is the same as that in the first embodiment.

According to the above processing, the user can view the processed image in which the article arranged in the predetermined area of the real space is virtually replaced with the article related to the article by capturing the image of the real space.

[Operation and effect of the second embodiment]
In the second embodiment, the target object detection unit 233 detects a target object arranged in a predetermined area from the original image.
In the second embodiment as described above, the server device 20 automatically replaces the article placed in the predetermined area in the original image with a virtual object of an arbitrary article by acquiring the original image. As a result, the user can easily replace the article in the image without the trouble of moving the article in the physical space or setting the degree of the impaired reality.

Further, in the second embodiment, the virtual object acquisition unit 232 acquires the virtual object of the article based on the target object, so that the user does not need to select the product in advance. Therefore, when there is an item to be renewed but the product to be purchased has not been decided, the server device 20 can receive a proposal of a product that can be a purchase candidate.

[Modification]
The present invention is not limited to each of the above-described embodiments, but also includes the following modifications within a range in which the object of the present invention can be achieved.

(Modification 1)
In each of the above-described embodiments, the case where the virtual object acquisition unit 232 acquires one virtual object has been described for simplification of description, but the present invention is not limited to this. That is, the above description can be applied even when a plurality of virtual objects are acquired.

For example, in the first embodiment, the virtual object acquisition unit 232 associates the real object with a plurality of virtual objects and associates the real object with a plurality of virtual objects when a certain real object has a similarity degree equal to or more than a threshold value with the plurality of virtual objects. A real object is detected as a target object. Then, the target object may be replaced with any virtual object associated with the target object, and the virtual object to be arranged may be switched at predetermined time intervals.
In addition, when there are a plurality of target objects associated with a plurality of virtual objects, a combination of virtual objects to be arranged may be created and the combination may be switched to a different combination every predetermined time.

Further, in the second embodiment, the number of virtual objects acquired by the virtual object acquisition unit 232 may be set in advance to an arbitrary number. Then, when a plurality of virtual objects are acquired, the virtual object that replaces the target object may be switched at regular intervals.
Such a modified example is particularly useful when the image processing system 1 constitutes a part of an electronic commerce system. For example, since the article in the original image can be sequentially replaced with the plurality of virtual objects without requiring the user's trouble, the user can easily compare and consider the plurality of products.

(Modification 2)
In the first embodiment, the real object whose similarity to the virtual object is equal to or higher than the threshold is detected as the target object, but the present invention is not limited to this.
For example, the target object detection unit 233 may analyze the acquired real object and determine the category to which the real object corresponds. Then, the target object detection unit 233 may detect a real object belonging to the same category as the virtual object as the target object.
In addition, the target object detection unit 233 can detect the target object using any other method such as machine learning.

(Modification 3)
In the first embodiment, the virtual object acquisition unit 232 acquires the virtual object of the user-selected product based on the product (user-selected product) information stored in the history information of the user database 221. The invention is not limited to this. For example, the virtual object of the product may be acquired based on the product information stored in the recommendation information of the user database 221.

(Modification 4)
In the second embodiment, in step S22, the virtual object acquisition unit 232 acquires the virtual object based on the target object, but the virtual object may be acquired based on the product information included in the user information.
Here, the product information included in the user information is, for example, product information registered in history information (browsing history, bookmark information, cart information, purchase history, etc.) or recommendation information.
According to such a modified example, similarly to the second embodiment, it is possible to receive a proposal of a product that can be a purchase candidate from the server device 20. Further, in such a modification, it is possible to receive proposals for articles in various categories regardless of the category of the target object.

(Modification 5)
In each of the above embodiments, the server device 20 exemplifies the configuration for performing the above-mentioned image processing, but the user terminal 10 may perform the above-mentioned image processing. That is, in the user terminal 10 in which the image processing program of the present invention is installed, the terminal control unit 14 reads out and executes the image processing program, so that the image acquisition unit, the virtual object acquisition unit, and the target object detection unit of the present invention. And may function as an image processing unit.
In addition, the user terminal 10 may perform a part of the processing in the server device 20. For example, the server device 20 may transmit the mask image and the virtual object to the user terminal 10, and the user terminal 10 may superimpose the received mask image and the virtual object on the original image. That is, the image processing apparatus of the present invention may be configured as a system including a plurality of communicable computers.

(Modification 6)
The article of the present invention is not limited to furniture. That is, the present invention can be applied to various articles, such as clothing, accessories, home appliances, and building equipment, which are expected to be exchanged or considered new.

(Modification 7)
In each of the above embodiments, the image processing system 1 is described as being a part of the electronic commerce system, but the present invention is not limited to this. That is, the image processing system 1 of the present invention may be a single system or may be combined with another system. Further, the article of the present invention is not limited to the product form, and may be a non-sale item or a transfer item.

In addition, the specific structure and procedure for carrying out the present invention can be appropriately changed to other structures and the like within the range in which the object of the present invention can be achieved.

DESCRIPTION OF SYMBOLS 1... Image processing system, 10... User terminal, 11... Terminal communication part, 12... Input operation part, 13... Terminal storage part, 14... Terminal control part, 15... Display, 16... Camera, 20... Server device, 21... Communication unit, 22... Storage unit, 221... User database, 222... Product database, 223... Virtual object database, 224... Image processing database, 23... Control unit, 231,... Image acquisition unit, 232... Virtual object acquisition unit, 233... Target object detection unit, 234... Image processing unit.

Claims (11)

A virtual object acquisition unit that acquires a virtual object of the article based on the article information included in the user information;
An image acquisition unit that acquires the original image captured by the imaging device,
A target object detection unit that detects a target object from the original image based on the information of the article,
And an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object. The image processing apparatus according to claim 1, wherein
The target object detection unit,
An image processing apparatus, characterized in that, among the physical objects existing in the original image, the physical object whose similarity to the article is equal to or more than a threshold value is detected as the target object. The image processing apparatus according to claim 2, wherein
The image processing apparatus, wherein the target object detection unit reduces the threshold value when the similarity between the physical object and the virtual object is smaller than the threshold value. An image acquisition unit that acquires an original image captured by an imaging device used by a user,
A target object detection unit that detects a target object arranged in a predetermined area from the original image,
A virtual object acquisition unit for acquiring a virtual object of an article,
And an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object. The image processing apparatus according to claim 4, wherein
The image processing apparatus, wherein the virtual object acquisition unit acquires the virtual object of the article based on the target object or based on information of the article included in the user information of the user. .. The image processing device according to any one of claims 1 to 5,
The image processing device, wherein the image processing unit sets an arrangement direction of the virtual object in the processed image based on an arrangement direction of the target object. The image processing apparatus according to any one of claims 1 to 6, wherein:
When the virtual object acquisition unit acquires a plurality of the virtual objects,
The image processing apparatus, wherein the image processing unit switches the virtual object to be replaced with the target object at predetermined time intervals. An image processing method for causing a computer to perform image processing,
A virtual object acquisition step of acquiring a virtual object of the article based on the article information included in the user information;
An image acquisition step of acquiring an original image imaged by the imaging device,
A target object detection step of detecting a target object from the original image based on the information of the article,
An image processing step of generating a processed image in which the target object in the original image is replaced with the virtual object. An image processing method for causing a computer to perform image processing,
An image acquisition step of acquiring an original image imaged by the imaging device used by the user,
A target object detecting step of detecting a target object arranged in a predetermined region from the original image,
A virtual object acquisition step of acquiring a virtual object of the article,
An image processing step of generating a processed image in which the target object in the original image is replaced with the virtual object. An image processing program that is read and executed by a computer,
The computer,
A virtual object acquisition unit that acquires a virtual object of the article based on the article information included in the user information,
An image acquisition unit that acquires the original image captured by the imaging device,
A target object detection unit that detects a target object from the original image based on the information of the article, and
An image processing program that functions as an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object. An image processing program that is read and executed by a computer,
The computer,
An image acquisition unit that acquires the original image imaged by the imaging device used by the user,
A target object detection unit that detects a target object arranged in a predetermined area from the original image,
A virtual object acquisition unit for acquiring a virtual object of an article, and
An image processing program that functions as an image processing unit that generates a processed image in which the target object in the original image is replaced with the virtual object.

Images