JPH09259252A - Picture processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To execute optional deformation to pictures inputted by a customer so as not to be unnatural pictures as the whole in on-line shopping. SOLUTION: Source pictures 500 and 510 are respectively separated into areas 520, 525, 530 and 535 for respective objects projected in the pictures. Then, for the respective corresponding objects, the intermediate picture 540 is generated from the areas 520 and 530 and the intermediate picture 550 is generated from the areas 525 and 535. Then, the generated intermediate pictures 540 and 550 are synthesized so as to make the picture element of higher priority be valid in the case that the picture elements are overlapped at the same position corresponding to the priority supplied beforehand and the final picture 560 is obtained.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method using a computer system connected to a network.

[0002]

BACKGROUND OF THE INVENTION The present invention mainly relates to online shopping in which a computer system connected to a network is used to sell or purchase a product, in which a consumer visits a retail store to view the product or consult with a clerk. The present invention relates to a system for performing a process of confirming a feeling of use of a product by receiving the image by image processing. The document “View Interpolation f” is related to the image processing method used in the present invention.
or Image Synthesis "Computer Graphics Proc. p
p.279-288, (1993) has been reported. This is for the purpose of realizing at a personal computer level without special hardware such as Z-buffer when changing the viewpoint in a virtual reality (VR) walkthrough, etc. This is a paper on a method of creating a mid-division image using a morphing distortion function prepared in advance using a set or a captured image.

In addition, the document "Feature-Based Image Me"
tamorphosis ”Computer Graphics Proc. vol.26, pp.
35-42, (1992) calculates the distortion field by designating the corresponding part between the images with a line segment in order to make a half-divided image from the two images. This is a paper on a morphing method in which a high-quality output image can be obtained although the amount of work is significantly smaller than that of the conventional mesh designation method.

[0004]

In recent years, online shopping has been performed in which information is received and transmitted using the function of the WWW (World Wide Web) server of the Internet, and goods are purchased as if walking in a shopping district. Services are starting up.

[0005] On the other hand, TV shopping and the like, in which products are introduced and sold on TV, are popular, but when introducing interior products such as figurines, not to mention the products themselves, but to see how they are placed indoors. It is often shown as a video.

Although it is considered that services such as online shopping will spread rapidly from now on, the current system is at most about sending an image of a product, and the consumer is about to send an image of the product and information about a commentary article. You must decide whether to buy using.

The present invention relates to an image processing method for performing a matching simulation between a product and a room, which is performed in television shopping, in online shopping and using a room or the like of each consumer.

The conventional method has the following problems.

In the first document, an image at the time of changing the viewpoint is created by morphing, but there is a limit to the images that can be used there, and it is necessary to prepare an image with almost no perspective. That is, here, no consideration is given to a phenomenon in which the positional relationship between objects changes, or the objects appear and disappear depending on the positional relationship. In the literature, CG is used to create an original image, the viewpoint is gradually changed within a range in which the positional relationship does not change so much, and morphing is performed between a large number of images thus created.

When such an image is created by morphing from several images, there are a plurality of objects in the image, so that the picture drawn on a stretchable rubber plate is distorted. The technique cannot make a natural image. Further, unlike the first document, it is not realistic to prepare a large number of photographed images without manually using the CG images and manually perform detailed association with them.

A first object of the present invention is online shopping using a plurality of computer systems connected by a network, which can be prepared by the consumer side without transferring a large amount of a large amount of image data between the computer systems. By sharing some of the image data and the data according to the present invention with each computer system, an image when a product is placed in a home or the like where various things are arranged in a complicated manner is present in each home. The objective is to prevent the occurrence of troubles such as different images after purchasing a product, by making it possible to observe by changing the interactive viewpoint on the computer system.

The second document is a method for simplifying the designation of the distortion field in morphing, but conventional morphing changes smoothly from, for example, a dog's face to a human's face, and intermediates between them. The goal was to create a perfect image, and the problem was how to obtain a high quality half-dog half-human image with a small amount of work. That is, the evaluation is only that the face picture changes accordingly, and even if the wall around the face may be distorted, it was not a serious problem. However, in online shopping, for example, when performing a dress-up simulation in which an image of clothes, which is a commodity, is deformed and combined with a person image showing a consumer, if only an image of a partial area of clothes is combined, The clothing that appears in the image and the clothing of the product may interfere with each other to cause an unnatural portion. Conversely, if the areas other than the clothing are also deformed and combined, the portions other than the clothing may be distorted unnaturally. .

A second object of the present invention is to perform online shopping using a plurality of computer systems connected by a network, and to store some images including the image of the consumer himself and data according to the present invention. By sharing the information with each computer system, the person can observe the image of the result of changing clothes and the like, and prevent the trouble that the clothes do not match after the purchase of the product.

The second document is a method for simplifying the designation of the distortion field in the morphing as described above, and there has been a problem of how to obtain a high quality intermediate image with a small amount of work. That is, the evaluation is only that the picture changes accordingly, and even if the three-dimensional model of the object shown in the picture sometimes collapses, it is not a big problem.

However, in the case of products appearing in television shopping, etc., even if only one product is shown in the image, the distortion of one part adversely affects the other part and the shape is changed. Then it's not convenient. Further, since the shape of the object shown in the image changes due to the distortion, in order to combine the CG and the intermediate image, the position of the CG must be adjusted each time the intermediate image is generated.

A third object of the present invention is online shopping using a plurality of computer systems connected by a network, in which some images and data according to the present invention are shared by each computer system, so It realizes the function of automatically synthesizing the intermediate image of CG and CG, and enables the computer system at home to simulate CG image of the made-to-order product and the scenery of the live-action image, which is different from the image after purchasing the expensive product. It is to prevent trouble from occurring.

[0017]

In order to achieve the above object, the present invention provides an image input device, an image storage device, and
A first device including an image processing device, an image display device, a data communication device, an image storage device, an image processing device, an image display device, and a data communication device. In the method of generating a processed image by processing the image input from the input device by the processing device, the image data input from the input device is transferred from the first device to the second device. A step of transferring, a step of storing the received data together with data that is an image of a product in the storage device of the second device, a step of dividing the image stored in the storage device into a plurality of areas, There is a step of generating a plurality of processed area images obtained by processing each area, giving a priority to each processed area image, and a request of the first device. In this case, the processing area image and the data of the order of priority and the document data for ordering products, a step of returning from the second device to the first device, and each processing area image returned, When pixels of different processed area images overlap at the same position in accordance with the given priority order, a step of generating a combined processed image so that the pixels of the processed area image with higher priority become effective, And a step of displaying the processed image and the order document of the product on the display device of the first device.

As a result, in each home where the first computer system is installed, a walk-through function for interactively observing the product and a change in clothes when changing the clothes in order to evaluate the usability of the product in advance. See images,
You will be able to use the simulation function that allows you to see the image when the product is placed indoors. That is, in online shopping where the user cannot actually go to the store to see or try on the product, it is possible for the consumer to sufficiently consider the product and reduce the trouble that the image does not match after purchase.

According to the image processing method of the present invention, the image data input from the input device is transferred from the first device to the second device, and the received data is stored in the second device together with the data which is the image of the product. For each individual object in the image stored in the storage device and stored in the storage device, the image stored in the storage device is divided into a plurality of regions, and each divided region is individually processed into a plurality of processes. A region image is generated, each processed region image is given a respective priority, and a region corresponding to an object appearing closer to the viewpoint in the image is given a higher priority. In this case, the processed area image, the data of those priorities, and the document data for ordering the products are returned from the second device to the first device, and each returned processed area image is subjected to the given priority order. , Different in the same position When the pixels of the processed area image overlap, by synthesizing so that the pixels of the processed area image with higher priority become effective, for example, when there are a plurality of objects in the room, It is possible to create an image when the image is taken at a position different from that at the time of shooting from the real image data of the indoor image, and to combine the product in the image. That is, by using a live-action image taken by the consumer, it is possible to customize the product to the taste of the consumer and provide an environment in which the product can be evaluated as in the case of shopping at an actual store.

[0020]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail.

First, a first embodiment of the present invention will be described.

The first embodiment uses image processing for generating an intermediate image from two real shot images, and sends a real shot image input to a computer device placed at home to a computer in a center, By transforming the image for each divided area, generating a composite image that combines the product images and returning it, and observing the product from various viewpoints with a home computer device and examining it, it is possible to It is to prevent troubles such as different images.

FIG. 1 shows the configuration of an apparatus for performing the processing of this embodiment.

In FIG. 1, a camera 100 is in a consumer's home and takes a still image or a moving image of a room or a person. Although only one camera 100 is shown in the figure, a plurality of cameras 100 are actually provided as needed. When the image captured by the camera 100 is output as digital data, the digital image data is directly stored in the storage device 120. On the other hand, when the image captured by the camera 100 is analog data, the analog data is converted into analog data.
When 0, the converted digital image data is stored in the storage device 120.

The communication control device 170 is a network connection device for transmitting and receiving image data and the like to and from the storage device of another computer device. Control pad 150
Is an input device of instruction information, and a user at home selects a menu, selects an image represented by digital data stored in the storage device 120 or a converted image, and receives other operator's instructions. The display device 140 is
The digital image stored in the storage device 120, the generated converted image, and other images are displayed. Processor 13
0 uses the data and programs stored in the memory 180 to perform image processing as a player, which will be described later. Data for performing the image processing of the processor 130, intermediate images obtained as a result of the image processing, and the like are also stored in the memory 180 and the storage device 120. Highway bus 16
0 connects the above-mentioned components and transfers data etc. mutually. In addition, the first device 190 including the above components 110 to 180 is installed in a home, and the network 200 such as a leased line or a public line is provided via the communication control device 170.
Connect with.

The storage device 220 stores the digital data received from the network via the communication control device 270 and the image of the product or the digital data by computer graphics which has been loaded in the memory 280 in advance by using an input device such as a scanner. To do. In addition, the display device 240 is
A display for showing an image represented by digital data or a converted image to a human being, and displaying an image to be processed when performing a process of dividing an area for performing image processing described later or when associating images. Provide to the operator.

The mouse 250 and the keyboard 255 are instruction information input devices, and are used by the user to select a menu or
In the work of associating images, an operator's instruction is accepted, and it becomes a computer device and a human information transmission medium.

The processor 230 uses the data and programs stored in the memory 280 to perform image processing for authoring described later. Data for performing the image processing of the processor 230, an intermediate image obtained as a result of the image processing, and the like are also stored in the memory 280 and the storage device 220. The high-speed bus 260 connects the above components and transfers data and the like to each other. Also, the above 220-
The second device 290 including the components of 280 is installed in the product sales center, and the conversion data and the like obtained as a result of the image processing of the processor 230 and the document data for purchasing the product are passed through the communication control device 270. Then, the data is transferred to the storage device 120 of the first device 190.

Next, FIG. 2 shows a data structure for realizing the image processing of this embodiment by the apparatus shown in FIG.

The data 300 is loaded on the memory 180 from the storage device 120, input via the communication control device 170, or created by the processor 130. The data 300 is the image data 310 of the photographed image input from the camera.
And distortion field data 330 representing distortion of each point of the image necessary for converting the image represented by the image data created by the processor 230 of the device 2 into the converted image, the generated converted image data 340, and the conversion The priority data 345 indicating the priority of images, the software program 350 defining the image processing of the player function of the present embodiment, the document data 360 of the product sales contract, and the like are stored.

The data 305 is loaded on the memory 280 from the storage device 220, input via the communication control device 270, or created by the processor 230.
The data 305 includes input image data 310 from the camera stored in the storage device 120 of the first device 190, region data 315 for specifying a region into which an image is divided, and priority data indicating the priority of each region. 320, corresponding point or line segment data 325 for associating images to be described later, and a distortion field representing distortion of each point of the image necessary for converting the image represented by the image data into a converted image. The data 330, a software program 355 that defines the image processing of the authoring function of this embodiment, document data 360 for selling products to customers, and the like are stored.

Image processing performed by the processor 130 executing the software program 350 on the memory 180 and image processing performed by the processor 230 executing the software program 355 on the memory 280 will be described below.

FIG. 3 shows the procedure of image processing of this embodiment.

FIG. 4 is a diagram for explaining the process of making an intermediate image by using two still images taken at almost the same place from two different viewpoints, and the result of the processing in each step of FIG. Is shown visually. The image taken from the viewpoint A is 500, and the viewpoint B moved to the right a little.
The image taken from is 510.

In this image processing, first, the first device 190
For example, an operator takes a picture inside the room using the camera 100 (step 400), and the data of the pictures taken from different viewpoints are digitized (step 40).
5) The digital data is stored in the storage device 120 (step 410).

Next, the digital data representing this image is transferred to the second device 290 connected to the network, and the digital data is stored in the storage device 220 of the second device 290 (415).

The digital data stored in the storage device 2 is the data of the images 500 and 510 in FIG. In step 420, each object included in these images is divided into regions. For example, the original image 500 in FIG. 4 is separated into a television image 525 placed inside the room and an image 520 other than the television. That is, in 525 and 535 of FIG. 4, there is no data in the blank portion other than the television in the image. On the contrary, when the image 520 and the image 525 are superposed, the original image 500 is completely restored, and when the image 530 and the image 535 are superposed, the original image 510 is completely restored.

When there are more objects, the area to be divided also increases in proportion thereto, and the number of separated images also increases accordingly.

Area division is performed by, for example, Japanese Patent Laid-Open No. 03-2185.
As described in Japanese Unexamined Patent Publication No. 81-81 "Image segmentation method", first, an image is differentiated, and a portion having a differential value higher than a predetermined threshold value is extracted as an edge and displayed on the display device 240. The mouse 250 is used to select the edge that constitutes the edge of the area to be divided from the operator.
Etc. Then, it can be realized by selectively extracting edges by inputting from the outside, connecting the selected edges, and enclosing the object in a closed region with the connected edges.

Further, in order to have a mask for distinguishing the divided areas corresponding to the image, a mask value for distinguishing the areas is given to the pixels inside the closed area, and the mask value and the mask type are described. The data is stored in the storage means 220 together with the table-formatted data.

When the division of the area is completed, the keyboard 25
5, the priority input by the operator is accepted for each of the separated areas. That is, when the object shown in 525 of FIG. 4 is closer to the viewpoint (camera) than the object shown in 520, the proximity from the viewpoint is used for the priority evaluation, and the object shown in 525 is 520.
Give it a higher priority than what is shown in. In this way, in step 425, the priority order corresponding to the mask value of each image is stored in the storage device 220.

Next, in step 430, for each of the separated areas, where to move each pixel in the area and how to distort the image are defined.

The method of defining this distortion is described, for example, in "Feature-Based Image Metamorphosi" described in the second document.
As described in “s”, the distortion field is calculated by designating corresponding portions between the images with line segments.

That is, first, for example, two images 525
And the image 535, as shown in FIG.
The line segment PQ is designated above, and the line segment P′Q ′ is designated on the image 535 to associate the images. The designation of the corresponding line segment is accepted for each set of the previously divided regions. In addition, the area where the corresponding line segment is not specified is
The area is not deformed.

Now, when the corresponding line segment is defined in this way, the movement vector d on the image 525 for transforming each area is obtained by the equation 1, and this movement vector is stored in the memory 280. This set of movement vectors represents a distortion field for transforming an image.

[0046]

[Equation 1]

However, P in FIG. 5 is a vector reaching the point P on the image 525, and similarly, Q and X are points Q and
A vector reaching point X, and P ′ and Q ′ are vectors reaching point P ′ and point Q ′, respectively. Further, X ′ is a vector that reaches the destination point X ′ when the point X is moved.

Perpendicular () represents a vertical vector having the same length as the vector in (). In addition, the parameter expressed in lower case represents the scalar quantity.

When a plurality of line segments are defined in the area, di is calculated for each line segment i, and the calculated di is, for example, the longer the line segment, the larger the distance from the line segment. A value obtained by dividing Wide by multiplying the weight Wi to be smaller by the sum of Wi is set as the movement vector D.

The movement vector D obtained by this weighted average can be expressed by equation 2.

[0051]

[Equation 2]

In the second document, W determined by the equation 3
i is used as a weight.

[0053]

(Equation 3)

In this way, in step 435, the strain field data consisting of the movement vector of each pixel in each region is created and stored in the storage device 220. That is, in the case of FIG. 4, since one image is divided into the partial area of the television and the partial area other than the partial area of the television, the images of the set of the image 525 and the image 535 are associated with each other, and The images of the image 530 are associated with each other, strain field data is created for each region, and the data is stored in the storage device 220.

Next, the strain field data 330 created by the second device 290 and the document data 36 relating to the sales contract of the product.
0 is returned to the first device 190 via the network 200. The first device 190 temporarily stores data on the memory 180.

At step 450, the input of the deformation amount of the image is accepted from the control pad 150. The image deformation amount is a value that is similar to the feature of the image 500 and how similar to the feature of the image 510 when an intermediate image between the image 500 and the image 510 is to be created. However, in direct terms, the image 50
When 0 is transformed into the image 510, if a pixel needs to be moved by D pixels, it is a value indicating what percentage of D is the movement amount D ′ required to generate an intermediate image.

The amount of deformation of the image is input in step 450, and for example, when an intermediate image having the same characteristics of both the image 500 and the image 510 is created, in step 455, the intermediate image 550 of the image 525 and the image 535 is first obtained. Produces
Next, from the image 520 and the image 530, the intermediate image 54 between them.
Generate 0. As the pixel value of the intermediate image 550, the pixel value of 525 or 535 may be used as it is, or the pixel values of the image 525 and the image 535 may be mixed in half. The pixel value of the intermediate image 540 is basically the same as that of the intermediate image 550.
20 and the image 530 have a defective portion when the television area in the image is separated, the image 520 having different viewpoints
And the image 530 are used to supplement the pixel in the defective portion.

When the intermediate image 540 is created, the image 52
By using the information of both 0 and 530, it is possible to obtain an image in which the defective portion is filled. However, when the defective portion is not filled by this processing alone, another defect repair method is used as necessary. .

A method of repairing defects is described in, for example, The Transactions of The Institute of Shinji, 86.
/ 1 Vol.1.J69-D No.1 pp.91-97 As described in "One method of restoration processing for lossy image", the pixel value around the loss area is referred to Fill the deficient area from the periphery to the center as if you were to stretch the skin. For a simple texture image or the like, there is also a method of analyzing a frequency component of an image when referring to surrounding pixel values and generating a periodic pattern in a defective portion.

When the intermediate image 550 of the object in the foreground and the intermediate image 540 of the background subjected to the restoration processing are created, next, in step 460, the intermediate images with lower priorities are sequentially ordered according to the priority data of the converted images. Superimpose and synthesize on. In the case of FIG. 4, since the image 550 has a higher priority than the image 540, the image 550 is superimposed on the image 540 and combined. The composite image 560 thus generated and the document relating to the sales contract of the product are displayed on the display device 140 in step 465. When a customer, who is a consumer, wants to buy the product, he / she inputs the identification code, the bank account number, the number of products to be purchased, and the like using the control pad 150 according to the comments in this document. The input sales contract information is temporarily stored in the memory 180, encrypted by the processor 130, and sent to the communication control device 170. On the other hand, the encrypted sales contract information sent is stored in the memory 280 and then decrypted by the processor 230 to restore the sales contract document data. After confirming the contents of this document, the sales center carries out procedures for shipping products and collecting charges, and sends the fact to the first device 190 by mail. In this way, when buying and selling online, there is a problem with confirmation of contract partners and security, but consumers need to output payment contract documents displayed on the display device in hard copy etc. If you fill out the items and send it to the sales center, you can purchase the product regardless of these problems. Furthermore, since the image of the product is attached to this document, it is possible to avoid a mistake in delivery in buying and selling.

The second embodiment of the present invention will be described below.

FIG. 6 is a diagram for explaining a mechanism for realizing a function of interactively observing a product from an arbitrary position in a home computer device.

Now, the consumer uses the camera 100 as four cameras 600, 605, 6 at positions vertically and horizontally shifted.
10 and 615 are arranged and a picture is taken. For example, assume that the camera 605 is to the left of the camera 600, the camera 610 is below the camera 600, and the camera 615 is below the camera 600. Here, an image captured by the camera 600 is 620, an image captured by the camera 605 is 625, an image captured by the camera 610 is 630, and an image captured by the camera 615 is 635.

At this time, a virtual camera 640 is assumed to be inside the quadrangle connecting the four cameras, and an image obtained by the virtual camera 640 is assumed to be
The present invention produces an intermediate image of four images.

The procedure for creating the intermediate image is performed by separating the regions of each image according to the process of FIG. 3 and defining the priority order for each region. Then, for each corresponding region, the corresponding portion between the images is associated with a line segment, and the movement vector D of each pixel is obtained for each region.

In this way, the image 620 is converted into the image 625.
A moving vector D1 for transforming into an image 620, a moving vector D2 for transforming the image 620 into an image 630, and an image 6
A movement vector D3 for transforming 20 into an image 635 is obtained and stored in the storage device 220 and the memory 180.

Next, when the intermediate image is generated, the position of the virtual camera 640 is input by the control pad 150 or the like. At this time, the movement vector D for generating an intermediate image from the image 620 is expressed by Equation 4.

[0068]

(Equation 4)

Similarly, a movement vector for generating an intermediate image is obtained from the images 625, 630 and 635.

When each pixel is moved from the four images by the movement vector, four pixel values correspond to each pixel on the intermediate image. Therefore, when determining the color of the pixel of the intermediate image, the color of the corresponding point in the four images is bilinearly interpolated. That is, S1, S2, S3 in FIG.
It is assumed that four cameras are arranged at the vertices of the quadrangle formed by S4, and the camera 600 is located at the vertices on the S1 side.
5 is the apex on the S2 side, camera 610 is the apex on the S3 side,
The camera 615 is at the apex on the S4 side, and the virtual camera 640
Is assumed to be at a position where four points S1, S2, S3 and S4 are in point contact. Further, S1 to S4 are positive numbers of 0 or more and 1 or less, and S1 + S2 + S3 + S4 = 1.

When the virtual camera 640 is at the position shown in FIG. 6, the color of the pixels of the image 620 is S4 and the image 62
The color of 5 is mixed in the ratio of S3, the color of the image 630 is mixed in the ratio of S2, and the color of the image 635 is mixed in the ratio of S1.

As described above, the device 2 installed in the sales center obtains the movement vector between each image by dividing the area in advance, so that the virtual computer interactively designated by the home computer device 1 can be obtained. According to the camera position, an intermediate image corresponding to the one shot at the position can be generated at high speed, and the consumer can check the usability of the product from various viewpoints when purchasing the product.

The third embodiment of the present invention will be described below.

FIG. 7 is an explanatory diagram showing an example of the flow of processing when a composite image is created by combining an image of a real image taken by the camera 100 and a product artificially created by computer graphics (CG). is there. For simplicity, the object in the image is stationary, and if only the camera 100 moves and the position changes, the image 700 and the image 71 are displayed.
0 is a photographed image corresponding to the image 500 and the image 510 in FIG. Now, distortion field data is generated in accordance with the process of FIG. 3, an intermediate image 720 is created, and the intermediate image 720 is stored in the memory 180.

On the other hand, for the images 700 and 710,
For example, a known shape such as a rectangular parallelepiped is used to associate the contours of the object shown in the image. An image 730 is obtained by fitting a rectangular parallelepiped to the image 700, and an image 740 is obtained by fitting the same rectangular parallelepiped to the image 710.

Then, if the same rectangular parallelepiped is associated with the image 730 and the image 740, and if the three-dimensional shape of the rectangular parallelepiped is known, the camera position at the time of capturing each image can be estimated. Further, if the camera position corresponding to each image is obtained, it can be estimated how the camera (viewpoint) has moved, and the camera position corresponding to the intermediate image between the image 700 and the image 710 can be estimated.

That is, the CG system 750 generates a CG image 760 obtained by projecting a three-dimensional model designed by CG and stores it in the memory 180. Then, the intermediate image and CG
Since it is considered that the viewpoints of the images are the same, a composite image 770 that does not cause a three-dimensional discomfort can be obtained by combining the images in accordance with the priority order, and the display device 140 can display a real scene or the like and a CG product. It is possible to observe a composite image in which Therefore, when ordering a product made to order by online shopping, first create the product with CG, combine it with a live-action image, customize it to the taste of the customer who is the consumer, and check the result. It is possible to prevent troubles such as different images after purchasing a product.

The fourth embodiment of the present invention will be described below.

FIG. 8 is an explanatory diagram when the creation of the intermediate image using the different-view still image in FIG. 4 is expanded to the creation of the intermediate moving image using the different-view moving image.

That is, the image 8 taken by the camera 800
One of the images 10 is the image 500 in FIG. 4, and one of the images 830 taken by the camera 820 is the image 510.
become. At this time, 810 and 830 photographed at the same time t
An image 840 that would be obtained when the virtual camera 850 of the intermediate viewpoint is used to generate the image is generated from the image. Further, when the processing shown in FIG. 3 is performed for the set of images at each time,
A continuous moving image 840 taken by the virtual camera 850 is obtained.

In order to select images at the same time, the camera 8
00 and the camera 820 must be recorded in synchronization, or a time counter or the like must be recorded at the same time as the video.

According to the present embodiment, similar to a still image, it is possible to observe a moving image equivalent to that seen from a position not actually photographed, and even when a moving car or the like is a product. , You can observe the product from various viewpoints regardless of the movement of the car in the video, and you can prevent troubles such as different images after purchasing the product.

The fifth embodiment of the present invention will be described below.

FIG. 9 shows image data prepared when performing a dress-up simulation for deforming and synthesizing an image of clothes, which is a commodity, with a person image showing a consumer in online shopping.

The image 905 is image data of a product prepared in advance in the storage device 220 of the second device 290 at the sales center. The image 905 shows the second device 29 from the beginning.
Data stored in the storage device 220 of 0, read into the memory 280 in step 415 according to the processing of FIG. 3, and area division is performed in step 420, and the value shown in the area division data 915 for each divided area. Is defined.

On the other hand, the image 900 is a person image input from the camera 100, is input when the consumer wants to perform online shopping, and is stored in the storage device 120 of the first device 190. Further, the digital data representing the person image is transferred to the memory 280 of the second device 290 according to the processing of FIG. Then, in step 420, area division is performed on the image 900, and the values shown in the area division data 910 are defined for the divided areas.

FIG. 10 is an explanatory diagram showing the procedure of a process for creating a dressed-up image using the images of the person and the clothes shown in FIG. 9 and their area division data.

First, the image 900 is the area division data 91.
According to the value given to each area of 0, the partial area image 92
0, partial area image 930, and partial area image 940. For the sake of simplicity, the areas having the values of 2, 3, and 4 in the area division data 910 are treated as the same area.

At step 425, the priority order of the divided partial area image 920, partial area image 930, and partial area image 940 is defined. Here, the partial area image 940 is assumed to have the highest priority, and the partial area image 930,
It is assumed that the priority order is lowered in the order of the partial area image 920.

Next, at step 430, for each corresponding region, the corresponding characteristic portions between the image 900 and the image 905 are made to correspond by the line segment used in FIG. That is, in the area division data 910 and 915, the areas of the image 900 and the image 905 in which the value of 0 is defined are associated with each other, and the areas of the image 900 and the image 905 in which the value of 1 is defined are associated with each other, Similarly, the areas in which the value of 2, the value of 3 or the value of 4 are defined are associated with each other. Then, in step 435, distortion field data, which means the amount of movement for each pixel in the region, is generated based on the equations (1) to (3).

Using this strain field data, step 4
After 55, the clothes are changed and the display device 140
To display the result.

That is, with respect to the clothing area whose value of the area division data is 1, the pixels of the clothing area in the image 905 are moved by the distortion field data of the area to generate the deformed partial area image 945. Stored in the memory 180.

In principle, the person image 900 is used as it is for the area of the face or arm of the person whose value of the area division data is 2 or 3 or 4. However, if it is only that, the sleeve part of the clothes and the part around the neck may be missing. Therefore, first, the pixels in the region are moved based on the corresponding strain field data, and the moved pixels are moved to the image 90.
Only the images that come in the area of 0 clothes are 90
It is stored in the memory 180 together with the area of the face and arm of the person of 0. However, when the color tones between the image 900 and the image 905 are different, the pixel value of the image 905 is not brought as it is, but, for example, the average pixel value of the portion of the image 900 in which the value of the region division data is 3 is used. The color tone is adjusted so that the average pixel values of the image 905 are the same. The same processing is performed for the areas where the values of the area division data are 2 and 4, and the partial area image 935 obtained as a result is stored in the memory 180.

Finally, with respect to the partial area image 920 in which the value of the area division data is 0, the defective portion is repaired by the above-mentioned method (Shingaku Gakkai 86/1 Vol.1.J69-D No.1 pp.99). -97
The method described in “One method of restoration processing of lost image”) is performed, and the resulting image 925 is stored in the memory 1.
Store in 80.

In step 460, the image 925, the image 935, and the image 945 generated in step 455 are superposed and combined in accordance with the priority order defined in step 425. That is, the image 935 is superposed on the image 925, the image 945 is further superposed on the combined image, and the dress-up image 950 is generated.

As described above, the image of the person is input according to the process of FIG. 3, the image of the person and the image of the clothes are divided into regions at the sales center, and a process of generating a transformed image by transforming each region is performed. By superimposing the deformed images of the above on the computer device at home according to the priority order, it becomes possible to confirm the state when the consumer changes clothes by purchasing before purchasing the clothes, and after purchasing the clothes. It is possible to prevent troubles such as not matching.

[0097]

According to the present invention, an image for evaluating matching between a room owned by a customer and a product is used by using an image of an object owned by a customer who is a purchaser of the product in product introduction in online shopping or the like. A processing method can be provided. Furthermore, by applying an arbitrary transformation to the image of this room so that it does not look unnatural as a whole, an image processing method that can evaluate the matching of products by changing the viewpoint as if walking around the room It is possible to provide the product, and it is possible to prevent a problem that the product does not match the room owned by the customer after the product is purchased. In addition, compared to creating and sending an image, using the strain field data can significantly reduce the network load and communication time.

Furthermore, according to the present invention, even for a target such as a room owned by a customer, which the seller of the product cannot easily obtain the three-dimensional model, if a plurality of images with different viewpoints can be prepared. It is possible to provide an image processing method capable of morphing these images and observing the generated intermediate images in accordance with the interactive operation of an operator at a remote place. Further, it becomes possible to observe the image of the product when the viewpoint is changed without sending the shape data of the product from the sales center.

Further, in online shopping for selling clothes and the like, the image of the customer's person and the image of the clothes may be modified according to the present invention to generate an image that would be obtained when the customer changes his clothes. It is possible to provide an image processing method capable of performing the above, and prevent troubles such as incompatibility after buying clothes from occurring.

Furthermore, these image processing methods are
It can be expanded to various uses other than online shopping.

[Brief description of drawings]

FIG. 1 is an explanatory diagram of an apparatus that performs image processing according to an embodiment of the present invention.

FIG. 2 is an explanatory diagram of data used for image processing according to the embodiment of this invention.

FIG. 3 is a flowchart showing a procedure of image processing according to the first embodiment of this invention.

FIG. 4 is an explanatory diagram visually showing a procedure of image processing according to the first embodiment of this invention.

FIG. 5 is an explanatory diagram illustrating a method of associating images according to the embodiment of this invention.

FIG. 6 is an explanatory diagram visually showing an outline of image processing according to a second embodiment of the present invention.

FIG. 7 is an explanatory diagram visually showing a procedure of image processing according to a third embodiment of the present invention.

FIG. 8 is an explanatory diagram visually showing an outline of image processing according to a fourth embodiment of the present invention.

FIG. 9 is an explanatory diagram showing data used for image processing according to a fifth embodiment of the present invention.

FIG. 10 is an explanatory diagram visually showing a procedure of image processing according to a fifth embodiment of the present invention.

[Explanation of symbols]

 500, 510 ... Original image, 520, 525, 530, 535 ... Region, 540, 550 ... Intermediate image, 560 ... Image.

Continuation of the front page (51) Int.Cl. ⁶ Identification number Office reference number FI Technical display location G09G 5/36 520 G06F 15/24 101

Claims (10)

[Claims] 1. A first device including an image input device, an image storage device, an image processing device, an image display device, and a data communication device, an image storage device, and an image processing device. A second device including a device, an image display device, and a data communication device, and online shopping by generating a composite image in which the image input from the input device and the image of the product are combined by the processing device In the information providing method for smoothly performing communication in, the step of transferring image data input from the input device from the first device to the second device, and the received data being an image of a product. Storing the data in the storage device of the second device together with the data, dividing the image stored in the storage device into a plurality of areas, and dividing each divided area. Generating a plurality of processed region images respectively processed, giving a priority to each of the processed region images, the processed region image and those when there is a request of the first device Of the priority order data and the document data for ordering the goods to the first device, and the returned processing area images are processed in different processing areas at the same position according to the given priority. When the pixels of the image overlap, the step of generating a processed image combined so that the pixel of the processed area image having a higher priority becomes effective, and the generated processed image and the order document of the commodity are the first device. And a step of displaying the image on the display device. 2. A first device including an image input device, an image storage device, an image processing device, an image display device, and a data communication device, an image storage device, and an image processing device. A device, an image display device, and a second device including a data communication device, and a deformation area image obtained by deforming a plurality of images stored in the storage device by the processing device, and the deformation area image. An information providing method for facilitating communication in online shopping by generating a composite image in which images of products are combined, wherein image data input from the input device is stored in the storage device of the first device. And a step of transferring the received data from the first device to the second device, and a step of storing the received data in the storage device of the second device together with data that is an image of a product. And a step of dividing the plurality of images stored in the storage device into a plurality of areas, a step of defining mutually corresponding areas between different images, and a selected image for each set of the corresponding areas. Generating a deformation area image of the other image having the characteristics of 1., storing deformation data for generating the deformation area image in a storage device, and giving a priority to each deformation area image. And, when there is a request of the first device, the step of returning the deformation data, the data of their priority order and the document data for ordering a product, from the second device to the first device, A deformed region image is generated by deforming the image data stored in the first device by each returned deformed data, and the deformed region image is generated at the same position in accordance with the given priority. When the pixels of the deformation area image overlap, the step of generating a combined image in which the pixels of the deformation area image having a higher priority become effective, and the generated combined image and the order document of the product are And a step of displaying on the display device of the device. 3. A first device including an image input device, an image storage device, an image processing device, an image display device, and a data communication device, an image storage device, and an image processing device. And an image display device, and a second device including a data communication device, and an intermediate image and a product having an intermediate feature of the features of the plurality of images stored in the storage device by the processing device. Transferring an image input from the input device from the first device to the second device, in an information providing method for smoothly performing communication in online shopping by generating a combined image in which the images are combined. A step of storing the received data in the storage means of the second device together with data that is an image of a product; and a plurality of images stored in the storage device in a plurality of areas. A step of dividing, a step of defining mutually corresponding regions between different images, and an intermediate region image having an intermediate feature of the image features of a plurality of regions belonging to the set for each set of corresponding regions A step of generating, a step of giving a priority to each of the intermediate area images, and a step of ordering the intermediate image, data of those priorities and products when requested by the first device Returning the document data from the second device to the first device, each returned intermediate region image, according to the given priority, when the pixels of different intermediate region images overlap at the same position, A step of generating an intermediate image combined so that the pixels of the intermediate area image having a higher priority become effective, and the generated intermediate image and the order document of the product are displayed on the display device of the first device. And a step of displaying the image on a display. 4. The pixel according to claim 1, wherein when the generated processed image has a defective portion in which a pixel is defective, a pixel whose value is determined based on the values of pixels around the defective portion is placed in the defective portion. An image processing method having a step of supplementing. 5. The pixel according to claim 2, wherein when a pixel defective portion is present in the generated composite image, a pixel whose value is determined based on the values of pixels around the defective portion is filled in the defective portion. An image processing method having a step of performing. 6. The method according to claim 3, wherein when a pixel defective portion is present in the generated intermediate image, a pixel whose value is determined based on the values of pixels around the defective portion is filled in the defective portion. An image processing method having steps. 7. The method according to claim 2, wherein, for each group of corresponding areas, in the step of generating a processed area image of another image having the characteristics of an image arbitrarily selected from the images belonging to the group, An image processing method in which each pixel of an image of a plurality of regions belonging to is associated with each pixel of a modified image, and the value of each pixel of the modified image is a value obtained based on the values of the corresponding plurality of pixels. 8. The method according to claim 3, wherein for each set of corresponding regions, an intermediate region image having an intermediate feature of the features of images of a plurality of regions belonging to the set is generated, and the intermediate region image belongs to each set. Each pixel of the images of the plurality of regions is associated with each pixel of the intermediate region image, and the value of each pixel of the intermediate region image is set as a value obtained by mixing the values of the corresponding plurality of pixels, and the pixel is associated with the pixel. When there is only one pixel in the area, the image processing method is set to the value of the corresponding pixel. 9. An image input device, an image storage device, an image processing device, an image display device, a first device equipped with a data communication device, computer graphics data, and others. The storage device stores a plurality of images, a processing device that processes the images to operate computer graphics, an image display device, and a second device that includes a data communication device. To facilitate communication in online shopping by generating a synthetic image by synthesizing an intermediate image having intermediate characteristics of a plurality of images stored in the device and a computer graphics image representing a product by computer graphics In the information providing method, the image input from the input device is transferred from the first device to the second device. Storing the received data together with the computer graphics data of the product in the storage means of the second device, and displaying a plurality of images stored in the storage device on the image. The step of dividing into a plurality of areas corresponding to the plurality of objects present, and the image of the corresponding object in each area,
Setting a known shape model so as to match the shape of the image of the object; setting a shape model of computer graphics on the set shape model in a plurality of images; Defining mutually corresponding regions, generating an intermediate region image having intermediate characteristics of images of a plurality of regions belonging to the set, for each set of corresponding regions; Generating an intermediate shape model having intermediate characteristics of the shape model, generating an intermediate computer graphics image from the intermediate shape model, and giving priority to each of the intermediate area image and the intermediate computer graphics image. If there is a step of giving a rank and the request of the first device, A step of returning the area image, the intermediate computer graphics image, the priority data thereof, and the document data for ordering the goods from the second device to the first device; the returned intermediate area image and the intermediate computer graphic If the pixels of different intermediate area images or intermediate computer graphics images overlap at the same position according to the given priority, the pixels of the intermediate area image or intermediate computer graphics image with higher priority are An image processing method, comprising: a step of generating an intermediate image that is combined so as to be valid; and a step of displaying the generated intermediate image and an order document for a product on a display device of the first device. 10. An image input device, an image storage device,
A first device including an image processing device, an image display device, and a data communication device, an image storage device, an image processing device, an image display device, and a data communication device. In the method of generating a processed image by processing the image input from the input device by the processing device, the image data input from the input device is converted from the first device to the second device. Transferring the received data to the storage device of the second device together with data that is an image of a product, dividing the image stored in the storage device into a plurality of areas, Each of the processed areas is processed to generate a plurality of processed area images, each processed area image is given a respective priority, and the request of the first device is made. In the case where the processing area images and their priority data are returned to the first device, each processing area image returned is processed differently at the same position according to the given priority. When the pixels of the area image overlap, a step of generating a processed image combined so that the pixels of the processed area image with higher priority become effective,
And a step of displaying the generated processed image on the display device of the first device.