JP4708397B2 - Information terminal and computer program - Google Patents

Description

  The present invention relates to an information terminal and a computer program that can capture and synthesize a captured image output from a photographing apparatus such as a digital camera with existing image data.

  In stores that handle fashion-related goods such as clothing and accessories, it is common to try on clothing before selling it. Even on the customer side, it will be rare to purchase items such as clothes without trying on.

  However, since it takes some time to try on, it is not possible to try on many products at once. Further, when a plurality of stores are installed in different places, for example, a situation may occur in which a product requested by a customer is not in a store in Tokyo but is in a store in Kobe. In such a case, a customer who visits a store in Tokyo cannot try on a product that is only in Kobe.

  For this reason, in recent years, for example, a technology has been developed in which a product image and a customer image are superimposed in a virtual space on a computer and can be tried on the image without actually trying on. I'm starting to show signs. However, in order to enable such a try-on in a virtual space, a large-scale mechanism is required, such as having to store 3D information in both product images and customer images. At that time, there is no easy way to adopt it easily in many stores.

  On the other hand, Patent Document 1 discloses a technique in which an image captured by a digital camera can be easily superimposed on previously stored image data.

JP 2004-312162 A

  As mentioned above, a system equipped with a technology that superimposes product images and customer images in a virtual space on a computer and allows you to try on the image without actually trying it on, has a high threshold, Currently it is not popular. In addition, in order to maintain the three-dimensional information for operating such a system, the customer must be photographed from various angles. Such an operation tends to make the customer feel resistance.

  On the other hand, as described in Patent Document 1, a technique has been developed in which an image captured by a digital camera can be easily superimposed on image data held in advance. However, according to such a technique, there is a problem that it is difficult to obtain a realistic and high-precision superimposed image in combination with the superimposition of both images after displaying two images side by side in one screen. .

  An object of the present invention is to obtain a realistic and highly accurate superimposed image while adopting a simple method when superimposing a model image and a captured image into one image.

An information terminal according to the present invention includes a display unit for displaying information, an operation input unit for operating and inputting information, and a plurality of lamps arranged in a two-dimensional space on the back of the display unit. A gaze board that selectively controls lighting , and (1) extracts a face portion in a model image included in image data stored in a storage unit, and (2) in the extracted model image (3 ) displaying a part of the image data on the display unit as an enlarged image including the face portion in the extracted model image, and (4) the extracted model image. A first filtering process is performed on the inner face part to make the display of the face part different from the display of the other part, and (5) the moving image being captured output by the photographing device is taken in and the face in the moving image is captured (6) The extracted portion is extracted according to the recognized face orientation. The direction of the face of the subject standing in front of the photographing apparatus is directed in a direction in which the face direction of the face part in the extracted model image and the face direction of the face part in the extracted moving image are relatively approximate (7) The face part in the extracted moving image is superimposed and displayed on the model image displayed on the display unit, and (8) the operation input unit By this shooting instruction operation, a composite image which is a still image in which the face portion in the moving image is replaced with the image data is generated.

The computer program of the present invention is installed in a computer including a display unit that displays information and an operation input unit that inputs information, and causes the computer to execute the processes (1) to (8) .

  According to the present invention, the face portion in the model image included in the image data is extracted to display an enlarged image centered on the face portion, and the display of the face portion is changed to the display of other portions. Since the face part extracted from the moving image being photographed by the photographing device is superimposed here, the model image and the photographed image are superimposed to form a single image while adopting a simple technique. Realistic and highly accurate superimposed images can be obtained.

  An embodiment of the present invention will be described with reference to FIGS.

  FIG. 1 is an external perspective view showing an information terminal. The information terminal 101 is configured by attaching a liquid crystal display 103 as a display unit to the upper surface of a tall body type main body unit 102 that accommodates each unit. A touch panel 105 as an operation input unit is laminated on the display surface 104 of the liquid crystal display 103. On the liquid crystal display 103, a camera 107 as a photographing device is also attached to the upper part of the frame 106.

  A gaze board 109 is fixed to the main body 102 so as to be positioned on the back side of the liquid crystal display 103. The gaze board 109 is configured by arranging a plurality of lamps 111 on a vertically arranged board 110 and further providing lighting circuits (not shown) for controlling the lighting of the lamps 111 individually. As an example of the lamp 111, an LED lamp with high luminance is used.

  FIG. 2 is a block diagram of electrical connection of the information terminal 101. As shown in FIG. 2, the information terminal 101 includes a microcomputer 201, and the microcomputer 201 drives and controls each unit. The microcomputer 201 functions as a work area or the like by storing a ROM 204 for storing fixed information such as a control program in advance via a bus line 203 and a ROM 204 for storing various information in a rewritable manner in a CPU 202 that controls each unit centrally. A RAM 205 is connected. Therefore, the microcomputer 201 constitutes an information processing unit that executes information processing. In FIG. 2, the CPU 202, the ROM 204, and the RAM 205 are displayed as a single component. However, the CPU 202, the ROM 204, and the RAM 205 may be divided into a plurality of solids. In particular, the component displayed as the RAM 205 includes at least a chip used as a work area and a chip constituting an image memory.

  An I / O interface IF is connected to the microcomputer 201 via the bus line 203. A touch panel controller 206, a video controller 207, an HDD 208, and a serial port 209 are connected to the I / O interface IF. The serial port 209 is connected to the camera 107 and the gaze board 109 described above, and is further connected to an RFID reader / writer 210.

  A touch panel 105 is connected to the touch panel controller 206. The touch panel controller 206 takes an input signal from the touch panel 105 into the microcomputer 201.

  A liquid crystal display 103 is connected to the video controller 207. The video controller 207 drives and controls the liquid crystal display 103 based on the image data, and causes the liquid crystal display 103 to display an image corresponding to the image data.

  The HDD 208 is provided with a program area 301, a product master area 302 (product database), a meta image storage area 303 (part of the image database), and a product ID link area 304 (other part of the image database). . In the program area 301, an OS, a computer program, and the like are installed. These OSs, computer programs, and the like are all or partially copied to the RAM 205 and accessed by the CPU 202 when the information terminal 101 is activated. The CPU 202 executes processing according to the OS and computer program copied in this way. The product master area 302 constructs a product database. Details thereof will be described later with reference to FIG. The meta image storage area 303 and the product ID link area 304 construct an image database. Details of the meta image storage area 303 will be described later with reference to FIG. 4, and details of the product ID link area 304 will be described later with reference to FIG.

  The camera 107 connected to the serial port 209 forms an image captured from the lens 108 on a two-dimensional CCD array (not shown), and outputs the CCD array output after amplification, A / D conversion, filtering, and the like. It is a photographing device.

  The RFID reader / writer 210 connected to the serial port 209 performs short-range wireless communication with the RFID chip by, for example, electromagnetic induction coupling. In the present embodiment, an RFID chip is attached to a product. That is, the product is attached with a wireless tag (not shown) that incorporates an RFID chip that stores a product code that identifies the product. Therefore, the RFID reader / writer 210 receives a product code specifying the product from the wireless tag attached to the product, and outputs the received product code. The microcomputer 201 takes in the product code output from the RFID reader / writer 210 from the serial port 209 via the I / O interface IF.

  As another embodiment, a product code that specifies a product may be read or input by another device and loaded into the microcomputer 201. As an example, a numeric keypad may be displayed on the liquid crystal display 103, and a product code may be input by inputting on the touch panel 105 in accordance with the displayed numeric keypad. As another example, a product code expressed as a code symbol such as a barcode or a two-dimensional code may be prepared, and the product code may be input by optically reading the code symbol with a code reader. .

  FIG. 3 is a schematic diagram showing record data of the product master area 302 (product database) recorded in the HDD 208 of the information terminal 101. The product master area 302 stores product names in association with product IDs (product codes) that specify individual products.

  FIG. 4 is a schematic diagram showing recorded data in the meta image storage area 303 (part of the image database) recorded in the HDD 208 of the information terminal 101. In the meta image storage area 303, binary data (image data) that is data of an image captured by the camera 107 and its path name are added to an image ID that identifies an image captured by the camera 107 and captured by the microcomputer 201. It is stored in association. The path name includes the data format of binary data. As another embodiment, the binary data itself is not stored in the meta image storage area 303 but is stored in one of the storage areas, and only the path name is stored in the meta image storage area 303. It is also possible. When such a data structure is adopted, a meta image can be searched using a path name as a key.

  FIG. 5 is a schematic diagram showing recorded data in the product ID link area 304 (another part of the image database) recorded in the HDD 208 of the information terminal 101. The product ID link area 304 stores the product ID in association with the image ID. Therefore, by referring to the product ID link area 304, for example, a corresponding image ID can be searched using the product ID as a key. If the image ID is found by the search, binary data can be acquired from the meta image storage area 303.

  FIG. 6 is a flowchart showing the flow of the image composition process. The computer program installed in the program area 301 of the HDD 208 includes an image composition processing program for generating a composite image in which the model image and the captured image are superimposed. This image composition processing program is copied to the RAM 205 and used at the time of activation. The processing of the flowchart shown in FIG. 6 shows the processing contents executed by the CPU 202 in accordance with the image data registration processing program copied to the RAM 205 in this way.

  The CPU 202 first executes a specific image selection process (step S101). As an example, this process derives a specific image including a captured image of the product from the specific product. That is, when a product is touch-designated by touch designation on the touch panel 105, the CPU 202 acquires the product ID. Therefore, the image ID is searched from the product ID link area 304 shown in FIG. 5 using the acquired product ID as a key. Then, the CPU 202 retrieves binary data from the meta image accumulation area 303 shown in FIG. 4 using the retrieved image ID as a key. This binary data is the image data itself of the specific image selected in step S101.

  FIG. 7 is a flowchart showing another example of the specific image selection process (step S101) in FIG. In another example, instead of deriving a specific image including a captured image of the product from the specific product, the image data stored in the meta image storage area 303 shown in FIG. This is processing for selecting desired image data as a specific image from the displayed thumbnail by touch designation on the touch panel 105. That is, the CPU 202 sequentially acquires image data that forms one image from the meta image storage area 303, and executes the process of extracting the face portion of the model image (step S101-1). The process of extracting the face portion of the model image will be described in detail later. Then, it is determined whether or not the face part of the model image has been extracted, in other words, whether or not the face part of the model image has been recognized (step S101-2). If not, the process proceeds to step S101-1 for the next image data. The process returns (N in step S101-2), and if possible, the process proceeds to step S101-3 (Y in step S101-2). In step S101-3, so-called thumbnail display processing for displaying a reduced image on the list screen displayed on the liquid crystal display 103 is executed. The user can select desired image data as a specific image from a thumbnail image displayed on the liquid crystal display 103 by touch designation on the touch panel 105 (step S101-4).

  The CPU 202 executes processing for extracting a face portion from the image data selected in step S101 as processing subsequent to the specific image selection processing in step S101 (step S102). That is, since the model image included in the image data includes a face, the face portion is extracted. Extraction of a face from image data can be executed using various known methods. As an example, in a human face, both eyes and mouth take a predetermined arrangement within a certain range. Therefore, a part that is likely to be both eyes and a part that is likely to be mouth are discriminated by image processing technology, and there is such an arrangement. If an arrangement predetermined in the range is taken, it is determined that this is a human face. The part which will be both eyes and the part which will be the mouth can be estimated from various feature values on the image.

  The CPU 202 also executes a process of acquiring the direction in which the face is facing from the image data as the process of step S102. Acquisition of the direction in which the face is facing can also be performed using various known methods. As an example, it can be estimated from the positional relationship between the part that will be both eyes and the part that will be the mouth, the left-right balance of the part that will be the chin with respect to the center of the face determined from such positional relationship.

  If the face portion cannot be recognized from the image data in Step S102 (N in Step S103), the CPU 202 does not proceed to the subsequent processing, and returns to the specific image selection processing in Step S101. This is because if the face portion cannot be recognized from the image data, the subsequent composite image generation processing itself cannot be executed.

  On the other hand, even if the face direction can not be recognized from the image data, if the face portion can be recognized from the image data, the subsequent composite image generation process can be performed. Therefore, as an example, when the face portion can be recognized from the image data, the recognition possibility determination in step S103 is passed even if the direction in which the face is facing cannot be recognized from the image data. However, as another example, even if the face portion can be recognized from the image data, it is possible not to pass the recognition possibility determination in step S103 when the direction in which the face is facing cannot be recognized.

  When the CPU 202 passes the recognition possibility determination in step S103 (Y in step S103), the CPU 202 is an enlarged image centered on the face portion in the model image extracted from the image data, and the face in the extracted model image A first filtering process is performed on the part to make the display of the face part different from the display of the other part, and a part of the image data is displayed on the liquid crystal display 103 (step S104). The first filter process at this time is a transparent process for the face portion. That is, the CPU 202 performs an enlarged display process and a transparent display process on the image data selected in step S101 to deform the image, and causes the video controller 207 to display the deformed image on the liquid crystal display 103. The signal is output. The enlarged display process and the transmissive display process can be executed using known methods in various places.

  FIG. 8 is a schematic diagram illustrating an example of an enlarged model image displayed on the liquid crystal display 103 of the information terminal 101. As shown in FIG. 8, it is assumed here that an image having a path name of “0001.jpg” is selected as the original image 401 in step S101. The CPU 202 performs an enlarged display process and a transmissive display process on the selected original image 401 to deform the image, and displays the deformed image on the liquid crystal display 103 as a model image 402 illustrated in FIG. Here, for convenience, a display image including the model image illustrated in FIG. In the image A immediately before photographing, a photographing button 403 and a cancel button 404 are displayed in addition to the model image 402. In the model image 402, the face portion is transparently displayed.

  Returning to the description of the flowchart shown in FIG. The CPU 202 refers to information on the direction in which the face recognized from the image data in step S102 is facing, and outputs a lighting command for one lamp 111 to the gaze board 109. That is, the CPU 202 stands in front of the camera 107 according to the recognized face orientation, but the user faces the orientation that substantially matches the face orientation of the extracted model image 402. The lamp 111 arranged at the position is turned on.

  FIG. 9 is a schematic view illustrating a light emission example of the lamp 111 on the gaze board 109. FIG. 9 shows a state where the lamp 111 located on the right side is lit. If the lamp 111 is lit, the user standing in front of the camera 107 is expected to watch the direction and turn his face to the direction. Therefore, the direction in which the user turns the face is controlled by turning on the lamp 111, and the user is directed to face the face of the extracted model image 402. As a result, it is expected that the composite image created in step S109, which will be described later, becomes more realistic and highly accurate.

  Returning to the description of the flowchart shown in FIG. Following the processing in step S105, the CPU 202 activates the camera 107 and executes moving image shooting processing (step S106). The camera 107 outputs the captured moving image to the serial port 209. As a result, the microcomputer 201 captures the moving image output from the camera 107.

  In the subsequent step S107, the CPU 202 extracts a face portion from the captured moving image and obtains a face image 405 (see FIG. 10). Image extraction in this case can also be executed by the same or similar process as the extraction process in step S102 described above. Then, the CPU 202 superimposes and displays the face image 405 of the face portion captured and extracted from the camera 107 on the image A immediately before shooting.

  FIG. 10 is a schematic diagram showing an example in which a face image 405 extracted from a moving image is superimposed on a model image 402 displayed on the liquid crystal display 103 of the information terminal 101 and displayed. Here, for convenience, the image illustrated in FIG. 10 is referred to as a shooting adjustment image B. In this shooting adjustment image B, a superimposed image 406 is displayed in which the face image 405 of the face portion captured and extracted from the camera 107 is superimposed on the model image 402 displayed in the image A immediately before shooting, and the shooting button 403 is further displayed. A cancel button 404 is displayed.

  Here, in the shooting adjustment image B, the face portion of the model image 402 is subjected to transparency processing, and the moving image captured from the camera 107 is the face image 405 of the face portion. It is easy to superimpose the face image 405 captured from the camera 107 on the face portion of the model image 402 that has been processed. That is, the user moves his / her face up / down / left / right in front of the camera 107 or moves the face in the perspective direction relative to the camera 107 so that the face image 405 overlaps the face portion of the model image 402. Good. At this time, since the face portion of the model image 402 has been subjected to transparency processing, it is very easy to superimpose the own face image 405 on this transparency portion.

  Returning to the description of the flowchart shown in FIG. The CPU 202 superimposes and displays the face image 405 captured and extracted from the camera 107 on the image A immediately before shooting (step S107), and then waits for determination of the presence or absence of a shooting operation instruction executed by touch designation of the shooting button 403. (Step S108). If there is no touch designation of the shooting button 403 (N in step S108), the process returns to the moving image capturing process from the camera 107 in step S106. In contrast, when the CPU 202 determines that the shooting button 403 is touched (Y in step S108), after starting the timer and waiting for a predetermined time, the CPU 202 holds the moving image captured from the camera 107 as a still image and stores it in the RAM 205. Temporary storage is performed (step S109). That is, the CPU 202 sets a predetermined timer time after specifying the shooting operation by specifying the touch of the shooting button 403. At this time, a voice synthesis circuit and a voice circuit (not shown) having a speaker are connected to the serial port 209, a countdown voice such as “3, 2, 1, 0” is generated and uttered from the speaker, The moving image captured from the camera 107 at the timing of “0” utterance may be held as a still image.

  After holding the moving image captured from the camera 107 as a still image (step S109), the CPU 202 performs a second filter process on the held still image (step S110). The second filter process is a process for approximately correcting the color (hue, lightness, saturation) and lightness (contrast) of the face part in the extracted model image and the face part in the extracted moving image. . Correction of these colors (hue, lightness, saturation) and shading (contrast) can be realized by various known methods.

  The CPU 202 performs the second filter process on the still image held by the process of step S109 (step S110), and then performs the second filter process on the face portion of the original image 401 (see FIG. 8). A composite image 407 is generated by replacing with an image. Then, the generated composite image 407 is displayed on the liquid crystal display 103 (step S111).

  FIG. 11 is a schematic diagram illustrating an example of the generated composite image 407. As shown in FIG. 11, the composite image 407 is not a composite image of the user's face image 405 captured from the camera 107 with the model image 402 enlarged from the original image 401, but a composite image of the face image 405 with the original image 401. It has become an image. As a result, the user can view the composite image 407 with an impression as if he / she is wearing clothing worn by the model included in the original image 401.

  As described above, according to the present embodiment, the face image 405 extracted from the moving image captured from the camera 107 can be superimposed and displayed on the face portion of the model image 402 based on the original image 401 ( Step S107 in FIG. 6). In this case, the model image 402 is displayed as an image that is enlarged with respect to the original image 401 centering on the face portion (see step S104 in FIG. 6), and thus the face captured from the camera 107 in the face portion of the model image 402. It becomes easy to accurately overlay the images 405. In addition, since the model image 402 is subjected to the first filter processing and the face portion is transparently displayed (see step S104 in FIG. 6), the face portion of the model image 402 is further captured from the camera 107. It is easy to accurately overlay the face image 405. For this reason, when the model image 402 and the face image 405 that is the captured image are superimposed to form a single image, a realistic and highly accurate superimposed image can be obtained while adopting a simple method. . A composite image, which is a still image generated by superimposing the face image 405 on the model image 402, is displayed on the liquid crystal display 103 with the same size as the original image 401 (see step S111 in FIG. 6). Can virtually try on clothing worn by the model included in the original image 401, and can easily grasp the impression of wearing the clothing.

It is an external appearance perspective view which shows the information terminal which is one Embodiment of this invention. It is a block diagram of the electrical connection. It is a schematic diagram which shows the recording data of the goods master area currently recorded on HDD of an information terminal. It is a schematic diagram which shows the recording data of the meta image storage area recorded on HDD of an information terminal. It is a schematic diagram which shows the recording data of the goods ID link area recorded on HDD of an information terminal. It is a flowchart which shows the flow of an image synthesis process. It is a flowchart which shows another example of the selection process (step S101) of the specific image in FIG. It is a schematic diagram which shows an example of the enlarged model image displayed on the liquid crystal display of an information terminal. It is a schematic diagram which illustrates the light emission example of the lamp | ramp in a gaze board. It is a schematic diagram which shows an example which superimposed and displayed the face part extracted from the moving image on the model image by which the enlarged display displayed on the liquid crystal display of the information terminal was carried out. It is a schematic diagram which shows an example of the produced | generated synthesized image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 103 ... Liquid crystal display (display part), 105 ... Touch panel (operation input part), 107 ... Camera (imaging device), 110 ... Gaze board, 111 ... Lamp, 208 ... HDD (memory | storage part)

Claims (7)

A display for displaying information;
An operation input unit for operating and inputting information;
A gaze board that is positioned on the back of the display unit and arranged with a plurality of lamps on a two-dimensional space, and selectively controls the lamps for lighting
Means for extracting a face portion in the model image included in the image data stored in the storage unit;
Means for recognizing the face orientation of the face portion in the extracted model image;
Means for displaying a part of the image data on the display unit in an enlarged image including a face portion in the extracted model image;
Means for performing a first filtering process on the face part in the extracted model image to make the display of the face part different from the display of other parts;
Means for taking a moving image being shot output from the photographing device and extracting a face portion in the moving image;
According to the recognized face direction, the photographing is performed in a direction in which the face direction of the face part in the extracted model image and the face direction of the face part in the extracted moving image are relatively approximated. Means for instructing lighting of the lamp at a position for directing the face of a subject standing in front of the apparatus;
Means for superimposing and displaying the face portion in the extracted moving image on the model image displayed on the display unit;
Means for generating a composite image that is a still image obtained by replacing a face portion in the moving image with the image data by a shooting instruction operation in the operation input unit;
An information terminal comprising:   The information terminal according to claim 1, wherein the first filter process is a filter process that transmits a face portion in the extracted model image.   The information terminal according to claim 1, further comprising means for setting a predetermined timer time after a photographing instruction operation at the operation input unit.   4. The apparatus according to claim 1, further comprising means for performing a second filter process for approximately correcting colors of a face part in the extracted model image and a face part in the extracted moving image. Information terminal.   5. The apparatus according to claim 1, further comprising means for performing a second filter process for approximately correcting light and shade of a face portion in the extracted model image and a face portion in the extracted moving image. Information terminal. Means for extracting from the image database storing a plurality of the image data image data capable of extracting a face portion of the model image and displaying the thumbnail on the display unit;
Means for selecting the image data selected and designated from the thumbnail display by the operation input unit as image data to be combined;
The provided information terminal according to any one of claims 1 to 5. Installed in a computer comprising a display unit for displaying information and an operation input unit for operating and inputting information,
A function of extracting a face portion in a model image included in image data stored in a storage unit;
A function of recognizing the face orientation of the face portion in the extracted model image;
A function of displaying a part of the image data on the display unit in an enlarged image including a face portion in the extracted model image;
A function of performing a first filtering process on the face part in the extracted model image to make the display of the face part different from the display of other parts;
A function of taking a moving image being shot output from the photographing device and extracting a face portion in the moving image;
A plurality of lamps are arranged in a two-dimensional space positioned on the back of the display unit, and the extracted information is selected according to the recognized face direction with respect to a gaze board that selectively controls the lamps to be turned on. Position for directing the face of the subject standing in front of the photographing apparatus in a direction in which the face direction of the face part in the model image and the face direction of the face part in the extracted moving image are relatively approximate A function of instructing lighting of the lamp,
A function of superimposing and displaying the face portion in the extracted moving image on the model image displayed on the display unit;
A function of generating a composite image that is a still image obtained by replacing a face portion in the moving image with the image data by a shooting instruction operation in the operation input unit;
A machine-readable computer program that executes the program.

Images