JP7095574B2 - Image processing programs, image processing methods, and image processing equipment - Google Patents

Description

The present invention relates to an image processing program, an image processing method, and an image processing apparatus.

For example, at airports and the like, a baggage handling system is used in which baggage checked by passengers is transported to an area where a container to be loaded on an aircraft on which a passenger is boarding is placed.

In the baggage handling system, for example, when baggage is checked in, identification information associated with the baggage owner's information is issued. Then, for example, a tag printed with a barcode encoding identification information is attached to the baggage. The baggage handling system then reads the identification information from the tags attached to the baggage with a barcode reader installed along the conveyor. Then, the baggage handling system controls the conveyor based on the read identification information to sort the baggage, and finally transports the baggage to the container to be loaded on the aircraft on which the baggage owner is boarding.

Further, for example, a technique of using image data for managing articles and detecting abandoned objects is known.

Japanese Unexamined Patent Publication No. 10-149434 Japanese Unexamined Patent Publication No. 2009-152748 International Publication No. 2018/012439

For example, while transporting baggage on a conveyor, the tag attached to the baggage may come off, making it impossible to identify the baggage owner's information from the tag. For example, in order to deal with such a situation, it is conceivable to store the baggage image showing the passenger's baggage in advance in association with the identification information. Thereby, for example, even when the baggage with the tag removed is found, the identification information can be specified by executing the image search using the image of the baggage with the tag removed as the search key. However, in this way, for example, when an image search is performed using an image showing a baggage such as baggage as a search key, a baggage image showing a baggage of a different color may be included in the upper part of the search result. As a result, the efficiency of finding a baggage image that matches the baggage that matches the baggage that appears in the image used as the search key may decrease.

In one aspect, it is an object of the present invention to provide a technique capable of efficiently finding a baggage image in which a baggage matching the baggage shown in the image as a search key is shown in an image search.

The image processing program of one aspect of the present invention is a plurality of baggage images in which different packages are captured when an image search request is received, and each of them is associated with identification information for identifying the owner of the baggage. Perform an image search for multiple baggage images using the image that shows the baggage included in the image search request as the search key, and each of the multiple baggage images is similar to the image that shows the baggage included in the image search request. Changed to reduce the similarity between the color of the image showing the baggage included in the image search request and the baggage image judged that the color does not meet the predetermined judgment conditions among multiple baggage images. Then, the image processing device is made to execute a process of returning the baggage image whose similarity after the change satisfies a predetermined condition and the identification information associated with the baggage image satisfying the predetermined condition.

When searching for an image, it is possible to efficiently find a baggage image that shows a baggage that matches the baggage that appears in the image used as the search key.

It is a figure which illustrates the system structure of the baggage handling system which concerns on embodiment. It is a figure explaining the flow of the baggage handling by the baggage handling system which concerns on embodiment. It is a figure which illustrates the block structure of the image processing apparatus which concerns on embodiment. It is a figure which illustrates the baggage management information which concerns on embodiment. It is a figure which illustrates the operation flow of the baggage image generation processing which concerns on embodiment. It is a figure explaining an example about the generation of a baggage image. It is a figure which illustrates the flow of the generation of the baggage image which concerns on embodiment. It is a figure which illustrates the operation flow of the image search process which concerns on embodiment. It is a figure exemplifying the color wheel. It is a figure which illustrates the change of the evaluation result of the degree of similarity. It is a figure which illustrates the generation of a plurality of baggage images from the baggage image which concerns on 1st modification. It is a figure which illustrates the operation flow of the baggage image generation processing which concerns on the 1st modification. It is a figure which illustrates the case where the background of an image taken by a shooting apparatus is different from the background of an image taken at another timing. It is a figure which shows the exemplary background image management information. It is a figure which illustrates the operation flow of the baggage image generation processing which concerns on the 2nd modification. It is a figure which illustrates the hardware configuration of the computer for realizing the image processing apparatus which concerns on embodiment. It is a figure which illustrates the hardware configuration of the computer for realizing the terminal which concerns on embodiment.

Hereinafter, some embodiments of the present invention will be described in detail with reference to the drawings. The corresponding elements may be designated by the same reference numerals in a plurality of drawings.

As mentioned above, at the airport, for example, passengers check in baggage other than baggage that they bring on board at the airline's check-in counter before boarding. At the time of check-in, the baggage is attached with a tag including a barcode in which identification information for identifying the baggage is coded, and is carried on a conveyor such as a belt conveyor. The identification information is associated with, for example, information about the baggage owner (hereinafter, may be referred to as baggage information), and may be used to identify the baggage owner. Baggage information may include, for example, information such as the name of the owner of the baggage, the gender of the owner, the airline of the aircraft on which the owner is aboard, and the flight number of the aircraft.

In addition, a reading device equipped with a barcode reader is installed along the conveyor. Then, the bar code of the tag is read by the reading device, and the baggage is finally sorted and transported by the belt conveyor to the make area where the container is loaded in the luggage compartment of the aircraft on which the owner is boarding according to the read identification information.

Here, the tag attached to the baggage may come off during the transportation of the baggage. For example, a tag may be attached to baggage with an adhesive sticker, but if the adhesive surface of the tag is exposed, the adhesive surface will stick to the wall, etc., and the baggage will be carried by the conveyor. It may be pulled and the tag may be cut off. If the tag is removed from the baggage in this way, the baggage information cannot be obtained from the identification information written on the tag, and it becomes difficult to specify where the baggage should be transported.

As one method for dealing with such a situation, for example, an image of a passenger's baggage (hereinafter, may be referred to as a baggage image or a baggage image) is associated with identification information and stored in a storage device in advance. It is possible to keep it. Then, for example, when the baggage with the tag removed is found, an image of the baggage with the tag removed is taken with a shooting device, and the shot image is used as a search key to search the baggage image stored in the storage device. do. As a result, it is possible to identify the baggage image corresponding to the baggage with the tag removed from the baggage images ranked high in the search and quickly acquire the identification information.

However, for example, as a result of an image search, a baggage image showing a baggage shown in an image used as a search key and a baggage image showing a baggage having a clearly different color may appear at the top.

For example, images based on local features used in image retrieval are compared based on the distribution of changes in brightness, so if the shapes are similar even if the colors are different, the similarity of the images will be determined. It may be highly evaluated. Therefore, it is desired to provide a technique that can suppress the appearance of baggage images that show baggage that is clearly different in color from the baggage that appears in the image input as the search key at the top of the image search and improve the search accuracy. There is.

In the embodiment described below, the feature value representing the color feature of the baggage is determined from each image of the baggage used as the search key in the image search and the baggage image stored in the storage device. Then, when the feature values of the two images are less than a predetermined degree of approximation and are not similar, the evaluation of the degree of similarity of the baggage images is lowered. As a result, the accuracy of the image search is improved, and it is possible to efficiently identify the baggage image in which the baggage matching the baggage in the image input as the search key is shown. Hereinafter, embodiments will be described in more detail.

First, the baggage handling system 100 according to the embodiment will be described.

(Baggage handling system)
FIG. 1 is a diagram illustrating a system configuration of the baggage handling system 100 according to the embodiment. The baggage handling system 100 may be, for example, a BHS (Baggage Handling System). The baggage handling system 100 includes, for example, a conveyor 101, a conveyor control device 102, a reading device 103, a controller 104, a management server 105, an image processing device 106, and a terminal 107.

The conveyor control device 102, the controller 104, the management server 105, the image processing device 106, and the terminal 107 may be connected to each other via, for example, the network 150. The network 150 may be, for example, a LAN (Local Area Network) and may include a wired network and a wireless network.

The conveyor 101 may be, for example, a belt conveyor. The conveyor 101 may be connected to, for example, a conveyor control device 102 having a communication function, is connected to the network 150 via the conveyor control device 102, and sorts and conveys baggage under the control of the management server 105.

The reading device 103 includes, for example, a barcode reader 131 and a photographing device 132. The bar code reader 131 of the reading device 103 reads the identification information from, for example, the bar code of the tag 260 attached to the baggage carried on the conveyor 101. Further, the photographing device 132 of the reading device 103 photographs the baggage carried on the conveyor 101, for example. The reading device 103 is connected to, for example, a controller 104 having a communication function, and transmits identification information to the management server 105 via the controller 104. Further, the reading device 103 transmits, for example, the identification information read by the barcode reader 131 and the captured image of the baggage captured by the photographing device 132 to the image processing device 106 via the controller 104.

The management server 105 controls, for example, the sorting and transportation of baggage by the conveyor 101 based on the identification information read by the bar code reader 131 of the reading device 103.

The image processing device 106 analyzes, for example, a photographed image of baggage carried on the conveyor 101 photographed by the photographing device 132 of the reading device 103, and generates a baggage image obtained by extracting an area in which the baggage is captured. Then, the image processing device 106 stores the baggage image in association with the baggage identification information reflected in the baggage image. Further, when the image processing device 106 receives an image search request from the terminal 107, for example, the image processing device 106 uses the image data included in the image search request as a search key to perform an image search for the stored baggage image.

The terminal 107 is a terminal having a communication function such as a tablet terminal or a smartphone. Further, the terminal 107 includes, for example, a photographing device 171 such as a camera.

The baggage handling system 100 shown in FIG. 1 is an example, and the embodiment is not limited to this, and the baggage handling system 100 may have other configurations. For example, the baggage handling system 100 may include a further configuration such as a terminal installed in a check-in counter described later.

Subsequently, the flow of handling baggage by the baggage handling system 100 according to the embodiment will be described. FIG. 2 is a diagram illustrating a flow of baggage handling by the baggage handling system 100 according to the embodiment. FIG. 2 shows, for example, a conveyor 101, a reading device 103, a management server 105, an image processing device 106, a check-in counter 201, a baggage inspection device 202, a container 270, and a make area 290.

Here, the passenger, for example, deposits the baggage 250 at the check-in counter 201 ((1) in FIG. 2). The terminal installed in the check-in counter 201, for example, issues identification information in association with baggage information regarding the owner of the checked baggage 250, and issues a tag 260 printed with a barcode encoding the identification information. The issued tag 260 is attached to the checked baggage 250 ((2) in FIG. 2).

Subsequently, the baggage 250 is carried on the conveyor 101 to the baggage inspection device 202, and the baggage inspection device 202 performs a baggage inspection on the baggage 250 and transmits the inspection contents to the management server 105 ((3) in FIG. 2 (3). )). The baggage inspection inspector inspects the baggage based on, for example, the inspection content of the baggage inspection device 202 received by the management server 105.

Subsequently, the baggage 250 is further carried on the conveyor 101 and passes through the reader 103. In the reading device 103, the barcode reader 131 reads the identification information from, for example, the tag 260 attached to the baggage 250 flowing on the conveyor 101. Further, the photographing device 132 of the reading device 103 photographs, for example, the baggage 250 flowing on the conveyor 101. The photographing device 132 of the reading device 103 may be installed so as to photograph on the conveyor 101 at a predetermined angle of view. Subsequently, the reading device 103 transmits, for example, the identification information read by the barcode reader 131 to the management server 105 via the controller 104 ((4) in FIG. 2).

For example, the management server 105 controls the conveyor control device 102 based on the identification information and the baggage information of the baggage 250 corresponding to the identification information ((5) in FIG. 2), and the conveyor 101 reaches the make area 290 where the container 270 of the loading destination is located. Sort and transport baggage 250. As a result, the baggage 250 is carried to the make area 290 where the baggage 250 is loaded into the container 270 loaded on the flight on which the owner is boarding (FIG. 2 (6)). In the make area 290, the worker reads the identification information from the tag 260 attached to the baggage 250, for example, using the terminal 107. Then, for example, the worker confirms the baggage information corresponding to the baggage 250 by using the identification information on the terminal 107, and loads the baggage 250 into the container 270 of the aircraft on which the owner is boarding.

Further, for example, when a suspicious object is found in the baggage 250 in the baggage inspection device 202, the baggage 250 is carried by the conveyor 101 to the avoidance inspection room and undergoes further inspection ((7) in FIG. 2). .. As described above, in the baggage handling system 100, the baggage 250 is individually managed by the identification information such as the barcode printed on the tag 260, and a large amount of the baggage 250 is sorted with high accuracy and loaded into the aircraft.

Further, in the embodiment, the reading device 103 may transmit, for example, a photographed image of the baggage 250 photographed by the photographing apparatus 132 and the identification information read by the barcode reader 131 to the image processing apparatus 106 in association with each other (FIG. FIG. 2 (8)). The image processing device 106 generates, for example, a baggage image obtained by extracting an area in which baggage is captured from a captured image in which baggage 250 captured by the photographing device 132 is captured. Then, the image processing device 106 stores the generated baggage image in the storage device in association with the identification information received from the reading device 103. Therefore, even if the tag 260 is removed from the baggage 250, the worker takes an image of the baggage 250 and uses the taken image as a search key to search the baggage image stored in the storage device for the baggage image. It becomes possible to acquire the identification information associated with.

For example, suppose a worker in charge of loading baggage into a container 270 in the make area 290 finds baggage 250 with the tag 260 removed. In this case, the identification information cannot be acquired from the tag 260. Here, for example, the worker takes an image of the baggage 250 with the tag 260 removed by the taking device 171 of the terminal 107, and requests the image processing device 106 to search the image using the taken image as a key (FIG. 2). (9)).

The image processing device 106 compares the baggage image stored in the storage device with the image of the baggage 250 received as the search key, and evaluates the similarity. Further, the image processing device 106 evaluates the degree of color approximation between the baggage image stored in the storage device and the image of the baggage 250 received as the search key, and determines that the colors are not similar based on the degree of approximation. Change the similarity of the baggage image to be low. After that, the image processing device 106 returns the baggage image similar to the baggage image with a degree of similarity satisfying a predetermined condition to the terminal 107 together with the identification information associated with the baggage image. The predetermined condition may be, for example, that the degree of similarity falls within a predetermined number of higher ranks. The worker can identify the identification information of the baggage 250 from which the tag 260 has been removed from the baggage image and the identification information received by the terminal 107. Then, if the identification information of the baggage 250 is known, the worker operates the terminal 107 to inquire the management server 105 about the identification information and the corresponding baggage information, so that the information such as the container 270 to be loaded with the baggage 250 can be obtained. Can be obtained.

Further detailed examples of the processing executed by the image processing apparatus 106 with respect to the flow of baggage handling by the baggage handling system 100 according to the embodiment described above will be described below.

FIG. 3 is a diagram illustrating a block configuration of the image processing device 106 according to the embodiment. The image processing device 106 includes, for example, a control unit 301, a storage unit 302, and a communication unit 303. The control unit 301 may operate as, for example, an evaluation unit 311, a change unit 312, a reply unit 313, and the like. The storage unit 302 may store, for example, the above-mentioned baggage image and the identification information in association with each other. The communication unit 303 connects to the network 150 according to the instruction of the control unit 301, communicates with the terminal 107, and communicates with the reading device 103 via the controller 104. The details of each of these units and the details of the information stored in the storage unit 302 will be described later.

FIG. 4 is a diagram illustrating the baggage management information 400 according to the embodiment. In the entry of the baggage management information 400, baggage information including, for example, an airline, a flight, a gate, and identification information is registered. The airline is, for example, information indicating the airline of the flight carrying the entry and the corresponding baggage 250. A flight is information indicating, for example, an airmail carrying an entry and a corresponding baggage 250. The gate is information indicating, for example, the gate at which the airmail carrying the entry and the corresponding baggage 250 is stopped. The identification information is, for example, the identification information issued for the baggage 250 corresponding to the entry. The identification information may be issued, for example, when the baggage 250 corresponding to the entry is checked in at the check-in counter 201, and the baggage 250 may be attached with a tag 260 containing the identification information. In the example of FIG. 4, the entry of the baggage management information 400 includes, but is not limited to, the airline, the flight, the gate, and the identification information. For example, the entry for baggage management information 400 may include other information such as the name of the baggage owner, the gender of the owner, and the like.

FIG. 5 is a diagram illustrating an operation flow of a baggage image generation process according to an embodiment. For example, the control unit 301 of the image processing device 106 may start the operation flow of FIG. 5 when an execution instruction of the baggage image generation process is input.

In S501, the control unit 301 determines whether or not a set of a photographed image of the baggage 250 taken from the reading device 103 and the identification information read from the tag 260 attached to the baggage 250 has been received. When the set of the captured image and the identification information is not received from the reading device 103 (S501 is NO), the flow repeats the process of S501. On the other hand, when the set of the captured image and the identification information is received from the reading device 103 (YES in S501), the flow proceeds to S502.

In S502, the control unit 301 generates a baggage image obtained by extracting an area in which the baggage 250 is captured from the photographed image based on the received photographed image and the background image stored in the storage unit 302. The background image may be, for example, an image taken by the photographing device 132 of the conveyor 101 as a background at a timing when the baggage 250 is not included in the angle of view, and is stored in advance in the storage unit 302 of the image processing device 106. You may be.

FIG. 6 is a diagram illustrating an example of the generation of the baggage image executed in S502. FIG. 6 shows a background image 601 obtained by photographing the conveyor 101 as a background, a photographed image 602 showing the baggage 250, and a baggage image 603. Based on the captured image 602 received from the reading device 103 and the background image 601 read from the storage unit 302, the control unit 301 removes the region where the background conveyor 101 is captured from the captured image 602, and the region where the baggage 250 is captured. The baggage image 603 extracted from the above is generated. In the baggage image 603 extracted in this way, most of the pixels in the image serve as information indicating the baggage 250, so that the accuracy of the image search can be improved. Further details of the generation of the baggage image 603 will be illustrated below.

FIG. 7 is a diagram illustrating a flow of generating a baggage image according to an embodiment. In (1) of FIG. 7, the control unit 301 of the image processing device 106 reads out the captured image of the baggage 250 captured by the imaging device 132 of the reading device 103 from the storage unit 302. In (2), the control unit 301 reads the background image from the storage unit 302, and obtains a difference image of the brightness between the background image and the photographed image in which the baggage 250 is captured. It should be noted that (2) to (7) of FIG. 7 are shown by black-and-white images showing brightness, in which black is dark and white is bright.

In (3), the control unit 301 removes noise such as a small white region from the obtained difference image, and then thickens the outline of the white region. In (4), the control unit 301 generates an edge image obtained by extracting an edge from the captured image in which the baggage 250 shown in (1) is captured, and creates a region where white pixels overlap between the edge image and the image obtained in (3). Extract.

In (5), the control unit 301 removes noise such as a small white region from the image obtained in (4), and then thickens the outline of the white region. In (6), the control unit 301 removes a small black region surrounded by a white region from the image obtained in (5). In (7), the control unit 301 rotates the image obtained in (6) by 180 degrees and superimposes it on the image before rotation. In (8), the control unit 301 performs rectangular fitting on the image obtained in (7), cuts out a baggage area, and saves it as a baggage image. In the fitting, the control unit 301 may cut out, for example, the smallest rectangular area surrounding the outer shape of the white area as a baggage image.

For example, as described above, the control unit 301 of the image processing device 106 can generate a baggage image from the captured image of the baggage 250 captured by the imaging device 132.

The acquisition of the baggage image described in FIG. 7 is an example, and the embodiment is not limited to this. For example, in another embodiment, the control unit 301 recognizes the area in which the baggage 250 is captured by a machine learning method using the captured image in which the area of the baggage 250 is specified and the background image as teacher data. May be created and a baggage image may be generated by the created learner.

Subsequently, in S503 of FIG. 5, the control unit 301 stores the baggage image generated in S502 in the storage unit 302 in association with the identification information received in S501, and the flow returns to S501. In one example, the control unit 301 may store the baggage image and the identification information in the storage unit 302 in association with each other by using the file name when storing the baggage image as the identification information. In another embodiment, the control unit 301 may manage the baggage image and the identification information in association with each other by registering an entry in which the baggage image and the identification information are associated with each other in the database. The set of the baggage image and the identification information stored in the storage unit 302 each time S503 of FIG. 5 is executed can be used as a search target for the image search, as will be described later.

(Image search process for baggage images)
Subsequently, the image search process of the baggage image according to the embodiment will be described. The image search process for the baggage image may be executed, for example, to identify the identification information of the baggage 250 when the baggage 250 with the tag 260 removed is found. In the image search, for example, in the image search, the baggage image showing the baggage whose color is clearly different from the baggage shown in the image used as the search key is ranked higher as the similar baggage image, for example, as in the image search based on the above-mentioned local feature amount. May appear in. Therefore, in the image search process described below, the control unit 301 evaluates the degree of color approximation between the baggage image used as the search key and the baggage image stored in the storage device, and the colors of the two images are similar. If not, lower the similarity rating of the baggage image. As a result, the accuracy of the image search is improved, and it becomes possible to efficiently identify the baggage image in which the baggage matching the baggage in the image input as the search key is shown.

FIG. 8 is a diagram illustrating an operation flow of the image search process according to the embodiment. For example, the control unit 301 of the image processing device 106 may start the operation flow of FIG. 8 when an execution instruction of the image search process of the baggage image is input.

In S801, the control unit 301 receives an image search request from the terminal 107. For example, suppose a worker in charge of loading baggage 250 into a container 270 in make area 290 finds baggage 250 with the tag 260 removed. In this case, the worker may use the terminal 107 to take an image of the baggage 250 from which the tag 260 has been removed, and send an image search request including the taken image from the terminal 107 to the image processing device 106. When the control unit 301 has not received the image search request from the terminal 107 in S801 (S801 is NO), the flow repeats the process of S801. On the other hand, when the control unit 301 receives the image search request from the terminal 107 in S801 (YES in S801), the flow proceeds to S802.

In S802, the control unit 301 acquires an image of the baggage 250 from which the tag 260 has been removed from the image search request, and uses the acquired image as a search key to search for a plurality of baggage images stored in the storage unit 302. do. The plurality of baggage images to be searched stored in the storage unit 302 may be, for example, the baggage images stored in the storage unit 302 in S503, and are associated with the identification information. For image search, for example, SIFT (Scale-Invariant Feature Transform), SURF (Speeded-Up Robust Features), BRISK (Binary Robust Invariant scalable Keypoints), BRIEF (Binary Robust Invariant Elementary Features), KAZE features, AKAZE (Accelerated-KAZE) Image comparisons based on local features such as features can be used. The control unit 301 may evaluate the similarity of images by, for example, image comparison based on these local features. Then, the control unit 301 may sort the baggage images in descending order of similarity, for example. For example, in the image comparison based on the local feature amount, the similarity can be expressed by a score, and the control unit 301 may rearrange the baggage images in order from the baggage image having the highest score.

Subsequently, in S803, the control unit 301 evaluates the degree of color approximation. For example, the control unit 301 determines a feature value representing the color feature of the baggage from an image of the baggage 250 included in the image search request. Further, the control unit 301 determines a feature value representing the color feature of the baggage for each of the plurality of baggage images stored in the storage unit 302. Then, the control unit 301 compares the feature value obtained from the image of the baggage 250 with the feature value of each baggage image stored in the storage unit 302, and determines the degree of approximation as to whether or not the colors are close to each other. evaluate.

As the feature value that can be used for evaluating the degree of approximation of color, for example, hue can be used. Hue is, for example, hue and hue, and is one of the three attributes of color together with saturation and lightness. Hue can be preferably used as an index for color comparison. Hereinafter, an example of evaluating the degree of color approximation using hue as an example will be described, but the embodiment is not limited to this, and the degree of color approximation may be evaluated using other color components.

In one embodiment, the control unit 301 may calculate a representative value representing the hue of the pixels in the image and use it as a feature value representing the color feature of the baggage. In one example, the representative value may be a value obtained by averaging the hues of the entire image. For example, the average hue can be calculated as follows.

FIG. 9 is a diagram illustrating the color wheel. As illustrated in FIG. 9, the hue circle is formed by arranging the whole hues in an orderly circle, and the hue can be represented by a value by an angle θ on the hue circle. In the following, the angle θ on the hue circle may be referred to as a hue value. The hue of one pixel in the image can be represented by an angle θ. Furthermore, for example, the average value of the hues of the pixels included in the image can be obtained by the following calculation.

First, the sum of the X-axis component and the Y-axis component of the hue of each pixel in the image is obtained by the following formula.

Here, θ is the hue of the pixels in the image. S is the sum of the Y-axis components of the hues of all the pixels in the image. Further, C is the sum of the X-axis components of the hues of all the pixels in the image. Then, the arctangent of S / C is calculated to obtain θ _average .

However, the arctangent output takes a value in the range of −90 ° <θ <90 °. Therefore, the average value of the hue is determined using the respective codes of S and C. For example, it is assumed that θ _average is 45 °. In this case, if the signs of S and C are positive, the average value of hue can be determined to be 45 °. On the other hand, if the signs of S and C are both negative, the average value of hue can be determined to be 225 °. Similarly, it is assumed that the θ _average is −45 °. In this case, if the sign of S is negative and the sign of C is positive, the average value of hue can be determined to be −45 °, that is, 315 °. On the other hand, if the sign of S is positive and the sign of C is negative, the average value of hue can be determined to be −45 °, that is, 135 °. For example, as described above, the average value of hue can be obtained.

Then, in the process of S803, the control unit 301 has a large angle difference between, for example, the average value of the hue obtained from the image of the baggage 250 and the average value of the hue of the baggage image stored in the storage unit 302. However, if the predetermined determination conditions are satisfied, it is determined that the colors are close to each other. On the other hand, the control unit 301 determines, for example, the magnitude of the angle difference between the average value of the hue obtained from the image of the baggage 250 and the average value of the hue of the baggage image stored in the storage unit 302. If the determination conditions are not satisfied, it may be determined that the colors are not similar. As an example, in the control unit 301, the magnitude of the angle difference between the average value of the hue obtained from the image of the baggage 250 and the average value of the hue of the baggage image stored in the storage unit 302 is a predetermined angle. If it is less than, it may be determined that the colors are close. On the other hand, for example, when the magnitude of the angle difference between the average value of the hue obtained from the image of the baggage 250 and the average value of the hue of each baggage image stored in the storage unit 302 is equal to or larger than a predetermined angle. If so, the control unit 301 may determine that the colors are not close. The predetermined angle may be, for example, an angle in the range of 45 ° to 135 °, and in one example, it may be 90 °.

In S804, the control unit 301 changes the baggage image determined that the colors are not close to each other so that the similarity obtained by the image search of S802 is low. For example, the control unit 301 may lower the score for the baggage image determined to be not close in color by subtracting a predetermined value from the score representing the similarity obtained as a result of the image search. As the predetermined value, for example, a value can be used so that the baggage images determined to be not close in color do not rank high in the order of arrangement according to the degree of similarity.

Subsequently, in S805, the control unit 301 returns the baggage image having the similarity satisfying a predetermined condition and the identification information of the baggage image to the terminal 107 as a search result in the changed similarity. The flow returns to S801. For example, the control unit 301 may return a predetermined number of baggage images having a high degree of similarity to the terminal 107 as a search result together with the identification information.

Therefore, the worker can easily find the baggage image that matches the baggage 250 with the tag 260 removed from the baggage image received by the terminal 107, and acquires the identification information associated with the baggage image. be able to. Then, for example, the worker notifies the management server 105 of the identification information associated with the baggage image from the terminal 107, so that the information of the entry of the baggage management information 400 associated with the identification information is transmitted from the management server 105. Can be obtained. Therefore, the worker can quickly identify, for example, the container 270 to be loaded with the baggage 250 from which the tag 260 has been removed.

Further, according to the operation flow of FIG. 8, the baggage image whose color does not resemble the image of the search key is evaluated to have a low degree of similarity. You can find it efficiently.

FIG. 10 is a diagram illustrating changes in the evaluation results of the similarity of the operation flow of FIG. 8 described above. As shown in (1), the control unit 301 executes an image search using an image of the baggage 250 that has been removed from the tag 260 as a search key (S802 in FIG. 8). In (2), the baggage images are arranged in order from the one with the highest similarity as the search result, and in the example of FIG. 10, the similarity is high in the order of baggage image A> baggage image B> baggage image C> baggage image D. Here, the baggage 250 shown in the baggage image A and the baggage image B are similar in shape but completely different in color, and are clearly different from the baggage in the image input as the search key. Thus, in local feature-based comparisons performed based on the distribution of changes in brightness, distinctly different colored baggage 250s may appear higher as similar baggage 250s.

However, in the embodiment, as shown in (3), the control unit 301 evaluates the degree of approximation of the colors and lowers the degree of similarity of the baggage images whose colors are not similar (S803 and S804 in FIG. 8). As a result, as shown in (4), the baggage image A and the baggage image B having dissimilar colors can be placed in the lower position, and the baggage image C having a high degree of similarity and similar colors can be placed in the upper position. Then, the control unit 301 may return a predetermined number of higher-level baggage images to the terminal 107 as a search result together with the identification information. As a result, the worker can easily find the baggage image that matches the baggage 250 with the tag 260 removed from the baggage image received by the terminal 107, and obtains the identification information associated with the baggage image. can do.

As described above, according to the embodiment, in order to evaluate the degree of color approximation, the accuracy of the image search is improved, and the baggage image showing the baggage matching the baggage shown in the image input as the search key is efficiently displayed. It will be possible to identify.

In the embodiment described above, an example in which an average value is used as a representative value of the hue of an image has been described, but the embodiment is not limited to this. For example, in another embodiment, other statistics such as mode can be used. Further, for example, in the above-described embodiment, an example of obtaining an average value of hue in the entire image has been described, but the embodiment is not limited to this. For example, when the worker takes a picture of the baggage 250 from which the tag 260 has come off with a photographing device, the baggage is likely to be moved to the center of the image and taken. Further, for example, in the baggage image generated by the process of S502, it is highly possible that the baggage is placed in the center of the image. That is, in these images, it is highly possible that the baggage color information can be obtained from the pixels near the center without including unnecessary information such as the background. Therefore, in another embodiment, the control unit 301 may obtain a representative value such as an average value of hues from pixels included in a region of a predetermined size near the center instead of the entire image.

(First modification)
Subsequently, a first modification of the above-described embodiment will be described. For example, when performing an image search on a baggage image, the evaluation accuracy of the similarity is improved when the orientation of the baggage 250 shown in the image is approximately the same for the image used as the search key and the baggage image. Therefore, in the first modification, the control unit 301 generates a plurality of baggage images from the baggage images.

FIG. 11 is a diagram illustrating the generation of a plurality of baggage images from the baggage images according to the first modification. In the example of FIG. 11, the control unit 301 rotates the baggage image 1101 by 90 degrees to generate three baggage images 1102. It should be noted that the generation of a plurality of baggage images from the baggage images according to the embodiment is not limited to this. For example, in another embodiment, the control unit 301 rotates the baggage image at another angle to generate a plurality of baggage images, such as rotating the baggage image by 30 degrees until it makes a full turn to generate a plurality of baggage images. May be good. Alternatively, in yet another embodiment, the control unit 301 may generate a plurality of baggage images, for example, by changing the scale of the baggage image or changing both the orientation and the scale.

FIG. 12 is a diagram illustrating an operation flow of a baggage image generation process according to the first modification. For example, the control unit 301 of the image processing device 106 may start the operation flow of FIG. 12 when an execution instruction of the baggage image generation process is input.

Subsequent processing of S1201 and S1202 may correspond to processing of S501 and S502, and the control unit 301 may execute the same processing as S501 and S502 in the processing of S1201 and S1202, for example.

In S1203, the control unit 301 generates a plurality of baggage images from the generated baggage images. For example, as described in FIG. 11, the control unit 301 may change the orientation of the baggage image to generate a plurality of baggage images.

In S1204, the control unit 301 stores each of the baggage image generated in S1202 and the plurality of baggage images generated from the baggage image in the storage unit 302 in association with the identification information received in S1201, and the flow returns to S1201. .. In one example, the control unit 301 associates the baggage image and the plurality of baggage images with the identification information by setting the file name for storing each of the baggage image and the plurality of baggage images as a name including the identification information. It may be stored in the storage unit 302.

As described above, according to the operation flow of FIG. 12, the control unit 301 generates a plurality of baggage images from the baggage images, and stores the plurality of baggage images in the storage unit 302 in association with the identification information. Therefore, for example, when the image search is executed in the operation flow of FIG. 8 described above, there is a possibility that the baggage image in which the direction of the baggage 250 shown is close to the image used as the search key is stored in the storage unit 302. Can be enhanced. As a result, the accuracy of image search can be improved.

(Second modification)
For example, as described with reference to FIG. 6, in the embodiment, the control unit 301 photographs the captured image of the baggage 250 captured by the photographing device 132 and the conveyor 101 at the timing when the baggage 250 does not enter the angle of view. A baggage image is generated based on the background image obtained.

However, for example, the surface of the conveyor 101 may be scratched by metal exposed on the outer surface of the baggage 250, or a sticker such as a tag 260 may stick to the conveyor 101. For example, in such a case, the background area other than the area where the baggage 250 is reflected in the photographed image may be different from the background imaged in the background image.

FIG. 13 is a diagram illustrating a case where the background of an image taken by the photographing device 171 is significantly different from the background of an image taken at another timing. In FIG. 13A, the tag 260 is attached to the conveyor 101 in the captured image 1301, and the tag 260 is reflected in the image. Then, it is assumed that the difference is taken between the captured image 1301 and the background image 1302 in which the tag 260 attached to the conveyor 101 is not shown. In this case, in addition to the area where the baggage 250 is captured, a large difference appears in the area where the tag 260 is captured, which may be an obstacle in defining the region where the baggage 250 is captured.

Therefore, in the modified example, as shown in FIG. 13B, the image processing apparatus 106 stores a plurality of background images 1302 in the storage unit 302 in which the conveyor 101 is photographed at various timings when the baggage 250 is not included in the angle of view. I remember. Then, for example, when the control unit 301 receives the captured image 1301 captured by the imaging device 132 from the reading device 103, the control unit 301 has a background image out of a plurality of background images 1302 according to the background of the received captured image 1301. Select. In one example, the control unit 301 generates a difference image obtained by taking the difference in brightness between the received captured image 1301 and each of the plurality of background images 1302, and the background in which the difference image having the smallest total difference is obtained. An image may be selected and used in subsequent processing. In the example shown in FIG. 13B, the background image 1302 in which the tag 260 appears is selected from the plurality of background images 1302.

As illustrated above, by taking a plurality of background images at various timings and storing them in the storage unit 302, it is possible to improve the extraction accuracy of the area where the baggage 250 is captured in the generation of the baggage image. The embodiment is not limited to the above examples. For example, a plurality of background images taken under various shooting conditions can be stored in the storage unit 302.

As an example, a plurality of background images having different light conditions at the time of shooting can be stored in the storage unit 302. The light conditions differ depending on, for example, when the photographing device 132 is installed in a building in which natural light is incident, the time of day, the weather, the angle of incidence of natural light according to the season, and the like. Therefore, the storage unit 302 may store a plurality of background images according to the time, weather, and season.

FIG. 14 is a diagram showing exemplary background image management information 1400. In the background image management information 1400, for example, an entry associated with a date, a weather, a time, and a background image is registered. The date is, for example, the date on which the background image corresponding to the entry was taken. The weather is, for example, the weather when the background image corresponding to the entry was taken. The time is, for example, the time when the background image corresponding to the entry was taken. The background image is, for example, a background image taken under the shooting conditions indicated by the entry date, weather, and time. Then, in one embodiment, when the control unit 301 generates the baggage image, the control unit 301 takes a background image taken at a season, a weather, and a time close to the season, the weather, and the time when the photographed image showing the baggage 250 is taken. May be obtained from the background image management information 1400. In one example, the control unit 301 captures the captured image on a month and day that is close to the shooting date within a predetermined date, the shooting time is close within a predetermined time, and the weather is similar. The background image taken in 1400 may be acquired from the background image management information 1400 and used. Hereinafter, the operation flow of the baggage image generation process according to the second modification will be described.

FIG. 15 is a diagram illustrating an operation flow of a baggage image generation process according to a second modification. For example, the control unit 301 of the image processing device 106 may start the operation flow of FIG. 15 when an execution instruction of the baggage image generation process is input.

In S1501, the control unit 301 determines whether or not a set of a photographed image of the baggage 250 taken from the reading device 103 and the identification information read from the tag 260 attached to the baggage 250 has been received. When the set of the captured image and the identification information is not received from the reading device 103 (S1501 is NO), the flow repeats the process of S1501. On the other hand, when the set of the captured image and the identification information is received from the reading device 103 (YES in S1501), the flow proceeds to S1502.

In S1502, the control unit 301 reads out a plurality of background images stored in the storage unit 302. For example, as shown in FIG. 13, the control unit 301 may read out a plurality of background images showing different backgrounds. Further, for example, in another embodiment, the control unit 301 reads out the season, the weather, and the background image taken at the season, the weather, and the time of the captured image received in S1501 from the background image management information 1400. May be good.

In S1503, the control unit 301 selects a background image to be used for generating the baggage image from the plurality of background images. For example, the control unit 301 may take a difference in the brightness of each pixel between the received captured image and each of the plurality of background images, select a background image having a small total difference, and use it in the subsequent processing. ..

Subsequent processing of S1504 and S1505 may correspond to, for example, processing of S502 and S503, and the control unit 301 may execute the same processing as S502 and S503 in S1504 and S1505.

According to the operation flow of FIG. 15 above, an appropriate background image is selected according to the received captured image, so that the accuracy of cutting out the area where the baggage 250 is captured when the baggage image is generated from the captured image is improved. be able to.

It is possible to execute the image search process described in FIG. 8 for the baggage image stored in the storage unit 302 in the first modification and the second modification, and the identification information and the set. ..

Although the embodiments have been illustrated above, the embodiments are not limited thereto. For example, the above-mentioned operation flow is an example, and the embodiment is not limited thereto. When possible, the operation flow may be executed by changing the order of processing, may include additional processing, or may omit some processing. For example, in the operation flow of FIG. 15, instead of the processing of S1505, the processing of S1203 and S1204 described in FIG. 12 is executed, a plurality of baggage images are generated from the baggage images, and stored in association with the identification information. You may.

Further, in the above-described embodiment, an example in which a barcode in which identification information is encoded is printed on the tag 260 is described, but the embodiment is not limited to this. For example, in another embodiment, the identification information may be encoded as a two-dimensional code. Alternatively, the tag 260 may be an RFID (radio frequency identifier) tag, and the reading device 103 may include an RFID reader and read the identification information by short-range wireless communication.

Further, in the above-described embodiment, the reading device 103 acquires the captured image and the identification information of the baggage 250 and notifies the image processing device 106, but the embodiment is not limited to this. .. For example, when the baggage 250 is checked in at the check-in counter 201, the terminal installed in the check-in counter photographs the baggage 250, and the photographed image is associated with the identification information issued to the baggage 250. The image processing device 106 may be notified.

Further, in the above-described embodiment, the baggage is described as an example, but the embodiment is not limited to this, and the embodiment may be applied to other luggage. For example, the management of packages according to the above-described embodiment may be used for sorting packages in a distribution center such as a distribution center.

In the above-described embodiment, for example, in the processing of S802, the control unit 301 of the image processing device 106 operates as the evaluation unit 311. Further, for example, in the processing of S803 to S804, the control unit 301 of the image processing device 106 operates as the change unit 312. For example, in the process of S805, the control unit 301 of the image processing device 106 operates as the reply unit 313.

FIG. 16 is a diagram illustrating a hardware configuration of a computer 1600 for realizing the image processing device 106 according to the embodiment. The hardware configuration for realizing the image processing device 106 of FIG. 16 includes, for example, a processor 1601, a memory 1602, a storage device 1603, a portable storage medium interface 1604, a communication interface 1606, and an input / output interface 1607. The processor 1601, the memory 1602, the storage device 1603, the portable storage medium interface 1604, the communication interface 1606, and the input / output interface 1607 are connected to each other via, for example, the bus 1608.

The processor 1601 may be, for example, a single processor, a multiprocessor, or a multicore. The processor 1601 uses the memory 1602 to execute, for example, a program describing the procedure of the operation flow of FIG. 5, FIG. 8, FIG. 12, or FIG. Provides the functionality of. For example, the processor 1601 may operate as the evaluation unit 311, the change unit 312, and the reply unit 313 described above by executing the program.

The memory 1602 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 1603 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device. RAM is an abbreviation for Random Access Memory. ROM is an abbreviation for Read Only Memory.

The portable storage medium interface 1604 accesses the portable storage medium 1605 according to the instructions of the processor 1601. The portable storage medium 1605 is, for example, a semiconductor device (USB memory or the like), a medium to which information is input / output by magnetic action (magnetic disk or the like), a medium to which information is input / output by optical action (CD-ROM, etc.). It is realized by DVD etc.). USB is an abbreviation for Universal Serial Bus. CD is an abbreviation for Compact Disc. DVD is an abbreviation for Digital Versatile Disk.

The storage device 1603 of the image processing device 106 may store, for example, the baggage image and the identification information in association with each other. The memory 1602, the storage device 1603, the portable storage medium 1605, and a combination thereof are examples of the storage unit 302 described above.

The communication interface 1606 transmits and receives data via the network according to the instructions of the processor 1601. The communication interface 1606 is an example of the above-mentioned communication unit 303. The input / output interface 1607 may be, for example, an interface between an input device and an output device. The input device is, for example, a device such as a keyboard or a mouse that receives an instruction from a user. The output device is, for example, a display device such as a display and a printing device such as a printer.

FIG. 17 is a diagram illustrating a hardware configuration of a computer 1700 for realizing the terminal 107 according to the embodiment. The terminal 107 may be, for example, a tablet terminal, a smartphone, a mobile phone, or the like. The hardware configuration for realizing the terminal 107 of FIG. 17 includes, for example, a processor 1701, a memory 1702, a communication device 1703, an input device 1704, a display device 1705, and a photographing device 171. The memory 1702, the communication device 1703, the input device 1704, the display device 1705, and the photographing device 171 may be connected to the processor 1701 via, for example, a signal line 1610 such as a bus.

The processor 1701 provides some or all of the functions of the terminal 107 by, for example, executing a program describing the procedure of the process executed by the terminal 107 by using the memory 1702.

The memory 1702 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The communication device 1703 transmits / receives data to / from the management server 105 and the image processing device 106 via the network according to the instruction of the processor 1701, for example. The input device 1704 may be, for example, an electrostatic touch panel, a pressure sensitive touch panel, or the like. The display device 1705 may be, for example, a display. In one example, the user may input via an input device 1704 such as a touch panel arranged along the display screen of the display device 1705, for example.

The imaging device 171 is, for example, a camera and may be used to capture an image of the baggage 250.

Each program according to the embodiment may be stored in advance in, for example, the storage unit 302 and the memory 1702. Alternatively, each program according to the embodiment may be provided from a server such as a program server via a network.

Further, the hardware configuration for realizing the image processing device 106 and the terminal 107 described with reference to FIGS. 16 and 17 is an example, and the embodiment is not limited thereto. For example, some or all of the functions of the image processing device 106 and the terminal 107 described above may be implemented as hardware by FPGA, SoC, or the like. FPGA is an abbreviation for Field Programmable Gate Array. SoC is an abbreviation for System-on-a-chip.

In the above, some embodiments will be described. However, the embodiments are not limited to the above embodiments and should be understood to include various variants and alternatives of the above embodiments. For example, it will be understood that various embodiments can be embodied by transforming the components within a range that does not deviate from the purpose and scope. It will also be appreciated that various embodiments can be implemented by appropriately combining the plurality of components disclosed in the above-described embodiments. Further, various embodiments are implemented by removing or replacing some components from all the components shown in the embodiments, or by adding some components to the components shown in the embodiments. Those skilled in the art will understand that it can be done.

100 Baggage handling system 101 Conveyor 102 Conveyor control device 103 Reading device 104 Controller 105 Management server 106 Image processing device 107 Terminal 131 Barcode reader 132 Imaging device 150 Network 171 Imaging device 201 Check-in counter 202 Baggage inspection device 250 Baggage 260 Tag 270 Container 290 Make area 301 Control unit 302 Storage unit 303 Communication unit 311 Evaluation unit 312 Change unit 313 Reply unit 1600 Computer 1601 Processor 1602 Memory 1603 Storage device 1604 Portable storage medium interface 1605 Portable storage medium 1606 Communication interface 1607 Input / output interface 1608 Bus 1610 Signal line 1700 Computer 1701 Processor 1702 Memory 1703 Communication device 1704 Input device 1705 Display device

Claims (7)

When an image search request is received, the image search is performed on a plurality of baggage images showing different packages, each of which is associated with identification information for identifying the owner of the baggage. An image search is performed using the image of the baggage included in the request as a search key, and the similarity between each of the plurality of baggage images and the image of the baggage included in the image search request is evaluated.
The degree of similarity of the baggage image determined that the color of each of the plurality of baggage images and the color of the image showing the baggage included in the image search request do not satisfy the predetermined determination conditions is changed.
The changed baggage image having the similarity satisfying a predetermined condition and the identification information associated with the baggage image satisfying the predetermined condition are returned.
An image processing program that causes an image processing device to execute processing. A predetermined determination condition is satisfied when the magnitude of the angle difference in the hue circle between the hue value representing the image in which the luggage is captured and the hue value representing each of the plurality of luggage images is equal to or greater than a predetermined angle. Judge not,
The image processing program according to claim 1, wherein the processing is further executed by an image processing apparatus. When the first identification information read by the reading device from the tag attached to the first baggage flowing on the conveyor and the first photographed image of the first baggage photographed by the photographing device are received. A region in which the first luggage is captured is extracted from the first captured image based on the background image captured by the photographing apparatus at a timing when the luggage is not included in the angle of view and the first captured image. Generate a baggage image of 1 and
The first baggage image is associated with the first identification information, included in the plurality of baggage images as a search target of the image search, and stored in a storage unit.
The image processing program according to claim 1 or 2, wherein the processing is further executed by an image processing apparatus. By rotating the first baggage image, a second baggage image is generated.
The second baggage image is associated with the first identification information, included in the plurality of baggage images as a search target of the image search, and stored in the storage unit.
The image processing program according to claim 3, wherein the processing is further executed by the image processing apparatus. Among the plurality of background images taken by the photographing device at the timing when the baggage is not included in the angle of view, the background image having the smallest difference in brightness from the first photographed image is selected.
The image processing program according to claim 3 or 4, wherein the processing is further executed by the image processing apparatus. When an image search request is received, the image search is performed on a plurality of baggage images showing different packages, each of which is associated with identification information for identifying the owner of the baggage. An image search is performed using the image of the baggage included in the request as a search key, and the similarity between each of the plurality of baggage images and the image of the baggage included in the image search request is evaluated.
Among the plurality of baggage images, the color of the image showing the baggage included in the image search request and the baggage image determined that the color does not satisfy the predetermined determination condition are changed so as to be low in similarity.
The changed baggage image having the similarity satisfying a predetermined condition and the identification information associated with the baggage image satisfying the predetermined condition are returned.
An image processing method performed by an image processing apparatus, including the above. When an image search request is received, the image search is performed on a plurality of baggage images showing different packages, each of which is associated with identification information for identifying the owner of the baggage. An evaluation unit that executes an image search using the image of the baggage included in the request as a search key and evaluates the similarity between each of the plurality of baggage images and the image of the baggage included in the image search request.
Among the plurality of baggage images, the color of the image showing the baggage included in the image search request and the baggage image determined that the color does not satisfy the predetermined determination condition are changed so as to be low in similarity. Department and
A reply unit that returns a package image whose similarity after the change satisfies a predetermined condition and the identification information associated with the package image satisfying the predetermined condition.
Image processing equipment, including.

Images