JP7019223B1 - Information processing equipment, information processing system and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To capture a three-dimensional object in a system in which an information processing apparatus acquires two-dimensional image data generated by photographing under the assumption that a three-dimensional object is photographed. It prevents inappropriate processing for 2D image data from being performed when the assumption is not observed. SOLUTION: A server 3 acquires 3D image data generated by scanning by a LiDAR scanner 6 executed at the same time as shooting by a camera 5 from a terminal via a network N, and shoots using at least the 3D image data. A three-dimensionality determining unit 13 for determining the presence or absence of three-dimensionality of an object is provided, and it is determined whether or not the assumption that a three-dimensional object is photographed is observed. [Selection diagram] Fig. 2

Description

The present invention relates to an information processing device for processing captured and generated data, an information processing system including the information processing device and a terminal, and an information processing method by the information processing device.

Conventionally, under the assumption that a three-dimensional object is photographed, an information processing device acquires two-dimensional image data (photographed image data) generated by photography via a network and executes processing based on the two-dimensional image data. The system to do is known. For example, in Patent Document 1, a panoramic photograph (two-dimensional image data) generated by taking a used car is acquired by a server (information processing device) via a network, and then the panoramic photograph is stored, and the used car is ex post facto. A system that allows you to check the status of scratches on a panoramic photograph is described.

Japanese Unexamined Patent Publication No. 2018-97679

However, the system of Patent Document 1 has the following problems. That is, the panoramic photo stored in the server is used to check the condition of the used car's scratches, but the panoramic photo sent to the server is different from the one in which the image of the used car without scratches is displayed. It may have been generated by taking a picture of the car or a display screen showing an image of a used car without scratches. In such a case, if the panoramic photograph is stored as data for confirming the scratch condition, a situation may occur in which the scratch condition of the used car is confirmed from the illegal panoramic photograph. Therefore, although it is not appropriate to store the panoramic photograph in the above case, the system of Patent Document 1 has no means for avoiding this.

The present invention has been made to solve such a problem, and information on two-dimensional image data generated by photography performed under the assumption that a three-dimensional object is photographed via a network. In the system acquired by the processing device, it is possible to prevent inappropriate processing for 2D image data from being performed when the assumption that a three-dimensional object is photographed is not observed. The purpose.

In order to solve the above-mentioned problems, the present invention networks not only the two-dimensional image data based on the shooting result of the camera but also the three-dimensional image data generated by the scanning by the ranging sensor executed at the same time as the shooting by the camera. It is acquired from the terminal via the terminal, and at least the three-dimensional image data is used to determine the presence or absence of the three-dimensionality of the shooting target.

According to the present invention configured as described above, not only the two-dimensional image data but also the three-dimensional image data generated at the same time as this data is acquired, and then the three-dimensional image data is used to capture the three-dimensional object. Since the presence or absence of sex is determined, it is possible to determine whether or not the image taken by the camera is performed on a three-dimensional object. Therefore, when shooting is not performed with a three-dimensional object as the shooting target, that can be detected through the above discrimination. For example, the two-dimensional image data generated by shooting a three-dimensional object is used as the processing target. Instead of performing the usual processing, it is possible to perform processing for dealing with the fact that a three-dimensional object has not been photographed. As a result, it is possible to prevent inappropriate processing for the two-dimensional image data from being performed when the assumption that a three-dimensional object is photographed is not observed.

It is a figure which shows the structural example of the information processing system which concerns on one Embodiment of this invention. It is a block diagram which shows the functional structure example of the terminal and the server which concerns on one Embodiment of this invention. It is a figure which shows the screen for shooting. It is a figure used for the explanation of the process of a three-dimensionality discriminating part. It is a flowchart which shows the operation example of the information processing system which concerns on one Embodiment of this invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration example of the information processing system 1 according to the present embodiment. As shown in FIG. 1, the information processing system 1 includes a terminal 2 and a server 3. Both the terminal 2 and the server 3 can be connected to the network N including the Internet.

The server 3 is a server device having the terminal 2 as one of the clients. In FIG. 1, the server 3 is represented by one block, but the server 3 does not have to be a single server device, and may be configured by, for example, a plurality of server devices, or is a part of a predetermined system. It may be. The server 3 is managed and operated by an operating company, and provides services described later. The server 3 corresponds to the "information processing device" in the claims.

The terminal 2 is a smartphone in which a touch screen 4 (display panel + touch panel) is provided in a wide area on the entire surface of a tablet-type housing. In particular, the terminal 2 according to the present embodiment is equipped with a camera 5 and a LiDAR scanner 6, has a function of generating two-dimensional image data based on the result of shooting by the camera 5, and has a scan result by the LiDAR scanner 6. It has a function to generate 3D image data based on. The LiDAR scanner 6 corresponds to the "distance measuring sensor" in the claims.

The two-dimensional image data is two-dimensional image data (so-called photograph), and holds information about colors for each pixel arranged in a matrix. The three-dimensional image data is point cloud data in which pixels whose distances have been measured by the LiDAR scanner 6 are arranged in a three-dimensional Cartesian coordinate system. In the three-dimensional image data, the surface of each object within the scanning range of the LiDAR scanner 6 (the surface whose distance is to be measured by the LiDAR scanner 6) is represented as a set of points in the three-dimensional Cartesian coordinate system.

In the present embodiment, the terminal 2 is equipped with the LiDAR scanner 6, but other mechanisms used for generating three-dimensional image data (for example, a stereo camera or an optical distance measurement different from the LiDAR method). It may be configured to include a sensor or the like). Terminals having a function of generating three-dimensional image data (smartphones, tablet terminals not having a mobile phone function, notebook computers, etc.) such as the terminal 2 according to the present embodiment are widely used at present and will be used in the future. It is expected that it will become more and more popular.

A predetermined application (hereinafter referred to as "dedicated application") is installed in the terminal 2 in advance by using the application download system. The function of the dedicated application will be described later. Hereinafter, the owner of the terminal 2 is referred to as a "user".

FIG. 2 is a block diagram showing a functional configuration example of the terminal 2 and the server 3. As shown in FIG. 2, the terminal 2 includes a terminal-side control unit 10 and a terminal-side communication unit 11 as a functional configuration. Further, the server 3 includes a server-side communication unit 12, a three-dimensionality determination unit 13, and a processing unit 14 as functional configurations. Each of the above functional blocks 10 to 14 is configured to include a CPU, RAM, ROM, etc. of a computer, and is realized by the CPU reading a program stored in a storage medium such as a ROM, a hard disk, or a semiconductor memory into the RAM and executing the program. Will be done. In particular, the terminal-side control unit 10 is executed by the CPU reading and executing a dedicated application (however, including an OS, an API provided by the OS, and other accompanying programs) stored in a storage medium. However, the method of realizing each of the above functional blocks 10 to 14 is not limited to the method exemplified in this embodiment, and may be realized by the cooperation of software and hardware by another method, or a DSP or the like. It may be realized only by hardware. Further, the server 3 includes a server-side storage unit 16 as a storage means.

The terminal-side communication unit 11 communicates with a device connected to the network N according to a predetermined protocol. In the following description, it is assumed that the communication between the terminal 2 and the device connected to the network N is appropriately performed by the terminal-side communication unit 11, and detailed description thereof will be omitted. Further, the server-side communication unit 12 communicates with the device connected to the network N according to a predetermined protocol. In the following description, it is assumed that the communication between the server 3 and the device connected to the network N is appropriately performed by the server-side communication unit 12, and detailed description thereof will be omitted.

Hereinafter, an operation example of the information processing system 1 according to the present embodiment in two scenes, an evaluation scene and an identity verification scene, will be described.

<Evaluation scene>
First, an operation example of the information processing system 1 in the evaluation scene will be described. In the evaluation scene, it is assumed that the article owned by the user is first photographed by the camera 5 of the terminal 2. The goods owned by the user are three-dimensional objects that the user wants to be evaluated (the specific contents of the evaluation will be described later), for example, a branded bag, a branded accessory, and the like. Other brand products, such as laptops, smartphones, game consoles, and other electronic devices. Hereinafter, the goods owned by the user are referred to as target goods. The target article corresponds to the "three-dimensional object to be evaluated" in the claims.

After the user takes a picture of the target article, the two-dimensional image data based on the pictured result is transmitted to the server 3. The server 3 evaluates the target article based on the two-dimensional image data. In the present embodiment, "evaluation" means evaluating the value of the target article and attaching a purchase price (price). Although detailed explanation is omitted, after the purchase price is attached to the target goods, if the user consents, the target goods are purchased by the operating company at the purchase price, and the money corresponding to the purchase price is given to the user. Will be paid. In particular, in the present embodiment, the target goods purchased by the operating company are leased to the user. Therefore, the user can obtain money equivalent to the purchase price without handing over the target item to the operating company (however, the obligation to pay the lease fee, etc. arises).

As described above, in the evaluation scene, it is assumed that the user photographs the target article (three-dimensional object) owned by the user, and the two-dimensional image data generated by photographing the target article is the server 3. It is expected to be sent to. However, instead of taking a picture of the target article, the user takes a picture showing an image of the article or a display screen (for example, a screen displayed on a display of a personal computer, a screen of a television, etc.) displaying an image of the article. However, there is a possibility that two-dimensional image data based on the result of the shooting will be sent to the server 3. Hereinafter, a flat medium on which a two-dimensional image of an article is displayed (in addition to the above-mentioned photographs and display screens, for example, there are paper media other than photographs) is referred to as a "flat medium".

As an example, a user may take a picture of a photo of a catalog with an image of a branded bag, even though they do not actually own the branded bag, or the brand. There is a possibility that the corresponding part of the web page on which the image of the bag of the object is displayed is photographed, and the two-dimensional image data based on the result of the photographing is sent to the server 3.

The above user actions are expected to be performed by the user as follows. That is, as described above, in the information processing system 1, the target article is evaluated based on the two-dimensional image data, the purchase price is set for the target article, and the money corresponding to the purchase price is paid to the user. Based on this, the above user behavior is an image of a branded bag recorded in two-dimensional image data (an image based on the shooting result of a catalog or a web page, not an image based on the shooting result of a three-dimensional object). ) Is evaluated for goods (goods that are not originally owned by the user), and it is expected that money equivalent to the purchase price will be paid by the operating company.

In addition, although the user owns the target item, in order to make the purchase price as high as possible, a photograph showing an item of the same model as the target item and in a new state is displayed. It is also possible to shoot. Then, the information processing system 1 according to the present embodiment executes the following processing based on the possibility that the user takes the above actions.

First, prior to the generation of the two-dimensional image data, the user activates the dedicated application of the terminal 2 and instructs the touch screen 4 to display the article type selection screen. When the instruction is given, the three-dimensionality determination unit 13 is notified, and the three-dimensionality determination unit 13 causes the terminal side control unit 10 to display the article type selection screen on the touch screen 4 in response to the notification. The article type selection screen is a screen in which each of the article types is displayed in a list so that they can be selected. The article type means the types of articles that can be evaluated when they are roughly classified according to their functions and uses. In the present embodiment, a notebook computer, a smartphone, a game machine, a bag, a watch, and the like are prepared in advance as the item type. The user performs a predetermined operation to select one article type from the article types listed on the article type selection screen. Hereinafter, the product type selected by the user is referred to as "selected product type".

When the article type is selected by the user, the three-dimensionality determination unit 13 controls the terminal side control unit 10 to guide the shooting of the target article by a method according to the selected article type, and to execute the shooting by the camera 5. Various shooting screens G3 (FIG. 3) for instructing are displayed. The user instructs the execution of shooting by the camera 5 according to the guidance of the shooting screen G3. In response to this instruction, shooting by the camera 5 is executed, and further scanning by the LiDAR scanner 6 is executed, two-dimensional image data based on the shooting result of the camera 5 and three-dimensional based on the scan result of the LiDAR scanner 6. Image data is generated. The terminal side control unit 10 acquires a combination of the two-dimensional image data and the three-dimensional image data.

Each figure of FIG. 3 is a diagram showing an example of a shooting screen G3 displayed when the selected article type is a notebook computer. Hereinafter, the operation of the information processing system 1 will be described with reference to FIG. 3 with a specific example of the contents of the photographing screen G3. The three-dimensionality determination unit 13 controls the terminal-side control unit 10 to display the shooting screen G3A of FIG. 3A on the touch screen 4. On the shooting screen G3A of FIG. 3A, a sentence instructing to shoot the upper surface with the notebook computer closed is displayed along with an illustration depicting the shooting of the notebook computer in the closed state. ..

Further, a button B3A instructing the start of shooting is displayed on the shooting screen G3A of FIG. 3A, and the user operates the button B3A when the shooting is ready. In response to the operation of the button B3A, the terminal side control unit 10 enables the camera 5 and the LiDAR scanner 6 of the terminal 2 by using the API provided by the OS, and further touches the shooting screen G3B of FIG. 3B as a touch screen. Display in 4. In the shooting screen G3B of FIG. 3B, a sentence instructing that the frame line and the upper surface of the notebook computer, which is the target article, are within the frame line on the image shot by the camera of the terminal 2 to be shot. Is displayed on top of each other. Further, on the shooting screen G3B of FIG. 3B, a button B3B instructing execution of shooting is displayed.

Basically, the user places the target article on a desk or the like, and operates the button B3B in a state where the image of the target article is within the frame line. Then, shooting is executed by the camera 5 to generate two-dimensional image data, and further scanning is executed by the LiDAR scanner 6 to generate three-dimensional image data. As described above, it is assumed that a notebook computer, which is a three-dimensional object, is photographed by the user at this stage. On the other hand, at this stage, the user may take a flat medium (photograph, display screen, etc.) on which a (two-dimensional) image of a closed notebook computer is displayed.

When shooting (and scanning) is executed according to the user's instruction, the stereoscopic determination unit 13 cooperates with the terminal side control unit 10 to convert the shooting screen G3C of FIG. 3C into the touch screen 4. indicate. The shooting screen G3C of FIG. 3C is a screen that encourages shooting of a notebook computer in an open state. When the button B3C of the shooting screen G3C is operated, the terminal side control unit 10 enables the camera 5 and the LiDAR scanner 6 and then displays the screen corresponding to the shooting screen G3B of FIG. 3 (B). .. The user operates a predetermined button on the screen corresponding to the shooting screen G3B to execute shooting by the camera 5 and scanning by the LiDAR scanner 6. As a result, two-dimensional image data and three-dimensional image data are generated.

As described above, as illustrated with reference to FIG. 3, the three-dimensionality determination unit 13 controls the terminal side control unit 10 to guide the shooting of the target article by a method according to the selected article type. Further, the terminal side control unit 10 acquires a combination of the two-dimensional image data and the three-dimensional image data (hereinafter, referred to as “data combination”) generated in response to the shooting by the camera 5 and the scanning by the LiDAR scanner 6. As in the example of FIG. 3, there are cases where a plurality of data combinations are generated.

After acquiring one or more data combinations, the terminal-side control unit 10 transmits all the data combinations to the three-dimensionality determination unit 13 of the server 3. The three-dimensionality determination unit 13 receives all the data combinations and executes the following three-dimensionality determination processing for each data combination. Hereinafter, the three-dimensionality determination process executed by the three-dimensionality determination unit 13 for a certain data combination will be described.

First, the stereoscopic determination unit 13 acquires the two-dimensional image data included in the data combination. Next, the stereoscopic determination unit 13 analyzes the two-dimensional image data and recognizes the position of the image of the article of the selected article type in the two-dimensional image data. The position of the image of the article of the selected article type in the two-dimensional image data is performed by, for example, pattern matching using template data prepared in advance for each selected article type, or other image processing technology (image recognition technology). In addition, a model learned by deep learning or other machine learning method may be used.

As described above, the image of the article of the selected article type in the two-dimensional image data is not limited to the image formed by photographing the three-dimensional target article, but the flat medium on which the image of the article is displayed is photographed. It may be an image formed by this.

Next, the three-dimensionality determination unit 13 superimposes the two-dimensional image data and the three-dimensional image data so that the space reflected in the two-dimensional image data and the space reflected in the three-dimensional image data match, and in the three-dimensional image data, The area corresponding to the "area of the image of the article of the selected article type in the two-dimensional image data" is specified. Hereinafter, the specified area is referred to as a "specific area". Next, in the stereoscopic determination unit 13, the image belonging to the specific region in the three-dimensional image data is planarly continuous with the surrounding image (continuous on the plane), and is a flat image (unevenness or only limited unevenness). It is determined whether or not it is an image without. This determination corresponds to the process of "determining the presence or absence of three-dimensionality of the object to be photographed" in the claims.

Each figure of FIG. 4 is a diagram used for explaining the process of the three-dimensionality determination unit 13. Each figure of FIG. 4 schematically shows three-dimensional image data in an embodiment suitable for explanation. In each of the drawings of FIG. 4, reference numeral 20 indicates an optical center, reference numeral 21 indicates a radial region scanned by the LiDAR scanner 6, and reference numeral 22 indicates a specific region. When the user photographs and scans a three-dimensional object, as shown in FIG. 4A, a three-dimensional image J1 (point cloud) independent of the surrounding image is formed in the specific area. It will be. On the other hand, when the user photographs and scans a flat medium on which an image of an article is displayed, as shown in FIG. 4B, a flat image J2 that is planarly continuous with the surrounding image in a specific area. (Point cloud) will be formed.

Based on the above, the stereoscopic determination unit 13 determines that the object to be photographed has no stereoscopic effect when a flat image that is planarly continuous with the surrounding image is formed in the specific region. On the other hand, in other cases, the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality. Here, the imaged object is a concept including an object article when a three-dimensional object article is photographed and an image of the article when a flat medium on which an image of the article is displayed is photographed.

The method for determining the presence or absence of three-dimensionality by the three-dimensionality determining unit 13 is not limited to the above method. For example, the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality when a three-dimensional image (point cloud) independent of the surrounding image is formed in the specific region, and in other cases, the three-dimensional object is three-dimensional. It may be configured to determine that there is no sex. Other examples will be described later.

The above processing is the three-dimensional presence / absence determination processing. By executing the above three-dimensional presence / absence determination process, when a certain data combination is not generated by photographing a three-dimensional target object but is generated by photographing a flat medium. , In the three-dimensional presence / absence determination process for the data combination, it is determined that there is no three-dimensionality of the object to be photographed.

When the processing unit 14 determines that the image capture target does not have three-dimensionality by the three-dimensionality determination unit 13 (when there are a plurality of data combinations, even one data combination determined that the image capture target does not have three-dimensionality) If it exists), execute the following processing. That is, the processing unit 14 controls the terminal-side control unit 10 to indicate that a three-dimensional article has not been photographed and a photograph or a display screen may have been photographed, so that the image should be photographed again. Is displayed on the touch screen 4 to inform the user to that effect. In this case, the processing unit 14 does not execute the object evaluation-related processing (described later).

The process executed by the processing unit 14 when the three-dimensionality determining unit 13 determines that the object to be photographed does not have three-dimensionality is not limited to the above-mentioned processing (however, the object evaluation-related processing is not executed). For example, it may be configured to notify the person in charge of the operating company. The notification may be performed by any method such as displaying information on a display device connected to the server 3 or sending an e-mail to a person in charge. In response to the notification, the person in charge takes measures such as visually confirming the two-dimensional image data and the three-dimensional image data by other means, and re-verifying whether or not the three-dimensional article is photographed. be able to.

On the other hand, when the processing unit 14 determines that the image shooting target has three-dimensionality by the three-dimensionality determination unit 13 (when there are a plurality of data combinations, it is determined that all the data combinations have the three-dimensionality of the image shooting target. If), execute the following processing. That is, the processing unit 14 executes the object evaluation-related processing.

In the object evaluation-related processing, the processing unit 14 evaluates the value of the photographed target article and determines the purchase price. More specifically, the processing unit 14 analyzes the two-dimensional image data by the image processing technology (image recognition technology), and refers to the target article as the deepest type (for example, a model for a notebook computer and a model number for a branded bag). (Type in which the target article is most finely classified) is specified. Further, the processing unit 14 accesses a predetermined database and acquires the market price of the specified deepest type of article. Furthermore, the two-dimensional image data is analyzed by the processing unit 14 and the image processing technology (image recognition technology), and the state of appearance deterioration due to scratches, stains, wear, dents, missing parts, etc. can be determined, and the authenticity of the article can be determined. I do. At that time, template data can be prepared for each deepest type, and pattern matching can be performed using the template data corresponding to the specified deepest type.

Then, the processing unit 14 evaluates the value of the target article by reflecting the market price of the target article, the state of deterioration, and the authenticity, and determines the purchase price. The above is the object evaluation related processing. A model machine-learned by a predetermined machine learning method (for example, deep learning) may be used for the analysis of the two-dimensional image data. Further, the object evaluation-related processing by the processing unit 14 may be performed in cooperation with an external device, may include a judgment by a human being in a part thereof, and may include 3D image data in addition to 2D image data. May be used. Further, in this embodiment, although simplified for convenience of explanation, the information and data necessary for evaluating the value of the target article in the object evaluation-related processing and determining the purchase price are provided by the user or the operating company. Properly provided.

By performing the above processing in the evaluation scene, the following effects are obtained. That is, the three-dimensionality determination unit 13 acquires not only the two-dimensional image data but also the three-dimensional image data generated at the same time as this data, and determines the presence or absence of the three-dimensionality of the object to be photographed by using the two-dimensional image data. Then, when shooting is not performed with a three-dimensional object as the shooting target, that is, when the user is shooting a flat medium instead of shooting the target article, this is detected and the object evaluation related processing is executed. Instead of this, a process for dealing with the fact that a three-dimensional object is not photographed is executed. As a result, when the assumption that a three-dimensional object is photographed is not kept, an inappropriate process is performed in which the purchase price is determined based on the two-dimensional image data generated by photographing the planar medium. It is possible to prevent it from being stored, and by extension, it is possible to prevent money from being handed over to the user.

<Identity verification scene>
Next, an operation example of the information processing system 1 in the identity verification scene will be described. In the identity verification scene, a three-dimensional identity verification material owned by the user is photographed by the camera 5 of the terminal 2, two-dimensional image data based on the photographed result is transmitted to the server 3, and the server 3 converts the three-dimensional image data into two-dimensional image data. Based on this, identity verification related processing (details of the processing will be described later) is executed. Identity verification materials are, for example, a driver's license, passport, My Number card, and the like. The identity verification material corresponds to "an object in which information for identity verification is recorded" in the claims.

As for the identity verification material, the user can take a picture showing an image of another person's identity verification material or a display screen displaying an image of another person's identity verification material instead of the identity verification material owned by the user. There is sex. As an example, there is a possibility that an identity verification material of another person is illegally photographed by a camera, a photograph displaying an image of the identity verification material of another person is printed out or displayed on a display screen, and the photograph or the display screen is photographed. Based on this, the information processing system 1 executes the following processing.

That is, when the user instructs to take a picture in the identity verification scene, the terminal side control unit 10 enables the camera 5 and the LiDAR scanner 6 of the terminal 2 to take a picture with the camera 5 and scan with the LiDAR scanner 6. Acquire 3D image data and 3D image data. The terminal side control unit 10 transmits the acquired two-dimensional image data and the three-dimensional image data to the three-dimensionality discrimination unit 13.

The stereoscopic determination unit 13 receives two-dimensional image data and three-dimensional image data (data combination). Next, the three-dimensionality determination unit 13 executes a three-dimensionality determination process for the data combination to determine the presence or absence of the three-dimensionality of the image shooting target. In the three-dimensional presence / absence determination process, the same process as in the evaluation scene is executed.

Briefly, the stereoscopic determination unit 13 analyzes the two-dimensional image data and identifies the area of the image of the identity verification material in the two-dimensional image data. For example, if the identity verification material is a driver's license, the stereoscopic determination unit 13 analyzes the two-dimensional image data and identifies the area of the driver's license image in the two-dimensional image data. For example, the three-dimensionality determination unit 13 recognizes the type of identity verification material (driver's license, passport, My Number card, etc.) by user input or other means, and acquires template data according to the type (identity verification material). Template data for each type is prepared in advance), and pattern matching using the template data is performed to specify the area of the image.

Next, the three-dimensionality determination unit 13 superimposes the two-dimensional image data and the three-dimensional image data, and in the three-dimensional image data, the area (specific area) corresponding to the "image area of the identity verification material in the two-dimensional image data". To identify. Next, in the stereoscopic determination unit 13, the image belonging to the specific region of the three-dimensional image data is planarly continuous with the surrounding image (continuous on the plane), and is a flat image (unevenness or only limited unevenness). It is determined whether or not the image is (= the presence or absence of three-dimensionality of the object to be photographed). When the three-dimensionality determination unit 13 forms a flat image that is planarly continuous with the surrounding image in the specific region, the three-dimensionality determination unit 13 determines that the object to be photographed has no three-dimensionality. On the other hand, in other cases, the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality.

When the three-dimensionality determination unit 13 determines that the object to be photographed does not have three-dimensionality, the processing unit 14 executes the corresponding processing without executing the identification-related processing. In the present embodiment, the processing unit 14 notifies the person in charge of the operating company that it is determined that the object to be photographed does not have three-dimensionality. Based on this notification, the person in charge confirms the 2D image data and the 3D image data by visual inspection or other means, and re-verifies whether or not the three-dimensional identity verification document has been taken. I do.

On the other hand, when the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality, the processing unit 14 executes the identification-related processing. In the present embodiment, the processing unit 14 executes a process of registering the two-dimensional image data in a predetermined database in association with the user's identification information as the identity verification related process. The database is stored in the server-side storage unit 16. However, the above-mentioned identity verification-related processing is merely an example, and the processing unit 14 may be configured to execute different processing as the identity verification-related processing. As an example, the processing unit 14 performs optical character recognition of the image of the identity verification document of the two-dimensional image data, extracts the text of the name and address, and registers the text of the name and address in the database as the identity verification related processing. Check the consistency with the pre-registered information based on the text of the name and address.

By performing the above processing in the identity verification scene, as in the case of the identity verification scene, when the assumption that a three-dimensional object is photographed is not kept, the two-dimensional medium is photographed and generated. It is possible to prevent the improper process of registering the image data in the database from being executed.

Next, an operation example of the information processing system 1 will be described using a flowchart. In the following description, the operation of the server 3 represents an example of an information processing method by the server 3 (information processing device). FIG. 5 is a flowchart showing an operation example of the information processing system 1 in the evaluation scene. In FIG. 5, (A) shows the operation of the terminal 2, and (B) shows the operation of the server 3.

As shown in FIG. 5A, the user activates the dedicated application of the terminal 2 and instructs the touch screen 4 to display the article type selection screen (step SX1). In response to the instruction, the terminal-side control unit 10 and the three-dimensionality determination unit 13 cooperate to display the article type selection screen on the touch screen 4 (steps SA1 and SB1). When the article type is selected by the user (step SX2), the terminal side control unit 10 and the three-dimensionality determination unit 13 cooperate to guide the shooting of the target article by a method according to the selected article type, and the camera 5 Various shooting screens G3 (FIG. 3) for instructing the execution of shooting by the camera are displayed (steps SA2 and SB2).

When shooting (and scanning) is instructed by the user (step SX3) and two-dimensional image data and three-dimensional image data (data combination) are generated, the terminal side control unit 10 acquires one or more data combinations. (Step SA3). Next, the terminal-side control unit 10 transmits all the data combinations acquired in step SA3 to the three-dimensionality determination unit 13 (step SA4).

The three-dimensionality determination unit 13 receives all the data combinations (step SB3), and executes the following three-dimensionality determination process for each data combination (step SB4). As described above, in the three-dimensional presence / absence determination process, it is determined whether or not the object to be photographed has three-dimensionality.

The processing unit 14 determines whether or not the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality (step SB5). When the three-dimensionality determination unit 13 determines that the object to be photographed has three-dimensionality (step SB5: YES), the processing unit 14 executes object evaluation-related processing (step SB6). On the other hand, when the processing unit 14 determines that the object to be imaged does not have three-dimensionality (step SB6: NO), the three-dimensional object is photographed without executing the object evaluation-related processing. The process corresponding to the fact that the object has not been executed is executed (step SB7).

As described in detail above, the server 3 according to the present embodiment is generated not only by the two-dimensional image data based on the shooting result of the camera 5, but also by the scanning by the LiDAR scanner 6 executed at the same time as the shooting by the camera 5. The three-dimensional image data is acquired from the terminal 2 via the network N, and at least the three-dimensional image data is used to determine the presence or absence of the three-dimensionality of the image shooting target.

According to this configuration, not only the two-dimensional image data but also the three-dimensional image data generated at the same time as this data is acquired, and then the presence or absence of the three-dimensionality of the object to be photographed is determined using the three-dimensional image data. Therefore, it is possible to determine whether or not the image taken by the camera is performed on a three-dimensional object. Therefore, when shooting is not performed with a three-dimensional object as the shooting target, that can be detected through the above discrimination. For example, the two-dimensional image data generated by shooting a three-dimensional object is used as the processing target. Instead of performing the usual processing, it is possible to perform processing for dealing with the fact that a three-dimensional object has not been photographed. As a result, it is possible to prevent inappropriate processing for the two-dimensional image data from being performed when the assumption that a three-dimensional object is photographed is not observed.

Although one embodiment of the present invention has been described above, the above-described embodiment is merely an example of the embodiment of the present invention, whereby the technical scope of the present invention is limitedly interpreted. It should not be done. That is, the present invention can be implemented in various forms without departing from its gist or its main features.

For example, in the above embodiment, the present invention is applied to a system in which the server 3 performs object evaluation-related processing based on two-dimensional image data, and a system in which the server 3 performs identity verification-related processing based on two-dimensional image data. Explained. However, the system to which the present invention is applied is not limited to the system exemplified in the above embodiment. That is, the present invention has a function of acquiring two-dimensional image data generated by shooting a three-dimensional object from a terminal via a network and processing the two-dimensional image data on the premise that a three-dimensional object is shot. It can be widely applied to an information processing device or a system including such a terminal and an information processing device. In particular, the present invention can be widely applied to a system that executes different processes in place of object evaluation-related processes or identity verification-related processes.

For example, the present invention can be applied to a system in which the purchase price is simply presented as an assessed amount instead of paying money corresponding to the purchase price determined in the object evaluation-related processing. Further, for example, the present invention can be applied to a system in which an article purchased by an operating company is delivered from the user to the operating company by a predetermined means, instead of being kept in the hands of the user by leasing. Further, the present invention can be applied to a system that simply stores or publishes two-dimensional image data when the three-dimensionality of the object to be photographed is confirmed, instead of evaluating the target article based on the two-dimensional image data. be.

Further, for example, the three-dimensionality determination process by the three-dimensionality determination unit 13 may be performed by a method other than the method shown in the above embodiment. As an example, in the above embodiment, the specific area is defined based on the two-dimensional image data, but such a process may not be performed and the specific area may be a predetermined area. That is, in the evaluation scene and the identity verification scene, it is assumed that the relative position of the shooting target with respect to the terminal 2 at the time of shooting by the camera 5 (and scanning by the LiDAR scanner 6) is fixed to some extent. Specifically, it is assumed that the shooting is performed with the shooting target located in the center of the frame line of FIG. 3B, and the terminal 2 and the shooting target are separated from each other by about 30 to 90 cm. Will be done. Based on this, as a specific area, a region in which the shooting target is located in the three-dimensional image data when shooting is performed in a normal manner is predetermined, and the stereoscopic determination unit 13 is set to the predetermined specific region. It may be configured to determine whether or not the appearing image is a flat image that is two-dimensionally continuous with the surrounding image.

As another example, the following configuration may be used (however, the scene is assumed to be an evaluation scene for convenience of explanation). That is, three-dimensional template data (point cloud data) is prepared for each article type. Then, the three-dimensionality discrimination unit 13 performs pattern matching using the template data corresponding to the selected article type in the three-dimensionality discrimination processing, and an image (point cloud) having a degree of similarity with the template data of a certain degree or more is a three-dimensional image. Determine if it exists in the data. When such an image exists, the stereoscopic determination unit 13 determines that the photographing target is three-dimensional, and when it does not exist, it determines that the photographing target is not three-dimensional. As will be described later, the selection item type can be further subdivided (for example, a model can be used for a personal computer or a model number for a branded bag), and this configuration can be used as a selection item in this way. This is effective when the types are classified more finely. This is because the template data is prepared for each of the selected article types that are finely divided, and it is possible to determine with higher accuracy whether or not the object to be photographed is three-dimensional.

As yet another example, the following configuration may be used. That is, a model is prepared in advance in which "both two-dimensional image data and three-dimensional image data" and the selected article type are input and the probability that the imaged object has three-dimensionality is output. This model is a model learned by deep learning and other machine learning methods. Then, the three-dimensionality determination unit 13 inputs "both two-dimensional image data and three-dimensional image data" and the selected article type into this model, and obtains the output to determine whether or not the object to be imaged is three-dimensional. It may be configured to determine whether or not. However, the input of the model is not limited to the above, and the three-dimensional image data and the selected article type may be input, and the selected article type may not be included in the input. As for the output, the probability of no three-dimensionality and the presence or absence of three-dimensionality may be output.

Further, in the above embodiment, the article types are classified into various types such as notebook computers, smartphones, game machines, bags, watches, etc., when the articles are broadly classified according to their functions and uses. However, the article types may be further subdivided. For example, for notebook computers, smartphones, and other electronic devices, a model (sometimes called a model number) may be used as the item type. Further, for example, for brand-name bags, accessories, clothes, and accessories, the model number may be used as the item type. The article type may be configured to be selected or input by the user, or may be configured in which the stereoscopic determination unit 13 analyzes and recognizes the two-dimensional image data.

Further, in the above embodiment, the terminal 2 is a smartphone, but the terminal 2 is not limited to a smartphone. For example, the terminal 2 may be a tablet terminal or a notebook computer having no terminal function.

1 Information processing system 2 Terminal 3 Server (information processing device)
5 Camera 6 LiDAR Scanner (distance measuring sensor)
10 Terminal side control unit 13 Three-dimensionality discrimination unit 14 Processing unit

Claims (12)

It is an information processing device that acquires two-dimensional image data generated by shooting with a camera for a shooting target from a terminal via a network under the assumption that a three-dimensional object on which information for identity verification is recorded is shot. ,
The presence or absence of the three-dimensionality of the imaged object is obtained by acquiring the three-dimensional image data generated by the scan by the distance measuring sensor executed at the same time as the image taken by the camera from the terminal via the network and using at least the three-dimensional image data. A three-dimensionality discriminator that discriminates between
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it is provided with a processing unit that does not execute the above-mentioned identity verification related processing.
The three-dimensionality discriminating unit analyzes the two-dimensional image data, recognizes the position of the image to be photographed in the two-dimensional image data, and is based on the position of the image to be photographed in the two-dimensional image data. It is characterized in that the presence or absence of the three-dimensionality of the imaged object is determined by specifying a specific area which is an area of the image of the imaged object in the three-dimensional image data and determining the three-dimensionality of the image belonging to the specific area. Information processing device. The three-dimensionality determination unit determines whether or not the image belonging to the specific region in the three-dimensional image data is a flat image that is planarly continuous with the surrounding image, and if it is a flat image, the three-dimensional image is described. The information processing apparatus according to claim 1, wherein it is determined that the object to be imaged has no three-dimensionality, while it is determined that the object to be imaged has three-dimensionality in other cases. The three-dimensionality determining unit determines that the imaged object has three-dimensionality when a three-dimensional image independent of the surrounding image is formed in the specific region, while the three-dimensionality is determined in other cases. The information processing apparatus according to claim 1, wherein the information processing apparatus is determined not to be present. The three-dimensionality discriminating unit recognizes the type of the object to be photographed, and processes the image of the object of the type to be photographed in the two-dimensional image data by processing by an image processing technique or processing using a machine-learned model. Claims 1 to 3 are characterized in that the position is recognized and the position of the image of the object of the type to be photographed in the two-dimensional image data is recognized as the position of the image of the image to be photographed in the two-dimensional image data. The information processing apparatus according to any one of the above items. It is an information processing device that acquires two-dimensional image data generated by shooting with a camera for a shooting target from a terminal via a network under the assumption that a three-dimensional object on which information for identity verification is recorded is shot. ,
The presence or absence of the three-dimensionality of the imaged object is obtained by acquiring the three-dimensional image data generated by the scan by the distance measuring sensor executed at the same time as the image taken by the camera from the terminal via the network and using at least the three-dimensional image data. A three-dimensionality discriminator that discriminates between
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it is provided with a processing unit that does not execute the above-mentioned identity verification related processing.
The three-dimensionality determination unit recognizes the type of the shooting target, performs pattern matching using the three-dimensional template data corresponding to the type of the shooting target, and based on the result of the pattern matching, the three-dimensional image of the shooting target. An information processing device characterized by determining the presence or absence of sex. It is an information processing device that acquires two-dimensional image data generated by shooting with a camera for a shooting target from a terminal via a network under the assumption that a three-dimensional object on which information for identity verification is recorded is shot. ,
The presence or absence of the three-dimensionality of the imaged object is obtained by acquiring the three-dimensional image data generated by the scan by the distance measuring sensor executed at the same time as the image taken by the camera from the terminal via the network and using at least the three-dimensional image data. A three-dimensionality discriminator that discriminates between
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it is provided with a processing unit that does not execute the above-mentioned identity verification related processing.
The three-dimensionality discriminating unit is
Two-dimensional image data, three-dimensional image data, and the type of the shooting target are input, three-dimensional image data and the type of the shooting target are input, or two-dimensional image data and three-dimensional image data. The input value required for the machine-learned model that outputs the information that can be used to determine the presence or absence of the three-dimensionality of the imaged object is input, and the output is obtained to obtain the output of the imaged object. An information processing device characterized by determining the presence or absence of three-dimensionality. An information processing system including a terminal and an information processing device capable of communicating with the terminal via a network.
The terminal is
Two-dimensional image data generated by shooting with a camera for the shooting target under the assumption that a three-dimensional object on which information for identity verification is recorded is shot, and distance measurement performed at the same time as shooting with the camera. It has a terminal-side control unit that transmits 3D image data generated by scanning by a sensor to the information processing device via a network.
The information processing device is
A three-dimensionality determining unit that acquires the two-dimensional image data and the three-dimensional image data from the terminal via a network and determines the presence or absence of the three-dimensionality of the imaged object by using at least the three-dimensional image data .
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it has a processing unit that does not execute the identity verification related processing.
The three-dimensionality discriminating unit analyzes the two-dimensional image data, recognizes the position of the image to be photographed in the two-dimensional image data, and is based on the position of the image to be photographed in the two-dimensional image data. It is characterized in that the presence or absence of the three-dimensionality of the imaged object is determined by specifying a specific area which is an area of the image of the imaged object in the three-dimensional image data and determining the three-dimensionality of the image belonging to the specific area. Information processing system. An information processing system including a terminal and an information processing device capable of communicating with the terminal via a network.
The terminal is
Two-dimensional image data generated by shooting with a camera for the shooting target under the assumption that a three-dimensional object on which information for identity verification is recorded is shot, and distance measurement performed at the same time as shooting with the camera. It has a terminal-side control unit that transmits 3D image data generated by scanning by a sensor to the information processing device via a network.
The information processing device is
A three-dimensionality determining unit that acquires the two-dimensional image data and the three-dimensional image data from the terminal via a network and determines the presence or absence of the three-dimensionality of the imaged object by using at least the three-dimensional image data .
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it has a processing unit that does not execute the identity verification related processing.
The three-dimensionality determination unit recognizes the type of the shooting target, performs pattern matching using the three-dimensional template data corresponding to the type of the shooting target, and based on the result of the pattern matching, the three-dimensional shape of the shooting target. An information processing system characterized by determining the presence or absence of sex. An information processing system including a terminal and an information processing device capable of communicating with the terminal via a network.
The terminal is
Two-dimensional image data generated by shooting with a camera for the shooting target under the assumption that a three-dimensional object on which information for identity verification is recorded is shot, and distance measurement performed at the same time as shooting with the camera. It has a terminal-side control unit that transmits 3D image data generated by scanning by a sensor to the information processing device via a network.
The information processing device is
A three-dimensionality determining unit that acquires the two-dimensional image data and the three-dimensional image data from the terminal via a network and determines the presence or absence of the three-dimensionality of the imaged object by using at least the three-dimensional image data .
When it is determined by the three-dimensionality determination unit that the object to be photographed has three-dimensionality, it is determined that there is no three-dimensionality while performing the identification-related processing related to the identification of the person based on the two-dimensional image data. If so, it has a processing unit that does not execute the identity verification related processing.
The three-dimensionality discriminating unit is
Input 2D image data, 3D image data, and the type of the shooting target, input 3D image data and the type of the shooting target, or input 2D image data and 3D image data. Is input, and the input value required for the machine-learned model that outputs the information that can be used to determine the presence or absence of the three-dimensionality of the imaged object is input, and the output is obtained to obtain the output of the imaged object. An information processing system characterized by determining the presence or absence of three-dimensionality. Information processing by an information processing device that acquires two-dimensional image data generated by shooting with a camera for the shooting target from a terminal on the assumption that a three-dimensional object on which information for identity verification is recorded is shot. It ’s a method,
The first step in which the three-dimensionality discrimination unit of the information processing apparatus acquires three-dimensional image data generated by scanning by a distance measuring sensor executed at the same time as shooting by the camera from the terminal via a network.
A second step in which the three-dimensionality determining unit of the information processing apparatus determines the presence or absence of three-dimensionality of the photographing target by using at least the three-dimensional image data
When the processing unit of the information processing apparatus determines that the imaging target has three-dimensionality by the three-dimensionality determination unit, the identification-related processing related to the identification of the person is executed based on the two-dimensional image data. On the other hand, if it is determined that there is no three-dimensionality, the third step of not executing the identity verification related process is included.
In the second step, the stereoscopic determination unit analyzes the two-dimensional image data, recognizes the position of the image to be photographed in the two-dimensional image data, and recognizes the position of the image to be photographed in the two-dimensional image data. By identifying a specific area that is a region of the image to be photographed in the three-dimensional image data based on the position and determining the three-dimensionality of the image belonging to the specific area, the presence or absence of the three-dimensionality of the image to be photographed is determined. An information processing method characterized by discrimination. Information processing by an information processing device that acquires two-dimensional image data generated by shooting with a camera for the shooting target from a terminal on the assumption that a three-dimensional object on which information for identity verification is recorded is shot. It ’s a method,
The first step in which the three-dimensionality discrimination unit of the information processing apparatus acquires three-dimensional image data generated by scanning by a distance measuring sensor executed at the same time as shooting by the camera from the terminal via a network.
A second step in which the three-dimensionality determining unit of the information processing apparatus determines the presence or absence of the three-dimensionality of the imaged object by using at least the three-dimensional image data.
When the processing unit of the information processing apparatus determines that the photographing target has three-dimensionality by the three-dimensionality determination unit, the identification-related processing related to the identification of the person is executed based on the two-dimensional image data. On the other hand, if it is determined that there is no three-dimensionality, the third step of not executing the identity verification related process is included.
The three-dimensionality discrimination unit recognizes the type of the shooting target, performs pattern matching using the three-dimensional template data corresponding to the type of the shooting target, and based on the result of the pattern matching, the three-dimensional shape of the shooting target. An information processing method characterized by determining the presence or absence of sex. Information processing by an information processing device that acquires two-dimensional image data generated by shooting with a camera for the shooting target from a terminal on the assumption that a three-dimensional object on which information for identity verification is recorded is shot. It ’s a method,
The first step in which the three-dimensionality discrimination unit of the information processing apparatus acquires three-dimensional image data generated by scanning by a distance measuring sensor executed at the same time as shooting by the camera from the terminal via a network.
A second step in which the three-dimensionality determining unit of the information processing apparatus determines the presence or absence of three-dimensionality of the photographing target by using at least the three-dimensional image data
When the processing unit of the information processing apparatus determines that the imaging target has three-dimensionality by the three-dimensionality determination unit, the identification-related processing related to the identification of the person is executed based on the two-dimensional image data. On the other hand, if it is determined that there is no three-dimensionality, the third step of not executing the identity verification related process is included.
In the second step, the stereoscopic determination unit inputs two-dimensional image data, three-dimensional image data, and the type of the shooting target, or three-dimensional image data and the type of the shooting target. , Or input the input value required for the machine-learned model that inputs 2D image data and 3D image data and outputs information that can be used to determine the presence or absence of the three-dimensionality of the object to be photographed. However, an information processing method characterized in that the presence or absence of three-dimensionality of the object to be photographed is determined by obtaining the output.

Images