JP7034448B1 - Mobile terminals and size measurement methods using mobile terminals - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a surface of an object to be photographed and measure the height of the object to be photographed in a mobile terminal and a size measuring method using the mobile terminal.
SOLUTION: A mobile terminal 100 is photographed by a photographing unit 180 for photographing a rectangular parallelepiped object OB to which a marker MK is attached, a depth sensor 190 for measuring depth information up to the rectangular parallelepiped object OB, and an photographing unit 180. The acquisition unit 111 for acquiring the captured image of the rectangular parallelepiped object OB to which the marker MK is attached and the depth information from the imaging unit 180 measured by the depth sensor 190 to the rectangular parallelepiped object OB, and the marker MK are attached. An extraction unit 112 that extracts the coordinates of the four corners of the rectangular parallelepiped object OB in the captured image and the back coordinates of the four corners based on the captured image and the depth information of the rectangular parallelepiped object OB, and the rectangular parallelepiped object. A measuring unit 113 for measuring the physical size of a rectangular parallelepiped object OB based on the coordinates of the four corners of the OB and the back coordinates of the four corners is provided.
[Selection diagram] Fig. 3

Description

The present invention relates to a mobile terminal that measures the physical size of an object using a mobile terminal having a depth sensor, and a size measuring method using the mobile terminal.

Conventionally, a device for measuring the size of an object using a depth sensor has been studied. For example, Patent Document 1 discloses a measuring device that measures the thickness of an object even if the surface of the object is not flat (see the summary of Patent Document 1).

The technique of Patent Document 1 is a measurement system that measures the size of a fish transported by a belt conveyor using a camera including a photographing unit and a sensor unit and a measuring device composed of a personal computer. In the measurement system described in Patent Document 1, a depth sensor is provided in the sensor unit.

As a result, the measurement system described in Patent Document 1 uses the information related to the spatial coordinates on the surface of the fish body in the detection area placed on the reference surface in the calculation unit to change from the reference surface to the detection device in the fish body. The portion having the thickest thickness in the direction toward the direction is detected, and the thickness of the detected portion is calculated as the thickness of the fish body (see Patent Document 1).

Japanese Unexamined Patent Publication No. 2020-16501

In the measurement system described in Patent Document 1, the calculation unit includes a thickness measurement unit and a length measurement unit. Since the processing of the thickness measuring unit and the length measuring unit performed in the calculation unit has a large processing load, a personal computer is used as a measuring device, and each measurement processing is performed.

On the other hand, in recent years, technological progress has been remarkable in mobile terminals, and high-speed and high-precision image processing is being realized for images taken by a camera. Therefore, it is conceivable to mount a depth sensor on the mobile terminal and add depth information to the captured image captured by the camera of the mobile terminal to measure the height of the captured object in the captured image.

However, when depth information is added to the captured image and the augmented reality (AR) function is applied, the mobile terminal can recognize the background surface, but can recognize the surface of the captured object. Have difficulty. Therefore, it is difficult to measure the height of the object to be photographed with a mobile terminal equipped with a depth sensor.

Therefore, in the mobile terminal and the size measuring method using the mobile terminal, it is an object to recognize the surface of the object to be photographed and measure the height of the object to be photographed.

That is, the above problem of the present invention is solved by the following configuration.
The mobile terminal according to the present invention has a photographing unit for photographing a rectangular parallelepiped object with a marker, a depth sensor for measuring depth information to the rectangular parallelepiped object, and the marker photographed by the photographing unit. The acquisition unit that acquires the captured image of the rectangular parallelepiped object and the depth information from the imaging unit measured by the depth sensor to the rectangular parallelepiped object, and the acquired image of the rectangular parallelepiped object. A plurality of boundary lines of objects are extracted by image processing, the intersections where the boundary lines intersect are specified, the coordinates of the four corners of the object of the rectangular parallelepiped are specified as the model coordinates of the photographing unit, and each of the coordinates of the four corners is specified. The back coordinates corresponding to are extracted as the model coordinates of the photographing unit, the depth information corresponding to each pixel of the captured image is applied, and the model coordinates of the photographing unit are converted into world coordinates in three dimensions. An extraction unit that extracts the coordinates of the four corners of the rectangular parallelepiped object in space and the back coordinates of the four corners as world coordinates, and the coordinates of the four corners of the rectangular parallelepiped object and the back coordinates of the four corners extracted by the extraction unit. Based on the above, a measuring unit for measuring the physical size of the rectangular parallelepiped object is provided.

According to the present invention, in a mobile terminal and a size measuring method using a mobile terminal, the surface of the object to be photographed can be recognized and the height of the object to be photographed can be measured.

It is explanatory drawing which showed the whole structure of the box size measurement system using the mobile terminal which concerns on this embodiment. It is explanatory drawing explaining the main configuration example of the mobile terminal which concerns on this embodiment. It is a functional block diagram which showed the function of the CPU of the mobile terminal which concerns on this embodiment. It is a flowchart which showed the process that the mobile terminal which concerns on this embodiment takes a picture of the rectangular parallelepiped object with a marker, and displays the physical size of the rectangular parallelepiped object. It is explanatory drawing which showed the concept of the process which the extraction part extracts the coordinates of the four corners of a rectangular parallelepiped object and the back coordinates of the four corners, and converts them into world coordinates in a photographed image. It is explanatory drawing which showed the method of measuring the physical size of the object of a rectangular parallelepiped on the display screen of a mobile terminal. It is explanatory drawing which showed the method of measuring the physical size of the object of a rectangular parallelepiped in a photographed image. It is explanatory drawing which showed the example which displayed the physical size of the object of a rectangular parallelepiped on the display screen of the display part of a mobile terminal.

Hereinafter, embodiments for carrying out the present invention will be described in detail. The embodiments described below are examples for realizing the present invention, and should be appropriately modified or modified depending on the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is described below. Is not limited to the embodiment of the above.

<The present embodiment>
[overall structure]
FIG. 1 is an explanatory diagram showing an overall configuration of a box size measuring system 1 using the mobile terminal 100 according to the present embodiment. As shown in FIG. 1, in this box size measuring system 1, an object OB is placed on a background surface 200.

The mobile terminal 100 according to the present embodiment photographs an object OB placed on a background surface 200. A marker MK is attached to the object OB. The object OB is composed of a rectangular parallelepiped, and the height from the background surface 200 is constant. Further, the mobile terminal 100 is provided so as to be parallel to the object OB and the background surface 200. Further, the mobile terminal 100 shoots from directly above the object OB.

The surface of the object OB is specified by attaching the marker MK to the object OB. That is, the marker MK identifies the surface of the object OB. In the present embodiment, the physical size of the object OB is indicated by, for example, horizontal X, vertical Y, and height Z (see FIG. 6).

The marker MK, as an example, shows a pattern written on a paper shape attached to a rectangular parallelepiped object OB, but is not limited thereto. For example, the marker MK may have a thickness in the marker MK itself, or may have a predetermined height. Further, the marker MK may be a pattern, a mark, or the like described by a writing tool or the like.

[Mobile device configuration]
FIG. 2 is an explanatory diagram illustrating a main configuration example of the mobile terminal 100 according to the present embodiment. The mobile terminal 100 includes a CPU (Central Processing Unit) 110, a storage unit 120, a ROM (Read Only Memory) 130, a RAM (Random Access Memory) 140, an input unit 150, a display unit 160, a photographing unit 180, a depth sensor 190, and the like. It is configured with an internal bus 195. The mobile terminal 100 can be configured by, for example, a smartphone (smartphone) or a tablet (tablet) terminal, and is not particularly limited in this embodiment.

The CPU 110, the storage unit 120, the ROM 130, and the RAM 140 function as an application 170 when the CPU 110 executes a control program (not shown) stored in the storage unit 120 or the ROM 130. That is, the mobile terminal 100 can be executed as an application 170 by installing a control program in the storage unit 120, for example.

The CPU 110 is a central processing unit, and embodies each process shown in FIG. 3 by executing a control program stored in the storage unit 120 or the ROM 130. Each process embodied by the CPU 110 will be described later with reference to FIG.

The storage unit 120 is a large-capacity storage device, and is composed of, for example, a non-volatile memory. The storage unit 120 stores, for example, a control program, a captured image, information on the physical size of the measured rectangular parallelepiped object OB, and the like.

The RAM 140 functions as a work area for temporarily storing various programs, input data, output data, parameters, etc. that are read from the ROM 130 and can be executed by the CPU 110 in various processes whose execution is controlled by the CPU 110.

The input unit 150 includes an input device including a cursor key, a number input key, various function keys, and the like. The input unit 150 outputs the pressing signal and the operating signal of the pressed key to the CPU 110 as an input signal. The CPU 110 executes various processes based on the pressing signal and the operation signal from the input unit 150.

The display unit 160 is composed of, for example, an LCD (Liquid Crystal Display) monitor. The display unit 160 displays various screens according to the instructions of the display signal input from the CPU 110. In the present embodiment, the display unit 160 displays the physical size of the rectangular parallelepiped object measured by the measurement unit 113 of the CPU 110, which will be described later. A touch panel display can also be used for the display unit 160 and the input unit 150.

The photographing unit 180 is composed of a camera. The photographing unit 180 photographs the rectangular parallelepiped object OB to which the marker MK is attached. By photographing the rectangular parallelepiped object OB placed on the background surface 200, the photographing unit 180 outputs a two-dimensional image of the rectangular parallelepiped object OB and the background surface 200 by visible light.

The depth sensor 190 measures depth information from the photographing unit 180 to the rectangular parallelepiped object OB. Specifically, the depth sensor 190 detects the distance (depth information) in the depth direction from the photographing unit 180 with respect to the background surface 200 and the rectangular parallelepiped object OB. The depth sensor 190 calculates information (depth information) regarding the distance from the photographing unit 180 to the rectangular parallelepiped object OB for each pixel by, for example, a TOF (Time of Flight) method using infrared rays. The depth sensor 190 may calculate information (depth information) regarding the distance from the photographing unit 180 to the rectangular parallelepiped object OB by a triangular distance measuring method based on the light emitting position and the light receiving position. The depth sensor 190 detects the depth information of the background surface 200 and the depth information of the rectangular parallelepiped object OB, and outputs information (depth information) regarding the spatial coordinates of the portion corresponding to the pixel for each pixel.

The internal bus 195 electrically interconnects each component of the mobile terminal 100.

Next, the CPU 110 processing (function) of the mobile terminal 100 according to the present embodiment will be described with reference to FIG. Hereinafter, the same members will be designated by the same reference numerals, and the description thereof will be omitted as appropriate.

FIG. 3 is a functional block diagram showing the functions of the CPU 110 of the mobile terminal 100 according to the present embodiment. The CPU 110 embodies the acquisition unit 111, the extraction unit 112, the measurement unit 113, and the distribution unit 114 shown in FIG. 3, for example, by executing a control program (not shown) stored in the storage unit 120. ..

The acquisition unit 111 acquires the captured image of the rectangular parallelepiped object OB with the marker MK photographed by the imaging unit 180 and the depth information from the imaging unit 180 to the rectangular parallelepiped object OB measured by the depth sensor 190. do. The acquisition unit 111 sends the acquired captured image with the marker MK and the depth information to the extraction unit 112.

The extraction unit 112 sets the coordinates of the four corners of the rectangular parallelepiped object OB in the captured image and the four corners based on the captured image and the depth information of the rectangular parallelepiped object OB with the marker MK acquired by the acquisition unit 111. And the back coordinates of. The extraction unit 112 can identify the surface and the background surface of the rectangular parallelepiped object OB in the captured image by the marker MK, and can extract the back coordinates of the four corners based on the depth information. Further, the extraction unit 112 converts the coordinates of the four corners of the rectangular parallelepiped object OB and the back coordinates of the four corners into world coordinates.

The measuring unit 113 measures the physical size of the rectangular parallelepiped object OB based on the coordinates of the four corners of the rectangular parallelepiped object OB extracted by the extraction unit 112 and the back coordinates of the four corners.

The display unit 160 displays the physical size of the rectangular parallelepiped object OB measured by the measurement unit 113.

The distribution unit 114 distributes the storage destination of the rectangular parallelepiped object OB based on the physical size of the rectangular parallelepiped object OB measured by the measurement unit 113. The distribution unit 114 can distribute the storage destination based on the volume of the rectangular parallelepiped object OB, for example. Specifically, the distribution unit 114 automatically distributes the storage destination of the rectangular parallelepiped object OB depending on whether or not the volume is 100 [cm ³ ] or more. Further, the distribution unit 114 can also distribute the storage destination based on, for example, the height of the object OB of the rectangular parallelepiped. Specifically, the distribution unit 114 automatically distributes the storage destination of the rectangular parallelepiped object OB depending on whether or not the height of the rectangular parallelepiped object OB is 20 [cm] or more.

[Processing of mobile devices]
Next, the processing of the CPU 110 of the mobile terminal 100 according to the present embodiment will be described.
FIG. 4 is a flowchart showing a process in which the mobile terminal 100 according to the present embodiment photographs a rectangular parallelepiped object OB to which a marker MK is attached and displays the physical size of the rectangular parallelepiped object OB. .. In addition, it will be described with reference to FIG. 5 as appropriate.

In FIG. 5, in the captured images GZ1 to GZ5, the extraction unit 112 extracts the coordinates of the four corners (intersection points P1 to P4) of the rectangular parallelepiped object image OBP and the back coordinates H1 to H4 of the four corners, and the world coordinates. It is explanatory drawing which showed the concept of the process which converts into.

First, the mobile terminal 100 receives an instruction by the user operating the input unit 150. The photographing unit 180 photographs the rectangular parallelepiped object OB to which the marker MK is attached by the operation of the user (step S001). In this case, the depth sensor 190 measures the depth information up to the rectangular parallelepiped object OB at the same time as the timing at which the photographing unit 180 photographs the rectangular parallelepiped object OB. The photographed image GZ1 shown in FIG. 5 shows a state in which the object OB is photographed.

As a result, the acquisition unit 111 of the CPU 110 has a captured image of the rectangular parallelepiped object OB with the marker MK photographed by the imaging unit 180, and the depth from the imaging unit 180 measured by the depth sensor 190 to the rectangular parallelepiped object OB. Information and can be obtained.

Next, the extraction unit 112 of the CPU 110 determines the four corners of the rectangular parallelepiped object OB in the captured image based on the captured image and the depth information of the rectangular parallelepiped object OB to which the marker MK is attached acquired by the acquisition unit 111. The coordinates and the back coordinates of the four corners are extracted.

Specifically, the extraction unit 112 extracts the edge (boundary line) of the rectangular parallelepiped object OB in the captured image GZ1 by image processing (step S003). The captured image GZ2 of FIG. 5 shows a state in which the edge of the rectangular parallelepiped object OB is extracted by image processing.

Next, the extraction unit 112 identifies the intersection of the surfaces of the rectangular parallelepiped object OB and extracts the back surface coordinates (step S005). In this case, as shown in the captured image GZ3 of FIG. 5, the extraction unit 112 detects the vertical line A and the horizontal line B indicating the edge, and identifies the intersecting point P1. As shown in the captured image GZ4 of FIG. 5, the extraction unit 112 applies the process of specifying the intersecting intersections to the four corners to specify the coordinates of the four corners P1 to P4. Further, the extraction unit 112 simultaneously extracts the back surface coordinates H1 to H4 at the four corners of the background surface 200. That is, the extraction unit 112 extracts the coordinates (backward coordinates H1 to H4) of the background surface 200 corresponding to each of the four intersections P1 to P4 based on the depth information corresponding to each pixel.

Next, the extraction unit 112 converts the model coordinates (camera coordinates) into world coordinates (step S007). The model coordinates refer to the coordinates peculiar to the mobile terminal 100 when the direction of the rectangular parallelepiped object OB is viewed from the position of the photographing unit 180. The world coordinates are the coordinates of the rectangular parallelepiped object OB in the three-dimensional space.

The extraction unit 112 applies augmented reality (AR) using the depth information of each pixel of the captured image to obtain a rectangular parallelepiped object OB in a three-dimensional space from model coordinates. Convert to coordinates. As shown in the captured image GZ5 of FIG. 5, the extraction unit 112 can display the coordinates of the four corners of the four intersections P1 to P4 on the virtual three-dimensional space, and measures the distance between the intersections P1 to P4. can do.

Next, the measurement unit 113 of the CPU 110 is a rectangular parallelepiped based on the coordinates (four intersections P1 to P4) of the four corners of the rectangular parallelepiped object OB extracted by the extraction unit 112 and the back coordinates H1 to H4 of the four corners. The physical size of the object OB is measured (step S009).

FIG. 6 is an explanatory diagram showing a method of measuring the physical size of a rectangular parallelepiped object OB on the display screen 161 of the mobile terminal 100. FIG. 7 is an explanatory diagram showing a method of measuring the physical size of a rectangular parallelepiped object OB in the captured image GZ6.

In FIGS. 6 and 7, the same members are designated by the same reference numerals, and the description thereof will be omitted as appropriate.

As shown in FIG. 6, the mobile terminal 100 measures the rectangular parallelepiped object OB, and measures the horizontal X, the vertical Y, and the height Z of the rectangular parallelepiped object OB. Further, in FIG. 7, the object OB in the captured image GZ6 is measured as a rectangular parallelepiped object OB on the background surface 200.

As shown in FIGS. 6 and 7, for example, the measuring unit 113 uses the average value of the distance between the intersection P1 and the intersection P2 and the distance between the intersection P3 and the intersection P4 as the length of the lateral X of the rectangular parallelepiped object OB. Measure. Further, the measuring unit 113 measures, for example, the average value of the distance between the intersection P1 and the intersection P3 and the distance between the intersection P2 and the intersection P4 as the length of the vertical Y of the rectangular parallelepiped object OB.

Further, the measuring unit 113 measures, for example, the distance between the intersection P2 and the back coordinate H2 as the height Z of the object OB of the rectangular parallelepiped. The measuring unit 113 is, for example, an average value of the distance between the intersection P1 and the back coordinate H1, the distance between the intersection P2 and the back coordinate H2, the distance between the intersection P3 and the back coordinate H3, and the distance between the intersection P4 and the back coordinate H4. May be calculated and used as the height Z of the object OB of the rectangular body. Further, since the object OB is a rectangular parallelepiped, the distance between the intersection P1 and the back coordinate H1 may be measured as the height Z of the rectangular parallelepiped object OB. The measuring method by the measuring unit 113 is not particularly limited.

Returning to FIG. 4, the explanation will be continued.
The measuring unit 113 displays the physical size of the rectangular parallelepiped object OB measured by the measuring unit 113 on the display screen of the display unit 160 (step S011).

FIG. 8 is an explanatory diagram showing an example in which the physical size of a rectangular parallelepiped object OB is displayed on the display screen 162 of the display unit 160 of the mobile terminal 100.

As shown in FIG. 8, the measurement result of the rectangular parallelepiped object OB is displayed on the upper part of the display screen 162. For example, the rectangular parallelepiped object OB is "length: 10.8 cm, width: 10.8 cm, Height: 10.3 cm "is displayed.

Returning to FIG. 4, the explanation will be continued.
The distribution unit 114 of the CPU 110 distributes the storage destination of the rectangular parallelepiped object OB based on the physical size of the rectangular parallelepiped object OB measured by the measurement unit 113 (step S013).

The distribution unit 114 can distribute the storage destination based on the volume of the rectangular parallelepiped object OB, for example. Specifically, the distribution unit 114 can automatically distribute the storage destination of the rectangular parallelepiped object OB depending on whether or not the volume is 1000 [cm ³ ] or more.

In the case of FIG. 8, since the volume of the rectangular parallelepiped object OB is about 1200 [cm ³ ], the distribution unit 114 distributes the rectangular parallelepiped object OB so as to be stored in the warehouse K. If the volume of the rectangular parallelepiped object is 950 [cm ³ ], the distribution unit 114 distributes the storage destination of the rectangular parallelepiped object OB so as to be stored in the warehouse L.

Further, the distribution unit 114 can also distribute the storage destination based on, for example, the height of the object OB of the rectangular parallelepiped. Specifically, the distribution unit 114 can automatically distribute the storage destination of the rectangular parallelepiped object OB depending on whether or not the height of the rectangular parallelepiped object OB is 20 [cm] or more.

In the case of FIG. 8, since the height of the rectangular parallelepiped object OB is about 10 [cm], the distribution unit 114 distributes the rectangular parallelepiped object OB so as to be stored in the warehouse M. If the volume of the rectangular parallelepiped object is 25 [cm], the distribution unit 114 distributes the storage destination of the rectangular parallelepiped object OB so as to be stored in the warehouse N.

When the distribution unit 114 distributes the storage destination of the rectangular parallelepiped object OB, the process shown in FIG. 4 ends.

As described above, the mobile terminal 100 according to the present embodiment includes a photographing unit 180, a depth sensor 190, an acquisition unit 111, an extraction unit 112, and a measurement unit 113.

The acquisition unit 111 acquires the captured image of the rectangular parallelepiped object OB with the marker MK photographed by the imaging unit 180 and the depth information from the imaging unit 180 to the rectangular parallelepiped object OB measured by the depth sensor 190. do. The extraction unit 112 sets the coordinates (intersection points P1 to P4) of the four corners of the rectangular parallelepiped object OB in the captured image and the coordinates of the four corners of the four corners based on the captured image and the depth information of the rectangular parallelepiped object OB to which the marker MK is attached. The back coordinates H1 to H4 are extracted. The measuring unit 113 measures the physical size of the rectangular parallelepiped object OB based on the coordinates (intersection points P1 to P4) of the four corners of the rectangular parallelepiped object OB and the backside coordinates H1 to H4 of the four corners.

Thereby, the mobile terminal 100 according to the present embodiment can recognize the surface of the object OB by the marker MK and measure the height of the object to be photographed.

In particular, since the acquisition unit 111, the extraction unit 112, and the measurement unit 113 can be automatically executed as the application 170, the user operates the input unit 150 to attach a marker MK photographed by the photographing unit 180. By simply acquiring the captured image of the rectangular parallelepiped object OB and the depth information from the photographing unit 180 measured by the depth sensor 190 to the rectangular parallelepiped object OB, the physical physical of the rectangular parallelepiped object OB can be easily obtained. Size can be measured.

In the present embodiment, the mobile terminal 100 is designed to photograph a rectangular parallelepiped object OB to which the marker MK is attached. For example, when the object OB has a symbol, the symbol is attached. May be regarded as a marker MK.

As a result, if the object OB has a pattern, the mobile terminal 100 can recognize the surface of the object OB and measure the height of the object to be photographed. Since the symbol can be a feature point of the object OB, the more symbols there are, the better the surface detection accuracy and the better the measurement accuracy of the physical size of the object OB.

Further, in the present embodiment, the marker MK attached to the object OB may specify the surface of the object OB in relation to the background surface 200.

Specifically, a border, a stripe, a checkered pattern, or the like is provided on the background surface 200, and the surface of the rectangular parallelepiped object OB is made plain. As a result, in the rectangular parallelepiped object OB placed on the background surface 200, the surface of the rectangular parallelepiped object OB is specified by the background surface 200, so that the height of the rectangular parallelepiped object OB can be measured. In this case, the marker MK in the rectangular parallelepiped object OB is composed of a solid color.

Similarly, the background surface 200 may be set to a solid color, and the marker MK attached to the object OB may be composed of a border, a stripe, a checkered pattern, or the like.

Further, the marker MK is not limited to a pattern, a mark, a solid color, a border, a stripe, a checkered pattern, or the like, and may be configured in the opposite color to the background surface 200. That is, it is sufficient that the marker MK in the present embodiment can specify the surface of the object OB in relation to the background surface 200.

Further, although the marker MK is attached to the surface of the rectangular parallelepiped object OB, it may be arranged at any one of the four corners, for example. In this case, since the height Z can be measured at the corner where the marker MK is arranged, the measurement accuracy of the height Z can be improved.

Further, the marker MK may be arranged at the center of the rectangular parallelepiped object OB. For example, when the rectangular parallelepiped object OB is composed of an empty box, it is assumed that the vicinity of the center of the empty box is dented. Therefore, by arranging the marker MK near the center of the empty box which is the object OB, The accuracy of identifying the surface of the object OB can be improved.

100 mobile terminal 110 CPU
111 Acquisition unit 112 Extraction unit 113 Measurement unit 114 Distribution unit 120 Storage unit 130 ROM
140 RAM
150 Input section 160 Display section 170 Application 180 Shooting section 190 Depth sensor 195 Internal bus OB Object (shooting target)
MK marker

Claims (5)

A shooting unit that shoots a rectangular parallelepiped object with a marker,
A depth sensor that measures depth information to the rectangular parallelepiped object,
An acquisition unit for acquiring a captured image of a rectangular parallelepiped object with the marker photographed by the imaging unit, and depth information from the imaging unit measured by the depth sensor to the rectangular parallelepiped object.
In the acquired captured image, a plurality of boundary lines of the rectangular parallelepiped object are extracted by image processing, intersections where the boundary lines intersect are specified, and the coordinates of the four corners of the rectangular parallelepiped object are set as a model of the imaging unit. Specified as coordinates,
The back coordinates corresponding to each of the coordinates of the four corners are extracted as the model coordinates of the photographing unit, and the coordinates are extracted.
The depth information corresponding to each pixel of the captured image is applied to convert the model coordinates of the photographing unit into world coordinates, and the coordinates of the four corners of the rectangular parallelepiped object in the three-dimensional space and the back surface of the four corners. An extractor that extracts coordinates as world coordinates,
A measuring unit that measures the physical size of the rectangular parallelepiped object based on the coordinates of the four corners of the rectangular parallelepiped object and the back coordinates of the four corners extracted by the extraction unit.
A mobile terminal characterized by being equipped with. The marker identifies the surface of the object and
The extraction unit
The marker identifies the surface and the background surface of the rectangular parallelepiped object in the captured image, and extracts the back coordinates of the four corners based on the depth information.
The mobile terminal according to claim 1. With more display
The display unit is
The physical size of the rectangular parallelepiped object measured by the measuring unit is displayed.
The mobile terminal according to claim 1 or 2, wherein the mobile terminal is characterized in that. Further, a distribution unit for distributing the storage destination of the object of the rectangular parallelepiped based on the physical size of the object of the rectangular parallelepiped measured by the measurement unit is provided.
The mobile terminal according to any one of claims 1 to 3, wherein the mobile terminal is characterized by the above. In the shooting section, the step of shooting a rectangular parallelepiped object with a marker,
The step of measuring the depth information to the object of the rectangular parallelepiped with the depth sensor, and
A step of acquiring a captured image of a rectangular parallelepiped object with the marker taken by the photographing unit and depth information from the photographing unit to the rectangular parallelepiped object measured by the depth sensor.
In the acquired image, a step of extracting a plurality of boundary lines of the rectangular parallelepiped object by image processing and
The intersections where the boundary lines intersect are specified, the coordinates of the four corners of the rectangular parallelepiped object are specified as the model coordinates of the photographing unit, and the back coordinates corresponding to the coordinates of the four corners are the model of the photographing unit. Steps to extract as coordinates and
The depth information corresponding to each pixel of the captured image is applied to convert the model coordinates of the photographing unit into world coordinates, and the coordinates of the four corners of the rectangular parallelepiped object in the three-dimensional space and the back surface of the four corners. Steps to extract coordinates as world coordinates,
A step of measuring the physical size of the rectangular parallelepiped object based on the coordinates of the four corners of the rectangular parallelepiped object and the coordinates of the back of the four corners,
A size measurement method using a mobile terminal including.

Images