JP2021177162A - Measuring apparatus, measuring program, and measuring method - Google Patents

Abstract

To provide a measuring apparatus, a measuring program, and a measuring method that are inexpensive and highly convenient for measuring the dimensions of an object to be measured.SOLUTION: The present invention relates to a measuring apparatus including imaging means and display means. The measuring apparatus generates a rectangular parallelepiped shaped virtual object for measuring, which is associated with a coordinate system of a real space in which an object to be measured is placed, synthesizes an image of the virtual object for measuring with a picked-up image to display the synthesized image on a screen of the display means, recognizes a surface of the object to be measured with which an optical axis of the imaging means is brought into contact as a target surface, determines any position of the target surface as a target point, determines a surface in a direction closest to the target surface on the virtual object for measuring as an object surface, adjusts dimensions of the object surface of the virtual object for measuring in a normal direction so that the object surface moves in a direction approaching the target point, and acquires and outputs a measurement parameter concerning a shape of the current virtual object for measuring.SELECTED DRAWING: Figure 1

Description

The present invention relates to a measuring device, a measuring program and a measuring method, and can be applied to, for example, dimensional measurement of a baggage transported during a business such as home delivery or moving.

When estimating transportation charges and transportation conditions during operations such as home delivery and moving, it is necessary to measure the dimensions of the object to be measured (length, width, height) such as packed boxes.

As a conventional technique for measuring the dimensions of an object to be measured as described above, there is a technique described in Patent Document 1.

In Patent Document 1, as a method of measuring the size of an object to be measured without a user performing complicated operations, the object to be measured is irradiated with a laser beam by using two laser light sources and a sensor equipped with a camera. It is described that each dimension of the object to be measured is measured from an image obtained by capturing an image of the object to be measured in a state of being irradiated with a laser beam.

Japanese Unexamined Patent Publication No. 2018-11521

However, even if the technique described in Patent Document 1 is applied, the work load on the user can be reduced to some extent, but a sensor equipped with both a laser light source and a camera must be prepared and carried by the worker. It cost a lot of money and was not convenient.

In view of the above problems, a measuring device, a measuring program, and a measuring method capable of measuring the dimensions of the object to be measured at low cost and with high convenience are desired.

The first measuring device of the present invention is a measuring device including an imaging means and a display means, in which (1) a virtual object for measuring a rectangular shape associated with a coordinate system in the real space in which an object to be measured is arranged. Is generated, and the image of the virtual object for measurement is combined with the image captured by the imaging means and displayed on the screen of the display means. (2) Editing of the virtual object for measurement is performed. It has an object editing means to be performed, (3) a measurement parameter output means for acquiring and outputting measurement parameters related to the current shape of the virtual object for measurement, and (4) the object editing means is the object to be measured. The surface on which the optical axis of the imaging means abuts is recognized as the target surface, any position of the target surface is determined as the target point, and the surface closest to the target surface on the measurement virtual object is the target surface. The measurement virtual object is characterized in that the dimension of the target surface in the normal direction is adjusted so that the target surface moves in a direction approaching the target point.

In the second measurement program of the present invention, a computer mounted on a measurement device including an imaging means and a display means is (1) a rectangular shape in which an object to be measured is associated with a coordinate system in the real space. The augmented reality processing means for generating the virtual object for measurement of the above, synthesizing the image of the virtual object for measurement with the image captured by the imaging means, and displaying the image on the screen of the display means, and (2) for the measurement. It functions as an object editing means for editing a virtual object and (3) a measurement parameter output means for acquiring and outputting measurement parameters related to the current shape of the virtual object for measurement, and (4) the object editing means is described above. The surface of the object to be measured that the optical axis of the imaging means abuts is recognized as the target surface, any position of the target surface is determined as the target point, and the object is most oriented with the target surface on the virtual object for measurement. A surface close to the target surface is determined as the target surface, and the dimension of the virtual object for measurement in the normal direction of the target surface is adjusted so that the target surface moves in a direction approaching the target point.

A third aspect of the present invention is a measuring method performed by a measuring device, wherein the measuring device includes (1) an image pickup means, a display means, an augmented reality processing means, an object editing means, and a measurement parameter output means (2). The augmented reality processing means generates a virtual object for measurement of a rectangular shape associated with a coordinate system in the real space in which an object to be measured is arranged, and the virtual object for measurement is added to an image captured by the imaging means. The images of are combined and displayed on the screen of the display means, (3) the object editing means edits the virtual object for measurement, and (4) the measurement parameter output means is the current virtual for measurement. The measurement parameters related to the shape of the object are acquired and output, and (5) the object editing means recognizes the surface of the object to be measured with which the optical axis of the imaging means abuts as a target surface, and any one of the target surfaces. Is determined as a target point, the surface closest to the target surface on the measurement virtual object is determined as the target surface, and the measurement is performed so that the target surface moves in a direction approaching the target point. It is characterized in that the dimension of the target surface of the virtual object in the normal direction is adjusted.

According to the present invention, it is possible to provide a measuring device, a measuring program, and a measuring method for measuring the dimensions of an object to be measured at low cost and with high convenience.

It is a figure (perspective view) which showed the measuring apparatus and the object to be measured which concerns on 1st Embodiment. It is a block diagram which shows the whole structure of the measuring apparatus which concerns on 1st Embodiment. It is a flowchart which showed the operation at the time of starting the measuring apparatus (measurement processing part) which concerns on 1st Embodiment. It is a figure which showed the transition of the operation screen displayed on the touch panel display when the measurement processing unit which concerns on 1st Embodiment is activated. It is a figure which showed the code which specifies each side and each surface which constitutes the virtual object for measurement which concerns on 1st Embodiment. It is a figure (the 1) which showed the structure of the collider arranged in the virtual object for measurement which concerns on 1st Embodiment. It is a figure (the 2) which showed the structure of the collider arranged in the virtual object for measurement which concerns on 1st Embodiment. It is a figure (the 3) which showed the structure of the collider arranged in the virtual object for measurement which concerns on 1st Embodiment. It is a flowchart which showed the process flow at the time of performing the fitting process by the measurement processing part which concerns on 1st Embodiment. It is a figure which showed the configuration example of the fitting processing screen which concerns on 1st Embodiment. It is a figure which showed the example of the screen transition at the time of performing the position movement processing in the fitting processing screen which concerns on 1st Embodiment. It is a figure which showed the example of the screen transition at the time of performing a rotation process in the fitting process screen which concerns on 1st Embodiment. It is a figure which showed the example (the 1) of the screen transition when the dimension change process is performed on the fitting process screen which concerns on 1st Embodiment. It is a figure which showed the example (the 2) of the screen transition when the dimension change process is performed on the fitting process screen which concerns on 1st Embodiment. It is a figure which showed the example (the 3) of the screen transition when the dimension change process is performed on the fitting process screen which concerns on 1st Embodiment. It is a figure (No. 1) which showed the example which the collider of the upper surface of the virtual object for measurement which concerns on 1st Embodiment is dragged, and the dimension is changed (reduced). It is a figure (No. 1) which showed the example which the collider of the upper surface of the virtual object for measurement which concerns on 1st Embodiment is dragged, and the dimension is changed (reduced). It is a figure which showed the configuration example of the attribute information input screen which concerns on 1st Embodiment. It is a figure which showed the configuration example of the selection screen which concerns on 1st Embodiment. It is a figure which showed the configuration example of the storage arrangement simulation screen which concerns on 1st Embodiment. It is a figure which showed the example (the 1) of the transition of the storage arrangement simulation screen which concerns on 1st Embodiment. It is a figure which showed the example (the 2) of the transition of the storage arrangement simulation screen which concerns on 1st Embodiment. It is a figure which showed the configuration example of the fitting processing screen which concerns on 2nd Embodiment. It is a flowchart which showed the flow of the process at the time of performing the fitting process which concerns on 2nd Embodiment. It is a flowchart which showed the flow of the 1st autofit processing which concerns on 2nd Embodiment. It is a figure (perspective view) which showed the arrangement example of the measuring apparatus and the object to be measured in the 1st autofit processing which concerns on 2nd Embodiment. It is a figure which showed the example of the fitting processing screen which is displayed when the 1st autofit processing is performed in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (plan view) which showed the initial positional relationship between the object to be measured and the virtual object for measurement at the time of performing the 1st autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (plan view) which showed the example (the 1) of the transition between the object to be measured and the virtual object for measurement at the time of performing the 1st autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (plan view) which showed the example (the 2) of the transition between the object to be measured and the virtual object for measurement at the time of performing the 1st autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (perspective view) which showed the example (the 3) of the transition between the object to be measured and the virtual object for measurement at the time of performing the 1st autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a flowchart which showed the flow of the 2nd autofit processing which concerns on 2nd Embodiment. It is a figure (perspective view) which showed the arrangement example of the measuring apparatus and the object to be measured in the 2nd autofit processing which concerns on 2nd Embodiment. It is a figure which showed the example of the fitting processing screen which is displayed when the 2nd autofit processing is performed in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (perspective view) which showed the example of the distribution of the feature point group acquired when the 2nd autofit processing is performed by the measuring apparatus which concerns on 2nd Embodiment. It is a figure (plan view) which showed the initial positional relationship between the object to be measured and the virtual object for measurement at the time of performing the 2nd autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a figure (plan view) which showed the example of the transition of the object to be measured and the virtual object for measurement at the time of performing the 2nd autofit processing in the measuring apparatus which concerns on 2nd Embodiment. It is a flowchart which showed the flow of the 3rd autofit processing which concerns on 2nd Embodiment. It is a figure which showed the structure of the collider arranged in the virtual object for measurement which concerns on the modification of 1st Embodiment (the 1: perspective view). It is a figure which showed the structure of the collider arranged in the virtual object for measurement which concerns on the modification of 1st Embodiment (the 2: top view). It is a figure which showed the example of the screen transition at the time of performing the dimension change processing in the fitting processing screen which concerns on the modification of 1st Embodiment. It is a figure which showed the example of the process which selects one from two surfaces adjacent to the operation side in the fitting process (dimension change process) which concerns on the modification of 1st Embodiment.

(A) First Embodiment Hereinafter, the first embodiment of the measuring device, the measuring program, and the measuring method according to the present invention will be described in detail with reference to the drawings.

(A-1) Configuration of First Embodiment FIG. 1 is a diagram showing a measuring device 10 and an object to be measured according to this embodiment.

The measuring device 10 is a device that measures parameters (for example, dimensions of each part) related to the outer shape of the object to be measured TO according to the operation of a user (hereinafter, simply referred to as “user”) who owns the measuring device 10.

FIG. 2 is a block diagram showing the overall configuration of the measuring device 10 of this embodiment.

As shown in FIG. 2, the measuring device 10 includes a touch panel display 11, a camera 12, a control unit 13, and a sensor unit 14.

The control unit 13 has a function of controlling the entire measuring device 10, and has a measurement processing unit 131 for measuring the external dimensions of the object to be measured TO.

The control unit 13 may be configured in software by installing a program (including a measurement program according to the embodiment) on a computer (not shown), or may be configured in hardware (for example, a dedicated semiconductor chip or the like). It may be configured by using. In this embodiment, the control unit 13 will be described as being configured in software using a computer.

The camera 12 is an imaging means capable of imaging the real space in which the object TO to be measured is arranged.

The touch panel display 11 has a display means (display function) capable of displaying (presenting) an image (for example, an image of an operation screen or the like) to the user, and an operation means (touch panel) capable of receiving an operation from the user. Function).

The sensor unit 14 is a group of sensors for detecting the movement of the measuring device 10. For example, a sensor group including an acceleration sensor and a gyro sensor can be applied to the sensor unit 14.

As described above, in terms of hardware, the measuring device 10 can be configured by using, for example, an information processing device including a touch panel such as a smartphone or a tablet PC, a camera, and various sensors. In this embodiment, the measuring device 10 will be described as being configured by using a smart phone. That is, the measurement processing unit 131 will be described as being an application program that can be installed on the smartphone.

Next, the outline of the measurement processing unit 131 will be described.

The measurement processing unit 131 is a virtual object in which the coordinates of the real space are linked to the image of the real space (hereinafter referred to as “captured image”) captured by the camera 12 by AR (Augmented Reality) technology. A process of generating a rendered (composited) image of (so-called AR object) and displaying it on the touch panel display 11 is performed. Here, the measurement processing unit 131 performs a process of generating and arranging a measurement virtual object MO, which is a virtual object for measuring the dimensions (for example, width, height, and depth) of the object to be measured TO. In addition, the measurement processing unit 131 edits the measurement virtual object MO according to the user's operation (for example, changing the shape accompanied by the movement of sides and vertices, changing the position and posture, and the like). At this time, the image displayed (rendered) on the touch panel display 11 includes an image of the object to be measured TO together with the virtual object MO for measurement depending on the position and orientation of the measuring device 10 (hereinafter, “object to be measured”). (Called "image GT") is also displayed. As a result, the measuring device 10 (measurement processing unit 131) receives from the user an operation of matching (fitting) the position / orientation and shape of the virtual object MO for measurement with the image GT of the object to be measured on the touch panel display 11. Can be done.

Therefore, when the position / orientation and shape of the virtual object MO for measurement match (fit) with the image GT of the object to be measured by the user's operation, the shape of the virtual object MO for measurement (shape in the real space) and the subject The shapes of the measured object image GT are equal.

Further, since the measuring device 10 (measurement processing unit 131) holds the shape of the virtual object MO for measurement in the real space (for example, the 3D coordinates of each vertex), the virtual object MO for measurement exists in the real space. It is possible to grasp the parameters related to the shape in the case (for example, width, height, depth, etc.). Therefore, the measuring device 10 (measurement processing unit 131) of this embodiment displays the parameters related to the shape when the virtual object MO for measurement exists in the real space on the touch panel display 11 in real time.

As described above, the measuring device 10 (measurement processing unit 131) accepts the editing of the virtual object MO for measurement by the user's operation, and in real time obtains the parameters related to the shape when the virtual object MO for measurement exists in the real space. By displaying with, it is possible to measure parameters (for example, width, height, depth, etc.) related to the shape of the object to be measured TO. The detailed processing of the measurement processing unit 131 will be described later.

The measurement processing unit 131 includes an AR processing unit 131a as a component for performing image processing, information processing, and the like related to AR. In the example of this embodiment, various AR systems (middleware / platform) can be applied to the AR processing unit 131a. Specifically, in the example of this embodiment, ARkit (registered trademark) will be described as an AR system applied to the AR processing unit 131a. The AR system applied to the AR processing unit 131a is not limited to the AR kit, and various environments can be applied. For example, if the OS of the measuring device 10 is Android (registered trademark), it is natural that ARCore (registered trademark) may be used. That is, in the measurement processing unit 131 (AR processing unit 131a), various AR systems can be applied to the basic functions of AR. Therefore, in the present specification, basic processing related to AR (for example, ARkit or ARCORE). The detailed description of the image processing that can be realized by using the above) itself will be omitted.

Further, the measurement processing unit 131 may be constructed by using an integrated development environment of various software (a software integrated development environment capable of software development using an AR system). In this embodiment, as an example, the measurement processing unit 131 will be described as being developed using Unity. Therefore, in the processing description of the measurement processing unit 131, an object defined in a library or the like on Unity (a collider or the like described later) may be used, but a similar object may be described by using another integrated development environment. Of course, you can build it.

The AR processing unit 131a executes various AR-related processes using information and images obtained from the sensor unit 14, the touch panel display 11, and the camera 12. It is assumed that the sensor unit 14 is provided with at least a sensor group (for example, a sensor group including an acceleration sensor and a gyro sensor) required for an AR system such as Arkit or ARCore, and detailed description thereof will be omitted.

(A-2) Operation of First Embodiment Next, the operation of the measuring device 10 of this embodiment having the above configuration (each procedure of the measuring method according to the embodiment) will be described.

Here, the operation (operation of the measurement program) of the measurement processing unit 131 when the user measures the dimensions of the object to be measured TO by using the measuring device 10 will be mainly described.

FIG. 3 is a flowchart showing an operation (starting sequence) when the measuring device 10 (measurement processing unit 131) is started.

Here, first, it is assumed that the measuring device 10 (control unit 13) has started the application corresponding to the measurement processing unit 131 in response to the user's operation (S101). Since basic operations of various smart phones and tablet PCs can be applied to the operation of starting the application, detailed description thereof will be omitted.

When the measurement processing unit 131 is activated, the AR processing unit 131a is used to detect a plane (for example, a plane such as a floor or the upper surface of a desk) from an image captured in the field of view of the camera 12 (hereinafter, “plane detection process””. (Called) (S102). Since libraries in various AR systems can be applied to the plane detection process itself, detailed description thereof will be omitted.

Next, the measurement processing unit 131 uses the AR processing unit 131a to display the marker object (3D model) on the detected plane (S103).

FIG. 4 is a diagram showing a transition of an operation screen displayed on the touch panel display 11 when the measurement processing unit 131 is activated.

Specifically, the measurement processing unit 131 uses the AR processing unit 131a to shoot a ray from the center of the screen of the touch panel display 11 in the optical axis direction of the camera to hit the object (the plane detected in step S102). The marker object shall be placed. At this time, it is assumed that the measuring device 10 is operated by the user so that the plane on which the object to be measured TO is arranged is located at the center of the screen of the touch panel display 11.

Here, as shown in FIG. 4A, the display is such that the marker object M (downward conical 3D model) is arranged on the floor surface F on which the object to be measured TO is installed.

Here, it is assumed that the screen operation (touch) of the touch panel display 11 is detected by the user (user's hand UH) (S104).

Then, the measurement processing unit 131 uses the AR processing unit 131a to arrange the measurement virtual object MO having a predetermined initial shape at the position of the marker M (S105). When the measurement processing unit 131 arranges the measurement virtual object MO of the initial shape, the shape of the measurement virtual object MO is displayed on the same screen in response to the user's operation. The process shifts to a process (hereinafter referred to as "fitting process") for matching (fitting) the shape (outer shape) of the TO).

In other words, the measurement processing unit 131 is set so that the shape of the measurement virtual object MO changes according to the user's operation (mainly the operation on the touch panel display 11).

Here, assuming that the measurement virtual object MO is a rectangular parallelepiped as shown in FIG. 4B, the measurement processing unit 131 is positioned (on the floor surface F) with respect to the measurement virtual object MO in the fitting process. Position), posture (direction on floor F), and each dimension (width, height, and depth) need to be accepted. The measurement processing unit 131 applies a GUI (GUI that accepts editing of the 3D model) capable of accepting changes in the positions, postures, and dimensions of various 3D models for the fitting process of the virtual object MO for measurement. Can be done. However, in a GUI that accepts editing of a general 3D model, it may be difficult to set the desired shape of the user at the desired position of the user. Therefore, the following configuration is applied to the measurement virtual object MO of this embodiment. The configuration of the virtual object MO for measurement is not limited to the following configuration.

In this embodiment, the shape of the virtual object MO for measurement is assumed to be a rectangular parallelepiped, but if the shape of the virtual object MO for measurement matches the shape of the object to be measured TO, another shape (for example, a conical shape) is used. , Cylindrical shape, etc.).

In FIG. 4B, the measurement virtual object MO is arranged at the position where the marker M is arranged in FIG. 4A. In the example of this embodiment, as shown in FIG. 4B, in the initial state, one side of the measurement virtual object MO faces the screen of the touch panel display 11 (facing the optical axis of the camera 12). It shall be arranged so as to.

In this embodiment, for convenience of illustration, each side of the measurement virtual object MO is drawn with a dotted line and each side is transparent, but a drawing method (drawing) of the sides and faces constituting the measurement virtual object MO The pattern) is not limited. For example, each surface of the virtual object MO for measurement may have a semi-transparent (transparent; mesh) pattern.

In the following, as shown in FIG. 4B, in the measurement virtual object MO, the surface in the foreground in the initial state is referred to as the front FF, and the upper surface of the front FF is referred to as the upper surface FU. Further, in the following, in the measurement virtual object MO, the lower surface of the front FF (the surface grounded to the floor surface F) is the lower surface FS, the surface facing the front FF is the back FB, and the left side when viewed from the front FF. The surface on the left side FL and the surface on the right side when viewed from the front FF are referred to as FR.

5 to 8 are diagrams showing an example of the configuration of the virtual object MO for measurement.

FIG. 5 is a diagram showing a reference numeral for specifying each side and each surface constituting the virtual object MO for measurement.

FIG. 5A is a perspective view of the measurement virtual object MO as viewed from the front FF side. Further, FIG. 5B is a perspective view of the measurement virtual object MO as viewed from the rear FB side.

As shown in FIG. 5, in the following, in the measurement virtual object MO, the side between the upper surface FU and the front surface FF is S_UF, the side between the upper surface FU and the back surface FB is S_UB, and the upper surface FU and the left surface FL. The side between them is S_UL, the side between the upper surface FU and the right side FR is S_UR, the side between the front FF and the left side FL is S_FL, the side between the front side FF and the right side FR is S_FR, and the side between the back side FB and the left side FL. The side between and S_BL, the side between the back FB and the right FR is S_BR, the side between the top FU and the front FF is S_UF, the side between the top FU and the back FB is S_UB, and the top FU. The side between the left surface FL is referred to as S_UL, and the side between the upper surface FU and the right surface FR is referred to as S_UR.

Further, as shown in FIG. 5, in the following, in the measurement virtual object MO, the horizontal dimension of the front FF / back FB is the width LW, the vertical dimension of the front FF / back FB is the height LH, and the left FL. / The horizontal dimension of the right side FR is called the depth LD.

During the fitting process, the measurement processing unit 131 is deformed by dragging (moving the finger to the target location while touching the screen on the touch panel display 11) for each side constituting the measurement virtual object MO. (Changes in width LW, height LH, and depth LD) can be accepted. However, each side constituting the virtual object MO for measurement has only the width of a line, and it is difficult to succeed in the drag operation (the operation of skipping a ray from the touched position and hitting the side). Therefore, in the measurement processing unit 131 of this embodiment, as shown in, a collider having a predetermined shape is provided around the side that accepts the drag operation in the measurement virtual object MO. A collider is an invisible object (operation object) that can accept an operation by a hit determination (collision determination when a ray is shot from a touched position). A collider is an object that can be defined in a development environment such as Unity / ARkit.

In this embodiment, it is assumed that colliders corresponding to each side of the measurement virtual object MO are arranged on each surface of the measurement virtual object MO (plane shape collider). For example, in the measurement virtual object MO, each collider may have a triangular shape with each side as the base on each surface of the measurement virtual object MO.

FIG. 6 is a diagram showing an example of a collider arrangement configuration in the measurement virtual object MO in this embodiment.

FIG. 6A is a perspective view of the measurement virtual object MO seen from the front FF side, and FIG. 6B is a perspective view of the measurement virtual object MO seen from the back FB side.

In the example of FIG. 6, it is assumed that a collider is formed on each surface of the virtual object MO for measurement by each region divided by a diagonal line (divided into four by two diagonal lines on a quadrangular surface). .. In other words, in this embodiment, the shape of each collider constituting the measurement virtual object MO is such that one side (bottom side) is formed by each side on each surface of the measurement virtual object MO, and the remaining two sides are further formed. It shall be in the shape of a triangle (isosceles triangle) formed by the diagonal lines of the surface.

Therefore, in this embodiment, in the measurement virtual object MO, colliders corresponding to each of the four sides are formed on each surface, so that a total of 24 colliders are formed.

As shown in FIG. 6, in the measurement virtual object MO, colliders C_FU, C_FS, C_FL, and C_FR corresponding to the sides S_UF, S_SF, S_FL, and S_FR are arranged on the front FF. Further, colliders C_BU, C_BS, C_BL, and C_BR corresponding to the sides S_UB, S_SB, S_BL, and S_BR are arranged on the back surface FB. Further, colliders C_UF, C_UB, C_UL, and C_UR corresponding to the sides S_UF, S_UB, S_UL, and S_UR are arranged on the upper surface FU. Furthermore, colliders C_SF, C_SB, C_SL, and C_SR corresponding to the sides S_SF, S_SB, S_SL, and S_SR are arranged on the lower surface FS. Further, colliders C_LU, C_LS, C_LF, and C_LB corresponding to the sides S_UL, S_SL, S_FL, and S_BL are arranged on the left FL. Further, C_RU, C_RS, C_RF, and C_RB corresponding to the sides S_UR, S_SR, S_FR, and S_BR are arranged on the right side FR. Each collider accepts a dimension change (dimension change of the measurement virtual object MO) in the direction orthogonal to the side corresponding to the collider on the plane on which the collider is formed in the measurement virtual object MO. Functions as an object (operation object).

FIG. 7 is a diagram showing a configuration example of a collider arranged in the virtual object MO for measurement.

The collider C_FR (triangular region surrounded by the diagonal line on the side S_FR and the front FF) corresponding to the side S_FR arranged on the FF is illustrated with a hatch (diagonal line pattern). In this case, since the collider C_FR is in the width direction on the formed front FF in the direction orthogonal to the corresponding side S_FR, the collider C_FR serves as a collider that accepts the dimension change of the width LH. It will work.

The shape and arrangement position of each collider is not limited as long as it is within the area surrounded by the corresponding sides and diagonal lines. For example, as shown in FIG. 8, each collider may be in the shape of another figure (polygon) such as a circle or a quadrangle as long as it is within the area surrounded by the corresponding side and the diagonal line.

Next, an example of processing when the measurement processing unit 131 performs the fitting processing will be described with reference to the flowchart of FIG.

When the measurement processing unit 131 starts the fitting process, first, a necessary object is arranged on the screen of the touch panel display 11 to form a fitting process screen (hereinafter, referred to as a “fitting process screen”) (S201).

FIG. 10 is a diagram showing a configuration example of a fitting processing screen.

In the fitting processing screen shown in FIG. 10, two buttons BL and BR for accepting the rotation (attitude change) of the virtual object MO for measurement and each parameter when the current virtual object MO for measurement exists in the real world ( Hereinafter, a field FP for displaying "virtual object parameter for measurement") and a button BFE for accepting the completion of the current fitting process are displayed.

In the example of this embodiment, the parameters of the width LW, the height LH, the depth LD, and the volume (LW × LH × LD) of the measurement virtual object MO are displayed as the measurement virtual object parameters. The number and combinations of measurement virtual object parameters displayed in the field FP are not limited, and various combinations can be applied. The measurement processing unit 131 calculates the measurement virtual object parameters (width LW, height LH, depth LD, and volume) when the measurement virtual object MO is in the real space, and displays it in the field FP. In the fitting processing screen shown in FIG. 10, each parameter related to the measurement virtual object MO in the initial state is displayed in the field FP in the initial state. Since the measurement processing unit 131 (AR processing unit 131a) grasps the coordinates (global coordinates / 3D coordinates) of the measurement virtual object MO in the real space, it is possible to calculate the measurement virtual object parameters.

After that, the measurement processing unit 131 waits until a predetermined operation on the touch panel display 11 is detected on the fitting processing screen (S202), and the fitting process according to the content of the operation (editing process of the virtual object MO for measurement). Move to.

Specifically, the measurement processing unit 131 of this embodiment performs fitting processing (editing processing of the virtual object MO for measurement) corresponding to the case where at least one of the following three operations is detected on the fitting processing screen. Move to.

The measurement processing unit 131 moves the position of the virtual object MO for measurement when a drag operation with two touches (two fingers) occurs on the fitting processing screen (hereinafter, referred to as “position movement processing”). ) (S203). The details of the position movement process will be described later. After executing the position movement process, the measurement processing unit 131 returns to step S202 and waits until the next operation detection.

Further, when the measurement processing unit 131 touches any of the rotation buttons BL and BR on the fitting processing screen, the measurement processing unit 131 rotates the posture of the measurement virtual object MO (rotates in the direction corresponding to the touched button). A process (hereinafter referred to as "rotation process") is performed (S204). The details of the rotation process will be described later. After executing the rotation process, the measurement processing unit 131 returns to step S202 and waits until the next operation detection.

Further, when the collider of any of the measurement virtual objects MO is dragged by the measurement processing unit 131 with one touch (one finger) on the fitting processing screen, any of the measurement virtual object MOs. A process of changing the dimension (hereinafter referred to as "dimension change process") is performed (S205). The details of the dimension change processing will be described later. After the dimension change process in step S205, the measurement processing unit 131 updates the measurement virtual object parameter based on the changed dimension, and causes the field FP to display the updated measurement virtual object parameter (S206). After the measurement virtual object parameter update process, the measurement processing unit 131 returns to step S202 and waits until the next operation detection.

Furthermore, when the button BFE is pressed on the fitting processing screen, the measurement processing unit 131 ends the fitting process for the current object TO to be measured, and provides information including the result of the fitting process for the object TO to be measured (hereinafter,). , Called "measurement object information") is registered (retained) (S207), and a series of processes relating to the measurement object TO is completed.

The measurement processing unit 131 performs the fitting process by repeatedly receiving the above operations from the user on the fitting process screen.

Next, the details of the position movement process will be described.

FIG. 11 is a diagram showing an example of screen transition when the measurement processing unit 131 performs the position movement processing on the fitting processing screen.

FIG. 11A shows the state of the fitting processing screen before the position movement processing, and FIG. 11B shows the state after the position movement processing is performed according to the operation of the user's hand UH. ..

As shown in FIG. 11, when a drag operation with two touches (two fingers) occurs on the fitting processing screen, the measurement processing unit 131 positions the measurement virtual object MO (for example, the measurement virtual object). The process of moving the MO center position) in the direction of the drag operation (the direction in which the finger is slid) according to the distance of the drag operation (the distance in which the finger is slid) is performed. In the case of the two-point touch, the measurement processing unit 131 performs the position movement processing of the measurement virtual object MO according to the movement of the position of the intermediate point between the two points.

Next, the details of the rotation process will be described.

FIG. 12 is a diagram showing an example of screen transition when the measurement processing unit 131 performs rotation processing on the fitting processing screen.

FIG. 12A shows the state of the fitting processing screen before the rotation processing, and FIG. 12B shows the state after the rotation processing is performed according to the operation of the user's hand UH.

As shown in FIG. 12, when either the rotation button BL or BR is pressed on the fitting processing screen, the measurement processing unit 131 rotates the measurement virtual object MO as a center point on the floor surface F. Performs the process of rotating in the direction of the button. The measurement processing unit 131 rotates the measurement virtual object MO to the left (clockwise) when the left rotation button BL is pressed, and rotates the measurement virtual object MO to the right when the right rotation button BR is pressed. Rotate (counterclockwise). At this time, the measurement processing unit 131 may rotate the measurement virtual object MO by a certain angle each time the rotation button is touched, or the amount corresponding to the time when the rotation button is continuously touched. The angle of, and the virtual object MO for measurement may be rotated.

Next, the details of the dimension change processing will be described.

In the following, the collider being dragged is referred to as an "operation collider", and the side corresponding to the operation collider is also referred to as an "operation side". Further, in the following, when viewed from the surface on which the operation collider is arranged, the surface adjacent to the operation side across the operation side is referred to as an "operation surface". The operation surface is a surface that moves when the dimensions are changed by the operation of the operation collider.

On the fitting process screen, when a drag operation by touching one point (one finger) occurs with any collider of the virtual object MO for measurement (a ray is skipped from the touched position and any collider (When a collision occurs), the measurement processing unit 131 determines the dimensions in the direction orthogonal to the side corresponding to the operation collider (direction orthogonal to the operation surface) on the surface on which the operated operation collider is arranged. A process is performed in which the dimension is changed in the direction of the drag operation (direction in which the finger is slid) according to the distance of the drag operation (distance in which the finger is slid). In other words, the measurement processing unit 131 performs a process of moving the operation side corresponding to the operation collider in a direction orthogonal to the operation surface (normal direction or direction opposite to the normal direction).

Next, the details of the dimension change processing will be described.

13, FIG. 14, and FIG. 15 are views showing an example of screen transition when the measurement processing unit 131 performs the dimension change processing on the fitting processing screen.

13, FIG. 14, and FIG. 15 show screen transitions when dimension change processing is performed for the measurement virtual object MO with respect to the dimension LW in the width direction, the dimension LH in the height direction, and the dimension LD in the depth direction, respectively. Shown.

13 (a), 14 (a), and 15 (a) show the state of the fitting processing screen before the dimension change processing, respectively, and FIGS. 13 (b), 14 (b), and FIG. Reference numeral 15 (b) shows a state after the dimension change processing is performed according to the operation of the user's hand UH.

First, the dimension change process when changing the dimension LW in the width direction will be described with reference to FIG. FIG. 13 shows a dimension change process when the collider C_FR corresponding to the side S_FR is dragged on the surface FF.

In FIG. 13A, the collider C_FR is dragged from the camera viewpoint where the surface FF faces the front side.

When the operation collider is the collider C_FR, the direction orthogonal to the side S_FR corresponding to the collider C_FR is the width direction on the front FF on which the collider C_FR is formed. It will function as an operation collider that accepts.

When the collider C_FR is dragged in a direction away from the center position (center position of the surface FF) of the virtual object MO for measurement (shown as + LW direction in FIG. 13A) with reference to the operation side S_FR, the measurement is performed. The processing unit 131 performs a dimension change process in which the dimension LW in the width direction of the measurement virtual object MO is increased by the distance corresponding to the drag operation distance (distance in which the finger is slid in the + LW direction) in the + LW direction. At this time, the position of the side S_FL facing the operation side S_FR on the front FF remains fixed.

When the collider C_FR is dragged in a direction approaching the center position (center position of the surface FF) of the virtual object MO for measurement (shown as -LW direction in FIG. 13A) with reference to the operation side S_FR, The measurement processing unit 131 reduces the dimension LW in the width direction of the virtual object MO for measurement by the distance corresponding to the drag operation distance (distance in which the finger is slid in the −LW direction) in the −LW direction. I do. Also at this time, the position of the side S_FL facing the operation side S_FR on the front FF remains fixed.

In other words, in this case, it can be said that the collider C_FR functions as a slider that fluctuates the dimension LW in the width direction of the virtual object MO for measurement.

Then, here, as shown in FIG. 13B, the collider C_FR is dragged in the −LW direction to the position at the right end of the image GT to be measured, and the dimension LW in the width direction of the virtual object MO for measurement is increased. It is assumed that the test object TO (measurement object image GT) is in the same state.

Next, the dimension change process when changing the dimension LH in the height direction will be described with reference to FIG. FIG. 14 shows a dimension change process when the collider C_UF corresponding to the side S_UF is dragged on the surface FF.

When the operation collider is the collider C_UF, the collider C_UF has a height LH in the direction orthogonal to the corresponding side S_UF on the front FF on which the collider C_UF is formed. Will function as a collider that accepts.

When the collider C_UF is dragged in a direction away from the center position (center position of the surface FF) of the virtual object MO for measurement (shown as + LH direction in FIG. 14A) with reference to the operation side S_UF, the measurement is performed. The processing unit 131 performs a dimension change process in which the dimension LH in the height direction of the virtual object MO for measurement is increased by the distance corresponding to the drag operation distance (distance in which the finger is slid in the + LH direction) in the + LH direction. ..

Then, here, as shown in FIG. 14B, the collider C_UF is dragged in the −LH direction to the position of the upper end of the image GT to be measured, and the dimension LH in the height direction of the virtual object MO for measurement is operated. Is in agreement with the object to be measured TO (image GT to be measured).

Next, the dimension change process when changing the dimension LD in the depth direction will be described with reference to FIG. FIG. 15 shows a dimension change process when the collider C_UB corresponding to the side S_UB is dragged on the upper surface FU.

When the operation collider is collider C_UB, the direction orthogonal to the side S_UF corresponding to the collider C_UB is the depth on the upper surface FU on which the collider C_UB is formed, so the collider C_UB changes the dimension of the depth LD. It will function as a collider to accept.

When the collider C_UB is dragged in the direction away from the center position (center position of the surface FU) of the virtual object MO for measurement (shown as + LD direction in FIG. 15A) with reference to the operation side S_UB, the measurement is performed. The processing unit 131 performs a dimension change process in which the dimension LD in the depth direction of the measurement virtual object MO is increased by the distance corresponding to the drag operation distance (distance in which the finger is slid in the + LD direction) in the + LD direction. At this time, the position of the side S_UF facing the operation side S_UB on the upper surface FU remains fixed.

Further, the collider C_UB was dragged in a direction approaching the center position (center position of the surface FU) of the virtual object MO for measurement (shown as the -LD direction in FIG. 15A) with the operation side S_UB as a reference. In this case, the measurement processing unit 131 reduces the dimension LD in the depth direction of the virtual object MO for measurement by the distance corresponding to the drag operation distance (distance in which the finger is slid in the −LD direction) in the −LD direction. Perform change processing. Also at this time, the position of the side S_UF facing the operation side S_UB on the upper surface FU remains fixed.

Then, here, as shown in FIG. 15B, the collider C_UB is dragged in the −LD direction to the position of the upper end of the image GT to be measured, and the dimension LD in the depth direction of the virtual object MO for measurement is changed. It is assumed that the test object TO (measurement object image GT) is in the same state. That is, in the state of FIG. 15B, it is shown that the virtual object parameter for measurement displayed in the field FP matches the object TO to be measured in the real space.

Next, the details of the deformation processing of the virtual object MO for measurement at the time of the dimension change processing will be described.

As described above, the measurement processing unit 131 determines the measurement virtual object MO according to the drag operation distance (finger-sliding distance) received by any of the colliders placed on the measurement virtual object MO. Increase or decrease either width, height, or depth. Further, as described above, when accepting the dimension change processing for the measurement virtual object MO, as shown in FIGS. 13 to 15, only the operation side corresponding to the operation collider is moved, and the side facing the operation side is It is easier to operate if the is fixed (it can accept dimensional changes that more closely match the user's intention). For example, as shown in FIG. 15, when the collider C_UB is operated to change the dimension of the measurement virtual object MO in the depth direction, the position of the side S_UF (front FF) facing the side S_UB is fixed. Leaving it as it is, only the side S_UB (rear FB) corresponding to the collider C_UB will be moved.

However, in the GUI that accepts the deformation of the 3D object, when the drag operation of the movement is accepted with respect to one side, the process of keeping the center position of the 3D object fixed is common. In that case, if the movement operation is accepted for one side while the center position of the 3D object is fixed, not only the side for which the movement operation is accepted but also the side facing the side will be moved by the same distance. .. For example, when a fitting GUI is configured using a library (hereinafter referred to as "3D object processing library") that is standardly used when processing a 3D object in ARkit or Unity, the same applies to one side of the 3D object. On the other hand, when the drag operation is accepted, basically, the process of keeping the center position of the 3D object fixed is performed. It is assumed that the measurement processing unit 131 of this embodiment also applies the 3D object processing library that performs the same processing.

Therefore, when the measurement processing unit 131 of this embodiment receives a dimension change operation (drag operation) for one side of the measurement virtual object MO, the measurement processing unit 131 changes the center position of the measurement virtual object MO according to the dimension change. It shall be processed to move to the position. Specifically, when the measurement processing unit 131 accepts a dimension change operation (drag operation) for one side of the measurement virtual object MO, the center position of the measurement virtual object MO corresponds to the operation collider C_UB. It is assumed that the process of moving the first side in the same direction as the moving direction (direction of changing the dimension) by half the distance (half the distance) of the moving distance of the first side is performed.

16 and 17 are views showing an example in which the collider C_UB of the upper surface FU of the virtual object MO for measurement is dragged to change (decrease) the dimension of the virtual object MO for measurement in the depth direction. FIG. 16 shows an example in which the center position is fixed when the dimension is changed, and FIG. 17 shows an example in which the center position is moved according to the dimension change when the dimension of the virtual object MO for measurement is changed. Is shown.

16 and 17 are views (plan views) of the measurement virtual object MO as viewed from above (upper surface FU), respectively. 16 (a) and 17 (a) show the state before the measurement virtual object MO is dragged, and FIGS. 16 (b) and 17 (b) show the measurement virtual object MO, respectively. , The state after being dragged in the inner direction (the direction in which the dimension in the depth direction decreases) is shown. In the examples of FIGS. 16 and 17, an example in which the measurement virtual object MO is dragged from the positions P201 to P202 is shown. In FIG. 16, the dimension in the depth direction of the measurement virtual object MO before the dimension change is LD1, and the drag-operated distance (distance from P201 to P202) is LD2.

Further, in FIGS. 16 and 17, the center position of the measurement virtual object MO before the drag operation is set to P101. Further, in the example of FIG. 17, the center position of the measurement virtual object MO is moved to the position of P102 with the drag operation.

In the example of FIG. 16, since the center position of the measurement virtual object MO is fixed before and after the drag operation, the drag operation is performed not only on the side S_UB corresponding to the operated collider C_UB but also on the opposite side S_UF. It is in a state of being moved inward by the distance LD2.

On the other hand, in the example of FIG. 17, when the side S_UB corresponding to the dragged collider C_UB moves inward by a distance LD2, the center position of the virtual object MO for measurement is only half the distance (LD2 / 2) of that distance LD2. It is moving in the same direction. As shown in FIG. 17B, the center position of the measurement virtual object MO after the drag operation (dimension change) is P102. As a result, in the example of FIG. 17, processing is performed using a general 3D object processing library (a library that performs processing that keeps the center position of the 3D object fixed when a drag operation for moving to one side is accepted). Even if this is done, the position of the side S_UF facing the side S_UB (the side corresponding to the dragged collider C_UB) remains fixed.

As described above, the measurement processing unit 131 repeats the fitting process (including the update process of the virtual object parameter for measurement) according to the user's operation according to the flowchart of FIG. It is possible to present the measured value related to the shape.

Next, the details of the process of completing the fitting process for the object to be measured TO by the measurement processing unit 131 and registering the information to be measured for the object to be measured TO (the process of step S207 described above) will be described.

When the button BFE for accepting the completion of the fitting process is pressed on the fitting process screen, the measurement processing unit 131 completes the fitting of the object TO to be measured that is currently being fitted, and the shape of the virtual object MO for measurement is changed. It is considered that it matches / corresponds to the object TO to be measured, and information on the shape (dimensions) of the current virtual object MO for measurement is acquired as information on the shape of the object TO to be measured (hereinafter, "shape information").

Here, the measurement processing unit 131 acquires at least the shape information of the virtual object MO for measurement and the information regarding the attributes of the object TO to be measured (hereinafter, referred to as “attribute information”).

Information on the attributes of the object to be measured TO may be estimated from the captured image (for example, estimated by image recognition processing using AI or the like) and acquired, or may be acquired by operation input from the user. You may do so.

For example, the measurement processing unit 131 displays an operation screen (hereinafter, referred to as “attribute information input screen”) for receiving the input of the attribute information of the object to be measured TO from the touch panel display 11, and the attribute information from the user. You may accept the input of.

FIG. 18 is a diagram showing a configuration example of the attribute information input screen.

In FIG. 18, buttons B101, B102, B103, B104, ... For receiving the selection of attribute information from the user are arranged. The buttons arranged on the attribute information input screen are buttons corresponding to the corresponding attribute information values. For example, FIG. 18 shows that button B101 functions as a "cardboard box", button B102 functions as a "refrigerator", button B103 functions as a "television", and button B104 functions as a button that accepts input of values corresponding to "sofa". There is. For example, when the button B101 is pressed on the attribute information input screen shown in FIG. 18, the input that the attribute information of the object to be measured TO is a "cardboard box" is accepted.

As described above, the measurement processing unit 131 acquires the shape information and the attribute information of the object to be measured TO. Here, the measurement processing unit 131 acquires information including shape information and attribute information of the object to be measured TO as the information to be measured.

As described above, the measurement processing unit 131 of this embodiment can acquire the measurement object information for one or a plurality of measurement object TOs.

Here, it is assumed that the measurement processing unit 131 can continuously perform fitting processing on a plurality of objects to be measured TO (for example, a set of a plurality of packages to be loaded in one container).

For example, when the measurement processing unit 131 completes the fitting process for one object TO to be measured and the acquisition of the information to be measured, the user shifts to the fitting process of the next TO to be measured, or all the objects to be measured TO An operation screen (hereinafter, referred to as a “selection screen”) for accepting the selection of whether to complete the fitting process may be displayed, and the user may accept the selection of the next operation.

FIG. 19 is a diagram showing a configuration example of the selection screen.

On the selection screen shown in FIG. 19, a button B201 for accepting the intention to perform the fitting process of the next object TO to be measured and a button B202 for accepting the completion of the fitting process for all the objects TO to be measured are arranged. It is assumed that it has been done.

When the button B201 is pressed on the selection screen, the measurement processing unit 131 returns to step S102 of the flowchart of FIG. 3 described above to resume the operation, shifts to the fitting process of the next object to be measured TO, and selects the selection screen. When B202 is pressed in, a process of simulating the arrangement when storing one or more measured objects TO that have undergone fitting processing in a storage (for example, a container for transporting luggage) (for example, a container for carrying luggage). Hereinafter, it will be referred to as "storage arrangement simulation processing").

Next, the storage arrangement simulation process by the measurement processing unit 131 will be described.

When the measurement processing unit 131 starts the container arrangement simulation process, the operation screen for simulating each object TO to be measured (measured object TO whose information to be measured is acquired by fitting processing) is stored in the storage. (Hereinafter referred to as a “storage arrangement simulation screen”) is displayed on the touch panel display 11.

FIG. 20 is a diagram showing a configuration example of a storage arrangement simulation screen.

As shown in FIG. 20, on the storage arrangement simulation screen, the 3D objects OB201 to OB204 corresponding to each of the four measured objects TO and the 3D object OB201 corresponding to the storage room to store the measured object TO are arranged. Has been done. In the storage arrangement simulation screen shown in FIG. 20, each 3D object is placed on the floor surface, and the image is a model of a state viewed from an obliquely upper viewpoint.

In this embodiment, for convenience of illustration, each side of the 3D object OB201 of the storage is drawn with a dotted line and each side is transparent, but the sides and faces constituting the 3D object OB201 of the storage are drawn. The method (drawing pattern) is not limited. For example, each surface of the 3D object OB201 in the storage may have a translucent (transparent; mesh) pattern.

Further, in this embodiment, it is assumed that the attribute information of the four objects to be measured TO is a cardboard box. Then, in the storage arrangement simulation screen shown in FIG. 20, in the 3D objects OB201 to OB204 of the object to be measured, an image corresponding to the attribute information (here, the texture of the corrugated cardboard image) is drawn (rendered) on the surface. As the texture image to be drawn on the 3D object of the object to be measured, the corresponding image for each attribute information may be retained in advance, or the image taken during the fitting process or the like may be applied. good.

The measurement processing unit 131 models (renders) the shape of each 3D object of the object to be measured TO according to the shape information (width, height and depth acquired by the fitting process) on the screen. Further, the measurement processing unit 131 may acquire information on the shape (for example, width, height and depth) of the 3D object of the storage to be modeled in advance, or may accept input from the user. You may.

FIG. 20 shows the initial state of the storage arrangement simulation screen. In FIG. 20, the 3D objects OB201 to OB204 of each object to be measured TO are arranged outside the 3D object OB101 of the storage.

21 and 22 are diagrams showing an example of transition of the storage arrangement simulation screen.

Then, on the storage arrangement simulation screen, it is possible to accept the movement of the 3D objects OB201 to OB204 of each object to be measured TO by an operation (for example, a drag operation) from the user, as shown in FIG. It is assumed that the 3D object in the storage can be moved inward.

Further, as shown in FIG. 22, it is possible to accept the change of the viewpoint (viewpoint when each 3D object is viewed) on the storage arrangement simulation screen.

At this time, the measurement processing unit 131 attempts to move (drag) the 3D object of any of the objects to be measured TO on the boundary between the inside and the outside of the 3D object OB101 in the storage. ), The position of the 3D object of the object to be measured TO is moved to a position deviated from the boundary of the 3D object OB101 of the storage (for example, the position before the drag operation (drop)). May be good.

As shown in FIGS. 20 to 22, in the storage storage arrangement simulation screen, whether or not all the objects to be measured TO (3D objects OB201 to OB204) fit inside the storage (3D object OB101) for the user. It is possible to provide a simulation environment such as the layout of each object TO to be measured in the storage.

As described above, the user can easily perform the fitting process for each object to be measured TO and operate on the storage storage arrangement simulation screen to easily determine the required number of storages and the measurement object TO in the storage. You can grasp the layout and so on.

(A-3) Effect of First Embodiment According to this embodiment, the following effects can be achieved.

In the measuring device 10 of this embodiment, the virtual object MO for measurement is displayed on the touch panel display 11 together with the captured image of the camera 12, and the virtual object MO for measurement is edited (fitting process) according to the user's operation. , It is possible to acquire and present the measured value related to the shape of the object to be measured TO to the user.

As a result, since the measuring device 10 accepts the fitting operation by the user's visual measurement, it is not possible to perform precise measurement using a tape measure or the like, but the cost is estimated based on the size of the cargo in the moving or home delivery business. Sufficient accuracy can be ensured when performing the above.

Further, since the measuring device 10 only installs the application on the smartphone, it can measure without using an instrument such as a tape measure, which is highly convenient.

Further, in the measuring device 10, if the virtual object MO for measurement can be imaged by the camera 12 from a plurality of viewpoints, even a large object (for example, a bookshelf, a refrigerator, etc.) should use a long tape measure. It is possible to measure easily without using.

(B) Second Embodiment Hereinafter, the first embodiment of the measuring device, the measuring program, and the measuring method according to the present invention will be described in detail with reference to the drawings.

(B-1) Configuration and operation of the second embodiment The overall configuration of the second embodiment and the configuration of the measuring device 10 of the second embodiment can also be shown with reference to FIGS. 1 and 2 described above. can.

Hereinafter, the differences between the second embodiment and the first embodiment will be described.

The second embodiment is different from the first embodiment in that a part of the processing of the measurement processing unit 131 is added.

Specifically, in the second embodiment, in the fitting process, the measurement processing unit 131 automatically processes a part of the process (hereinafter, referred to as “autofit process”) (hereinafter, “autofit function”). ”) Is added.

FIG. 23 shows a configuration example of the fitting processing screen in the second embodiment.

The fitting processing screen of the second embodiment is different from the first embodiment in that a button BA for accepting on / off switching of the autofit function is added. FIG. 23A shows a state in which the autofit function is off (a state in which “off” is displayed in the button BA) on the fitting processing screen, and FIG. 23B shows a state in which the fitting processing screen is displayed. , Indicates a state in which the autofit function is on (a state in which "on" is displayed in the button BA).

Hereinafter, an example of the fitting process performed by the measurement processing unit 131 of the second embodiment will be described with reference to the flowchart of FIG. 24.

FIG. 24 is a flowchart showing an example of processing when the measurement processing unit 131 performs the fitting processing in the second embodiment.

In FIG. 24, the same step numbers (reference numerals) are assigned to the steps for performing the same processing as in the first embodiment (FIG. 9).

The flowchart of the second embodiment is different from the first embodiment in that the autofit process (step S208) when the autofit function is on is added.

That is, in the flowchart of FIG. 24, when the operation of turning on the autofit function is detected by the operation detection in step S202 (for example, when the autofit function is turned on by pressing the button BA), the autofit process is started. , Step S208 is performed.

The element that triggers the on / off of the autofit function is not limited to the above buttons, and various methods can be applied. For example, the measurement processing unit 131 may accept on / off of the autofit function when a part of the measurement virtual object MO is pressed and held for a long time. In the following, in the measurement processing unit 131, the state in which the autofit function is turned on is referred to as an "autofit on state", and the state in which the autofit function is turned off is referred to as an "autofit off state". And.

Since the measurement processing unit 131 of the second embodiment can turn on / off the autofit function as described above, the autofit processing is applied only to the fitting processing of a part of the surface of the virtual object MO for measurement. , Manual processing may be applied to the fitting processing of other surfaces.

A plurality of methods of autofit processing by the measurement processing unit 131 can be mentioned. In the following, three examples of autofit processing by the measurement processing unit 131 will be given.

[First autofit processing]
First, the first autofit processing by the measurement processing unit 131 will be described with reference to the flowchart of FIG.

When the autofit-on state is set, the measurement processing unit 131 performs a process of trying to search for a representative point on a plane (a plane of the object TO to be measured; that is, a plane other than the floor surface F) (S301), and the search for the representative point is performed. If it succeeds, it operates from step S301 described later, and if the search fails, it returns to the beginning and loops, and tries to search for the representative point again. Specifically, the measurement processing unit 131 shoots a ray from the center of the screen of the touch panel display 11 in the optical axis direction of the camera, and a plane (a plane of the object TO to be measured detected by the AR system; that is, a plane other than the floor surface). ) Is hit, the center position of the plane is acquired as a representative point.

At this time, the measurement processing unit 131 estimates the plane with the lowest height (the plane having the smallest Y coordinate) among the planes detected by the plane detection of the AR system as the floor surface F, and processes (excludes from the plane search process). ) May be done.

When the measurement processing unit 131 succeeds in searching for the representative point of the virtual object MO for measurement, the measurement processing unit 131 has a target surface (hereinafter referred to as “target surface”) and dimensions when the virtual object MO for measurement is enlarged / reduced (moved). (Hereinafter referred to as "enlargement / reduction direction") is set (S302).

Here, the measurement processing unit 131 determines the plane whose representative point is searched for in step S301 as the target plane, and expands the direction (direction orthogonal to the target plane) about the normal direction of the plane determined as the target plane. It shall be decided as the reduction direction.

Next, the measurement processing unit 131 determines a surface (hereinafter, referred to as an “operation surface” in this embodiment) to be an operation target (movement target) in the measurement virtual object MO (S303).

For example, the measurement processing unit 131 may select a surface closest to the target surface (close to the rotation angle; small difference in rotation angle) as the operation surface from the measurement virtual object MO.

For example, the measurement processing unit 131 converts the normal vector of the target surface (for example, the unit vector of the world coordinate system) into the local vector of the virtual object MO for measurement (the vector of the local coordinate system in the virtual object MO for measurement). , The surface corresponding to the largest component may be determined as the operation surface. Further, for example, the measurement processing unit 131 calculates the inner product of the normal vector of each surface of the virtual object MO for measurement with the normal vector of the target surface, and the surface having the largest inner product value (that is, the operation surface). A surface having a similar orientation) may be selected as the operation surface.

Next, the measurement processing unit 131 determines whether or not the current target surface is the object to be measured TO (whether or not it is appropriate as a target) (S304), and is the object to be measured TO (as a target). If it is determined that it is appropriate), the operation is performed from step S305 described later, and if not, the process returns to step S301 and loops.

Judgment processing as in step S304 is not essential, but by inserting it, autofit processing is performed even for objects that the user does not intend as fitting targets (for example, luggage and walls around the non-measurement object TO). This is to avoid getting rid of it.

For example, the measurement processing unit 131 may determine that the current target surface is the object to be measured TO when the current target surface is the plane closest to the measurement virtual object MO. Specifically, for example, the measurement processing unit 131 rays from the center position of the virtual object MO for measurement in a direction corresponding to the enlargement / reduction direction determined in step S302 (for example, the same direction as the normal direction of the target surface). If the plane that hits first is the target plane, it may be determined that the target plane is the object TO to be measured (appropriate as a target).

When it is determined that the current target surface is the object to be measured TO (valid as a target), the measurement processing unit 131 rotates the measurement virtual object MO (rotates in the horizontal direction) to measure the measurement virtual object. The directions of the operation surface of the MO and the target surface of the object to be measured TO are aligned (parallel) (S305).

For example, the measurement processing unit 131 rotates the measurement virtual object MO so that the direction of the normal vector of the operation surface coincides with the direction of the normal vector of the target surface.

In step S305, when the measurement processing unit 131 tries to rotate the measurement virtual object MO, if the rotation amount (rotation angle) is less than or equal to a predetermined value (for example, 2 degrees or less), the rotation processing is performed. May be canceled (cancelled) to move to the next process. This is because the measurement processing unit 131 suppresses excessive frequency of rotation processing (for example, rotation processing based on an error generated by user's camera shake or the like) to perform stable and efficient autofit processing.

Next, the measurement processing unit 131 moves (operates) the operation surface of the measurement virtual object MO in the enlargement / reduction direction (adjusts (enlarges or reduces) the dimensions of the measurement virtual object MO in the enlargement / reduction direction). , The position of the operation surface and the target surface of the object to be measured TO are matched (S306), and the process returns to step S301 described above to loop.

In step S306, the measurement processing unit 131 determines the measurement virtual object MO so that, for example, the representative point of the target surface (the representative point searched in step S301 described above) is located on the operation surface or an extension of the operation surface. The operation surface may be moved in the enlargement / reduction direction.

In step S306, when the measurement processing unit 131 tries to move the operation surface, if the movement amount (adjustment amount, operation amount) is less than or equal to a predetermined value (for example, equivalent to 2 cm or less in the real space), The movement process (adjustment process, operation process) may be canceled (cancelled) to move to the next process. This is to suppress an excessively frequent movement process (for example, a rotation process based on an error caused by a user's camera shake) by the measurement processing unit 131 to perform a stable and efficient autofit process.

As described above, in the first autofit processing, the plane of the object to be measured TO reflected in the center of the measuring device 10 (camera 12, touch panel display 11) is set as the target surface, and the virtual object MO for measurement corresponds to the target surface. Select one surface as the operation surface, rotate the virtual object MO for measurement so that the target surface is parallel to the operation surface, and the positions of the target surface and the operation surface match (the operation surface and the target surface are aligned on a straight line). The operation surface is repeatedly moved (adjusting the dimensions in the enlargement / reduction direction of the virtual object MO for measurement) so as to move (including the above).

Next, a specific example will be described when the measurement processing unit 131 performs the first autofit processing according to the flowchart of FIG. 25 described above.

Here, an example of autofit processing in the case where the object to be measured TO and the measuring device 10 (camera 12) have a positional relationship as shown in FIG. 26 will be described.

In FIG. 26, it is assumed that the optical axis of the measuring device 10 (camera 12) is directed to one side surface of the object to be measured TO. In FIG. 26, it is assumed that the optical axis of the measuring device 10 (camera 12) is directed to the front surface T_FF of the object to be measured TO. In the following, the surface facing the front T_FF is the back T_FB, the surface adjacent to the upper side of the front T_FF is the upper surface T_FU, the surface adjacent to the lower side of the front T_FF is the lower surface T_FS, and the surface adjacent to the left side toward the front T_FF is the left surface. The surface adjacent to the right side of T_FL and the front surface T_FF is referred to as the right surface T_FR.

In FIG. 26, the optical axis of the measuring device 10 (camera 12) is directed slightly obliquely with respect to the front surface T_FF of the object to be measured TO. Here, it is assumed that when the object to be measured TO and the measuring device 10 (camera 12) are in the positional relationship as shown in FIG. 26, the measuring processing unit 131 starts the fitting process and is in the auto-fit-on state. ..

At this time, the fitting processing screen displayed on the measuring device 10 (touch panel display 11) is in the state as shown in FIG. 27.

At this time, in the process of step S301, when the measurement processing unit 131 shoots a ray from the center of the screen of the touch panel display 11 in the optical axis direction of the camera, it hits the front T_FF of the object to be measured TO, so that the measurement is performed as a representative point. The center position PR of the front T_FF of the object TO will be acquired.

At this time, when the object TO to be measured and the virtual object MO for measurement are viewed from above, the state is as shown in FIG. 28.

FIG. 28 shows the initial positional relationship between the object TO to be measured and the virtual object MO for measurement in the autofit process.

In FIG. 28, the arrow of the normal vector of the front T_FF of the object to be measured is shown as V_T_FF, and the arrow of the normal vector of the front FF of the virtual object MO for measurement is shown as V_FF.

Therefore, at this time, in the process of step S302, the measurement processing unit 131 sets the front surface T_FF as the target surface and selects the normal direction of the front surface T_FF (the direction of the normal vector V_T_FF) as the axis of the enlargement / reduction direction. become.

Further, at this time, in the process of step S303, the normal vector V_T_FF (unit vector of the world coordinate system) of the front surface T_FF, which is the target surface, is acquired, and the local vector of the virtual object MO for measurement (vector of the local coordinate system). Is converted to, and the surface corresponding to the largest component is determined as the operation surface.

In FIG. 28, the local coordinate system in the measurement virtual object MO is shown in the lower right.

In the local coordinate system of the virtual object MO for measurement, the depth direction is the Z axis, the width direction is the X axis, and the height direction is the Y axis when viewed from the center position PC. Further, in the local coordinate system of the virtual object MO for measurement, the direction toward the front T_FF is the + Z direction (the direction in which the Z coordinate value increases) and the direction toward the back T_FB is the -Z direction (Z coordinate) when viewed from the center position PC. The direction toward the upper surface T_FU is the + Y direction (the direction in which the Y coordinate value increases), the direction toward the lower surface T_FS is the -Y direction (the direction in which the Y coordinate value decreases), and the direction toward the left surface T_FL. It is assumed that the −X direction (the direction in which the value of the X coordinate decreases) and the direction toward the right surface T_FR are set as the + X direction (the direction in which the value of the X coordinate increases). In FIG. 28, since the view is viewed from above for the sake of simplicity, the Y-axis in the height direction is omitted.

Then, at this time, the measurement processing unit 131 sets the normal vector V_T_TF of the front surface T_FF (a unit vector having an absolute value of 1 in the world coordinate system) to the coordinates of the local coordinate system of the virtual object MO for measurement (width shown in FIG. 28). As a result of conversion to (coordinates represented by each coordinate value of X, height Y, and depth Z), the vector V_T_TF has a value of width X = -0.5, height Y = 0, and depth Z = +0.87. It shall be. In this case, since it is found that the + Z direction is the largest component in the vector V_T_TF, the measurement processing unit 131 selects the front FF corresponding to the + Z direction as the operation surface.

From FIG. 28, among the normal vectors of each surface of the virtual object MO for measurement, the surface corresponding to the one closest to the normal vector V_T_FF of the target surface (for example, the one having the largest inner product) may be selected. Similarly, it can be seen that the result is front FF.

Further, at this time, in the process of step S304, the measurement processing unit 131 is directed from the center position PC of the measurement virtual object MO to the direction corresponding to the enlargement / reduction direction determined in step S302 (for example, the normal direction of the target plane). When the ray is blown to, the plane that first abuts becomes the front surface T_FF of the object to be measured, so the current operation surface is truly the plane of the object to be measured (closest to the virtual object MO for measurement and is subject to measurement). It is determined that it is suitable as a surface of the measurement object TO), and the process proceeds to the process after step S305. Here, since the central position PC of the virtual object MO for measurement is located inside the object TO to be measured, the front surface T_FF is the most even if another plane (for example, a wall or the like) exists around the object TO to be measured. It will be a close plane.

Next, the measurement processing unit 131 makes the normal vector V_FF of the front surface FF, which is the operation symmetric plane, coincide with the normal vector V_T_FF of the front surface T_FF, which is the target surface, from the state of FIG. 28 by the process of step S305. In (so that they are parallel), the measurement virtual object MO is rotated.

At this time, the positional relationship between the object TO to be measured and the virtual object MO for measurement is as shown in FIGS. 28 to 29.

In FIG. 29, the normal vector V_FF of the front FF, which is the operation symmetric plane, coincides (parallel) with the normal vector V_T_FF of the front T_FF, which is the target plane.

Next, the measurement processing unit 131 moves (moves in the direction of reducing) the front FF, which is the operation surface of the virtual object MO for measurement, in the enlargement / reduction direction (direction of the normal vector V_FF / V_T_TF) by the process of step S306. ), As shown in FIG. 30, the representative point of the target surface (representative point PR on the front surface T_FF of the object to be measured TO) is positioned on the operation surface (front surface FF) of the virtual object MO for measurement.

As described above, the measuring device 10 (camera 12) is operated in the auto-fit-on state, and the front T_FF of the object to be measured TO is arranged so as to be in the center (so that it is reflected in the center of the touch panel display 11). As shown in FIG. 30, the front FF of the virtual object MO for measurement and the front T_FF of the object TO to be measured can be automatically matched.

After that, the user's operation grips the measuring device 10 and arranges each surface of the object to be measured TO (right surface T_FR, rear surface T_FB, left surface T_FL, and upper surface T_FU) so as to be centered in FIG. 31. As shown, all surfaces of the virtual object MO for measurement can be automatically fitted to the object TO to be measured.

[Second autofit processing]
Next, the second autofit processing by the measurement processing unit 131 will be described.

In the first autofit processing, the outer shape of the object to be measured TO was recognized by using the plane detection process of the object in the AR system, but in the second autofit process, the feature point detection process of the object in the AR system was used. The outer shape of the object to be measured TO shall be recognized.

The feature point in the AR process is an element that functions as a marker for tracking an object from a camera image or the like, and examples thereof include an acute-angled shape / pattern. In a general AR system, a group (point cloud) of feature points (3D coordinates of feature points) is detected from camera images, sensor data, etc., and a desired object is tracked from the point cloud of the detected feature points. Is possible.

For example, in Unity / ARkit, ARFoundation exists as a framework for AR development, and in this ARFoundation, there is a class called ARPPointCloudManager as a class for processing a point cloud of feature points from a camera image or the like. In various AR systems, there are classes that can detect a point cloud of such feature points. In this embodiment, the specific method for detecting the feature points is not limited.

That is, in the measurement processing unit 131, the point cloud of the feature points detected by the AR system (hereinafter referred to as “feature point cloud”) is a position along the outer shape (surface) of the object existing in the field of view of the camera 12. Is shown. Therefore, in the second autofit process, the feature points on the surface of the object to be measured TO are estimated from the acquired feature point group and acquired as a target (target point), and the representative point is used as a target for measurement. Autofit processing shall be performed by manipulating the corresponding surface in the virtual object MO. Here, the AR processing unit 131a is equipped with an AR system capable of detecting a group of feature points in the vicinity of the measuring device 10 based on the image captured by the camera 12 and the sensor data of the sensor unit 14. It is explained as.

The second autofit process as described above can be shown, for example, as shown in the flowchart of FIG. 32.

Next, the second autofit processing by the measurement processing unit 131 will be described with reference to the flowchart of FIG. 32.

When the autofit-on state is set, the measurement processing unit 131 acquires the data of the feature point group (3D coordinates of each feature point, etc.) recently acquired by the AR processing unit 131a (AR system) (S401).

Next, the measurement processing unit 131 selects a suitable feature point on the surface / outer shape of the object to be measured TO from the acquired feature point group and acquires it as a target point (S402).

The method by which the measurement processing unit 131 selects the target point from the feature point group is not limited, but for example, the following processing may be applied. For example, the measurement processing unit 131 may select the feature point closest to the optical axis of the camera 12 (the feature point closest to the central portion on the fitting processing screen) as the target point. At this time, the measurement processing unit 131 excludes the feature points presumed to be on the floor surface F and the feature points existing outside the field of view currently displayed on the fitting processing screen from the candidates for the target points. Is desirable. For example, the measurement processing unit 131 estimates that the plane with the lowest height (the plane with the smallest Y coordinate) among the planes detected by the plane detection of the AR system is the floor surface F, and is the same as or within a predetermined range of the floor surface F. A feature point of the inner height (for example, the difference between the floor surface F and the Y coordinate is equivalent to 3 cm or less) may be recognized as a feature point on the floor surface F.

Next, the measurement processing unit 131 selects an operation surface from the measurement virtual object MO (S403).

The method by which the measurement processing unit 131 selects the operation surface is not limited, but for example, the following processing may be applied.

For example, the measurement processing unit 131 selects the surface of the virtual object MO for measurement existing in the direction opposite to the optical axis direction of the camera 12 (that is, the direction toward the front side from the target point when viewed from the measurement device 10) as the operation surface. You may do so. For example, if the measurement virtual object MO has a position size that includes the object to be measured TO in the initial state, the measurement processing unit 131 performs the above processing and is appropriate to correspond to the representative point. The operation surface can be acquired.

Next, the measurement processing unit 131 selects the enlargement / reduction direction for the measurement virtual object MO (S404).

The method by which the measurement processing unit 131 selects the enlargement / reduction direction is not limited, but for example, the following processing may be applied.

For example, the measurement processing unit 131 may set the operation surface on the side opposite to the optical axis of the camera 12 (the front side when viewed from the camera 12) in the enlargement / reduction direction. For example, if the measurement virtual object MO has a position / size that includes the object to be measured TO in the initial state, the measurement processing unit 131 is optimal for autofit processing by the above processing. The enlargement / reduction direction can be acquired.

Further, for example, the measurement processing unit 131 determines in which direction the operation surface needs to be moved in order to position the target point on the enlargement / reduction target surface or on the extension line of the operation surface. Then, the enlargement / reduction direction may be selected according to the determined direction. For example, when the target point exists on the front side of the operation surface when viewed from the measuring device 10 (camera 12), the measurement processing unit 131 measures in the direction of moving the operation surface toward the front side (in the direction orthogonal to the operation surface). (Direction for expanding the dimensions of the virtual object MO) is selected as the enlargement / reduction direction, and if the target point exists on the back side of the operation surface when viewed from the measuring device 10 (camera 12), the operation surface is moved to the back side. The direction in which the measurement is to be performed (the direction in which the dimension of the virtual object MO for measurement is reduced in the direction orthogonal to the operation surface) may be selected as the enlargement / reduction direction.

Next, the measurement processing unit 131 confirms whether or not the camera 12 faces any surface of the virtual object MO for measurement (S405), and if it determines that the camera 12 faces any surface, step S406 described later. If not, the process returns to step S401 described above and loops.

For example, the measurement processing unit 131 shoots a ray from the center of the screen of the touch panel display 11 in the optical axis direction of the camera 12, and when the ray hits the virtual object MO for measurement, the ray and the surface hit by the ray are formed. When the angle (hereinafter referred to as "intersection angle") is acquired and the acquired intersection angle is within the range of 90 degrees ± α (α is, for example, 3 degrees to 5 degrees), the camera 12 determines the measurement virtual object MO. It may be determined that it faces either side.

Then, when it is determined that the camera 12 faces any surface of the virtual object MO for measurement, the measurement processing unit 131 moves (measures) the operation surface of the virtual object MO for measurement in the selected enlargement / reduction direction. The dimension of the virtual object MO for measurement is adjusted in the enlargement / reduction direction), the positions of the operation surface and the target point of the object to be measured TO are aligned (S406), and the process returns to step S401 described above to loop.

Next, an example in which the measurement processing unit 131 performs the second autofit processing according to the flowchart of FIG. 32 described above will be described.

Here, an example of autofit processing in the case where the object to be measured TO and the measuring device 10 (camera 12) have a positional relationship as shown in FIG. 33 will be described.

In FIG. 33, it is assumed that the optical axis of the measuring device 10 (camera 12) is directed to one side surface of the object to be measured TO. In FIG. 33, it is assumed that the optical axis of the measuring device 10 (camera 12) is oriented so as to substantially face the front surface T_FF of the object to be measured TO.

Here, it is assumed that when the object to be measured TO and the measuring device 10 (camera 12) are in the positional relationship as shown in FIG. 33, the measuring processing unit 131 starts the fitting process and is in the auto-fit-on state. ..

At this time, the fitting processing screen displayed on the measuring device 10 (touch panel display 11) is in the state as shown in FIG. 34.

At this time, when the measurement processing unit 131 acquires the feature point group in the process of step S401, it is assumed that the feature points have a distribution as shown in FIG. 35.

Then, the measurement processing unit 131 acquires the point PT on the surface T_FF as the target point as the feature point closest to the optical axis of the camera 12 (the feature point closest to the central portion on the fitting processing screen).

At this time, when the object TO to be measured and the virtual object MO for measurement are viewed from above, the state is as shown in FIG. 36.

FIG. 36 shows the initial positional relationship between the object TO to be measured and the virtual object MO for measurement in the autofit process.

Further, at this time, in the measurement virtual object MO, the direction opposite to the optical axis direction of the camera 12 (that is, the direction toward the front side from the target point when viewed from the measuring device 10) is the front FF. Therefore, in the process of step S403, the measurement processing unit 131 selects the front FF existing in the direction opposite to the optical axis direction of the camera 12 as the operation surface.

Further, at this time, since the target point PT exists on the back side of the operation surface (front FF) when viewed from the measuring device 10 (camera 12), the measurement processing unit 131 sets the operation surface (front FF) in step S404. The direction of moving to the back side (the direction of reducing the dimension in the depth direction) is selected as the enlargement / reduction direction.

Furthermore, at this time, when a ray is blown from the center of the screen of the touch panel display 11 in the optical axis direction of the camera 12, the ray corresponds to the operation surface (front FF), and the angle formed by the ray and the front FF is It is assumed that the temperature is within the predetermined range (for example, 90 degrees ± 3 degrees). Then, in step S405, the measurement processing unit 131 determines that the camera 12 faces the operation surface (front FF) of the virtual object MO for measurement, and proceeds to the process of step S406.

Then, in step S406, the measurement processing unit 131 moves the front FF, which is the operation surface of the measurement virtual object MO, in the selected enlargement / reduction direction (direction in which the dimension in the depth direction of the measurement virtual object MO is reduced). As a result, as shown in FIG. 37, the positions of the target point of the object to be measured TO and the operation surface (front FF) of the virtual object MO for measurement are matched (fitted).

After that, the user's operation grips the measuring device 10 and arranges each surface of the object to be measured TO (right surface T_FR, rear surface T_FB, left surface T_FL, and upper surface T_FU) so as to be centered in FIG. 31. As shown, all surfaces of the virtual object MO for measurement can be automatically fitted to the object TO to be measured.

[Third autofit processing]
As described above, in the first autofit processing, the autofit processing using the plane detection result is performed, and in the second autofit processing, the autofit processing using the detection result of the feature point group is performed.

In general, the feature point cloud detection process can be executed at a higher speed with a smaller amount of processing than the plane detection, so that the second autofit process has a smaller amount of autofit processing and is faster. (Efficient processing; processing in a shorter time) becomes possible.

Further, in the second autofit processing, since the plane detection result is not used, the shape of the object to be measured TO is not a polyhedron (for example, an object having a curved surface formed in part or all), or the shape is complicated to some extent. Even if there is, auto-fit processing can be performed.

Further, the second autofit processing has a problem that the accuracy is lower than that of the first autofit processing, and the autofit processing is attempted for a target object other than the object TO to be the target of the autofit processing. The second autofit process has lower processing accuracy). This is a problem that occurs because the feature point group is not originally an element for detecting a plane.

Therefore, in the third autofit processing, by adopting a method of properly using the first autofit processing and the second autofit processing, the merits of both the first autofit processing and the second autofit processing can be obtained. Can be generated.

As described above, in the second autofit processing, although the recognition speed is high, the accuracy is inferior to that of the first autofit processing. Therefore, for example, in the third autofit process, the second autofit process is applied when the plane of the object to be measured TO cannot be detected, and the first auto is applied when the plane of the object to be measured TO can be detected. A fit process may be adopted.

Specifically, for example, the measurement processing unit 131 may switch between the first autofit processing and the second autofit processing in the third autofit processing based on the flowchart of FIG. 38.

FIG. 38 is a flowchart showing an example of an operation when the measurement processing unit 131 performs the third autofit processing.

In this case, when the measurement processing unit 131 is in the autofit-on state, the measurement processing unit 131 attempts to detect the plane by the AR system (S501).

At this time, if the plane detection cannot be performed, the measurement processing unit 131 returns to the processing of step S501 and loops. At this time, if the floor surface is detected in the center of the screen of the touch panel display 11, the process proceeds to step S502 described later. Further, at this time, when the measurement processing unit 131 detects a plane other than the floor surface in the center of the screen of the touch panel display 11, the plane can be estimated as the plane of the object to be measured TO. The transition is performed and the first autofit processing is performed.

On the other hand, when the floor surface is detected in the center of the screen of the touch panel display 11 in step S501, the measurement processing unit 131 confirms whether or not the camera 12 faces any surface of the virtual object MO for measurement. (S502) If it is determined that the objects are facing each other, the process proceeds to step S504 described later to perform a second autofit process, and if not, the process returns to step S501 described above and loops.

At this time, regarding the process of determining whether or not the camera 12 faces any surface of the measurement virtual object MO in the measurement processing unit 131, the second autofit process (process of step S405 described above). Since the same processing as in the above can be applied, detailed description thereof will be omitted.

(B-2) Effect of Second Embodiment According to the second embodiment, the following effects can be obtained.

The measuring device 10 (measurement processing unit 131) of the second embodiment can reduce the operational burden on the user by supporting the autofit processing.

Further, in the measuring device 10 (measurement processing unit 131) of the second embodiment, by applying any of the first to third autofit modes, the merits of each of the above-mentioned autofit modes can be generated. Can be done.

(C) Other Embodiments The present invention is not limited to each of the above embodiments, and modified embodiments as illustrated below can also be mentioned.

(C-1) In the above embodiment, the measuring device 10 includes the touch panel display 11 as the operating means and the display means, but other devices may be used. For example, a display may be applied as a display means, and various pointing devices such as a trackpad and a mouse may be applied as an operation means.

(C-2) In the above embodiment, the object to be measured TO is placed on a flat surface (for example, the floor surface F), but the object to be measured TO is placed on an object not placed on the flat surface (for example, in the air). It may be a floating object).

Further, in the above embodiment, the shape of the object to be measured TO is a rectangular parallelepiped cardboard box or the like for simplification of explanation, but the shape of the object to be measured TO is not limited to the rectangular parallelepiped and other shapes (for example, a sofa or the like). Of course, it may be a globe, etc.).

(C-3) In each of the above embodiments, the collider is provided on each surface of the measurement virtual object MO, but as shown in FIGS. 39 to 42, each of the measurement virtual object MOs is provided. A collider having a predetermined cross-sectional shape may be provided around the side. Here, the length of the collider corresponding to each side (dimension in the direction of the corresponding side) and the shape of the cross section of the collider are not limited, but the shape of the virtual object MO for measurement (each dimension) is not limited. It may be a corresponding (proportional) dimension. In this embodiment, the cross section of the collider is described as being a rectangle having (proportional) dimensions corresponding to the shape (each dimension) of the virtual object MO for measurement, but the cross section of the collider is circular or the like. It may be in the shape of.

In the following, the modified examples shown in FIGS. 39 to 42 will be referred to as "this modified example".

In this modification, the colliders corresponding to the sides S_UF, S_UB, S_UL, S_UR, S_FL, S_FR, S_BL, S_BR, S_SF, S_SB, S_SL, and S_SR are colliders corresponding to C_UF, C_UB, C_UL, C_UR, C_FL, and C_FR, respectively. , C_BR, C_SF, C_SB, C_SL, C_SR.

39 and 40 are views showing the configuration of the collider C_FR arranged around the side S_FR. In FIGS. 39 and 40, the illustration of colliders other than the collider C_FR is omitted for the sake of simplicity.

FIG. 39 is a view (top view) when the measurement virtual object MO is viewed from the front FF side, and FIG. 40 is a view (top view) when the measurement virtual object MO is viewed from the top surface FF. It has become.

As shown in FIGS. 39 and 40, the collider C_FR is a rod-shaped object (a rod shape having a rectangular cross section) centered on the corresponding side S_FR. In FIGS. 39 and 40, the height direction of the collider C_FR is shown as LCH, the width direction is shown as LCW, and the depth direction is shown as LCD.

As shown in FIGS. 39 and 40, the collider C_FR is installed excluding the portions at both ends of the side S_FR (that is, the periphery of the apex). This is for the purpose of improving operability and simplifying programming (reducing programming cost) of the measurement processing unit 131, and has three dimensions (width LW, height LH, and height LH) related to the measurement virtual object MO in one operation. This is a measure for facilitating the acceptance of operations for only one dimension of the depth LD). For example, in the virtual object MO for measurement, when a drag operation (vertex movement operation) of the vertex portion is accepted, the shape is not deformed as desired by the user depending on the mounting method, but rather the shape is fitted to the image GT of the object to be measured. May be difficult.

Therefore, as described above, the colliders on each side are installed excluding the parts at both ends. The length of the collider to be installed on each side is not limited, but may be, for example, about 80% of the length of the corresponding side. In this case, gaps (sections in which colliders are not installed) having a length of about 10% of the sides are generated at both ends of the sides.

Further, the dimensions of the cross section of the collider to be installed are not limited for each side, but the dimensions may be set according to the sides parallel to each other. For example, the cross-sectional dimensions of the collider may be about 10% of the parallel sides. Specifically, for example, since the collider C_FR shown in FIGS. 39 and 40 is a side in the height direction, the dimension LCH in the height direction is 80% of the dimension LH, and the dimension LCW in the width direction is the dimension LW. The dimension in the depth direction is 10%, which is 10% of the dimension LCD.

Next, the dimension change processing in this modification will be described.

On the fitting processing screen of this modification, when a drag operation by touching one point (one finger) occurs on any collider of the virtual object MO for measurement (a ray is skipped from the touched position and either one). When a collision occurs in the collider), the measurement processing unit 131 selects one of the two surfaces adjacent to the collider, and the direction of the drag operation (with respect to the dimension in the direction orthogonal to the selected surface). A process is performed in which the dimension is changed according to the drag operation distance (distance in which the finger is slid) in the direction in which the finger is slid. In other words, the measurement processing unit 131 selects one of the two surfaces adjacent to the collider and moves the selected surface in the direction orthogonal to the surface (normal direction or direction opposite to the normal direction). Will be processed.

In the fitting processing screen of this modification, when the collider is dragged, it is examined which of the two surfaces adjacent to the collider the user intends to change the dimension corresponding to. In this embodiment, it is presumed that the user is trying to change the dimension corresponding to the side in the foreground (the side facing more) when viewed from the current camera viewpoint.

When any of the colliders is operated, the method of recognizing the front side of the first surface and the second surface adjacent to the collider when viewed from the camera viewpoint is not limited.

For example, the measurement processing unit 131 may perform an arbitrary position (here, here) on the side corresponding to the operated collider from the current camera viewpoint (for example, the origin on the camera model of the camera 12 / the center point of the touch panel display 11). , Which is the midpoint of the side), the angle formed by the ray and the extension line of the first surface (hereinafter referred to as "angle θ1"), and the ray and the first The angles formed by the extension lines of the two surfaces (hereinafter referred to as "angle θ2") are compared, and the surface corresponding to the larger angle is recognized as the surface existing in the foreground and selected (hereinafter, this selection is performed). The surface may be referred to as a "selected surface").

FIG. 41 is a diagram showing an example of screen transition when the measurement processing unit 131 performs the dimension change processing on the fitting processing screen.

FIG. 41 shows the screen transition of the measurement virtual object MO when the dimension change processing is performed for the dimension LW in the width direction, the dimension LH in the height direction, and the dimension LD in the depth direction.

Then, FIG. 41 (a) shows the state of the fitting processing screen before the dimension change processing, and FIG. 41 (b) shows the state after the dimension change processing is performed according to the operation of the user's hand UH. ing.

First, the dimension change process when changing the dimension LW in the width direction will be described with reference to FIG. 41. FIG. 41 shows a dimension change process when the collider C_FR arranged around the side S_FR between the surface FF and the surface FR is dragged.

In the following, the collider being dragged is referred to as an "operation collider", and the side corresponding to the operation collider is also referred to as an "operation side".

In FIG. 41 (a), the collider C_FR is dragged from the camera viewpoint where the surface FF faces the front side.

At this time, when the angles θ1 and θ2 are calculated with the first surface as the surface FF and the second surface as the surface FR, as shown in FIG. 42, it corresponds to the first surface FF which is the front surface when viewed from the viewpoint. The angle θ2 corresponding to the second surface FR corresponding to the side surface is larger than the angle θ1 to be formed when viewed from the viewpoint.

FIG. 42 illustrates the angles θ1 and θ2 on the plane including the ray when the side S_FR ray (PE-S_FR) is skipped from the camera viewpoint PE in FIG. 41 (a).

Therefore, in the example of FIGS. 41A and 42, the measurement processing unit 131 has a surface of the surface FF (first surface) and the surface FR (second surface) from the comparison with the angles θ1 and θ2. FR is selected as the selection surface, and the dimension change processing is performed for the dimension LW in the direction (width direction) orthogonal to the selection surface FR.

When the collider C_FR is dragged in a direction away from the center position (center position of the surface FF) of the virtual object MO for measurement (shown as + LW direction in FIG. 41 (a)) with reference to the operation side S_FR, the measurement is performed. The processing unit 131 performs a dimension change process in which the dimension LW in the width direction of the measurement virtual object MO is increased by the distance corresponding to the drag operation distance (distance in which the finger is slid in the + LW direction) in the + LW direction. At this time, the position of the side S_FL facing the operation side S_FR on the selection surface FF remains fixed.

When the collider C_FR is dragged in a direction approaching the center position (center position of the surface FF) of the virtual object MO for measurement (shown as the -LW direction in FIG. 41A) with reference to the operation side S_FR, The measurement processing unit 131 reduces the dimension LW in the width direction of the virtual object MO for measurement by the distance corresponding to the drag operation distance (distance in which the finger is slid in the −LW direction) in the −LW direction. I do. Also at this time, the position of the side S_FL facing the operation side S_FR on the selection surface FF remains fixed.

In other words, in this case, it can be said that the collider C_FR functions as a slider that fluctuates the dimension LW in the width direction of the virtual object MO for measurement.

Then, here, as shown in FIG. 41 (b), the collider C_FR is dragged in the −LW direction to the position at the right end of the image GT to be measured, and the dimension LW in the width direction of the virtual object MO for measurement is increased. It shows that the state is in agreement with the measured object TO (measured object image GT).

10 ... Measuring device, 11 ... Touch panel display, 12 ... Camera, 13 ... Control unit, 131 ... Measurement processing unit, 131a ... AR processing unit, and 14 ... Sensor unit.

Claims (5)

In a measuring device including an imaging means and a display means,
A virtual object for measurement of a rectangular shape associated with a coordinate system in the real space in which the object to be measured is arranged is generated, and an image of the virtual object for measurement is synthesized with an image captured by the imaging means. Augmented reality processing means to be displayed on the screen of the display means,
An object editing means for editing the virtual object for measurement and
The object editing means has a measurement parameter output means for acquiring and outputting measurement parameters related to the current shape of the virtual object for measurement, and the object editing means targets a surface of the object to be measured with which the optical axis of the imaging means abuts. The target surface is determined as a target point, the surface closest to the target surface is determined as the target surface on the measurement virtual object, and the target surface is set as the target point. A measuring device characterized in that the dimension of the target surface of the virtual object for measurement in the normal direction is adjusted so as to move in an approaching direction. The object editing means is a storage corresponding to one or a plurality of the measurement virtual objects edited by the object editing means and a storage for storing the measurement object corresponding to each of the measurement virtual objects. A claim further comprising a simulation providing means for displaying a storage object and providing a simulation screen capable of confirming whether or not each of the measurement virtual objects is housed inside the storage object. Item 1. The measuring device according to item 1. The measurement parameter output means is characterized in that, as the measurement parameter, one or more values of height, width, depth, and volume when the virtual object for measurement exists in the real space are output. The measuring device according to 1 or 2. A computer mounted on a measuring device including an imaging means and a display means.
A virtual object for measurement of a rectangular shape associated with a coordinate system in the real space in which the object to be measured is arranged is generated, and an image of the virtual object for measurement is synthesized with an image captured by the imaging means. Augmented reality processing means to be displayed on the screen of the display means,
An object editing means for editing the virtual object for measurement and
It functions as a measurement parameter output means that acquires and outputs measurement parameters related to the shape of the current measurement virtual object.
The object editing means recognizes a surface of the object to be measured that the optical axis of the imaging means abuts as a target surface, determines any position of the target surface as a target point, and determines the position of the target surface as a target point on the virtual object for measurement. The surface closest to the target surface is determined as the target surface, and the normal dimension of the target surface of the virtual object for measurement is adjusted so that the target surface moves in the direction approaching the target point. A measurement program characterized by that. In the measuring method performed by the measuring device
The measuring device includes an imaging means, a display means, an augmented reality processing means, an object editing means, and a measurement parameter output means.
The augmented reality processing means generates a virtual object for measurement of a rectangular shape associated with a coordinate system in the real space in which the object to be measured is arranged, and the virtual object for measurement is added to an image captured by the imaging means. The images of are combined and displayed on the screen of the display means.
The object editing means edits the virtual object for measurement, and the object editing means edits the virtual object for measurement.
The measurement parameter output means acquires and outputs measurement parameters related to the current shape of the measurement virtual object, and outputs the measurement parameters.
The object editing means recognizes a surface of the object to be measured that the optical axis of the imaging means abuts as a target surface, determines any position of the target surface as a target point, and determines the position of the target surface as a target point on the virtual object for measurement. The surface closest to the target surface is determined as the target surface, and the normal dimension of the target surface of the virtual object for measurement is adjusted so that the target surface moves in the direction approaching the target point. A measurement method characterized by that.

Images